Indexing Again?

I'd like a really good heads up about when to expect the next Beta so that I can purchase a couple of resources and add them to my computer before all the necessary indexing begins.

I'd also like my version of Christian History and Biography to ship so that I can add it before the next upgrade.

I am having the same issue also. I recieved Ariel's Genesis and updated v.3 on both of my computers.  i wasn't going to reindex until the next beta.  When  I turned off the computers and later fired them up and restarted v.4, they both went into the preparation phase and indexing.  I left both of them on over night. They appeared to finish and Ariel's Genesis is now available in v.4.  Now, everytime I start v.4 it immediately goes into preparation phase (about 2 minutes or less) and begins reindexing (forever).  Somewhere in the vacinity of about 4 to 5 hours it stops indexing and the blue icon disappears from the tray.  When I start v.4, it starts the process all over again from the beginning.  With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?

With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?

The answer is: "Shouldn't but might".

It shouldn't shorten the life of your computer because it's designed to compute.  Your computer manufacturer is supposed to design and stress test their systems so that they adequately cool under load.  The heat generated is, as you suggest a potential killer; that's why computers have noisy fans. 

The reality is that not all manufacturers design their systems well nor test them under extreme loads.  Most 3-5 year old through modern BIOS (don't worry if you don't know what that is) have features that shut off your computer to prevent damage in case of overheating. 

I have a notebook that is ill designed and if the slightest buildup of dust occurs on it, overheating from indexing will shut it down.  Thus I periodically fire up my compressor and blow the dust out of it.  That's just good maintenance though I suggest a can of compressed air instead of a compressor which can generate water vapor.  Do as I say, not as I do.  :-)

So the heavy CPU usage Might, but probably Won't damage your system.  One of our brothers here has found he must put a room fan on the computer to keep it from shutting down, but the good news is that the shut down feature is protecting his computer.

If you're not getting sudden power downs, don't worry about the overheating.  If you are concerned you should blow the dust out of your system.  If you're really concerned put a room fan on it during indexing, if you're uber concerned buy a cooling unit and have a geek wire it in for you. 

But in all honesty don't worry about it.

 

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

 

Bob, you guys REALLY  need to rethink this indexing scheme.

The good news: We're optimizing it as much as we can, and you'll see improvements in Beta 3.

The work Logos is doing to improve this is appreciated... The big issue for some of us seems to be CPU thrashing.....

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

...so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

......

We're planning to do A.

so I think A is the better alround solution

(I should also point out that how "heavy" the indexing feels has a lot to do with particular machine configurations. Some machines will see the processor being burdened, and feel other applications be slowed. Machines with multiple cores will not feel the processor load as heavily. And the more memory, the less disk usage.)

I don't want you designing the software to be optimized for 5 year old technology....but any gains that can be made in indexing for users in that situation would be appreciated....if the issue of having to re-start indexing after a computer shutdown can be overcome then option A followed by B might be a better solution for users in that scenario if they can leave their machines logged into the one profile for that long.... but we will have to see the improvements in b3  and whether the CPU thrashing issue is in any way alleviated before really makiing firm thoughts on some of these things....whatever improvements you are able to make thought I certainly am no looking for you to make a single core machine behave like a dual core....

...so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..

I'm not sure I understand this...Indexing doesn't have to be done to use the resource.  You just can't search the whole library. 

I just purchased Luther's Galatians from Logos a couple hours ago.  Of course I had to reindex. I didn't try using it right away, but I still have 1210 resources left to index and I'm reading the new resource now, plus I can search my library and get results for the books that are indexed.

For those of us who have older technology and CPU's thrashing at close to 100% you really can't do anything else...even though in theory you can....

Interestingly, on my 5 yr old single-core Pentium Thinkpad, the indexer is averaging around 85% CPU.  The computer is quite usable.

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

I like this idea if I control when the master indexing is done. I'd be happy to run the master indexing after very major purchases (say NICOT/NT). But if I purchase a series of very small resources, it could well be a few months before I wish to update the major index. A nag screen once a week on startup would be bearable.

To me, the simple selling point is that the computer is not in a continuous state of re-indexing. I can't see any new users making many additional purchases if they have to go through the process of re-indexing every time. You've heard the outcry from "power users" - imagine what the regular person will think.

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

I'd rather do it all in one hit when I want.

I'm still not sure why we also cannot permanently have multiple indexes. Why are bibles indexed together with other resources? It would save me considerable time indexing if bibles were not included every time (it's years since I bought a new bible).

Reverse interlinear indexing is the only really slow part of Bible indexing, and that's been dramatically sped up for Beta 3. So indexing Bibles separately doesn't help that much, since they're not the largest part of the library

Thanks for the response Bob.

I was pretty sure that the Interlinear bibles were amongst the largest files in my library - with only a couple of exceptions. While I await Beta 3 and its promise of a dramatic improvement in indexing these resources, I still wonder whether they wouldn't be better off separate to commentaries and monographs.

(Though we may look into that post 4.0 for other reasons.)

It could improve searches on these titles, couldn't it?

This beta program is full of the minority who buy lots of books -- and we love you!

Thanks Bob. It''s also why we press hard on issues important to us.

(It's even hard to "pause" the index and restart it. We get much of the speed it does have from using a huge amount of memory. If you've got nearly a gig -- or 4 gigs, on a bigger machine -- of the index in memory, and you want to be able to turn off the machine and restart later, you've got to write code and take the time to write that gig of memory to the hard drive, then reload it when you start, etc. 

And if you want to be able to shut down anytime, quickly, we've got to be keeping all that on the hard drive, not in memory, and that just makes it even slower....

This is a real concern..... My little netbook doesn't want it to take any longer than it already takes...

There is no magic bullet in indexing. There are optimizations, which we're doing. There are also tricks that make it feel less painful, which we're exploring.

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

Bob,

Thanks for all the effort you guys are making in order to speed up the indexing. I am one of the ones whose older laptop is constantly working for 1-2 days to get the titles indexed. I would go for A at this point. I would like to be able to use the laptop, while a painless indexing is taking place. I am sure that there will be better ways in the future and that you will continue to see how to expediate this process. Just wanted you to know that I appreciate all the hard work that Logos is doing.

Charlene

I am excited to hear these possibilities. The first, to have a separate index for new books until they are put into the full index sounds like a great new "feature".

The second sounds good if it works. I know it will work fine for the powerful new computers, but will it work with the more limited netbooks (such as EeePC)? They have a hard time handling too much at once. It would be great if there was a slow and steady method to get the book indexed without stopping the computer.

I figure most users (including myself) will want instant access to their new book(s) (idea A), but also aren't going to want to have to wait a day or more to use their computer (thus some version of idea . These sound like a win, win, especially if they work with the netbooks.

Personally, I don't mind if it takes a while to be fully indexed as long as I can access it immediately (or at least very soon) and it doesn't take me away from my other computer needs.

Thank you for being so amazingly responsive to our needs. I know it is good for business, but the way you all handle it is fantastic.

Thank you,

Rob

I nominate Thomas' computer to do all my
indexing!

Each customer has to run the index on the same resources.  Seems like such a waste.  My opinion is that Logos ought to deliver index files with resources and merge them into the customer's larger index.  That way it is done once, which i am sure Logos does anyway, and not done again for each customer.  Am i not seeing this clearly?

Joe,

So far I've gathered that since most people's libraries are slightly different, each library will need a different index, so it has to be custom built on the computer.  Earlier, Bob mentioned that they are considering (or will?) ship indexes of collections as they are, so scholars library will ship with a scholars library index.  And somewhere else, he mentioned that merging indexes would actually make them much larger than doing it the way their' doing it. 

I don't truly understand the indexing process myself but there is a good post from Bob on it at the top of the third page in this thread.

The program has been indexing my resources for several hours now since installing beta 3. During this time it seems to have also slowed my computer down quite a bit. What is the purpose of this indexing?

My Computer is an HP with Vista Premium 64 bit with SP2/ 2GB Ram/AMD Turion 64 X2 1.6ghz processor.