Indexing, indexing, indexing.....

I just started up Logos to do some reading and Bible study and had 873MB download. When Logos was restarted the dreaded indexing started. And now, for the next 4 hours (4h 17m remaining) all searches are returning meaningless results. For example, doing a morphological search for a New Testament word (κῆτος, in this case) returns Malachi 3.1 - obviously wrong.

Each time I see "Logos Bible Software is indexing" my heart sinks. I used Bibleworks for years and never had to wait 4 hours after applying an update before being able to accurately search again.

I really, really, really would like Logos to attempt something to improve this situation - keep the old indices, merge them, build them smarter, etc. For example, why is it necessary to re-index everything when only a few new resources have been added? Why can each resource not be indexed (once, say) and the indices merged into a larger "master index"? Then only the per-resource indices for new or updated need be rebuilt when resources are added/updated and the "master" index rebuilt.

These sorts of problems have been solved many times before in IT. I cannot understand why such a "brute-force" approach continues to be Logos' preferred approach, especially given all of the issues and discontent it causes.

Yes, I know I'm ranting but how hard can this be?

Yours patiently (only 4h 12m to go before I can run the search I wanted to),

Stuart.

Find more posts tagged with

Comments

Fred Chapman

I just started up Logos to do some reading and Bible study and had 873MB download. When Logos was restarted the dreaded indexing started. And now, for the next 4 hours (4h 17m remaining) all searches are returning meaningless results. For example, doing a morphological search for a New Testament word (κῆτος, in this case) returns Malachi 3.1 - obviously wrong.

Each time I see "Logos Bible Software is indexing" my heart sinks. I used Bibleworks for years and never had to wait 4 hours after applying an update before being able to accurately search again.

I really, really, really would like Logos to attempt something to improve this situation - keep the old indices, merge them, build them smarter, etc. For example, why is it necessary to re-index everything when only a few new resources have been added? Why can each resource not be indexed (once, say) and the indices merged into a larger "master index"? Then only the per-resource indices for new or updated need be rebuilt when resources are added/updated and the "master" index rebuilt.

These sorts of problems have been solved many times before in IT. I cannot understand why such a "brute-force" approach continues to be Logos' preferred approach, especially given all of the issues and discontent it causes.

Yes, I know I'm ranting but how hard can this be?

Yours patiently (only 4h 12m to go before I can run the search I wanted to),

Stuart.

I am assuming the download is what is the full view reverse interlinears that were released to the Beta group. What are the specs on your computer? I downloaded 875Mb yesterday and on my laptop it only took about 15 minutes to index. Did you execute a rebuild index or merge index (which I found out last week was the same thing as rebuilding) command?

As for using the program while indexing I have found (and others have said on the forums) that you should expect performance issues and erroneous search results until the indexing is complete.

Stuart Robertson

https://community.logos.com/discussion/comment/119215#Comment_119215

I have a Thinkpad T61 (3GB RAM) - Intel Core 2 2.2GHz - not an awesome machine but not ancient either.

Only 3h 48m till indexing is done... :-(

Kevin Becker

Stuart,

Part of Beta-testing includes the possibility of receiving new files, requiring indexing. This is especially the case with the new full-view reverse interlinears. We were warned at the beginning of this beta cycle. If you can't stand indexing set "Automatically Download Updates" to No and only download updated files when you can afford the indexing time. Also, Logos does not rebuild the entire index every time updates happen; this only happens if the supplemental index gets too big. 4.0a had index merging but there were some bugs in it that frequently resulted in corrupted indexes crashing Logos and causing lots of heartache resulting in complete re-indexing anyway; Logos removed the faulty merge pending the fixing of all those bugs. It was the right decision to make. I am confident that the merge, when it reappears, will greatly streamline maintaining an up-to-date index.

Jason Ray Nall

https://community.logos.com/discussion/comment/119228#Comment_119228

Kevin,

I have the same heart dropping experience when Logos has to index. It is usually a 4hr or more event and it is overworking my laptop. The fan was working really hard to keep things cool so I check the activity monitor and it goes between 99%-147% of computer usage.

i have a mac laptop with 2.4GHz intel core 2 Duo and 2GB 667Mhz of RAM. It is impossible to use the computer and I am worried about my laptop.

Not happy.

JimTowler

https://community.logos.com/discussion/comment/135870#Comment_135870

For your dual-core machine, one work-around is to set the indexer process to only use one of the cores. It will continue to run two copies, but run them on only one core, leaving the other idle. The indexing will of course take longer, but the overall CPU die may run a little cooler and maybe cope better.

We have asked Logos before, to add a pacing control to the index process, for the sake of laptops and netbooks that really can't handle it otherwise. So far, no answer or agreement to do this, as far as I know. Most users will of course leave it to run at 100% and complete ASAP, but for some, being able to set it to only index on one core and at never more than 20% (for example) would allow it to complete without frying the CPU with heat, and allow the system a little more responsiveness if required.

Also, the indexing needs to be better at keeping state, so it can restart at any point, unlike now.

I'm OK with needing to reindex at times, as searches take 1-3 seconds, not 1 minute. But it would be nice if the indexer could be made to BACK OFF if the user asks. For now, all we can do is STOP it for 4 hours. Thats not enough control.

Please Logos.

Jason Ray Nall

https://community.logos.com/discussion/comment/135879#Comment_135879

Thanks Jim for your input - agreed!

Rosie Perera

https://community.logos.com/discussion/comment/135879#Comment_135879

I'm OK with needing to reindex at times, as searches take 1-3 seconds, not 1 minute. But it would be nice if the indexer could be made to BACK OFF if the user asks. For now, all we can do is STOP it for 4 hours. Thats not enough control.

Yes! That would be much better. I'd love to be able to tell the Logos indexer to lower its priority but still get work accomplished during idle cycles, so I could go on using the computer for other things while it was indexing in the background.

I don't know why indexing in Logos is such an ordeal but Windows indexes my entire hard disk (or at least every file under my user folder) constantly and I never notice it's doing it (it is very well-behaved), yet I can always find whatever files I want to open very quickly by searching on their contents in the File Open dialog from any application.

Stuart Robertson

https://community.logos.com/discussion/comment/135914#Comment_135914

I recently purchased a 17" i7 MacBook Pro (quad core) and installed Logos v4 (alpha 20) for Mac. The indexing load that the Mac version produces makes the Windows version look like a walk in the park - 4+ hours with all 4 cores running at almost 100%, draining battery so fast I had to switch to external power.

I updated to Alpha 21 on Friday and am still trying to get it indexed - pausing for 4 hours regularly so I can get real work done.

I know I probably sound negative and ungrateful (and while I may be seen to be negative I'm certainly not ungrateful for Logos v4), but I truly cannot understand why such long and heavy indexing is required. I don't understand the product internals, but I have worked with Bibleworks (up to version 7) for a long time and it provides blindingly fast searches without the need for any indexing (at least not indexing that each user is required to run).

At the risk of redundancy, I'd like to echo the sentiment already expressed - please Logos, come up with a better solution - or at least provide some means of controlling indexing so that it doesn't take over the entire machine.

Also, the current behavior (deleting indexes before building new ones) means that I have to complete indexing before Logos is usable again - once indexing is started all of my visual filters, etc are misaligned and Bible searches return complete garbage (that is, are completely misaligned, returning verses that don't match the search criteria at all). At the worst, can the current indexes not be preserved until the new ones have been built?

Regards,

Stuart.

Dave Hooton

https://community.logos.com/discussion/comment/135921#Comment_135921

Also, the current behavior (deleting indexes before building new ones) means that I have to complete indexing before Logos is usable again

A user requested Rebuild will do this, but the Indexer only does this when necessary. And the situation will improve now that true Merge has been re-introduced in beta (less need to rebuild).

Regarding heating of the CPU, I direct cooling air from a household fan toward the laptop and let it index at 100%!

Whyndell Grizzard

https://community.logos.com/discussion/comment/135922#Comment_135922

Purchase a cooling table for less than $25 that helps enormously, the fan idea works well to. The key is ventilation, give the laptop a 1" space to get rid of the heat, elevate the rear of the laptop.

Stuart Robertson

https://community.logos.com/discussion/comment/135927#Comment_135927

:-) It's the first time I've been told to purchase a cooling table and add a fan so that a modern laptop can keep up cool when running a Bible program. I do a lot of development and am used to running fairly heavy-weight builds in Java and have never (yet) needed to invest in special cooling paraphernalia. Perhaps a high-speed RAID array, a 16-core desktop and 32GB RAM would do the trick...

JimTowler

https://community.logos.com/discussion/comment/135919#Comment_135919

Yes! That would be much better. I'd love to be able to tell the Logos indexer to lower its priority but still get work accomplished during idle cycles ...

Rosie,

Re the HEATING ISSUE

If you could set the indexer to the lowest priority every invented, it would still run at 100% CPU if there was nothing else to do, so that does not help with the heating issue. Short of fixing the cooling of laptops, the other answer is for the Logos Indexer to have a setting so it can "rest" for 80% of the time, and only work for 20% of the time. So then, an idle PC will never run over 20% busy, and hence run cooler if the CPU slows down as for laptops.

The coding for such a thing would require the current indexing program to "sleep" for x milliseconds every second or so. A tight loop would still be 100% CPU, so the program would have to really sleep - go idle. Of course, most users would always let it run at max. The answer is only useful for laptops or netbooks, but that is EXACTLY where the problem is. Users reporting their laptops freaking out with heat issues.

The other possible solution for a laptop is to go into the the Power Plans, and create a new one that sets the MAX CPU to, for example, 50%. This will limit the speed the CPU is clocked at, which is DIRECTLY related to power usage and heat generation. Its also important to set any fan speed options to max, for cooling.

An example I made just now follows. It will of course make the index process take longer, but the idea here is to limit heat problems, not reduce index times.

JimTowler

https://community.logos.com/discussion/comment/135936#Comment_135936

OK, running with power supply connected, but using the plan above, I get the results as shown below.

The system bus is just under 50%, and the dual-CPU-cores are running at 31%, so if Logos4 was reindexing it would be going between 30% to 50% of normal, excluding disk speed. It will take longer to complete, but I would expect less or no issue with cooling.

I offer this idea as a possible work-around to those with heating issues during an index rebuild. I welcome feedback on results so this might become a wiki page and one recommended soution if its useful to those where heating is the problem.

Dave Hooton

https://community.logos.com/discussion/comment/135939#Comment_135939

OK, running with power supply connected, but using the plan above, I get the results as shown below

Thanks for the suggestion and the Utility, Jim. Changing CPU speed on the laptop was simple thanks to its
proprietary power utility and the reported speed indicated its effect.