Beta 9 Indexing: Crazy Slow!

An hour or so after telling Logos to merge my indexes for the newly downloaded B9 resources, the estimated time remaining on the index is steadily increasing. In the early beta days, this particular computer could index my entire library in somewhere between 12 and 24 hours depending on use.

Now...

Is there a problem with the time estimator or a problem with the indexer?

The option for background processing is not available upon right click?

Is this background processing (I don't notice any slowdown in computer function, but I only ever really perceived that on my netbook anyway)?

Am I able to restart now without needing to restart?

Find more posts tagged with

Performance

Comments

Ken McGuire

The problem with the time index is that you might believe it. As we are all aware, Reverse Interlinears are the slowest books to index. The time estimate seems to be based on looking at the time it has taken to index the resources it has already looked at as well as the number remaining, and so especially at the beginning when it is grinding trough those slow files, the estimate is wildly off. The time estimate should go down as it realizes that typical resources are faster.

The good thing is that it actually estimates how long it will take AFTER all the files have been indexed when it is combining everything into the master.

Jacob Hantla

https://community.logos.com/discussion/comment/21780#Comment_21780

You're right. It's now at 15 hours.

Bradley Grainger (Logos)

Is there a problem with the time estimator or a problem with the indexer?

The estimation can be pretty rough, especially after the indexer has just started. Perhaps the biggest problem is that it assumes all resources are the same; like Kenneth said, if the first 3 it indexes are reverse interlinears (which take a long time), it's going to assume that the rest of your library would take as long. (And if you did have 1000+ reverse interlinears, it probably would take 143 hours!)

I hope we can improve it in future releases. Perhaps we can (anonymously) collect statistics when indexes are being built and report # of books, # of CPUs, and actual time vs estimated time (sampled at 5 minute intervals, say). This would be a small amount of data per computer, but aggregated over thousands of machines, it should really help us to fine tune the initial estimates. Other improvements could be to treat different types of resources differently when estimating the amount of work done so far, and the amount remaining; or remembering the actual indexing times previously recorded on your hardware and use those to set the initial estimate.

Jacob Hantla

https://community.logos.com/discussion/comment/21793#Comment_21793

That seems like a good idea, Bradley. I now only have 6 hours left.

Jason

https://community.logos.com/discussion/comment/21793#Comment_21793

Perhaps we can (anonymously) collect statistics when indexes are being built and report # of books, # of CPUs, and actual time vs estimated time (sampled at 5 minute intervals, say). This would be a small amount of data per computer, but aggregated over thousands of machines, it should really help us to fine tune the initial estimates. Other improvements could be to treat different types of resources differently when estimating the amount of work done so far, and the amount remaining

This would be a great idea. Anything to help get better estimates...just remember to turn it off before going live unless you want even better stats [;)]