PROBLEM: "Update Resources" is exceedingly slow

I bought some Vyrso books this evening. Logos didn't automatically recognize that there were new resources for me to download after a restart and waiting a while, so I ran "update resources". It took the better part of half an hour to check for new resources and report back to me that there were new ones available.

Upon examining the log file, I found that most of that time was taken up doing what looked like very redundant checking for overlapping lists of 100 hidden resources at a time (an order N-squared algorithm): for example, first a list of 100 items would show up starting with LLS:DDBDPOLEID, LLS:DDBDPOLUND, LLS:DDBDPOMASADA, LLS:DDBDPOMEDINMADI and going through LLS:DDBDPPGOT, LLS:DDBDPPGRAD. Then another list of 100 items would be shown starting from LLS:DDBDPOLUND, LLS:DDBDPOMASADA, LLS:DDBDPOMEDINMADI and going through LLS:DDBDPPGOT, LLS:DDBDPPGRAD, LLS:DDBDPPGRAUX. Note the list shifted by one item, but repeated all the rest of the ones in between. And so on, over and over through all the hidden files I have (hundreds of them, no doubt).

In between each grouping of 100 output lines is a line that I think points to the flaw in the algorithm: "UpdateManager Requesting all resource files newer than '2011-09-30T07:32:11.5970000'" and then that exact line will show up again for the next grouping, only with a slightly later timestamp, e.g.: "UpdateManager Requesting all resource files newer than '2011-09-30T07:32:19.1300000'." And so on, hundreds of times.

This all starts at 20:06:22.9422 in the log file and continues to 20:33:24.5859 when it finally says "No more resource file info objects retrieved." That means the whole operation took 27 minutes and it hadn't even gotten to checking for updates on my non-hidden resources (thank God that took a lot less time). First of all, why does the UpdateManager have to check for updates on hidden resources at all? And second, and more important: Why can't the UpdateManager check once for all resource files newer than one particular timestamp? This N-squared looping thing is not efficient and seems unnecessary to me. Could someone in development who understands algorithm efficiency computations look into this and figure out if it's really being done as well as it could be?

Log file attached:

8233.SlowUpdateResources.zip

EDIT: I'm running 4.3 SR-4. Also note in my sig that I've got a mongo huge/fast machine so performance times like this are outrageous. Everything else about Logos 4 is lickety split for the most part these days.

Find more posts tagged with

Comments

BillS

I, too, have noticed that it takes a LONG time to check for Resource updates, compared to the time it takes to check for ALL updates... probably the same cause, though I haven't your expertise to diagnose the cause. My machine isn't super fast (specs in signoff), but this is noticeably MUCH slower.

Melissa Snyder

I will submit your report to development. Thanks.

Richard Koons

I have the same probem it is almost like the processes are stuck in a loop I looked at the logging file and can se t goes through the processes of verifying the collection of books I have. My CPU is running at high usage which isn't typical when running Logos. It doesn't show indexing but the performance is as if it is. hope ths helps

Bradley Grainger (Logos)

The "update resources" code would incorrectly determine the next batch of resources to download if the current batch contained many hidden resources. This caused the same information to be downloaded multiple times. (The server doesn't currently track which resources you've hidden; instead the client gets information about all resources that are available (this is pretty small) then ignores the information for hidden resources.)

This will be fixed in the next 4.5 beta.

Allen Browne

The "update resources" code would incorrectly determine the next batch of resources to download if the current batch contained many hidden resources. ... This will be fixed in the next 4.5 beta.

Great news. Thanks, Bradley. Will be glad to have this issue addressed also (in time).

Bradley Grainger (Logos)

This bug should be fixed in 4.5 Beta 12.

Rosie Perera

Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.