PROBLEM: "Update Resources" is exceedingly slow
I bought some Vyrso books this evening. Logos didn't automatically recognize that there were new resources for me to download after a restart and waiting a while, so I ran "update resources". It took the better part of half an hour to check for new resources and report back to me that there were new ones available.
Upon examining the log file, I found that most of that time was taken up doing what looked like very redundant checking for overlapping lists of 100 hidden resources at a time (an order N-squared algorithm): for example, first a list of 100 items would show up starting with LLS:DDBDPOLEID, LLS:DDBDPOLUND, LLS:DDBDPOMASADA, LLS:DDBDPOMEDINMADI and going through LLS:DDBDPPGOT, LLS:DDBDPPGRAD. Then another list of 100 items would be shown starting from LLS:DDBDPOLUND, LLS:DDBDPOMASADA, LLS:DDBDPOMEDINMADI and going through LLS:DDBDPPGOT, LLS:DDBDPPGRAD, LLS:DDBDPPGRAUX. Note the list shifted by one item, but repeated all the rest of the ones in between. And so on, over and over through all the hidden files I have (hundreds of them, no doubt).
In between each grouping of 100 output lines is a line that I think points to the flaw in the algorithm: "UpdateManager Requesting all resource files newer than '2011-09-30T07:32:11.5970000'" and then that exact line will show up again for the next grouping, only with a slightly later timestamp, e.g.: "UpdateManager Requesting all resource files newer than '2011-09-30T07:32:19.1300000'." And so on, hundreds of times.
This all starts at 20:06:22.9422 in the log file and continues to 20:33:24.5859 when it finally says "No more resource file info objects retrieved." That means the whole operation took 27 minutes and it hadn't even gotten to checking for updates on my non-hidden resources (thank God that took a lot less time). First of all, why does the UpdateManager have to check for updates on hidden resources at all? And second, and more important: Why can't the UpdateManager check once for all resource files newer than one particular timestamp? This N-squared looping thing is not efficient and seems unnecessary to me. Could someone in development who understands algorithm efficiency computations look into this and figure out if it's really being done as well as it could be?
Log file attached:
EDIT: I'm running 4.3 SR-4. Also note in my sig that I've got a mongo huge/fast machine so performance times like this are outrageous. Everything else about Logos 4 is lickety split for the most part these days.
Comments
-
I, too, have noticed that it takes a LONG time to check for Resource updates, compared to the time it takes to check for ALL updates... probably the same cause, though I haven't your expertise to diagnose the cause. My machine isn't super fast (specs in signoff), but this is noticeably MUCH slower.
Grace & Peace,
Bill
MSI GF63 8RD, I-7 8850H, 32GB RAM, 1TB SSD, 2TB HDD, NVIDIA GTX 1050Max
iPhone 12 Pro Max 512Gb
iPad 9th Gen iOS 15.6, 256GB0 -
I will submit your report to development. Thanks.
0 -
I have the same probem it is almost like the processes are stuck in a loop I looked at the logging file and can se t goes through the processes of verifying the collection of books I have. My CPU is running at high usage which isn't typical when running Logos. It doesn't show indexing but the performance is as if it is. hope ths helps
0 -
The "update resources" code would incorrectly determine the next batch of resources to download if the current batch contained many hidden resources. This caused the same information to be downloaded multiple times. (The server doesn't currently track which resources you've hidden; instead the client gets information about all resources that are available (this is pretty small) then ignores the information for hidden resources.)
This will be fixed in the next 4.5 beta.
0 -
Bradley Grainger said:
The "update resources" code would incorrectly determine the next batch of resources to download if the current batch contained many hidden resources. ... This will be fixed in the next 4.5 beta.
Great news. Thanks, Bradley. Will be glad to have this issue addressed also (in time).
0 -
This bug should be fixed in 4.5 Beta 12.
0 -
Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.
0 -
Rosie Perera said:
Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.
It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.
0 -
Bradley Grainger said:Rosie Perera said:
Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.
It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.
I didn't mean to sound like I was asking for even better performance. I understand that it's a rare command and am thrilled that you were able to speed it up by a factor of almost 5. Very nice to see some of these old bugs getting fixed in the latest releases! Keep up the good work!
0 -
Bradley Grainger said:
It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.
Ah: thanks, Bradley.
I've been using update resources to force it to find newly purchased books: clearly inefficient.
The wiki article on Logos 4 Commands does state this quite clearly (at the end), but your post clarifed it. Thanks.
0 -
Allen Browne said:
I've been using update resources to force it to find newly purchased books: clearly inefficient.
It's much more efficient to close Logos and open it again and wait a few moments. A popup should come up shortly saying you have new resources to download. It should be just a matter of seconds, not minutes.
0 -
Rosie Perera said:Allen Browne said:
I've been using update resources to force it to find newly purchased books: clearly inefficient.
It's much more efficient to close Logos and open it again and wait a few moments....
Yes, Rosie. Had found that to be the case.
0 -
Other option is toggling "Automatically Download Updates" in Program Settings since changing from No to Yes causes check for updates.
Keep Smiling [:)]
0 -
Bradley Grainger said:Rosie Perera said:
Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.
It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.
Is there some major difference between Mac and Windows going on here, because
- a) update now never seems to find anything at all for me, unless there's a software update waiting. I've pretty much given up using it.
- b) my little Mini has never taken anything near half an hour for update resources. 6 minutes sounds a lot more like it.
Rosie Perera said:It's much more efficient to close Logos and open it again and wait a few moments. A popup should come up shortly saying you have new resources to download. It should be just a matter of seconds, not minutes.
That's with your machine. I doubt I could close and reopen Logos in the same time an update resources takes.
Mac Pro (late 2013) OS 12.6.2
0 -
Dave
===Windows 11 & Android 13
0 -
fgh said:
Is there some major difference between Mac and Windows going on here, because
- a) update now never seems to find anything at all for me, unless there's a software update waiting. I've pretty much given up using it.
I always thought that's what update now was supposed to do, only check for program updates, not resources. I thought you had to do update resources (or restart) to check for resource updates. But maybe I've misunderstood all along. Also, I think there have been fewer mass updates of resources (fixing typos and such) in the past year than there used to be, so maybe it's just that there really haven't been any updates when you've tried the command.
fgh said:- b) my little Mini has never taken anything near half an hour for update resources. 6 minutes sounds a lot more like it.
Rosie Perera said:It's much more efficient to close Logos and open it again and wait a few moments. A popup should come up shortly saying you have new resources to download. It should be just a matter of seconds, not minutes.
That's with your machine. I doubt I could close and reopen Logos in the same time an update resources takes.
I wonder if that has anything to do with the comparative sizes of our libraries? I've got 8295 resources, plus more than 2000 hidden resources. And maybe the command is indeed faster on the Mac than on Windows. Anyway, I don't mind waiting the 6 minutes. I really don't do it that often, only when I think there's something wrong and a resource I bought hasn't been downloaded for me, which happens extremely infrequently.
0 -
Rosie Perera said:
maybe it's just that there really haven't been any updates when you've tried the command
No. I've never used it unless I was waiting for something. And when it didn't work, I always tried the other one instead. In fact, shortly after writing the above, I had a download coming and decided to try it out in case something had changed recently, but no: update now didn't find anything, update resources did.
Rosie Perera said:I wonder if that has anything to do with the comparative sizes of our libraries?
Yes, your library is bigger, but mine is pretty big too by now. And your computer power exceeds mine by far more than your library does, so you'd think that would win out.
No, I'm not too bothered by that time either. I'd appreciate a faster start up time, though...
Mac Pro (late 2013) OS 12.6.2
0
