PROBLEM: "Update Resources" is exceedingly slow

Page 1 of 1 (18 items)
This post has 17 Replies | 0 Followers

Posts 18867
Rosie Perera | Forum Activity | Posted: Tue, Oct 4 2011 9:13 PM

I bought some Vyrso books this evening. Logos didn't automatically recognize that there were new resources for me to download after a restart and waiting a while, so I ran "update resources". It took the better part of half an hour to check for new resources and report back to me that there were new ones available.

Upon examining the log file, I found that most of that time was taken up doing what looked like very redundant checking for overlapping lists of 100 hidden resources at a time (an order N-squared algorithm): for example, first a list of 100 items would show up starting with LLS:DDBDPOLEID, LLS:DDBDPOLUND, LLS:DDBDPOMASADA, LLS:DDBDPOMEDINMADI and going through LLS:DDBDPPGOT, LLS:DDBDPPGRAD. Then another list of 100 items would be shown starting from LLS:DDBDPOLUND, LLS:DDBDPOMASADA, LLS:DDBDPOMEDINMADI and going through LLS:DDBDPPGOT, LLS:DDBDPPGRAD, LLS:DDBDPPGRAUX. Note the list shifted by one item, but repeated all the rest of the ones in between. And so on, over and over through all the hidden files I have (hundreds of them, no doubt).

In between each grouping of 100 output lines is a line that I think points to the flaw in the algorithm:  "UpdateManager   Requesting all resource files newer than '2011-09-30T07:32:11.5970000'" and then that exact line will show up again for the next grouping, only with a slightly later timestamp, e.g.: "UpdateManager  Requesting all resource files newer than '2011-09-30T07:32:19.1300000'." And so on, hundreds of times.

This all starts at 20:06:22.9422 in the log file and continues to 20:33:24.5859 when it finally says "No more resource file info objects retrieved." That means the whole operation took 27 minutes and it hadn't even gotten to checking for updates on my non-hidden resources (thank God that took a lot less time). First of all, why does the UpdateManager have to check for updates on hidden resources at all? And second, and more important: Why can't the UpdateManager check once for all resource files newer than one particular timestamp? This N-squared looping thing is not efficient and seems unnecessary to me. Could someone in development who understands algorithm efficiency computations look into this and figure out if it's really being done as well as it could be?

Log file attached:

8233.SlowUpdateResources.zip

EDIT: I'm running 4.3 SR-4. Also note in my sig that I've got a mongo huge/fast machine so performance times like this are outrageous. Everything else about Logos 4 is lickety split for the most part these days.

Posts 3690
BillS | Forum Activity | Replied: Wed, Oct 5 2011 8:29 AM

I, too, have noticed that it takes a LONG time to check for Resource updates, compared to the time it takes to check for ALL updates... probably the same cause, though I haven't your expertise to diagnose the cause. My machine isn't super fast (specs in signoff), but this is noticeably MUCH slower.

Grace & Peace,
Bill


MSI GF63 8RD, I-7 8850H, 32GB RAM, 1TB SSD, 2TB HDD, NVIDIA GTX 1050Max
Samsung S9+, 64GB
Fire 10HD 64GB 7th Gen

Posts 4077
Melissa Snyder | Forum Activity | Replied: Wed, Oct 5 2011 8:34 AM

I will submit your report to development. Thanks.

Posts 64
Richard Koons | Forum Activity | Replied: Wed, Oct 5 2011 4:50 PM

I have the same probem it is almost like the processes are stuck in a loop I looked at the logging file and can se t goes through the processes of verifying the collection of books I have. My CPU is running at high usage which isn't typical when running Logos. It doesn't show indexing but the performance is as if it is. hope ths helps

Posts 8227
LogosEmployee

The "update resources" code would incorrectly determine the next batch of resources to download if the current batch contained many hidden resources. This caused the same information to be downloaded multiple times. (The server doesn't currently track which resources you've hidden; instead the client gets information about all resources that are available (this is pretty small) then ignores the information for hidden resources.)

This will be fixed in the next 4.5 beta.

Posts 1663
Allen Browne | Forum Activity | Replied: Fri, Dec 16 2011 5:07 PM

Bradley Grainger:
The "update resources" code would incorrectly determine the next batch of resources to download if the current batch contained many hidden resources. ... This will be fixed in the next 4.5 beta.

Great news. Thanks, Bradley. Will be glad to have this issue addressed also (in time).

Posts 8227
LogosEmployee

This bug should be fixed in 4.5 Beta 12.

Posts 18867
Rosie Perera | Forum Activity | Replied: Sun, Jan 1 2012 2:06 PM

Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.

Posts 8227
LogosEmployee

Rosie Perera:

Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.

It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.

Posts 18867
Rosie Perera | Forum Activity | Replied: Sun, Jan 1 2012 4:44 PM

Bradley Grainger:

Rosie Perera:

Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.

It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.

I didn't mean to sound like I was asking for even better performance. I understand that it's a rare command and am thrilled that you were able to speed it up by a factor of almost 5. Very nice to see some of these old bugs getting fixed in the latest releases! Keep up the good work!

Posts 1663
Allen Browne | Forum Activity | Replied: Sun, Jan 1 2012 7:26 PM

Bradley Grainger:
It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.

Ah: thanks, Bradley.

I've been using update resources to force it to find newly purchased books: clearly inefficient.

The wiki article on Logos 4 Commands does state this quite clearly (at the end), but your post clarifed it. Thanks.

Posts 18867
Rosie Perera | Forum Activity | Replied: Sun, Jan 1 2012 7:34 PM

Allen Browne:
I've been using update resources to force it to find newly purchased books: clearly inefficient.

It's much more efficient to close Logos and open it again and wait a few moments. A popup should come up shortly saying you have new resources to download. It should be just a matter of seconds, not minutes.

Posts 1663
Allen Browne | Forum Activity | Replied: Sun, Jan 1 2012 11:22 PM

Rosie Perera:
Allen Browne:
I've been using update resources to force it to find newly purchased books: clearly inefficient.

It's much more efficient to close Logos and open it again and wait a few moments....

Yes, Rosie. Had found that to be the case.

Posts 15805
Forum MVP
Keep Smiling 4 Jesus :) | Forum Activity | Replied: Mon, Jan 2 2012 2:13 AM

Other option is toggling "Automatically Download Updates" in Program Settings since changing from No to Yes causes check for updates.

Keep Smiling Smile

Posts 8893
fgh | Forum Activity | Replied: Mon, Jan 2 2012 4:05 PM

Bradley Grainger:

Rosie Perera:

Woot! Thanks! It is much faster now. Still takes about 6 minutes for my library of 8243 resources (and many hundreds of hidden resources), but that's far better than nearly half an hour.

It's not intended that "update resources" be run frequently; it should only be necessary in the rare circumstance that the local database has become out-of-sync with the server. In almost all circumstances "update now" will accomplish the same results much more quickly.

Is there some major difference between Mac and Windows going on here, because

  • a) update now never seems to find anything at all for me, unless there's a software update waiting. I've pretty much given up using it.
  • b) my little Mini has never taken anything near half an hour for update resources. 6 minutes sounds a lot more like it.

Rosie Perera:
It's much more efficient to close Logos and open it again and wait a few moments. A popup should come up shortly saying you have new resources to download. It should be just a matter of seconds, not minutes.

That's with your machine. I doubt I could close and reopen Logos in the same time an update resources takes.

"The Christian way of life isn't so much an assignment to be performed, as a gift to be received."  Wilfrid Stinissen

Mac Pro OS 10.9.

Posts 25661
Forum MVP
Dave Hooton | Forum Activity | Replied: Mon, Jan 2 2012 4:15 PM

fgh:
I doubt I could close and reopen Logos in the same time an update resources takes

Ouch!

Dave
===

Windows 10 & Android 8

Posts 18867
Rosie Perera | Forum Activity | Replied: Mon, Jan 2 2012 5:32 PM

fgh:

Is there some major difference between Mac and Windows going on here, because

 

  • a) update now never seems to find anything at all for me, unless there's a software update waiting. I've pretty much given up using it.

I always thought that's what update now was supposed to do, only check for program updates, not resources. I thought you had to do update resources (or restart) to check for resource updates. But maybe I've misunderstood all along. Also, I think there have been fewer mass updates of resources (fixing typos and such) in the past year than there used to be, so maybe it's just that there really haven't been any updates when you've tried the command.

fgh:

  • b) my little Mini has never taken anything near half an hour for update resources. 6 minutes sounds a lot more like it.

Rosie Perera:
It's much more efficient to close Logos and open it again and wait a few moments. A popup should come up shortly saying you have new resources to download. It should be just a matter of seconds, not minutes.

That's with your machine. I doubt I could close and reopen Logos in the same time an update resources takes.

I wonder if that has anything to do with the comparative sizes of our libraries? I've got 8295 resources, plus more than 2000 hidden resources. And maybe the command is indeed faster on the Mac than on Windows. Anyway, I don't mind waiting the 6 minutes. I really don't do it that often, only when I think there's something wrong and a resource I bought hasn't been downloaded for me, which happens extremely infrequently.

Posts 8893
fgh | Forum Activity | Replied: Tue, Jan 3 2012 11:05 AM

Rosie Perera:
maybe it's just that there really haven't been any updates when you've tried the command

No. I've never used it unless I was waiting for something. And when it didn't work, I always tried the other one instead. In fact, shortly after writing the above, I had a download coming and decided to try it out in case something had changed recently, but no: update now didn't find anything, update resources did. 

Rosie Perera:
I wonder if that has anything to do with the comparative sizes of our libraries?

Yes, your library is bigger, but mine is pretty big too by now. And your computer power exceeds mine by far more than your library does, so you'd think that would win out. 

No, I'm not too bothered by that time either. I'd appreciate a faster start up time, though...

"The Christian way of life isn't so much an assignment to be performed, as a gift to be received."  Wilfrid Stinissen

Mac Pro OS 10.9.

Page 1 of 1 (18 items) | RSS