Still unacceptably slow: populating Bible Search dropdown

Mark Barnes said:

Thanks. I think that suggests yours is working OK. The slow first time was caused mainly by Logos updating community metadata whilst you were waiting for the dropdown menu. Often the new metadata caused it to start recalculating again.

Thanks Mar, this is  interesting. I looked through the log, and updating 100 resources can take from 4 milliseconds to 400 milliseconds. I assume the variation is due to internet performance? I have a fast internet, that could easily add 5+ seconds for someone.

Mark Barnes said:

Let's test the hypothesis: Users with slow dropdowns have lots of tags, users with fast dropdowns have few tags?

Could someone define how we're counting tags? Is it number of distinct tag values, irrespective of how many resources to which they're applied?

If so, I'm sure I have a lot. I tag every new resource with at at least the date I purchased it. That's going to account for at least 100 distinct tags, I'd guess. I'm not at my Logos PC right now. What's the easiest way to get that count?

Thanks,

Donnie

Graham Criddle said:

Donnie Hale said:

What's the easiest way to get that count?

I ended up sorting my Library by "MyTags" and counting the entries manually.

I count the number of entries on one screen and then paged down and multiplied the number of entries per page by the number of pages (plus the extras).

I have 2600 resources not hidden. 162 collections and 111 unique tags.  Near instantaneous dropdown for me.

I also never use an asterisk wildcard in any of my collections. (I haven't seen this mentioned on this thread yet, but it's a performance killer)

For what it is worth, I have the same issue on an i7 Windows box,  It takes on average between 25-30 seconds.  I have 3,297 resources and 22 collections. 

You know I was thinking...when I was programming in Windows, we would often populate and display something in a drop box before the data fetching would finish, when there was  a lot to populate - as the person likely won't scroll faster than you can populate it anyway. In other words - get about 10 entries for the dropdown then display it/refresh the UI  - get 10 more - refresh the display -- it's more UI display code, but it will look almost instantaneous. Often we want to type something in like "ESV", and the population is meaningless anyway. 

Just a thought if it helps ...who knows the code may work like this now anyway.

Rosie Perera said:

So, having lots of tags doesn't appear to be the main issue.

Because I only have 10 different tags, I will agree.  What I do have lots of are notes.  My notes db file is over 2G.

Rosie Perera said:

Tom, you mentioned in a previous post that you have an older computer. What are the specs on that computer (what processor, how much RAM, how fast is the hard disk)? That is more likely the cause of your menu taking nearly 4 minutes (!) to populate. Yikes. I would have given up by then.

There is no doubt having a six year old computer (I tend to keep mine for 8 years before upgrading) has a lot to do with the slowness of L4.

I have an Intel Core(TM)2 Duo CPU @ 2.0 GHz,

4 GB of Ram

and 500 GB Hard drive  @7200

I was wondering if Logos was doing things with things with my notes in the background.

But I have concluded that my notes are not the issue because if I use the "same" drop down box to filter my library, it is spontaneous. 

tom said:

I have an Intel Core(TM)2 Duo CPU @ 2.0 GHz,

4 GB of Ram

and 500 GB Hard drive  @7200

I just tried the Bible dropdown on one of my older PCs. It's still a better spec than yours, but not by that much. Mine is an Intel Core 2 Quad Q8300 @2.5 Ghz, 6GB RAM, and a 7,200rpm HDD.

My menu populates in under 9 seconds the first time of asking, and in about 1.5s subsequently.

It's bizarre that for some, performance is quite acceptable, whilst its unusable for others with apparently similar setups. Tom, apart from your notes, is there anything else about your configuration that you think might be unusual?

Bradley Grainger (Logos) said:

For example, tom's situation doesn't seem to be covered by these two issues.

Here is my log file.  I started L5, open up the search window, change to one of my collections, and then did a search on the word grace. 0654.Logos.log

tom said:

I started L5, open up the search window, change to one of my collections, and then did a search on the word grace

You have two or three very inefficient collections:

• * ANDNOT type:media ANDNOT type:lectionary
• *
• *

They are really not helping, and are probably the biggest cause of your slowdown.

It's not clear from the logs whether the two collections that just have * in the rule are two separate collections, or whether you're including the * collection inside another collection.

To remove this problem, at least, all you need to do is replace the * with rating:<6. That should make a noticeable difference for you.

Dave Hooton said:

It's not that simple! Your CPU has a higher Turbo clock speed  3.9 vs 3.6, a higher nominal clock of 3.5 vs  3.33 and is less thermally stressed which means that it is more likely to run a  thread/core at Turbo speed with other cores active. Resource monitor actually shows yours running at 4.3 GHz (overclocked?) for most of the operation whereas Rosie's took a while to ramp up to 100%.

I did say that I'd simplified the example!

What I was trying to say was that generally speaking if two processors performed equally well when using all their cores, then in a single-threaded operation, the processor with the fewer cores will perform better. I know that there are many additional factors involved (architecture, cache, clock speed, turbo speed, and so on), but assuming that the 6-core and 4-core have the same overall performance, the 6-core will have inferior single-threaded performance.

Mine and Rosie's CPUs are an example of that. Rosie's processor beats mine on Cinebench multi-threaded (8.91 vs 7.91). Now I know there are other factors involved, but given that performance you would expect my processor to beat hers in single threaded mode, because 8.91÷6 is 33% less than 7.91÷4. And, indeed, when you do the benchmark in single threaded mode, mine is 33% faster (1.24 vs 1.66). The maths doesn't quite work because of other factors.

As for the 4.3GHz in Task Manager - that's not accurate. My CPU is not overclocked, and according to Intel's Extreme Tuning Utility, it never gets above 3.9Ghz.

tom said:

The menu now responds in about one minute (it reduce the time by aprox 75%)

I'm glad that improves things. Hopefully Bradley can speed things up still further.

The other rules you have won't make much difference on their own - it's just that you have a lot of collections, and nest collections quite a lot.

That said, one you might experiment with is title:("study bible", "sfl") OR subject:(q, papyri, targum) OR mytag:no OR author:("Hayford, Jack W") OR title:"(Latin)" OR title:"(Greek)" OR "Perseus" OR title:("Bible Studies") OR title:(SAMPLE).

I presume that's a collection of resources you're not that interested in, which put in the 'exclude' section of many other collections. Consequently, it's called 90 times, each time taking an average of 0.2 seconds to calculate, which makes it one of the slowest of your remaining collections to calculate.

I don't know why it's on the slow side, but personally, I rate resources like this with 1 or 2 stars, and then creating a collection with the simple rule rating:<3. That seems to be pretty efficient.

Normally I wouldn't both suggesting tweaking a rule that 'only' took 0.2 seconds to run, but as this one is called 90 times, if you can half its calculation time, you'll save yourself at least another 9 seconds.

Incidentally, this does indicate another area in which some optimisation could be done by Bradley and the team. It appears that nested collections are still being calculated multiple times instead of being cached.

Bradley Grainger (Logos) said:

I have seen specs (but no code yet) for better tag management in the Library. It is currently planned (no guarantees, all plans subject to change) for a release in the near future.

This is really good news. I had all but abandoned tagging due to the present inelegant management system. This is very welcome news!

Thanks, Bradley. 

[:D]

Rosie Perera said:

It's helpful if you include some context of what you were responding to, especially when your response comes way later in the thread than the post you are replying to. You can quote the person you're replying to by selecting the relevant portion of their post when you're in Compose mode, and clicking Quote.

I believe Rick was responding to Mark's question which was specifically addressed to him on the 7th. You can find that out by clicking on the 'replied' link in the heading of his post (you probably knew that already).

(edit: sorry I see Mark replied while I was typing my post)