A plea for speeding up the "select resource" drop down in search

When I'm conducting a search, it currently takes between 10.3s (cached) and 18.5s (uncached) to populate the dropdown menu where I can specify which resources I want to search in. The menu lists Open Resources, Collections, Tags, Ratings, and Series, but the code is clearly also reading in every book from my library to make the filtering quicker.

Looking at ProcessMonitor, pretty much all the time is spent querying catalog.db, but:

It should only take a fraction of a second to run a query on catalog.db
catalog.db is already cached to display the library. Cannot this cache be consulted?

Find more posts tagged with

Comments

Rich DeRuiter

When I'm conducting a search, it currently takes between 10.3s (cached) and 18.5s (uncached) to populate the dropdown menu

This takes about 1s on my system. (My library shows 4561 resources.)

Just wondering: does your anti-virus do active scanning of files before opening? I set the entire Logos4 directory (with subdirectories) in my 'Exclusions' list in Avast for its "Real Time Scanning."

Keep Smiling 4 Jesus :)

[Y] Logos User Voice suggestion => Improve Logos 4 Menu Responsiveness

Keep Smiling [:)]

Mark Barnes

This takes about 1s on my system. (My library shows 4561 resources.)

Wow. I have a few more (6,622), but that doesn't explain the differences. Maybe there's some kind of bug where 5,000 or 6,000 resources pushes it over the edge and it gets horribly slow.

Can I ask what happens to Logos' memory usage when you click that menu? Mine goes up an additional 380Mb every time I click on it (the memory is released when I execute a search).

Mark Smith

When I'm conducting a search, it currently takes between 10.3s (cached) and 18.5s (uncached) to populate the dropdown menu where I can specify which resources I want to search in.

I've noticed this, too. When doing a Bible search it takes about 16 seconds to populate the list so I can choose a Bible. I don't know what can be done, but perhaps cache the last list generated and only reset the cache if I change collections or add resources. Something, at any rate to make this almost instantaneous.

Rich DeRuiter

Can I ask what happens to Logos' memory usage when you click that menu? Mine goes up an additional 380Mb every time I click on it (the memory is released when I execute a search).

It's kind of hard to tell, because it goes so fast, but it doesn't look like the Private jumps more than about 10MB, and the Commit much more than 20GB. My entire Private RAM usage on the Resource Monitor in Win7 is 381GB, and in Commit is 458GB (both with L4 idle), Working Set is at about 485GB.

My physical memory is at about 65% (of 6GB RAM), because I have a lot running at the same time at the moment.

Are you running any guides, or anything else that might be using the library cache? (Another wild guess.)

Bradley Grainger (Logos)

When I'm conducting a search, it currently takes between 10.3s (cached) and 18.5s (uncached) to populate the dropdown menu where I can specify which resources I want to search in.

I'm not able to reproduce this delay (on a test system with thousands of resources and several hundred custom collections).

Are you in Basic, Bible, or Morph Search? (Roughly) How many collections, unique tags, open resources do you have?

Mark Smith

I reran this a few times and noticed the only time the delay comes up is the first time I open the Search dialog box. after opening Logos After that caching seems to occur and the drop-down list populates almost instantaneously. Sometimes upon opening Logos 4 I have almost no delay in the list being populated from the start. There must be something in the background causing the delay when I notice it. I did not monitor what else was going on in the background.

Fred Chapman

Mark my experience is about the same as Richard. I wonder if it has more to do with the number of collections and tags you have.

Rosie Perera

I've noticed this recently too. Here's a log file from 4.5a Beta2 when it took 13 seconds just to populate the list of Bibles in a new Bible Search tab: 800067.Logos4.log

Disk hits are going through the roof during this time. Why? Shouldn't this list be created and cached for future use shortly after boot-up during some idle time, as it's one of the most likely-to-be used lists in Logos, and isn't likely to change often.

Mark Barnes

I'm not able to reproduce this delay (on a test system with thousands of resources and several hundred custom collections).

Are you in Basic, Bible, or Morph Search? (Roughly) How many collections, unique tags, open resources do you have?

The problem occurs in Basic or Bible Search. Morph Search is near-instantaneous (<1s).
The problem occurs regardless of how many resources are open (the tests above are with no resources open).
I have 84 collections, but I don't think the problem is with collections, because Resource Manager shows the time being spent accessing catalog.db
I only have 14 unique tags, though 1,985 resources are tagged.
The only things that may be slightly unusual about my library (apart from its size) is that almost every one of my resources are rated (6,432 resources), that I have a lot of PB Bibles (approaching 100), and that I have higher than average hidden resources (368 of them).

When I click on the drop-down menu I experience the following:

RAM Usage jumps from around 200Mb to around 600Mb (the RAM is regained only when I do a search or select a resource from the menu; if I just click off the menu without selecting, the RAM is not regained).
CPU Usage jumps to 50% (shared between the two cores) whilst the menu is being populated
Process Monitor shows thousands of reads from catalog.db

Thanks for looking into this. It really is quite painful at the moment.

Rich DeRuiter

Disk hits are going through the roof during this time. Why?

I saw this and went to check on my system. I saw a similar amount of disk hits when I was doing nothing in Logos (just booted it up and literally didn't click on anything. So I decided to close Firefox and Thunderbird (the only apps I have running - except for background/tray apps like anti-virus, etc.). Disk hits went way down.

Then I ran a search. There was some disk activity, but not much. Activating the Logos window (by clicking on it to make it the foreground app), I saw a huge jump in network activity. I let that die down and clicked on the search icon and the on the "Entire Library" text. There was minimal disk activity; some but nothing like what you're showing.

Getting back to another theory mentioned, how many collections do you have? (I just counted 48 in my system)

Also how many tags? (Sorting my Library by tags, I count 26)

Mark Barnes

I'm not able to reproduce this delay (on a test system with thousands of resources and several hundred custom collections).

Are you in Basic, Bible, or Morph Search? (Roughly) How many collections, unique tags, open resources do you have?

Very interesting. I just tried this on my other desktop (similar specs to this one), and have a near instantaneous response. So there's something particular on this machine that's causing the problem. The only thing I can think that's different is all those PB Bibles I have.

Rosie Perera

The only thing I can think that's different is all those PB Bibles I have.

I have one PB Bible on this machine. I wonder if that has anything to do with it.

Rich DeRuiter

The only thing I can think that's different is all those PB Bibles I have.

I have one PB Bible on this machine. I wonder if that has anything to do with it.

I do not have any PB Bibles. (If that helps.)

EDIT: I believe the PB's have their own index (right?). If so, can that index be rebuilt (and so optimized)?

Bradley Grainger (Logos)

Very interesting. I just tried this on my other desktop (similar specs to this one), and have a near instantaneous response. So there's something particular on this machine that's causing the problem. The only thing I can think that's different is all those PB Bibles I have.

I don't have a lot of PB Bibles on my test system either, which may be why I can't reproduce the bug. I suppose I can try building 100 different copies of Tyndale to see if that slows things down...

Rosie Perera

Very interesting. I just tried this on my other desktop (similar specs to this one), and have a near instantaneous response. So there's something particular on this machine that's causing the problem. The only thing I can think that's different is all those PB Bibles I have.

I don't have a lot of PB Bibles on my test system either, which may be why I can't reproduce the bug. I suppose I can try building 100 different copies of Tyndale to see if that slows things down...

I only have one PB bible and am seeing this issue. Try building lots of collections instead, using tagging.

Bradley Grainger (Logos)

Very interesting. I just tried this on my other desktop (similar specs to this one), and have a near instantaneous response. So there's something particular on this machine that's causing the problem. The only thing I can think that's different is all those PB Bibles I have.

I don't have a lot of PB Bibles on my test system either, which may be why I can't reproduce the bug. I suppose I can try building 100 different copies of Tyndale to see if that slows things down...

PB Bibles are the culprit. (Some of the automatically-generated metadata is extremely poor, which causes this performance problem when it's read back from catalog.db.) You'll probably have to rebuild all the Bibles (once) to regenerate this metadata once the bug is fixed.