Another plea for optimising the Parallel Resource menu

Page 2 of 3 (45 items) < Previous 1 2 3 Next >
This post has 44 Replies | 3 Followers

Posts 26810
Forum MVP
Dave Hooton | Forum Activity | Replied: Tue, Sep 28 2010 12:55 AM | Locked

Russ Quinn:
1. Is there a way to execute the search as a background process in anticipation of clicking the button instead of waiting on the user to click the button?
Perhaps the list could pre-populate for the current reference or headword whenever scrolling pauses for 5 seconds or so?

You're on track with what could be accomplished:-

  • pre-populate for current reference if there is a pause (and sync'ing doesn't coincide!); and then
  • cache the results so the search does not have to be performed next time for the same reference.

It may not be possible to avoid a delay for the first access of PRS' ie. if you wait 5 secs you've already introduced a delay, even if the PRS is completed in that time!

Russ Quinn:
2. Could the results of the search be listed in the resource window as they are found instead of waiting for the entire search to be completed before any of the relevant resources are displayed? 

That would be messy because the resources need to appear in order of prioritization ie. after all resources have been found.

Dave
===

Windows 10 & Android 8

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Sep 28 2010 12:57 AM | Locked

Dave Hooton:
All Parallel Resources lists only the resources that contain the reference (the New American Standard Dictionary is not in my Collection). My PRS lists the 11 Greek lexicons from the collection of 18, so it has done some filtering! However, 6 lexicons don't even have an Index for Strong's Greek.

I have the opposite problem - almost nothing in my PRS for that reference! Certainly some kind of bug - perhaps with ESL itself?

I might do some more work on this and take it to the other thread if I have time.

Posts 26810
Forum MVP
Dave Hooton | Forum Activity | Replied: Tue, Sep 28 2010 1:35 AM | Locked

Fred Greco:
I also do not use this feature, even though it would be of great benefit almost every time I use Logos. It makes collections more usable, workspaces cleaner, RAM usage less (since less tabs need to be open) and makes commentaries much more valuable.

It's part of the price for having dynamic collections, but I don't think it's a necessary price if we allow a few compromises  ie. don't assume our collections will change every second and even if their content did change in the course of a session is that necessarily bad!? I'm not aware of metadata changes that affect  my collections until something/someone brings it to my attention! But if my collections were indexed at startup this would vastly improve the efficiency of generating PRS's for a given reference. Possible triggers to perform a re-index would include editing a collection and resource metadata changes.

Note that I'm using the word "index" for any pre-defined structure that holds information on PRS's. It has more organisation than a typical cache. Some of your documents are indexed and indexing takes about 2s to 9s from various logs I've seen.

Dave
===

Windows 10 & Android 8

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Sep 28 2010 2:29 AM | Locked

Dave Hooton:

Russ Quinn:
1. Is there a way to execute the search as a background process in anticipation of clicking the button instead of waiting on the user to click the button?
Perhaps the list could pre-populate for the current reference or headword whenever scrolling pauses for 5 seconds or so?

You're on track with what could be accomplished:-

  • pre-populate for current reference if there is a pause (and sync'ing doesn't coincide!); and then
  • cache the results so the search does not have to be performed next time for the same reference.

It may not be possible to avoid a delay for the first access of PRS' ie. if you wait 5 secs you've already introduced a delay, even if the PRS is completed in that time!

This was my initial thought until I started running the tests. But I think the problem is that PRS wasn't designed for Logos 4 out of the box. If you remember, it only got added after we all said how much we missed it. As a consequence the way indexing currently works is highly inefficient for PRS (though I have to say also for BWS, but we're more used to a wait there and run it much less often).

Pre-populating is a good idea, but if each datatype had its own index, rather than being spread over thousands of separate databases, that would solve the problem. (There's a small worry it might not be scalable, but I think it would scale to at least 20,000 records without problems, and more than that should only require a little lateral thinking.)

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Sep 28 2010 3:05 AM | Locked

Dave Hooton:
Needs amending

I lost my edit rights before I posted the last collection, unfortunately.

Dave Hooton:
As I've shown resources that don't even have an index for the reference are included in the PRS.

I think we need to be careful to differentiation between bugs and features by design. I'll take the problems with ESL to your other thread.

Dave Hooton:
What is Stage 7 now?

The file access that I called stage 7, is happening as part of stage 4. So stage 4 should now read "Fourth, for every similar resource with this index type", rather than "Fourth, for every resource with this type".

As you point out, I'd confused resource type with data type. In fact Logos appears to look at both resource type and data type. So Bibles aren't searched when you're looking at commentaries, even though they have the same data-type. But commentaries are searched if you're looking at dictionaries provided the commentary has an English headword index, which most don't.

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Sep 28 2010 3:18 AM | Locked

Dave Hooton:
It's part of the price for having dynamic collections,

That's only a tiny fraction of the problem. We lose about 1-2s because of the dynamic collections.

The reason this is much worse than L3 is because of the "All parallel resources" menu which wasn't there in L3. This means L4 searches ALL our resources, whilst in L3 it only searched resources we'd manually added to a parallel association.

Dave Hooton:
But if my collections were indexed at startup this would vastly improve the efficiency of generating PRS's for a given reference.

It collections queries were cached, the maximum time saving would still only be 2-3 seconds on the first run, and less than 1 second on subsequent runs.

For those who are wondering what happens 'behind the hood', I created this little video demonstrating which files Logos is accessing during a PRS lookup. You'll have to be very quick to see the collections searching going on (look for ResourceCollectionManager.db flashing up a few times between 31s and 37s).

Posts 1559
PL | Forum Activity | Replied: Tue, Sep 28 2010 4:16 AM | Locked

Hi Mark,

Wow, that's really slow for your large library.  Thanks for posting a video clip to show that.  Two thoughts:

1) Perhaps not everyone needs the "All parallel resources" drop down - for those who set up their own resource collections (like yourself), perhaps offering the option to not show "All parallel resources" is a quick interim fix?

2) You mentioned the possibility of adding another index with all the head words as a long-term fix to this problem.  I'm concerned that this will further increase the indexing time AND frequency for us.

Thanks,

Peter

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Sep 28 2010 4:48 AM | Locked

Peter Li:
1) Perhaps not everyone needs the "All parallel resources" drop down - for those who set up their own resource collections (like yourself), perhaps offering the option to not show "All parallel resources" is a quick interim fix?

Yes, that would speed things up and you would imagine be very easy to achieve. Even the largest of my collections would only represent about a quarter of "All parallel resources".

Peter Li:
2) You mentioned the possibility of adding another index with all the head words as a long-term fix to this problem.  I'm concerned that this will further increase the indexing time AND frequency for us.

I'm not sure it needs another index (although without fully understanding the workings of Logos I can't be sure). I'm suggesting replacing the existing indexes with new, larger ones.

Currently, each resource has it's own index file. Within that file are multiple indexes, one for each headword/datatype supported by that resource. So commentaries normally have a Bible index and a volume/page index, within that one file. 1,000 commentaries means 1,000 files. I'm suggesting having two files instead. One Bible index file, and one volume/page file. Each file would store one index. So instead of running 1,000 queries, taking perhaps 0.05s each (50 seconds), you'd only need to run one query which would take (very roughly) about 0.25seconds. (You'd also need additional files for all the other datatypes, so you'd end up with perhaps a few dozen files instead of a few thousand.)

Because I'm suggesting these indexes replace the existing ones (it's exactly the same data, just stored in a different way), I don't believe it would increase indexing time or frequency.

Posts 26810
Forum MVP
Dave Hooton | Forum Activity | Replied: Tue, Sep 28 2010 5:18 AM | Locked

Mark Barnes:
Currently, each resource has it's own index file. Within that file are multiple indexes, one for each headword/datatype supported by that resource. So commentaries normally have a Bible index and a volume/page index, within that one file. 1,000 commentaries means 1,000 files. I'm suggesting having two files instead. One Bible index file, and one volume/page file. Each file would store one index. So instead of running 1,000 queries, taking perhaps 0.05s each (50 seconds), you'd only need to run one query which would take (very roughly) about 0.25seconds.

Isn't that fulfilled by the LibraryIndex?

EDIT:  Ok  - you mean the mstidx file in ResourceManager\logos!

Dave
===

Windows 10 & Android 8

Posts 10826
Forum MVP
Jack Caviness | Forum Activity | Replied: Tue, Sep 28 2010 5:35 AM | Locked

Peter Li:
1) Perhaps not everyone needs the "All parallel resources" drop down - for those who set up their own resource collections (like yourself), perhaps offering the option to not show "All parallel resources" is a quick interim fix?

That would be good for me. I never use the "All Parallel Resources". I am only interested in the collections where I have checked use as PRS.

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Sep 28 2010 6:01 AM | Locked

Dave Hooton:
Isn't that fulfilled by the LibraryIndex?

I'm not able to understand the format of LibraryIndex (I suspect it's proprietary). But LibraryIndex is used for searching, and is entirely separate from the Datatype/Headword indexing which is used for lookups. I think this is one of the reasons why it's not straightforward to add headword:term into the default search syntax.

So if you do a search for <HebrewStrongs = Strong’s Hebrew #1288> it will use LibraryIndex. But if you do a PRS lookup on the same number it will use the .mstidx files. I don't think LibraryIndex differentiates between headwords and occurrences within the text.

Posts 9137
LogosEmployee
Bradley Grainger (Faithlife) | Forum Activity | Replied: Tue, Sep 28 2010 10:23 AM | Locked

Mark,

Thanks for the performance analysis (which is spot on) and for sparking a conversation showing us that this feature is clearly useful and important to many users. Many problems could be solved (as you point out) by creating a global keylink index; in fact, we have a case in our bug tracking system about this that predates (I think) the PRS feature. As with any addition to an already complex system, there are some complicating factors that would make it a little tricky to implement. It wouldn't be as difficult as creating the Sentence Diagrammer, or optimising phrase searching, but certainly not something that could just be whipped out in a day. As the missing features list gets smaller, I hope we will have more time to focus on performance; it's clear that this is an important issue for many of you.

Posts 13419
Mark Barnes | Forum Activity | Replied: Tue, Sep 28 2010 10:49 AM | Locked

Thanks for the feedback, Bradley. We trust you guys to come up with good solution once you're able to focus on this sort of thing.

Posts 9488
Forum MVP
Mark Smith | Forum Activity | Replied: Tue, Sep 28 2010 11:02 AM | Locked

Bradley Grainger:
it's clear that this is an important issue for many of you.

Yes Thanks for letting us know you will take a look at this.

Thanks to Mark for championing this cause!

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 3578
steve clark | Forum Activity | Replied: Tue, Sep 28 2010 11:06 AM | Locked

Thanks Mark for bringing this to the foreground.

Thanks Bradley for letting us know it is not forgotten.

i use PRS frequently, which prevents me from having too many resources open.

QLinks, Bibl2, LLR, Macros
Dell Insp 17-5748, i5, 1.7 GHz, 8G RAM, win 8.1

Posts 26810
Forum MVP
Dave Hooton | Forum Activity | Replied: Tue, Sep 28 2010 10:35 PM | Locked

Bradley Grainger:

Mark,

Thanks for the performance analysis (which is spot on) and for sparking a conversation showing us that this feature is clearly useful and important to many users. Many problems could be solved (as you point out) by creating a global keylink index;

Yes. It's only when you run the tests Mark has performed that you realise the nature of the inefficiencies with the current system. So thanks to Mark, and to Bradley for his confirmation and indication of a performance improvement in the (near) future.

Dave
===

Windows 10 & Android 8

Posts 2237
Damian McGrath | Forum Activity | Replied: Tue, Sep 28 2010 10:58 PM | Locked

Bradley Grainger:
As the missing features list gets smaller, I hope we will have more time to focus on performance; it's clear that this is an important issue for many of you.

As someone who doesn't miss the missing features, I'm all for performance improvements. It's my number one issue (and, I notice, the third most important issue on uservoice)

 

I would also like to thank Mark for his efforts here. It's been fascinating, truly.

Posts 179
Trey Selman | Forum Activity | Replied: Wed, Sep 29 2010 6:29 AM | Locked

Bradley, Thank you for your reply.

As Mark said, we trust you to come up with a good solution.That trust comes easy from the years of faithful work.

Each day I look forward to seeing how you make a great tool better.

Again Mark, thank you very much for your diligence, love for God, care for the tool that aids in knowing Him more, and your care and love for others.

Posts 363
Wyn Laidig | Forum Activity | Replied: Wed, Sep 29 2010 11:01 AM | Locked

I agree that PRS is nearly unusable as it is now.  I used this ALL THE TIME in L3, and is one of  the two major reasons I have a hard time accepting L4.  (The other reason is the slowness of the editor when writing notes!)  I hope Logos will invest time to solve these issues quickly.

Posts 386
Clinton Thomas | Forum Activity | Replied: Wed, Sep 29 2010 8:44 PM | Locked

Wyn Laidig:
I agree that PRS is nearly unusable as it is now.

The word 'nearly' should not be in that sentence!

In June we were told in another thread that "Development is working on optimizing the parallel resource sets. I'll add this thread to the case. Thanks."

They have not done a good job.

Page 2 of 3 (45 items) < Previous 1 2 3 Next > | RSS