Indexing: time to think of a better approach

I had the same experience. The Wiersbe book was a pain to index.

What if:

Message from Logos: there are new or updated resources ready to load - Is this a good time to load them?  

- if you answer no then they will not be loaded at this time

- if you answered no then open the library there will be a banner stating that there are resources waiting to be loaded.

Message from Logos: Logos needs to run the index program - is this a good time? 

- if you answer no then indexing will not run at this time

- if you did not let the indexer run then all searches will have the banner stating the not all results will be shown.

Why: You just entered the dock [pulpit] and while giving your opening remarks and prayer you opened Logos  

Whoops - big down load and index start.  And you just wanted to give your congregation the revelations that God had given you. [Your sermon]

At that time you do not need the new resources nor indexing of them.  Maybe this evening.   And maybe then you will rerun the search you did for that sermon just to see what new insights were added.

David Ames said:

Message from Logos: Logos needs to run the index program - is this a good time? 

David Ames said:

Message from Logos: there are new or updated resources ready to load - Is this a good time to load them?  

I like these ideas David

Also what might be helpful in conjunction with these ideas is if rather than simply being able to nominate certain hours during the day in which downloads occur as it the current option in settings, but be able to setup a weekly schedule of hours when dowloads can occur.  Using the scenario you raised David.it might be desireable for a preacher to be able to setup a schedule where they never download any updates at anytime on a Sunday and for example's sake lets say Thursday because on Thursday you are doing the bulk of your sermon preparation but on Mondays and Tuesdays you are either in staff meetings or out doing visitations so you are fine with downloads occuring at anytime on these days becuase whatever access you need to your library, your tablet and or smart phone will suffice on those days if your libary is indexing. On the other days of the week you might to want to block out half of the day.

Equally for different reasons a professor / teacher or a student may want to prevent certain day / hours duriing the week where downloads should not occur but are ok with it happening at any other time.

Mark Barnes said:

Rick Ausdahl said:

if Faithlife does not already do this, would it be possible to run the indexer as a separate executable that could have it's CPU priority individually set (in Logos program settings)

They already do both of these things. The indexer is a separate executable. All the threads begun by the indexer run at low priority.

The problem isn't even CPU for most users. CPU is unlikely to reach full utilisation (and therefore will have spare cycles for other uses). For the majority of users, it's SSD/HDD usage that will get maxed out and cause slow down elsewhere.

My library is small (apx. 2500 resources) compared to the libraries many other have, so I've never been hit as hard by the indexer as those with large libraries.  But even with a smallish library, on the 6 year old hand-me-down laptop I inherited from my wife after getting her a new one, the time required for indexing and the impact on system response was painful.  Installing a SSD on the old machine did help general Logos performance considerably, but even then, with the older AMD Turion 64 x2 processor in that machine, the SSD upgrade was not enough to take the pain out of the performance hit when the indexer kicked in.  Things are much better (with my small library) after biting the bullet and purchasing a new machine for myself, but I feel the pain others are still going through.

I realize the storage drive--especially a traditional HDD--is the real bottleneck when it comes to indexing, but even so, I thought that if the CPU priority could be set low enough, that it might still reduce the impact on the drive, thereby increasing overall Logos performance.

Since the indexer is already a separate executable set to low priority and people still feel the pain, would it be possible to reduce the impact even further by having options to have an "intermittent" setting by which the indexer would only engage/run at set time intervals to slow it down even further?  Or... perhaps for those who are more concerned with Logos performance than say the performance of other apps like email and web browsing, to have an option that would only allow the indexer to run when the Logos app is not running.

In any event, my thoughts are not currently for myself, but for others who are more impacted by the indexer than I am.

Rick Ausdahl said:

Since the indexer is already a separate executable set to low priority and people still feel the pain, would it be possible to reduce the impact even further by having options to have an "intermittent" setting by which the indexer would only engage/run at set time intervals to slow it down even further?

At least one of the problems here is that a large part of the time in merging is running a single SQLite query through a third-party library. If it was multiple queries, Logos could space them out to reduce the bottleneck. But there's no easy way of reducing the performance impact of a single command.

Theoretically, reducing the dependence on SQLite would give Faithlife many more options, but it's unrealistic to think that Faithlife developers on their own could develop a more efficient database system than hundreds of SQLite developers — at least without a massive budget.

A more realistic option would be to switch to another third party library such as Sphinx, Lucene or Xapian. (Bradley has also shown interest in Gigablast in the past.) I don't know enough about those libraries to know how realistic that is, although Lucene has been ported to .NET, which might help. Faithlife do have developers with understanding in these areas, so we can only hope that they're working on it, or at least thinking about it. But it would mean a complete re-architecturing of the indexing system, which would be a major investment that may not be possible.

Mark Barnes said:

But it would mean a complete re-architecturing of the indexing system, which would be a major investment that may not be possible.

i think it is very unlikely that Faithlife would re-architect its SQL data structure on the PC-based perpetual licensed product. There is too much product risk and Faithful doesn't even tag resources without getting enough pre-orders to cover at least some of the development costs. I think a more likely approach with be to use multi-model databases in the cloud to provide better search. However, this means that those improvements will be limited to Logos Now subscription customers. Such is the problem of using a version 7 product.

IMHO.

Agape,

Steve

Mark Barnes said:

Rick Ausdahl said:

Since the indexer is already a separate executable set to low priority and people still feel the pain, would it be possible to reduce the impact even further by having options to have an "intermittent" setting by which the indexer would only engage/run at set time intervals to slow it down even further?

At least one of the problems here is that a large part of the time in merging is running a single SQLite query through a third-party library. If it was multiple queries, Logos could space them out to reduce the bottleneck. But there's no easy way of reducing the performance impact of a single command.

Theoretically, reducing the dependence on SQLite would give Faithlife many more options, but it's unrealistic to think that Faithlife developers on their own could develop a more efficient database system than hundreds of SQLite developers — at least without a massive budget.

A more realistic option would be to switch to another third party library such as Sphinx, Lucene or Xapian. (Bradley has also shown interest in Gigablast in the past.) I don't know enough about those libraries to know how realistic that is, although Lucene has been ported to .NET, which might help. Faithlife do have developers with understanding in these areas, so we can only hope that they're working on it, or at least thinking about it. But it would mean a complete re-architecturing of the indexing system, which would be a major investment that may not be possible.

Thanks, Mark--that explains a lot--probably even explains the reason behind Denise's observation that although the indexer's priority may be set low, it still seems to have an all-or-nothing impact on the CPU (and the SSD/HDD), thereby hammering performance for the user.

That leaves me with a question regarding the way the indexer currently operates, and depending on the answer, with one more possible thought regarding options for how/when the indexer is initiated.

The Question

It's been my understanding that once the indexer starts, exiting Logos before indexing is finished can corrupt the index, even if the indexer has been paused.  Is that true?  If so, that means once the indexer starts, the user has to keep Logos open until indexing is complete and perhaps makes the thought below more relevant.  

The Thought

Since the indexer is already a separate executable, would it be possible for Faithlife to provide the user with a separate icon for it that could be submitted outside of the regular Logos app, and have a corresponding option in Logos to not automatically start the indexer when Logos is running.  Not that the indexer couldn't still be manually initiated from within Logos if desired, but for those who would rather have system performance impacted while doing other things such as email or web browsing, this would allow them to initiate the indexer from outside Logos.  This would have the side benefit of freeing up system resources otherwise allocated to the main Logos app while the indexer runs--resources that could be used by the OS and other apps.

Again, I'm getting by with things the way they are, but I know that's not the case for everyone.

Rick Ausdahl said:

It's been my understanding that once the indexer starts, exiting Logos before indexing is finished can corrupt the index, even if the indexer has been paused.

I've never experienced that, and can't see how exiting Logos would cause problems with indexing.

Rick Ausdahl said:

Since the indexer is already a separate executable, would it be possible for Faithlife to provide the user with a separate icon for it that could be submitted outside of the regular Logos app, and have a corresponding option in Logos to not automatically start the indexer when Logos is running.

Because we can already pause the indexer as soon as it starts, personally I'm not sure that delaying its start would make much of a difference.

I also have never had a problem with corrupt indexes, and I often open and close Logos itself, and pause and unpause indexing without giving it a second thought.

I used to write software using a "backend" database that was notorious for becoming corrupt if somehow the PC was summarily powered down while indexing was taking place. I can see how that might happen for almost any indexing system. I wonder if that's what leads to mangled Logos indexing.

Andrew Batishko (Faithlife) said:

If the computer is shut down while the index is in the middle of building, then the indexing process starts over from scratch next time. Closing the Logos application does not interfere in any way with indexing.

Thanks, Andrew!  Very nice to have official confirmation.

Can you clarify one point though?  Does the indexing only have to restart if the computer is shut down during indexing?  I.e.  Will the indexing continue if Logos is shut down but the computer is left on?

Andrew Batishko (Faithlife) said:

Exiting the Logos application has no effect on the indexer.

Awesome!  [:)] [:)] [:)]

For me, the problem likely seems much worse than it is, or perhaps is worse than it seems, I cannot be sure which is more true.  Why?

I am on-the-go a lot.  So, I jump on my desktop, check email, grab a freebie book, start Logos and ugghhhh.  Index purgatory.  So I shut off indexing so I can get something done but then I can't find what I want in my library.  No problem!  Being in a hurry, I have a great idea!  

I have a large laptop for extended stops when traveling.  I open it, smugly start Logos, but alas!  I forget to disconnect the internet, so down comes my new freebie.  Grrr.  Pause indexing.  Search results incomplete.  Well, it's time to go, so I grab my small laptop and head out to teach a class.  

On the train I open my iPhone, hunt for a quote I think is in The Imitation of Christ (which I am needing by now), but alas!  Data is downloading.  Then it is indexing.  OK.  use another app.  Trouble is, on my phone the indexing stops when I stop the app (at least I think so),  So it takes a few days to index (because I give up each time I can't do what I want) and by then it needs to download more data.  So...undaunted, I press on faithfully.

Arriving in class, I have that thought I want to share from The Imitation of Christ, so I open my iPad, but alas, it has connected to wifi (why didn't I think to put it in airplane mode temporarily!), and so much for Thomas a Kempis's great ideas. Owell, on with classes.

I open the small laptop, connect to the big screen, open Logos.  No! No! No!  here comes my freebie.  Now entering index purgatory.  Mash down on the "Stop Indexing thingy.  Library lags, can't urp up what I want.  I think I'm gonna hurl myself.  Patience.  Where is The Imitation of Christ when I need it!  Restart indexing, tell a funny story or so, indexing complete and classes go swimmingly.  And I LOVE Logos for what it does, how it works and how I can use it to teach.  With some twinge of hypocrisy, I recommend Logos heartily to my students. But I do love it and I truly mean what I say.  Logos may not have a bigger fan, a more deep-hearted fan than me.  

--------------

Probably nothing nearly that bad has actually happened.  But sometimes it is how I feel. The problem is not Logos, or indexing, it is me.  I have all these devices using Logos, so I have indexing on a desktop, 2 laptops, 2 phones and 3 iPads.  Of course I don't access them all at once, but if I access half of them in an hour or so (which might realistically happen) that is index-purgatory X 4. 

Thanks for letting me vent in what I hope is a whimsical and constructive way--a sort of reality show as an example.  I truly love Logos and if it stayed the way it is forever, I would be a VERY happy camper. My hat is off to one and all at Faithlife and the MVP's and all the good people who use Logos.

Gao Lu said:

So, I jump on my desktop, check email, grab a freebie book, start Logos and ugghhhh.  Index purgatory.  So I shut off indexing so I can get something done but then I can't find what I want in my library.

The scenario resonates with me — but there's no problem just pausing indexing. All that will happen is that search won't turn up results for that specific resource. Nothing else will be affected. So long as you don't need search results from the resource(s) that have just download, there's no issue.