Logos 6 Best Feature: NO INDEXING !!!!!!!!!!

Page 5 of 14 (263 items) « First ... < Previous 3 4 5 6 7 Next > ... Last »
This post has 262 Replies | 5 Followers

Posts 1523
Forum MVP
Fr Devin Roza | Forum Activity | Replied: Fri, Oct 4 2013 7:42 AM

Bob Pritchett:

Everybody has hundreds (or thousands) of different books making up their library. The permutations of ownership of 36,000 different books make it impossible to store every possible index and then just download it. While people who buy the same base package, and nothing else, in the same window of a few months might all have the same index, that's still a lot of combinations, and within a very short time someone will add a book and then need their own special index.

We can't just index each book separately, because if we did A) ranked results across your library would be impossible to compute with any accuracy, and B) we'd have to search each book, dramatically increasing the number of seeks on your hard drive.

Most indexes are basically like database tables. I don't know the technical details of how the Logos indexes work, but if they do work like database tables then the index of each book could theoretically be downloaded and then merged into the users general index - like merging the rows of a smaller db into a larger one. When a book is updated, its database rows get downloaded with the book, deleted from the general index and re-imported.

Posts 2589
Lee | Forum Activity | Replied: Fri, Oct 4 2013 7:45 AM

What users on high-end systems need to understand is that good programming practices may help lower-end users, with hardly any negative effect on the high-end systems.

Posts 13415
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 7:54 AM

Lee:
The thread priority / %CPU usage for indexing needs to be lowered. Users report better responsiveness when thread priority is manually lowered. Why is this even happening?

Although the indexer has its process priority set to normal, individual threads are already run at the lowest priority - certainly on Windows.

Lee:
Indexing can provide "failover" functionality. The user runs on the old library (plus temporarily provisioned searches on new resources), until the updates and indexing are all finished, then boom! the switch is thrown and the user emerges in the new library. Tagging, highlighting and stuff can be synched, or alternatively, suspended under some kind of temporary, restricted functionality mode.

This mostly happens already - that is, indexing takes place in three stages, and the second and third of those stages takes place whilst Logos is running. Once those stages are finished, then the user is already using the new resources.

Posts 13415
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 8:00 AM

Fr Devin Roza:
Most indexes are basically like database tables. I don't know the technical details of how the Logos indexes work, but if they do work like database tables then the index of each book could theoretically be downloaded and then merged into the users general index - like merging the rows of a smaller db into a larger one. When a book is updated, its database rows get downloaded with the book, deleted from the general index and re-imported.

Indexes are not like database tables. Databases have tables and indexes, and you can only search databases tables quickly if there is a separate index. Database indexes, especially full-text indexes, are slow to create, but quick to use.

An index has to be created in exactly the right order (that's why it's quick to sort/search). So when you merge two indexes, you have to rewrite the whole index, to ensure that the correct order is maintained. That's inevitably a slow, CPU intensive process.

Posts 2589
Lee | Forum Activity | Replied: Fri, Oct 4 2013 8:00 AM

Mark Barnes:

Although the indexer has its process priority set to normal, individual threads are already run at the lowest priority - certainly on Windows.

This mostly happens already - that is, indexing takes place in three stages, and the second and third of those stages takes place whilst Logos is running. Once those stages are finished, then the user is already using the new resources.

Hmmm... something looks broken then. If these features are in place and working properly, we should not be getting these user reports.

Posts 13415
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 8:10 AM

Lee:
If these features are in place and working properly, we should not be getting these user reports.

It would be very interesting to have indexing logs and disk/cpu passmark for each report.

Posts 2589
Lee | Forum Activity | Replied: Fri, Oct 4 2013 8:43 AM

Mark Barnes:

Lee:
If these features are in place and working properly, we should not be getting these user reports.

It would be very interesting to have indexing logs and disk/cpu passmark for each report.

Logos needs to take a pro-active approach to this, otherwise it could fuel one of those confirmation-bias disasters.

Posts 251
Rod | Forum Activity | Replied: Fri, Oct 4 2013 8:56 AM

Would it be possible to schedule indexing?  I have a new computer but when I index, everything slows to a crawl.  i realize that I have the option to delay it for 4 hours but then it pops back on unannounced and things again grind to a halt and I then delay it another 4 hours.

Being able to schedule indexing to happen in the dead of night would be beneficial.

Alternatively, having indexing only kick in when the computer is not being used would also be huge and probably a better solution.

Posts 1674
Paul N | Forum Activity | Replied: Fri, Oct 4 2013 9:29 AM

Along with scheduling, could we somehow control our desktop/laptop indexing remotely through the Logos App?  We could receive a text that an update is available on a computer currently running Logos, and a quick reply could trigger the download remotely and automatic index.  We could even set our desktop Logos to an "away" setting that would permit access to downloading/indexing via the Logos app (or some control app)

Posts 9947
George Somsel | Forum Activity | Replied: Fri, Oct 4 2013 9:52 AM

I would support being able to schedule indexing of the library in the same manner we can schedule downloads.  It would be good if we could schedule the download and have the rest of the process automatically execute in the same manner as updates from Microsoft function (they do give the option of delaying the restart).  In that way it would be possible to schedule downloading, installation and indexing for the off hours—problem solved (Personally, I don't have a problem with the current method since I barely notice it.  The only time it was a problem was when I installed L5 on a new laptop and had to download, install and index 5,000 resources which took hours.).  Otherwise, I think we should leave it to Logos and trust that they know what they're doing.

george
gfsomsel

יְמֵי־שְׁנוֹתֵינוּ בָהֶם שִׁבְעִים שָׁנָה וְאִם בִּגְבוּרֹת שְׁמוֹנִים שָׁנָה וְרָהְבָּם עָמָל וָאָוֶן

Posts 77
Erik DiVietro | Forum Activity | Replied: Fri, Oct 4 2013 10:16 AM

Bradley Grainger (Logos):

mike:

I'm sure those smart programmers guys at Logos will know how to fix this (I'm praying for a miracle)

One possibility is to move the index from your computer to our servers (similar to what is done for the mobile apps right now). Would it be an acceptable solution to require an Internet connection for searching to work?

Bradley, I think cloud storage of the index would not be a bad idea - as long as a local copy was downloaded as well. The desktop client needs to function without internet access.

Posts 1955
Donovan R. Palmer | Forum Activity | Replied: Fri, Oct 4 2013 10:35 AM

alabama24:
I think it was more of a rhetorical question…

My answer was tongue in cheek to a rhetorical question...

Regardless, this suggestion has been discussed seriously before. Logos' heavy dependency on the internet is already a challenge, even for my parent's use in a rural US location. Hence the suggestion about a user configurable option.

Posts 29195
Forum MVP
JT (alabama24) | Forum Activity | Replied: Fri, Oct 4 2013 10:53 AM

Donovan R. Palmer:

alabama24:
I think it was more of a rhetorical question…

My answer was tongue in cheek to a rhetorical question...

Regardless, this suggestion has been discussed seriously before. Logos' heavy dependency on the internet is already a challenge, even for my parent's use in a rural US location. Hence the suggestion about a user configurable option.

Yes Smile

macOS, iOS & iPadOS | Logs |  Install

Posts 1523
Forum MVP
Fr Devin Roza | Forum Activity | Replied: Fri, Oct 4 2013 11:00 AM

Mark Barnes:

Fr Devin Roza:
Most indexes are basically like database tables. I don't know the technical details of how the Logos indexes work, but if they do work like database tables then the index of each book could theoretically be downloaded and then merged into the users general index - like merging the rows of a smaller db into a larger one. When a book is updated, its database rows get downloaded with the book, deleted from the general index and re-imported.

Indexes are not like database tables. Databases have tables and indexes, and you can only search databases tables quickly if there is a separate index. Database indexes, especially full-text indexes, are slow to create, but quick to use.

An index has to be created in exactly the right order (that's why it's quick to sort/search). So when you merge two indexes, you have to rewrite the whole index, to ensure that the correct order is maintained. That's inevitably a slow, CPU intensive process.

Thanks for clearing that up! Apart from the technical imprecision, I guess the heart of my question was whether Logos downloads an already pre-prepared index with the book, and then merges that index with the user's index, or whether Logos downloads the text and then both creates and merges the index.

Posts 3733
BillS | Forum Activity | Replied: Fri, Oct 4 2013 11:41 AM

fgh:
The solution -- the only currently available one -- is to never let it start, unless it's reasonably convenient.

I'm having MOSTLY good result by using the Details tab of Task Manager to set the priority of Indexer to Lower than Normal. By mostly good results I mean that I'm mostly NOT experiencing the lockup for minutes on end during the most intense phase of Indexer that I was experiencing before. There may be several times when the system seems "hesitant", but I don't think it has actually locked up like it did before.

/on soapbox.

I don't call my workaround a solution, because I don't believe any one app should have the RIGHT to demand that much of my system resources. A solution needs to come from Logos, & I agree with the main points I've seen in this thread so far:

  1. We like 1 second all-library searches.
  2. We like updated resources as soon as they're available.
  3. We don't like the impact of indexing, afterward, as currently implemented. This really does need a redesign--not just tweaking. Long term, since Logos has total control (via partnerships & ownership of PBB) of the sea of resources, Logos should be able to come up with a way to allow the new index to be downloaded with the new or updated resources. Short term, it may be ok to ask users to "manage" the problem via methods that Dave Hooton & others suggest.

/off soapbox.

 

Grace & Peace,
Bill


MSI GF63 8RD, I-7 8850H, 32GB RAM, 1TB SSD, 2TB HDD, NVIDIA GTX 1050Max
iPhone 12 Pro Max 512Gb
Fire 10HD 64GB 7th Gen

Posts 9054
LogosEmployee

Fr Devin Roza:

I guess the heart of my question was whether Logos downloads an already pre-prepared index with the book, and then merges that index with the user's index, or whether Logos downloads the text and then both creates and merges the index.

Logos downloads just the text and creates an index locally. This is typically very fast; for example, most Bibles take under 6s to index on my computer, and indexing almost 2000 Bibles and morph resources took about 24m. This created almost 3GB of indexing data, which is about 2MB/sec. You'd have to have a pretty fast sustained Internet connection (and/or a much slower computer) to make downloading quicker overall.

(This was also a decision decision influenced by the first Logos 4 DVDs--yes, even just a few years ago, most customers were receiving a physical copy of the software--which were large enough to contain all the text, but not large enough to contain all the additional indexes. A multi-DVD install process would be complicated, and we didn't want to make customers who purchased on DVD have to also download indexes.)

Posts 2875
Mike Childs | Forum Activity | Replied: Fri, Oct 4 2013 1:32 PM

Obviously, most indexing problems would be solved if we just stopped buying any new books.

So, what's the problem?

"In all cases, the Church is to be judged by the Scripture, not the Scripture by the Church," John Wesley

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 1:58 PM

I bought seven new WordSearch books yesterday and added them to my library of over 1700 books. There was no indexing! This is a new feature with the latest WordSearch (which I got for free). Apparently WordSearch has discovered the Binary Tree Index. If this sounds like Geek Speak, check out Wikipedia; they have a very understandable article on the subject.

With a B.T. index, each book is individually indexed. The individual index for each book is downloaded with the book, and then it is placed in just the right location in the master index. It's analogous to hanging an ornament on a Christmas tree. And B.T indexes are fast! There is no need to separately look at the individual indexes for each book. Once a book's index is placed in just the right location in the B.T. master index, it becomes seamlessly integrated into the whole.

Now I don't know that the Microsoft libraries have an object to do this for you. You might just have to do some actual programming.

Also, I don't know anything about how you accomplish your search ranking. Personally I can do without it.

Doug Witmer

A customer, but not a fan boy.

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 2:05 PM

The problem is that every time Logos does an "update" (which for me is just about every time I fire up the program) it wants to re-index whether I've bought any new books or not.

Posts 13415
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 2:51 PM

Doug Witmer:
The problem is that every time Logos does an "update" (which for me is just about every time I fire up the program) it wants to re-index whether I've bought any new books or not.

Logos won't re-index if the program is updated, but it will do some indexing if resources are updated. It won't do a full re-index, it will just index the newly added books and merge them in.

Page 5 of 14 (263 items) « First ... < Previous 3 4 5 6 7 Next > ... Last » | RSS