Logos 6 Best Feature: NO INDEXING !!!!!!!!!!
Comments
-
As I've said before, indexing is a pain, but I understand why it is necessary. I'd MUCH rather be bothered with indexing after a download than with slow searches. And as Bob implied, having something like a Google search result is untenable. "We found your restaurant. It's on Earth." Start computing permutation formulae with a thirty-six thousand in there, and you'll need a mainframe computer to not choke on the numbers.
As for anyone for whom slow indexing becomes inconvenient, I sympathize. My computer is pretty much useless for that half-hour too. But as for anyone for whom slow indexing 'cripples my life', well, you really ought to find something else to do with your time.
Eating a steady diet of government cheese, and living in a van down by the river.
0 -
mike said:
I really couldn't use the computer & I always play the waiting game whenever I want to use Logos.
I suggest that you just use the "Pause for 4 hours" option if indexing is slowing you down too much. Once you stop having to do high CPU activity you can restart the indexing and pause it as many times as necessary. However, I rarely find indexing a problem and can keep on working just fine. I just close other CPU hogs like iTunes for example.
0 -
DavidS said:
That is an option but it still involves an indexing process. I have SSD so it is not that much of a problem for me. But there are many that can not afford to upgrade there computers.
I DON'T HAVE SSD, but I don't find the indexing to be a problem at all. It slows things down, but it only lasts a short time.
george
gfsomselיְמֵי־שְׁנוֹתֵינוּ בָהֶם שִׁבְעִים שָׁנָה וְאִם בִּגְבוּרֹת שְׁמוֹנִים שָׁנָה וְרָהְבָּם עָמָל וָאָוֶן
0 -
I've been watching this thread and thinking it through a bit. May I ask, Bob/Logos, what kind of indexing data structure do you use? As a Logos user and a developer, I imagine there's a smoother way of accomplishing what everyone wants, and I'd like to think through it a little more deeply.
Here's a thought: what if we kept the local index as is but also offered the option of the online supplement? You already do this for my iPad, right? I do a Basic Search on a word, it returns with results from my library in about a second. If I search for that same term on the desktop, it can take 30 seconds to retrieve the final output. (just tried it out now, and that was my experience)
Of course, not everyone is on the internet all the time, but some kind of a synthesized/hybrid search option may give us the best of both worlds.
As far as indexing is concerned, there has to be a way to lower the priority. I'm not talking about CPU prioritization--I know that's already reduced so that it's not supposed to get in the way. But I've got to tell you, the indexer hogs the system until it's done. I imagine it's due to the use of all sorts of resources like I/O in addition to the CPU. I have developed a strategy for Logos indexing, but frankly, I shouldn't have to do that. I shouldn't have to worry about it. I don't care if it takes longer to finish, I want to be able to productively use my computer while Logos indexes.
Finally, have you considered a drop-in product for the indexing and searching? Some really smart guys have already done a lot of work like this, and they offer they work freely. What would it look like if Logos incorporated Lucene?
Bill
0 -
In Logos 2, indexes came on disk, and the application just merged them in (and therefore cut indexing time).
In Logos 3, indexes came on disk and were kept separate so that indexing wasn't required. Without one joint index though, searching was slow.
In Logos 4 and 5, no indexes were supplied, and the application creates and merges them.
It would be relatively easy for Logos to go back to the Logos 2 days, when indexes were created by Logos and then downloaded. That could could indexing in half, but increase downloading by 50% or so.
That said, by the time Logos 6 comes out we'll all have SSDs and superfast processors, and we won't mind if indexing taking place. I have a very large library, and books are frequently purchased and updated. I also have an i7 processor and pro-level SSD. I regularly have indexing taking place whilst I'm working, and I barely notice it.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Bob Pritchett said:
We can't just index each book separately, because if we did A) ranked results across your library would be impossible to compute with any accuracy, and
we'd have to search each book, dramatically increasing the number of seeks on your hard drive.So why do we still have .mstidx files [:P]? Isn't it time they were optimized too?
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Bob Pritchett said:
We can't just index each book separately, because if we did A) ranked results across your library would be impossible to compute with any accuracy, and
we'd have to search each book, dramatically increasing the number of seeks on your hard drive.Bob,
I am confused. I used to program computers years ago. If each book was indexed and the results sorted [at Logos] then they are all merged into a library wide index [on user computers] then you would have no long indexing. With today's processing speeds this should be really fast. Am I missing something here?
Thanks for your time to communicate with us. It really helps to understand Logos.
0 -
I don't notice a difference in performance when my computer is indexing vs when its not. But my computer is not old either, nor does it lack for memory and has an SSD.Bob Pritchett said:Back in 2009 we posted a lot of technical info on why indexing 'is what it is.' While there may still be some room to optimize, a large part of the required time is literally required in order to do what we do.
Everybody has hundreds (or thousands) of different books making up their library. The permutations of ownership of 36,000 different books make it impossible to store every possible index and then just download it. While people who buy the same base package, and nothing else, in the same window of a few months might all have the same index, that's still a lot of combinations, and within a very short time someone will add a book and then need their own special index.
We can't just index each book separately, because if we did A) ranked results across your library would be impossible to compute with any accuracy, and
we'd have to search each book, dramatically increasing the number of seeks on your hard drive.Physical hard drives are much faster reading in sequence than 'seeking' to new locations. Wikipedia reports an average of 9 milliseconds for a seek on a desktop hard drive. That means that to load hits for "term" we have to spend 9 ms just getting to the list of hits, which can then be read continuously from that location. If you have 1,000 separate books with 1,000 separate pre-built indices, that'll be 9 ms x 1,000 = 9 seconds just in hard drive seek. Even super-smart coding can't argue with physics. And worse, you can't find the location of the hits in a single seek -- that seek just gets you there. You have to do other seeks (at least one) to load the index of where the term's hit lists are.
So we build a single index of your library -- but building that index involves 'reading' every book and storing the hits. We can use more memory to minimize seeks for writing the index file, but it's still going to take a lot.
This is why SSD's make a huge difference -- much faster seeking. (None, really -- it's direct access to memory.)
So how does Google be so fast even over the Internet?
A) Google has only one index for the whole Internet, not a separate index for every permutation of books a user owns. (We could pre-build a single index of all 36,000 books, but for most of you we'd keep returning results in books you don't own. Filtering out the books you don't own would slow you down. And you might not want us to re-download that multi-gig index to your hard drive every night as new books are released. We could leave it on the Internet, but then you'd have to be online.) Which leads to...
Google makes you use an index in the cloud, and does not support off-line use, like we do.C) Google never 'seeks.' Google stores the entire index to the Internet in memory. Lots of memory. On lots of computers. Using lots of power. "Build my data center next to the dam" kind of power.
It's possible we could hand-optimize our indexing process and squeeze 10% or more out of it. (I'm not sure, but it's possible -- anything is possible.) But that's 6 minutes out of an hour, and may be literally the end of what we could do without changing behavior / functionality.
But laptops are quickly moving to SSDs, and there you'll see much bigger improvements than code will ever offer. If you're on a desktop, adding an SSD as an extra drive, and then moving resources and indexing there, is another option.
Sorry I don't have an easy solution, like "Bradley, stay up all night and optimize this!". :-) But I hope the explanation helps...
-- Bob
However Bob - my thinking was that you have 36,000 individual indexes - one for each book.
But my computer should only have one index. So when I buy a new book, the book downloads, and then the index for that book is added to the master index of books i own on my computer. As in ALL book indexes (for which I own a license) get added to the same file as they do now, however instead of my computer doing the indexing, why couldn't the index just be appended to my existing index?
Does that make more sense? This way most of the problems you outlined don't exist, and people with slower computers don't have to deal with indexing.
McAffee calls this "delta updates".
This way if you have version 1 of the virus definitions and they are on version 3, rather than download vs 2, and then version three, you simply download the things that version 2, and version 3 have that are new, and they are dynamically inserted into the virus definition file.
L2 lvl4 (...) WORDsearch, all the way through L10,
0 -
Why not simply incorporate/expand the scheduler for downloading and indexing with power options after the dnload/index is complete?
IOW something like this:
Check for downloads Every day / Every Sun/Mon/Tue/Wed/Thu/Fri/Sat between xx:xx am/pm and zz:zz am/pm.
Shut down / Reboot / Put Computer to Sleep / Do Nothing / Index after Downloading.
Shut down / Reboot / Put Computer to Sleep / Do Nothing after Indexing.
Logos should also be able to wake the computer from a sleep state to perform this dnload/index function (If it isn't already able to do so).
And, it should send Notification emails with log files attached upon any errors and/or a successful dnload/index report upon completion.
Instead of Artificial Intelligence, I prefer to continue to rely on Divine Intelligence instructing my Natural Dullness (Ps 32:8, John 16:13a)
0 -
mike said:
this constant indexing really cripple my computer and my life regardless whether I change the schedule of updates in setting or not.
My Windows 7 laptop also gets crippled during indexing. Everything running on my computer slows to a halt.
The solution? I go to Windows Task Manager, go to the Logos Indexer process, and Set Priority to Low.
Boom. No more slow computer.
My solution? The Logos Indexer should be set to Priority Low by default. It's set to Normal by default, which means it fights with every other program running, including Logos Bible Software. If it is set to Low, then it will run in the background and not take any CPU cycles unless the other processes aren't using them.
0 -
elnwood said:
My Windows 7 laptop also gets crippled during indexing. Everything running on my computer slows to a halt.
The solution? I go to Windows Task Manager, go to the Logos Indexer process, and Set Priority to Low.
Boom. No more slow computer.
[:D] Are you an NBA play by play announcer?
macOS, iOS & iPadOS |Logs| Install
Choose Truth Over Tribe | Become a Joyful Outsider!0 -
abondservant said:
But my computer should only have one index. So when I buy a new book, the book downloads, and then the index for that book is added to the master index of books i own on my computer. As in ALL book indexes (for which I own a license) get added to the same file as they do now, however instead of my computer doing the indexing, why couldn't the index just be appended to my existing index?
It's not a simple append operation. If you just had a bunch of appended indexes, you would still have a very slow search. The index is most-likely some type of tree (e.g. b-tree), so you have your master index, which is a huge tree, and a book index, which is a relatively small tree. You can't just pin the small tree wherever you want onto the big tree and have it work the same. You basically need to traverse both trees and "splice" them together. That takes quite a bit of computing power, and the Logos indexer is building that first, smaller tree and then merging the two together. In other words, it already works pretty close to the way you suggest, but it sounds like a pre-computed index for the book is not downloaded with the book as of right now.
So, you already do have just one index, but building and maintaining that index is a lot of work. Not nearly as much work as a pure search through all your materials would be, but when your resource count starts to get into the thousands (millions of words), it's still a bit of heavy lifting for your computer to manage that one index.
0 -
alabama24 said:elnwood said:
My Windows 7 laptop also gets crippled during indexing. Everything running on my computer slows to a halt.
The solution? I go to Windows Task Manager, go to the Logos Indexer process, and Set Priority to Low.
Boom. No more slow computer.
Are you an NBA play by play announcer? The question should be "Are you now or have you ever been …" [;)]
george
gfsomselיְמֵי־שְׁנוֹתֵינוּ בָהֶם שִׁבְעִים שָׁנָה וְאִם בִּגְבוּרֹת שְׁמוֹנִים שָׁנָה וְרָהְבָּם עָמָל וָאָוֶן
0 -
mike said:
The best part the requirement system for Logos doesn't have to be mid or high, low spec could also run Logos because indexing is run outside (not simultaneously with power-to-search)
I think you vastly underestimate Logos' need for good specs.
Example 1: My little 2010 Mac Mini can already do most searches in a second or two. It's displaying the results that takes time, often half a minute or more. (I got 5 times faster internet in July. I thought my browser would fly. I could hardly notice the difference. That's when I really realized how much a good graphics card means!)
Example 2: When I tag books, I have to restart Logos every 15-20 minutes or so. By that time it has eaten up all my 8 MB's of RAM and become slow as molasses.
Indexing or not, Logos needs good specs. Very good specs. Especially on Mac, which I believe is less optimized than the PC version.
Richard DeRuiter said:indexing is much faster now than it was when Logos4 was first released
That may be true for PC. For me it seems to be getting slower and slower.
Doc B said:As for anyone for whom slow indexing becomes inconvenient, I sympathize. My computer is pretty much useless for that half-hour too.
If you believe indexing takes half an hour for those of us who complain about it, you are not sympathizing at all. I'm happy if mine is finished in three hours. Often it's much more. (And that's with Logos and almost everything else off.) I'd be dancing the jig if it took half an hour. That's nothing!!!
Keith Larson said:I suggest that you just use the "Pause for 4 hours" option if indexing is slowing you down too much.
Logos doesn't release the RAM. Once the Indexer has reached the merging phase, Pause cripples the computer almost as much as no Pause -- and much longer.
The solution isn't to pause. The solution -- the only currently available one -- is to never let it start, unless it's reasonably convenient.
abondservant said:But my computer should only have one index. So when I buy a new book, the book downloads, and then the index for that book is added to the master index of books i own on my computer. As in ALL book indexes (for which I own a license) get added to the same file as they do now, however instead of my computer doing the indexing, why couldn't the index just be appended to my existing index?
Does that make more sense? This way most of the problems you outlined don't exist, and people with slower computers don't have to deal with indexing.This seems to be a common misunderstanding on this thread. Even on my slow computer, most books index in seconds. Indexing isn't the problem. It's exactly the merging of the new book's index with the existing index that's the problem. That's what's taking all the time. (Only a very large Bible download, like the one this week -- or possibly a base package upgrade --can make the indexing take longer than the merging, but that's an exceptionally rare occurrence.)
Mac Pro (late 2013) OS 12.6.2
0 -
Two suggestions that would help me, and two that sound like they would help many others:
• Optimize the Mac Indexer to be as fast as the PC one seemingly is.
• Release as much RAM as possible when the Indexer is paused.
• Add a 'Day' setting to Program Settings, so that pastors (and others, of course) can set Logos to "download automatically between 8PM Sunday and 7AM Tuesday, but don't even notify me about downloads at other times, unless I ask for them", or "check for downloads only at 8AM Mondays".
• Include two indexing options on the Notification bar, and in the menu bar icon dropdown:
- Fast -- use all available CPU and RAM.
- Background -- leave plenty of CPU and RAM for the user, and always prioritize the program[s ] he's actively using. (This would also be suitable for those who have problems with over-heating).
In general, I prefer a local index over a download.
(If we want Bob to read this -- which I presume we do -- may I suggest we don't fill the thread with OT posts.[;)])
Mac Pro (late 2013) OS 12.6.2
0 -
fgh said:
(If we want Bob to read this -- which I presume we do -- may I suggest we don't fill the thread with OT posts.
)Good point. [Y]
fgh said:• Add a 'Day' setting to Program Settings, so that pastors (and others, of course) can set Logos to "download automatically between 8PM Sunday and 7AM Tuesday, but don't even notify me about downloads at other times, unless I ask for them", or "check for downloads only at 8AM Mondays".
Really good idea.
macOS, iOS & iPadOS |Logs| Install
Choose Truth Over Tribe | Become a Joyful Outsider!0 -
fgh said:
Two suggestions that would help me, and two that sound like they would help many others:
• Optimize the Mac Indexer to be as fast as the PC one seemingly is.
• Release as much RAM as possible when the Indexer is paused.
• Add a 'Day' setting to Program Settings, so that pastors (and others, of course) can set Logos to "download automatically between 8PM Sunday and 7AM Tuesday, but don't even notify me about downloads at other times, unless I ask for them", or "check for downloads only at 8AM Mondays".
• Include two indexing options on the Notification bar, and in the menu bar icon dropdown:
- Fast -- use all available CPU and RAM.
- Background -- leave plenty of CPU and RAM for the user, and always prioritize the program[s ] he's actively using. (This would also be suitable for those who have problems with over-heating).
In general, I prefer a local index over a download.
(If we want Bob to read this -- which I presume we do -- may I suggest we don't fill the thread with OT posts.
)[Y] [Y] [Y]
(quoting you over and over can keep your post from getting lost in the shuffle)
I have Logos on a Mac and a Windows computer, and I believe the "performance of indexing" has deteriorated on both since Logos 5 was first released. I have an SSD on the Mac (2009) and a separate Logos HD for the PC (2012), but I've gone from letting the indexer run to needing to strategize when it can run because it gets in the way of operating both. I like your indexing options suggestion, as the MBP burns me as it indexes right now.
0 -
Bradley Grainger (Logos) said:
One possibility is to move the index from your computer to our servers (similar to what is done for the mobile apps right now). Would it be an acceptable solution to require an Internet connection for searching to work?
Not to be rude, but if you go to Africa for a month and use it, I think you would have your own answer... [:)]
There are a lot of places that the Bible is being studied using Logos that does not have good internet...
But, if this were a user configurable option... use Logos servers or your own, I could see a win/win for all.
0 -
Donovan R. Palmer said:
Not to be rude, but if you go to Africa for a month and use it, I think you would have your own answer
I think it was more of a rhetorical question…
macOS, iOS & iPadOS |Logs| Install
Choose Truth Over Tribe | Become a Joyful Outsider!0 -
Donovan R. Palmer said:
But, if this were a user configurable option... use Logos servers or your own, I could see a win/win for all.
I totally agree with you [Y]
"No man is greater than his prayer life. The pastor who is not praying is playing; the people who are not praying are straying." Leonard Ravenhill
0 -
Ahh. OK, that makes sense.William Gabriel said:abondservant said:But my computer should only have one index. So when I buy a new book, the book downloads, and then the index for that book is added to the master index of books i own on my computer. As in ALL book indexes (for which I own a license) get added to the same file as they do now, however instead of my computer doing the indexing, why couldn't the index just be appended to my existing index?
It's not a simple append operation. If you just had a bunch of appended indexes, you would still have a very slow search. The index is most-likely some type of tree (e.g. b-tree), so you have your master index, which is a huge tree, and a book index, which is a relatively small tree. You can't just pin the small tree wherever you want onto the big tree and have it work the same. You basically need to traverse both trees and "splice" them together. That takes quite a bit of computing power, and the Logos indexer is building that first, smaller tree and then merging the two together. In other words, it already works pretty close to the way you suggest, but it sounds like a pre-computed index for the book is not downloaded with the book as of right now.
So, you already do have just one index, but building and maintaining that index is a lot of work. Not nearly as much work as a pure search through all your materials would be, but when your resource count starts to get into the thousands (millions of words), it's still a bit of heavy lifting for your computer to manage that one index.
I saw somewhere that I have platinum + 3563 books. Additionally the Spurgeon collection (which I own) is somewhere around 24m words I'm told.
I wonder if downloading the prebuilt index is what Bob was talking about when he said they could gain another 5% or so.L2 lvl4 (...) WORDsearch, all the way through L10,
0 -
Bradley Grainger (Logos) said:
One possibility is to move the index from your computer to our servers (similar to what is done for the mobile apps right now). Would it be an acceptable solution to require an Internet connection for searching to work?
For me No!!! you would lose customer..
Never Deprive Anyone of Hope.. It Might Be ALL They Have
0 -
No! Please dont take away local index lookups for PC and Mac etc.
But maybe there could be a mid-point: Let people build a Collection that becomes their small or local index. This rebuilds always if its member titles are updated, just as for the current Logos 5 indexes.
The full index can be left unbuilt for weeks or until such time as someone can allow thr time and effort to wait for it to run overnight or whatever they like.
Now, if the the Logos 5 client is asked to search and the full index is not current, it can offer to do an online search. Given the Logos servers already have all the web services and api calls etc to support mobil clients, the lpumbing is there and waiting.
So, I could just have a few hundreds of titles in my "small" index collection, and they will rebuild in under 30 mins or whatever. If I don't want th 4 hour rebuild of the other 10,000 titles, I can do an online search if I really need to.
0 -
fgh said:
• Add a 'Day' setting to Program Settings, so that pastors (and others, of course) can set Logos to "download automatically between 8PM Sunday and 7AM Tuesday, but don't even notify me about downloads at other times, unless I ask for them", or "check for downloads only at 8AM Mondays".
An other great suggestion.
Perhaps we need to split the download and the index functions. As in maybe I want to read the new books now but wait until I go to sleep before updating the index. Maybe ask if we want to index now or wait before starting. Don't wake indexing up rather then putting it to sleep.
0 -
David Ames said:
Perhaps we need to split the download and the index functions. As in maybe I want to read the new books now but wait until I go to sleep before updating the index.
I <think> you can do that now! You just don't have full search capabilities. Of course, I have been wrong before. [:O]
EDIT: I think I misunderstood, although I don't know what the benefit would be. You were not concerned about the reading part, but about the delay in indexing part. Right?
macOS, iOS & iPadOS |Logs| Install
Choose Truth Over Tribe | Become a Joyful Outsider!0 -
Said this before, 'll say it again...
- The thread priority / %CPU usage for indexing needs to be lowered. Users report better responsiveness when thread priority is manually lowered. Why is this even happening?
- Indexing can be scheduled and sessioned.
- Indexing can provide "failover" functionality. The user runs on the old library (plus temporarily provisioned searches on new resources), until the updates and indexing are all finished, then boom! the switch is thrown and the user emerges in the new library. Tagging, highlighting and stuff can be synched, or alternatively, suspended under some kind of temporary, restricted functionality mode.
- Expecting userbase to adopt SSD in a big way is too much to ask. Some users want to keep running programs off spinners. There are good reasons for this.
I don't expect "Bradley, hammer out a physically impossible coding optimization over the weekend", but I do see a good return on "go back to the drawing board and think through what can be done".
0 -
Talking about priority, I assume the question is complex. I have a W7 with Logos on the SSD drive. I really don't see much (actually 'any') impact on the other programs during indexing. I'm using fairly high-end paint packages and synths and there's no hiccup, nor is the CPU above 15% or so.
This is not to say there's no problem; only that there seems to be a lot of variables relative to a PC configuration.
Now fgh's chewing up RAM and not releasing it is a big issue; when Logos gets to 60% I have to reboot the PC to fix Logos. That a recent problem Logos introduced.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
Bob Pritchett said:
Everybody has hundreds (or thousands) of different books making up their library. The permutations of ownership of 36,000 different books make it impossible to store every possible index and then just download it. While people who buy the same base package, and nothing else, in the same window of a few months might all have the same index, that's still a lot of combinations, and within a very short time someone will add a book and then need their own special index.
We can't just index each book separately, because if we did A) ranked results across your library would be impossible to compute with any accuracy, and
we'd have to search each book, dramatically increasing the number of seeks on your hard drive.Most indexes are basically like database tables. I don't know the technical details of how the Logos indexes work, but if they do work like database tables then the index of each book could theoretically be downloaded and then merged into the users general index - like merging the rows of a smaller db into a larger one. When a book is updated, its database rows get downloaded with the book, deleted from the general index and re-imported.
0 -
What users on high-end systems need to understand is that good programming practices may help lower-end users, with hardly any negative effect on the high-end systems.
0 -
Lee said:
The thread priority / %CPU usage for indexing needs to be lowered. Users report better responsiveness when thread priority is manually lowered. Why is this even happening?
Although the indexer has its process priority set to normal, individual threads are already run at the lowest priority - certainly on Windows.
Lee said:Indexing can provide "failover" functionality. The user runs on the old library (plus temporarily provisioned searches on new resources), until the updates and indexing are all finished, then boom! the switch is thrown and the user emerges in the new library. Tagging, highlighting and stuff can be synched, or alternatively, suspended under some kind of temporary, restricted functionality mode.
This mostly happens already - that is, indexing takes place in three stages, and the second and third of those stages takes place whilst Logos is running. Once those stages are finished, then the user is already using the new resources.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Fr Devin Roza said:
Most indexes are basically like database tables. I don't know the technical details of how the Logos indexes work, but if they do work like database tables then the index of each book could theoretically be downloaded and then merged into the users general index - like merging the rows of a smaller db into a larger one. When a book is updated, its database rows get downloaded with the book, deleted from the general index and re-imported.
Indexes are not like database tables. Databases have tables and indexes, and you can only search databases tables quickly if there is a separate index. Database indexes, especially full-text indexes, are slow to create, but quick to use.
An index has to be created in exactly the right order (that's why it's quick to sort/search). So when you merge two indexes, you have to rewrite the whole index, to ensure that the correct order is maintained. That's inevitably a slow, CPU intensive process.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
Although the indexer has its process priority set to normal, individual threads are already run at the lowest priority - certainly on Windows.
This mostly happens already - that is, indexing takes place in three stages, and the second and third of those stages takes place whilst Logos is running. Once those stages are finished, then the user is already using the new resources.
Hmmm... something looks broken then. If these features are in place and working properly, we should not be getting these user reports.
0 -
Lee said:
If these features are in place and working properly, we should not be getting these user reports.
It would be very interesting to have indexing logs and disk/cpu passmark for each report.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:Lee said:
If these features are in place and working properly, we should not be getting these user reports.
It would be very interesting to have indexing logs and disk/cpu passmark for each report.
Logos needs to take a pro-active approach to this, otherwise it could fuel one of those confirmation-bias disasters.
0 -
Would it be possible to schedule indexing? I have a new computer but when I index, everything slows to a crawl. i realize that I have the option to delay it for 4 hours but then it pops back on unannounced and things again grind to a halt and I then delay it another 4 hours.
Being able to schedule indexing to happen in the dead of night would be beneficial.
Alternatively, having indexing only kick in when the computer is not being used would also be huge and probably a better solution.
0 -
Along with scheduling, could we somehow control our desktop/laptop indexing remotely through the Logos App? We could receive a text that an update is available on a computer currently running Logos, and a quick reply could trigger the download remotely and automatic index. We could even set our desktop Logos to an "away" setting that would permit access to downloading/indexing via the Logos app (or some control app)
0 -
I would support being able to schedule indexing of the library in the same manner we can schedule downloads. It would be good if we could schedule the download and have the rest of the process automatically execute in the same manner as updates from Microsoft function (they do give the option of delaying the restart). In that way it would be possible to schedule downloading, installation and indexing for the off hours—problem solved (Personally, I don't have a problem with the current method since I barely notice it. The only time it was a problem was when I installed L5 on a new laptop and had to download, install and index 5,000 resources which took hours.). Otherwise, I think we should leave it to Logos and trust that they know what they're doing.
george
gfsomselיְמֵי־שְׁנוֹתֵינוּ בָהֶם שִׁבְעִים שָׁנָה וְאִם בִּגְבוּרֹת שְׁמוֹנִים שָׁנָה וְרָהְבָּם עָמָל וָאָוֶן
0 -
Bradley Grainger (Logos) said:mike said:
I'm sure those smart programmers guys at Logos will know how to fix this (I'm praying for a miracle)
One possibility is to move the index from your computer to our servers (similar to what is done for the mobile apps right now). Would it be an acceptable solution to require an Internet connection for searching to work?
Bradley, I think cloud storage of the index would not be a bad idea - as long as a local copy was downloaded as well. The desktop client needs to function without internet access.
0 -
alabama24 said:
I think it was more of a rhetorical question…
My answer was tongue in cheek to a rhetorical question...
Regardless, this suggestion has been discussed seriously before. Logos' heavy dependency on the internet is already a challenge, even for my parent's use in a rural US location. Hence the suggestion about a user configurable option.
0 -
Donovan R. Palmer said:alabama24 said:
I think it was more of a rhetorical question…
My answer was tongue in cheek to a rhetorical question...
Regardless, this suggestion has been discussed seriously before. Logos' heavy dependency on the internet is already a challenge, even for my parent's use in a rural US location. Hence the suggestion about a user configurable option.
[Y] [:)]
macOS, iOS & iPadOS |Logs| Install
Choose Truth Over Tribe | Become a Joyful Outsider!0 -
Mark Barnes said:Fr Devin Roza said:
Most indexes are basically like database tables. I don't know the technical details of how the Logos indexes work, but if they do work like database tables then the index of each book could theoretically be downloaded and then merged into the users general index - like merging the rows of a smaller db into a larger one. When a book is updated, its database rows get downloaded with the book, deleted from the general index and re-imported.
Indexes are not like database tables. Databases have tables and indexes, and you can only search databases tables quickly if there is a separate index. Database indexes, especially full-text indexes, are slow to create, but quick to use.
An index has to be created in exactly the right order (that's why it's quick to sort/search). So when you merge two indexes, you have to rewrite the whole index, to ensure that the correct order is maintained. That's inevitably a slow, CPU intensive process.
Thanks for clearing that up! Apart from the technical imprecision, I guess the heart of my question was whether Logos downloads an already pre-prepared index with the book, and then merges that index with the user's index, or whether Logos downloads the text and then both creates and merges the index.
0 -
fgh said:
The solution -- the only currently available one -- is to never let it start, unless it's reasonably convenient.
I'm having MOSTLY good result by using the Details tab of Task Manager to set the priority of Indexer to Lower than Normal. By mostly good results I mean that I'm mostly NOT experiencing the lockup for minutes on end during the most intense phase of Indexer that I was experiencing before. There may be several times when the system seems "hesitant", but I don't think it has actually locked up like it did before.
/on soapbox.
I don't call my workaround a solution, because I don't believe any one app should have the RIGHT to demand that much of my system resources. A solution needs to come from Logos, & I agree with the main points I've seen in this thread so far:
- We like 1 second all-library searches.
- We like updated resources as soon as they're available.
- We don't like the impact of indexing, afterward, as currently implemented. This really does need a redesign--not just tweaking. Long term, since Logos has total control (via partnerships & ownership of PBB) of the sea of resources, Logos should be able to come up with a way to allow the new index to be downloaded with the new or updated resources. Short term, it may be ok to ask users to "manage" the problem via methods that Dave Hooton & others suggest.
/off soapbox.
Grace & Peace,
Bill
MSI GF63 8RD, I-7 8850H, 32GB RAM, 1TB SSD, 2TB HDD, NVIDIA GTX 1050Max
iPhone 12 Pro Max 512Gb
iPad 9th Gen iOS 15.6, 256GB0 -
Fr Devin Roza said:
I guess the heart of my question was whether Logos downloads an already pre-prepared index with the book, and then merges that index with the user's index, or whether Logos downloads the text and then both creates and merges the index.
Logos downloads just the text and creates an index locally. This is typically very fast; for example, most Bibles take under 6s to index on my computer, and indexing almost 2000 Bibles and morph resources took about 24m. This created almost 3GB of indexing data, which is about 2MB/sec. You'd have to have a pretty fast sustained Internet connection (and/or a much slower computer) to make downloading quicker overall.
(This was also a decision decision influenced by the first Logos 4 DVDs--yes, even just a few years ago, most customers were receiving a physical copy of the software--which were large enough to contain all the text, but not large enough to contain all the additional indexes. A multi-DVD install process would be complicated, and we didn't want to make customers who purchased on DVD have to also download indexes.)
0 -
Obviously, most indexing problems would be solved if we just stopped buying any new books.
So, what's the problem?
"In all cases, the Church is to be judged by the Scripture, not the Scripture by the Church," John Wesley0 -
I bought seven new WordSearch books yesterday and added them to my library of over 1700 books. There was no indexing! This is a new feature with the latest WordSearch (which I got for free). Apparently WordSearch has discovered the Binary Tree Index. If this sounds like Geek Speak, check out Wikipedia; they have a very understandable article on the subject.
With a B.T. index, each book is individually indexed. The individual index for each book is downloaded with the book, and then it is placed in just the right location in the master index. It's analogous to hanging an ornament on a Christmas tree. And B.T indexes are fast! There is no need to separately look at the individual indexes for each book. Once a book's index is placed in just the right location in the B.T. master index, it becomes seamlessly integrated into the whole.
Now I don't know that the Microsoft libraries have an object to do this for you. You might just have to do some actual programming.
Also, I don't know anything about how you accomplish your search ranking. Personally I can do without it.
Doug Witmer
A customer, but not a fan boy.
0 -
The problem is that every time Logos does an "update" (which for me is just about every time I fire up the program) it wants to re-index whether I've bought any new books or not.
0 -
Doug Witmer said:
The problem is that every time Logos does an "update" (which for me is just about every time I fire up the program) it wants to re-index whether I've bought any new books or not.
Logos won't re-index if the program is updated, but it will do some indexing if resources are updated. It won't do a full re-index, it will just index the newly added books and merge them in.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Michael Childs said:
Obviously, most indexing problems would be solved if we just stopped buying any new books.
So, what's the problem?
Ha, if only. Logos is far too gracious and offers us resource updates on what we already own, and that triggers a re-index too!
[;)]
0
