Too Many Books
Hello folks,
Clearly I have too many books (23,000+). I say this because it takes forever to index the books even with an I7 processor, 16GB of Ram and a SSD. Strange as this sounds, Is there a way to reduce the number of books that I have to improve my Logos 7 software performance?
Thanks!
Comments
-
Open up Program Settings and drag the book(s) that you do not want into the "Hidden Resources" section. They will no longer be stored on your hard disk (and by implication, will no longer be indexed).
Another option would be to transfer your licenses away (e.g. sell your books to someone else or return them to FaithLife).
0 -
Another option is choosing when to download updates - either a scheduled time window OR turn off automatic updates so can manually choose when to download and index (ideally when not using computer for several hours).
Keep Smiling [:)]
0 -
A variaton of Joshua's suggestion is that you can also hide resources in the library by (Mac) alt-right clicking. I presume that there is a windows equivalent. This becomes much faster if you select multiple resources first (thus hiding resources in batches). In the back of my mind, I seem to remember that there was some sort of limit to the number of resources that you can hide in each batch (about 10) ... but I could be wrong ... experimentation is the key :-)
2017 15" MBP, iPad Pro
0 -
Someone ought to point out that hiding is not as easy to reverse. It's one book at a time with a few seconds delay meaning some people have taken weeks to recover hidden books they decided they do in fact want! Personally I used a script for this task...
גַּם־חֹשֶׁךְ֮ לֹֽא־יַחְשִׁ֪יךְ מִ֫מֶּ֥ךָ וְ֭לַיְלָה כַּיּ֣וֹם יָאִ֑יר כַּ֝חֲשֵׁיכָ֗ה כָּאוֹרָֽה
0 -
it takes forever to index the books even with an I7 processor, 16GB of Ram and a SSD.
Hi Father Gregory,
prior to offering advice I'd like to see logs (especially indexer.log, if available). It shouldn't take "forever" to index a large library on your machine - and it shouldn't bother you either, since it runs in the background. And even if it slowed down your machine a bit, you wouldn't worry since reindexing the whole library normally doesn't happen (only if the index is broken). So there are some questions, like
- is indexing of the library complete, or does it try over and over?
- is the "forever" indexing an incremental index of updated books?
- If so, how long does it really take and does it complete?
- Is indexing really what bothers you, or is it preparing the library upon startup, where building the collections really takes the time (and if you have all the ca. 100 canonical commentary collections plus some, then that takes a bit, even on a fast machine)?
In case it really is the number of books (impressive!) and you want to reduce it, hiding books one-by-one does not sound very helpful. It really comes down to bulk hiding as explained above (luckily there's no 10 books limit). So you can filter the library such as lang:latin or filter for Perseus and then bulk-hide the resulting many books. Just remember: there's no bulk unhide.
Have joy in the Lord!
0 -
Just remember: there's no bulk unhide.
Also, there are no filtering or sorting options. All hidden resources are in a single, long alphabetical list. 😣
0 -
- Is indexing really what bothers you, or is it preparing the library upon startup, where building the collections really takes the time (and if you have all the ca. 100 canonical commentary collections plus some, then that takes a bit, even on a fast machine)?
How much time does "building the collections" take? [[That is how do many collections affect start up time?]]
If we rarely use a collection should we keep its rules in a document file of collection rules and rebuild the collection as needed?
0 -
Version 7.14, which is in beta right now, includes a significant improvement to indexing. My library of ~60,000 resources went from indexing in 5h 21m to 2h 12m, a 59% reduction or 2.45x faster. Not all computers will see the same improvements, but the average increase may be around 2x.
We're also exploring selective download and cloud resources, which would let you choose what to install locally (and index) and what to leave in the cloud and have access to only when online.
We're considering some improvements to resource hiding, too.
0 -
Version 7.14, which is in beta right now, includes a significant improvement to indexing. My library of ~60,000 resources went from indexing in 5h 21m to 2h 12m, a 59% reduction or 2.45x faster. Not all computers will see the same improvements, but the average increase may be around 2x.
We're also exploring selective download and cloud resources, which would let you choose what to install locally (and index) and what to leave in the cloud and have access to only when online.
We're considering some improvements to resource hiding, too.
This is big news, Phil. Even though it might not score as well in the marketing department as other flashier features, I'm sure many users greatly appreciate these speed improvements!
(PS: Selective download would make me be more willing to buy more resources without fear of making the software unusable [i.e. slow] on my computer.)
0 -
We're also exploring selective download and cloud resources, which would let you choose what to install locally (and index) and what to leave in the cloud and have access to only when online.
Phil, these would be very significant improvements to the programs that would help a wide range of users in different situations. For example, I have and use the Perseus library. Due to space constraints, I have hidden the Greek and Latin volumes, but there are rare occasions on which I'd like to access them. It would be very helpful if I could get to them via the web app while not downloading them to the hard drive.
More users would benefit from being able to separate the downloading of resources from program updates. I encourage you to make these improvements.
0 -
0
-
Clearly I have too many books (23,000+). I say this because it takes forever to index the books even with an I7 processor, 16GB of Ram and a SSD.
I have a similar size library, and a similar spec PC, but I rarely notice indexing as an issue. Perhaps there's something deeper going on? A description of exactly the symptoms, together with logs and more detailed spec might help us track that down, if you're interested.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
We're also exploring selective download and cloud resources, which would let you choose what to install locally (and index) and what to leave in the cloud and have access to only when online.
If this setting would be 'per computer', it would help a lot of users who have low disk space on their laptop, etc., but don't really want, or can't afford, a new secondary computer.
macOS (Logos Pro - Beta) | Android 13 (Logos Stable)
0 -
Version 7.14, which is in beta right now, includes a significant improvement to indexing. My library of ~60,000 resources went from indexing in 5h 21m to 2h 12m, a 59% reduction or 2.45x faster. Not all computers will see the same improvements, but the average increase may be around 2x.
Do the improvements also affect the merge of newly indexed resources into the existing index? On my PC the merge takes far longer than the indexing itself.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Version 7.14, which is in beta right now, includes a significant improvement to indexing. My library of ~60,000 resources went from indexing in 5h 21m to 2h 12m, a 59% reduction or 2.45x faster. Not all computers will see the same improvements, but the average increase may be around 2x.
Do the improvements also affect the merge of newly indexed resources into the existing index? On my PC the merge takes far longer than the indexing itself.
I always thought merging and indexing were the same thing. How can you merge without indexing?
Using adventure and community to challenge young people to continually say "yes" to God
0 -
I didn't know about merging... How do I do that? Is there a benefit?
0 -
That's all good news, Phil! I hope it all (including improvements to resource hiding/unhiding) makes it into Logos soon.Version 7.14, which is in beta right now, includes a significant improvement to indexing. My library of ~60,000 resources went from indexing in 5h 21m to 2h 12m, a 59% reduction or 2.45x faster. Not all computers will see the same improvements, but the average increase may be around 2x.
We're also exploring selective download and cloud resources, which would let you choose what to install locally (and index) and what to leave in the cloud and have access to only when online.
We're considering some improvements to resource hiding, too.
0 -
I always thought merging and indexing were the same thing. How can you merge without indexing?
You can't merge without indexing, but you can index without merging (by re-creating the whole index, rather than creating a supplement index and merging it into the main one). The example Phil gave was of indexing without merging (i.e. a complete rebuild).
Looking at an old log (which happens to be indexing 16 new resources in 7.12 beta 3), it took 15 seconds to create the supplemental index, but 15 minutes to merge them.
Looking at a new log (indexing 2 new resources in 7.14 beta 2), it took 4 seconds to create the supplemental index, but 16 minutes to merge them.
In other words, there appears to be no improvement. That suggests the improvements are only in the 'create the index' stage. If so there will be no benefit to Logos users other than when they download/install for the first time — which means we're all getting over excited.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
I didn't know about merging... How do I do that? Is there a benefit?
Merging happens automatically, as part of indexing. It used to be a secondary operation in Logos 4.0, so you'd get two sets of search results — from the main index and then the supplemental index. But merging has happened by default for many releases now.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
You can't merge without indexing, but you can index without merging (by re-creating the whole index, rather than creating a supplement index and merging it into the main one).
That totally makes sense. I just viewed them as the same thing because I don't think of it unless it causes problems with my PC but, as you point out, they are not.
In other words, there appears to be no improvement. That suggests the improvements are only in the 'create the index' stage. If so there will be no benefit to Logos users other than when they download/install for the first time — which means we're all getting over excited.
It sure does sound like we're getting overly excited about something that apparently will not help unless we in the creating stage. Maybe the merging stage will be improved in the future.
Thanks for the explanation Mark.
Using adventure and community to challenge young people to continually say "yes" to God
0 -
We're also exploring selective download and cloud resources
This is very welcome news and I hope it'll get implemented sooner rather than later!
We're considering some improvements to resource hiding, too
The implementation I mentioned here would (in my mind) solve both issues at once. Create a 'master library' or 'manage resources' page on our faithlife.com/logos.com account. Having the solution web-based would allow a user to keep from ever downloading certain resources (e.g. after a bundle purchase, simply log-in and filter/find the resources in question and toggle the switch!).
It would be nice if a resource (or selected resources) would have 3 possible values for each desktop installation: hidden/online/local.
0 -
Version 7.14, which is in beta right now, includes a significant improvement to indexing. My library of ~60,000 resources went from indexing in 5h 21m to 2h 12m, a 59% reduction or 2.45x faster. Not all computers will see the same improvements, but the average increase may be around 2x.
We're also exploring selective download and cloud resources, which would let you choose what to install locally (and index) and what to leave in the cloud and have access to only when online.
We're considering some improvements to resource hiding, too.
wow. I can't wait for this
0 -
Looking at a new log (indexing 2 new resources in 7.14 beta 2), it took 4 seconds to create the supplemental index, but 16 minutes to merge them.
In other words, there appears to be no improvement. That suggests the improvements are only in the 'create the index' stage. If so there will be no benefit to Logos users other than when they download/install for the first time — which means we're all getting over excited.
Measure it again in 7.14 Beta 3.
0 -
Measure it again in 7.14 Beta 3.
- beta 1: 18m 20.2s
- beta 2: 17m 5.0s
- beta 2: 18m 43.8s
- beta 2: 16m 25.5s
- beta 3: 13m 59.7s
- beta 3: 15m 13.2s
- beta 3: 11m 46.9s
Averaging that out:
- Betas 1 & 2: 17m 38.6s (1,059s)
- Beta 3: 13m 39.9s (820s)
So an improvement in beta 3 of 22%. Not to be sniffed at. (The times quoted are those given at the end of the indexer log, for the total time to create the supplemental index, merge into the main index, and build the term/document info index. I omitted the beta 2 indexing log that reindexed all the reverse-interlinears, as that would be an unfair comparison.)
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Phil,
Is the indexing identical for everyone, or is some of indexing based on feature usage?
For example do user-added tags get indexed, so a user heavily tags resources will see indexing take longer than someone who does little or no user-tagging with the same resources.
If this is true could FL list features that based on the user having impacts on indexing?
0 -
Is the indexing identical for everyone, or is some of indexing based on feature usage?
The indexing we're talking about here is pretty much identical for everyone. It occurs when books are added to your library, or updated by Logos. The time it takes is dependent on hardware spec, library size, and the number of resources being updated.
For example do user-added tags get indexed, so a user heavily tags resources will see indexing take longer than someone who does little or no user-tagging with the same resources.
I'm not sure what you mean by user-added tags.
If you're talking about tagging books in your library, then doesn't affect the indexing we're talking about here. It does affect the time taken to go through "Updating Library Catalog". That time is dependent on how many resources you have, and how many tags, ratings and collections you have.
If you're talking about tags added to highlighting/notes, or labels added to parts of resources, these are not stored in the main index but in a personal documents index. For most users this index is small, and is updated fairly quickly.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
I was happy with my 2804 books but after reading this thread I feel sort of inadequate.
0 -
I was happy with my 2804 books but after reading this thread I feel sort of inadequate.
Some of us feel that we should have stopped there.
0
