Consistently very slow startups
Hello fellow brothers and sisters,
In the past months, Logos has started using up more than 100% CPU for approx. 5 minutes on each startup before being usable by yours truly, my perplexed self.
Upon each startup, for a few minutes, its screen is kind of blank; after that, for a few minutes, I see my panels appear and their contents all seem ready, but the waiting cursor keeps me from working (Apple's rolling beach ball of patience-or-go-do-something-else). After those minutes where the CPU is kept hot, the app starts to be usable (but still often slowish to answer user interactions).
Through the months, I have tried lots of things: starting up in an empty layout (and I think I remember the issue was less of a problem, but still a problem), etc. Also, note that this isn't an indexing issue. It's consistent even when indexes are up-to-date. Nor is it a hardware issue: my laptop's specs are a blessing: It's a 2013 MacBook Pro almost fully upgraded: i7, 16GB, ultra-fast SSD.
The only answer that will make sense of it all is the one that'll come out of those logs that I attach here, where I started Logos with 13 of my favorite panels open.
Thank you very much for your help!
Comments
-
I am unsure what you mean by "up-to-date." If a index has become corrupted, it doesn't matter if it is "up to date."Daniel Jomphe said:Also, note that this isn't an indexing issue. It's consistent even when indexes are up-to-date.
I am not a full fledged techie, but I did notice something. Your logs are FULL of things like this:
I am not sure what to make of it. As I said, I am not a full fledged techie, and only have a limited ability to read and understand logs. Your logs, however, are so full of these kinds of statements that it is nearly all I see when I scroll. I am leaning towards believing that you have a corrupted LibraryCatalog Index... but I would rather get confirmation before advising you.Logs said:2016-05-26 07:01:53.9301 | INFO | 35 | LibraryCatalogIndex | Searching for all records matching: Author:("Bence, Clarence", "Bence, Philip", "Black, Robert", "Bounds, Chris", "Brecheisen, Jerry", "Carter, Charles W.", "Cockerill, Gareth Lee", "Conrad, Chris", "Dongell, Joseph", "Drury, Keith", "Eckley, Richard", "Heer, Ken", "Holdren, David W.", "Holmes, Mark", "Lennox, Stephen J.", "McClung, Ronald", "Schenck, Kenneth", "Walters, J. Michael", "Williams, Wilbur Glenn", "Wilson, Earle", "Wilson, Norman G.", "Woolsey, Warren")
macOS, iOS & iPadOS |Logs| Install
Choose Truth Over Tribe | Become a Joyful Outsider!0 -
In my own logs I have assumed those entries represented building the collections.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
MJ. Smith said:
In my own logs I have assumed those entries represented building the collections.
That thought crossed my mind as well... Did you see how much of it was in his logs?
macOS, iOS & iPadOS |Logs| Install
Choose Truth Over Tribe | Become a Joyful Outsider!0 -
alabama24 said:MJ. Smith said:
In my own logs I have assumed those entries represented building the collections.
That thought crossed my mind as well... Did you see how much of it was in his logs?
Quite a lot, and VERY long. One took 5s to process. Many author:(... rules are missing a closing parentheses, which affects the results.
Daniel, what are you trying to achieve with these rules? Some appear to be duplicated.
Dave
===Windows 11 & Android 13
0 -
Thank you all so, so much!
Wow, I saw what you mean in the logs! Those search requests must have been quite costly to run, seeing them spelled out like that in the logs!
I did configure a lot of suggested custom Collections something like one year ago (theological collections and denominational collections coming out of some Community project here in those forums). It may be that those would have gotten corrupted somehow a few months ago, and started to affect my startup times more since then!?
I almost never used them. At first they seemed like a great idea. It took me something like 10 to 15 minutes to manually delete each one of them. After that I restarted Logos and instead of waiting for some number of minutes, I only waited for a few dozen seconds before I could scroll through my panels! Wow, what an improvement! And to think that all this time I thought that it was my Base Package upgrade and my Logos Now subscription that were affecting the software's performance so negatively... LOL Had I known this custom feature could be so costly in startup upkeep, I would never have added all those custom collections that I added!
Now I wonder if it would be useful to rebuild my indexes to clean up.
Again, thank you so much, I'm so happy and I definitely thank God as it will again be much more thinkable to quickly fire up Logos to check something up on the fly!
0 -
Daniel Jomphe said:
I almost never used them. At first they seemed like a great idea. It took me something like 10 to 15 minutes to manually delete each one of them.
The number/size of collections may be inefficient for the database!
Daniel Jomphe said:Had I known this custom feature could be so costly in startup upkeep, I would never have added all those custom collections that I added!
A long rule of authors would cover many that do not apply to your library but it would be better to understand what type of books are being selected e.g.
type:commentary ANDNOT author:(.......) ----> may be more efficient.
But I tend to rate commentaries irrespective of author. You could rate them because of the author! e.g.
type:commentary myrating:>3
Additionally, manually tag them as belonging to an appropriate category e.g.
type:commentary mytag:historical myrating:>3
But the examples at https://wiki.logos.com/Example_Collections would give you an approach that avoids manual tagging and rating to a large extent. Modify them to suit your library e.g. modify specific titles and authors.
Dave
===Windows 11 & Android 13
0 -
Dave Hooton said:
The number/size of collections may be inefficient for the database!
At one time in the distant past I wrote SQL queries. Finding the best way of writing one maybe an art. The ones that work best for a small database are often different from those that work best on a large database. [Often the order of sub queries needed to change and yes, we reran the large queries on small databases and they were not the fastest on the small databases] We sometimes resorted to pulling the keys into a temp database, sorting the temp database as needed and then using those keys [from the temp] to pull records from the main database and then deleting the temp database. [[we also found that the queries could be data dependent - two same size databases but with different data required different queries to find similar data in the same time]]
0 -
Thanks for the post, Daniel. Having taken a quick look at the logs you provided, it looks like there was a problem with some of the collections you had that cost a few seconds each on start-up. It would be good if someone at Logos could look at this.
The collections that slowed down your start-up were running more than once, sometimes taking just a couple of milliseconds, but then running again and taking up to 12.2 seconds to run. (As the slower runs were not the first ones, this can't be because the queries were learning and improving.) There also appears to have been some corruption of your collections, which slowed down some of them, but I'm not sure that this explains the whole problem. When running properly, these collections were taking much less than a second, and usually just a few milliseconds.
To read logs like this one, it may be helpful to copy and paste the data into a spreadsheet. Select the data in the spreadsheet, and use 'TextToColumns' with | as a delimiter. You may also want to create a custom format "mm:ss.000" to see the milliseconds part of the data.
Hope it's now running as fast as it should be.
0 -
Daniel Jomphe said:
Now I wonder if it would be useful to rebuild my indexes to clean up.
If you haven't done so in the last 12 months, or if you have held a Logos Now subscription for 6 months or so then a rebuild would be beneficial for searches and recovering some disk space.
Dave
===Windows 11 & Android 13
0 -
Daniel Jomphe said:
Now I wonder if it would be useful to rebuild my indexes to clean up.
I don't believe there was any index corruption - and even if there was, it wouldn't speed up this problem. Collections are created from the LibraryCatalog, not the index.
I suspect that the problem was the way the collections were set up. It sounds like you had dozens of similar collections, many of which were very complex (denominational). If you want to create multiple collections all based on the same denominational tagging, you should use compound collections, rather than pasting in the same denominational criteria every time. That way the denominational query only needs to be executed once.
For example, if you wanted a collection for Methodist Commentaries, you would do the following:
- Create a collection for Methodist authors in the normal way.
- Create a collection for everything except for Methodist authors. You do this by entering rating:<6 in the criteria, and then adding the Methodist author collection to the Minus these resources panel.
- Create a collection for Methodist Commentaries, by entering type:bible-commentary in the criteria, and then adding the "Not Methodist Authors" collection to the Minus these resources panel.
Then just repeat step 3 for all other Methodist collections you need (e.g. Methodist dictionaries, etc.). What you don't want to do is to have more than one collection with the criteria: Author:(Author1, Author2, Author3...).
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark Barnes said:
by entering rating:<6 in the criteria
Use: *
It's simpler to remember and no less efficient.
0 -
Bradley, are collections generated anew with each startup?
I have a similar problem to Daniel and experience very long startup times especially when running offline
0 -
toughski said:
Bradley, are collections generated anew with each startup?
Collections recalculated at startup and when the library catalog changes. This increases startup time slightly, but speeds up Logos thereafter.
Without logs it would be impossible to determine what's causing the slow start up for you.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Andrew Baguley said:
Thanks for the post, Daniel. Having taken a quick look at the logs you provided, it looks like there was a problem with some of the collections you had that cost a few seconds each on start-up. It would be good if someone at Logos could look at this.
The collections that slowed down your start-up were running more than once, sometimes taking just a couple of milliseconds, but then running again and taking up to 12.2 seconds to run. (As the slower runs were not the first ones, this can't be because the queries were learning and improving.) There also appears to have been some corruption of your collections, which slowed down some of them, but I'm not sure that this explains the whole problem. When running properly, these collections were taking much less than a second, and usually just a few milliseconds.
To read logs like this one, it may be helpful to copy and paste the data into a spreadsheet. Select the data in the spreadsheet, and use 'TextToColumns' with | as a delimiter. You may also want to create a custom format "mm:ss.000" to see the milliseconds part of the data.
Hope it's now running as fast as it should be.
Yes, it's fast like a new one now.
Ok it seems to confirm what I remember: when I added those dozens of collections, my software didn't slow down that much (otherwise I'd have removed them just as soon as I would have observed this marked slow down). It may be that through the months that followed, some Logos upgrades corrupted them and I then got stuck with that legacy. This makes me just a bit fearful of investing too much in building myself complex custom collections.
And thanks for the spreadsheet tip, I think I'll try it next time I go check my logs.
0 -
Mark Barnes said:
I suspect that the problem was the way the collections were set up. It sounds like you had dozens of similar collections, many of which were very complex (denominational). If you want to create multiple collections all based on the same denominational tagging, you should use compound collections, rather than pasting in the same denominational criteria every time. That way the denominational query only needs to be executed once.
The group who made those collections wrote them all up with completely independent queries so that users may pick and choose which denominational collections to copy into their documents.
In any case, all that denominational information was may too much for my curiosity. I'm still wondering why I ever thought it would be useful to me to use all those collections. Where I live (Quebec, Canada), we're only really aware of something like less than 5 denominations, I would think.
0 -
Daniel Jomphe said:
This makes me just a bit fearful of investing too much in building myself complex custom collections.
Question: How often does one need a 'complex custom collection'? If not needed very often why not save the rules string in the text or doc type file and only have the collection active when needed? Inconvenient when you have to rebuild it when needed but if only needed once a year or so the rest of the time you have faster startups.
0
