Consistently very slow startups

Hello fellow brothers and sisters,

In the past months, Logos has started using up more than 100% CPU for approx. 5 minutes on each startup before being usable by yours truly, my perplexed self.

Upon each startup, for a few minutes, its screen is kind of blank; after that, for a few minutes, I see my panels appear and their contents all seem ready, but the waiting cursor keeps me from working (Apple's rolling beach ball of patience-or-go-do-something-else). After those minutes where the CPU is kept hot, the app starts to be usable (but still often slowish to answer user interactions).

Through the months, I have tried lots of things: starting up in an empty layout (and I think I remember the issue was less of a problem, but still a problem), etc. Also, note that this isn't an indexing issue. It's consistent even when indexes are up-to-date. Nor is it a hardware issue: my laptop's specs are a blessing: It's a 2013 MacBook Pro almost fully upgraded: i7, 16GB, ultra-fast SSD.

The only answer that will make sense of it all is the one that'll come out of those logs that I attach here, where I started Logos with 13 of my favorite panels open.

Thank you very much for your help!

3782.LogosLogs.dj.20160526-070404.zip

Find more posts tagged with

Comments

JT (alabama24)

Also, note that this isn't an indexing issue. It's consistent even when indexes are up-to-date.

I am unsure what you mean by "up-to-date." If a index has become corrupted, it doesn't matter if it is "up to date."

I am not a full fledged techie, but I did notice something. Your logs are FULL of things like this:

2016-05-26 07:01:53.9301 | INFO | 35 | LibraryCatalogIndex | Searching for all records matching: Author:("Bence, Clarence", "Bence, Philip", "Black, Robert", "Bounds, Chris", "Brecheisen, Jerry", "Carter, Charles W.", "Cockerill, Gareth Lee", "Conrad, Chris", "Dongell, Joseph", "Drury, Keith", "Eckley, Richard", "Heer, Ken", "Holdren, David W.", "Holmes, Mark", "Lennox, Stephen J.", "McClung, Ronald", "Schenck, Kenneth", "Walters, J. Michael", "Williams, Wilbur Glenn", "Wilson, Earle", "Wilson, Norman G.", "Woolsey, Warren")

I am not sure what to make of it. As I said, I am not a full fledged techie, and only have a limited ability to read and understand logs. Your logs, however, are so full of these kinds of statements that it is nearly all I see when I scroll. I am leaning towards believing that you have a corrupted LibraryCatalog Index... but I would rather get confirmation before advising you.

MJ. Smith

https://community.logos.com/discussion/comment/823359#Comment_823359

In my own logs I have assumed those entries represented building the collections.

JT (alabama24)

https://community.logos.com/discussion/comment/823411#Comment_823411

In my own logs I have assumed those entries represented building the collections.

That thought crossed my mind as well... Did you see how much of it was in his logs?

Dave Hooton

https://community.logos.com/discussion/comment/823415#Comment_823415

In my own logs I have assumed those entries represented building the collections.

That thought crossed my mind as well... Did you see how much of it was in his logs?

Quite a lot, and VERY long. One took 5s to process. Many author:(... rules are missing a closing parentheses, which affects the results.

Daniel, what are you trying to achieve with these rules? Some appear to be duplicated.

Daniel Jomphe

https://community.logos.com/discussion/comment/823439#Comment_823439

Thank you all so, so much!

Wow, I saw what you mean in the logs! Those search requests must have been quite costly to run, seeing them spelled out like that in the logs!

I did configure a lot of suggested custom Collections something like one year ago (theological collections and denominational collections coming out of some Community project here in those forums). It may be that those would have gotten corrupted somehow a few months ago, and started to affect my startup times more since then!?

I almost never used them. At first they seemed like a great idea. It took me something like 10 to 15 minutes to manually delete each one of them. After that I restarted Logos and instead of waiting for some number of minutes, I only waited for a few dozen seconds before I could scroll through my panels! Wow, what an improvement! And to think that all this time I thought that it was my Base Package upgrade and my Logos Now subscription that were affecting the software's performance so negatively... LOL Had I known this custom feature could be so costly in startup upkeep, I would never have added all those custom collections that I added!

Now I wonder if it would be useful to rebuild my indexes to clean up.

Again, thank you so much, I'm so happy and I definitely thank God as it will again be much more thinkable to quickly fire up Logos to check something up on the fly!

Dave Hooton

https://community.logos.com/discussion/comment/823461#Comment_823461

I almost never used them. At first they seemed like a great idea. It took me something like 10 to 15 minutes to manually delete each one of them.

The number/size of collections may be inefficient for the database!

Had I known this custom feature could be so costly in startup upkeep, I would never have added all those custom collections that I added!

A long rule of authors would cover many that do not apply to your library but it would be better to understand what type of books are being selected e.g.

type:commentary ANDNOT author:(.......) ----> may be more efficient.

But I tend to rate commentaries irrespective of author. You could rate them because of the author! e.g.

type:commentary myrating:>3

Additionally, manually tag them as belonging to an appropriate category e.g.

type:commentary mytag:historical myrating:>3

But the examples at https://wiki.logos.com/Example_Collections would give you an approach that avoids manual tagging and rating to a large extent. Modify them to suit your library e.g. modify specific titles and authors.

David Ames

https://community.logos.com/discussion/comment/823487#Comment_823487

The number/size of collections may be inefficient for the database!

At one time in the distant past I wrote SQL queries. Finding the best way of writing one maybe an art. The ones that work best for a small database are often different from those that work best on a large database. [Often the order of sub queries needed to change and yes, we reran the large queries on small databases and they were not the fastest on the small databases] We sometimes resorted to pulling the keys into a temp database, sorting the temp database as needed and then using those keys [from the temp] to pull records from the main database and then deleting the temp database. [[we also found that the queries could be data dependent - two same size databases but with different data required different queries to find similar data in the same time]]

Andrew Baguley

https://community.logos.com/discussion/comment/823501#Comment_823501

Thanks for the post, Daniel. Having taken a quick look at the logs you provided, it looks like there was a problem with some of the collections you had that cost a few seconds each on start-up. It would be good if someone at Logos could look at this.

The collections that slowed down your start-up were running more than once, sometimes taking just a couple of milliseconds, but then running again and taking up to 12.2 seconds to run. (As the slower runs were not the first ones, this can't be because the queries were learning and improving.) There also appears to have been some corruption of your collections, which slowed down some of them, but I'm not sure that this explains the whole problem. When running properly, these collections were taking much less than a second, and usually just a few milliseconds.

To read logs like this one, it may be helpful to copy and paste the data into a spreadsheet. Select the data in the spreadsheet, and use 'TextToColumns' with | as a delimiter. You may also want to create a custom format "mm:ss.000" to see the milliseconds part of the data.

Hope it's now running as fast as it should be.

Dave Hooton

https://community.logos.com/discussion/comment/823461#Comment_823461

Now I wonder if it would be useful to rebuild my indexes to clean up.

If you haven't done so in the last 12 months, or if you have held a Logos Now subscription for 6 months or so then a rebuild would be beneficial for searches and recovering some disk space.

Mark Barnes

https://community.logos.com/discussion/comment/823638#Comment_823638

Now I wonder if it would be useful to rebuild my indexes to clean up.

I don't believe there was any index corruption - and even if there was, it wouldn't speed up this problem. Collections are created from the LibraryCatalog, not the index.

I suspect that the problem was the way the collections were set up. It sounds like you had dozens of similar collections, many of which were very complex (denominational). If you want to create multiple collections all based on the same denominational tagging, you should use compound collections, rather than pasting in the same denominational criteria every time. That way the denominational query only needs to be executed once.

For example, if you wanted a collection for Methodist Commentaries, you would do the following:

Create a collection for Methodist authors in the normal way.
Create a collection for everything except for Methodist authors. You do this by entering rating:<6 in the criteria, and then adding the Methodist author collection to the Minus these resources panel.
Create a collection for Methodist Commentaries, by entering type:bible-commentary in the criteria, and then adding the "Not Methodist Authors" collection to the Minus these resources panel.

Then just repeat step 3 for all other Methodist collections you need (e.g. Methodist dictionaries, etc.). What you don't want to do is to have more than one collection with the criteria: Author:(Author1, Author2, Author3...).

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/823643#Comment_823643

by entering rating:<6 in the criteria

Use: *

It's simpler to remember and no less efficient.

toughski

https://community.logos.com/discussion/comment/823703#Comment_823703

Bradley, are collections generated anew with each startup?

I have a similar problem to Daniel and experience very long startup times especially when running offline

Mark Barnes

https://community.logos.com/discussion/comment/823948#Comment_823948

Bradley, are collections generated anew with each startup?

Collections recalculated at startup and when the library catalog changes. This increases startup time slightly, but speeds up Logos thereafter.

Without logs it would be impossible to determine what's causing the slow start up for you.

Daniel Jomphe

https://community.logos.com/discussion/comment/823625#Comment_823625

Thanks for the post, Daniel. Having taken a quick look at the logs you provided, it looks like there was a problem with some of the collections you had that cost a few seconds each on start-up. It would be good if someone at Logos could look at this.

The collections that slowed down your start-up were running more than once, sometimes taking just a couple of milliseconds, but then running again and taking up to 12.2 seconds to run. (As the slower runs were not the first ones, this can't be because the queries were learning and improving.) There also appears to have been some corruption of your collections, which slowed down some of them, but I'm not sure that this explains the whole problem. When running properly, these collections were taking much less than a second, and usually just a few milliseconds.

To read logs like this one, it may be helpful to copy and paste the data into a spreadsheet. Select the data in the spreadsheet, and use 'TextToColumns' with | as a delimiter. You may also want to create a custom format "mm:ss.000" to see the milliseconds part of the data.

Hope it's now running as fast as it should be.

Yes, it's fast like a new one now.

Ok it seems to confirm what I remember: when I added those dozens of collections, my software didn't slow down that much (otherwise I'd have removed them just as soon as I would have observed this marked slow down). It may be that through the months that followed, some Logos upgrades corrupted them and I then got stuck with that legacy. This makes me just a bit fearful of investing too much in building myself complex custom collections.

And thanks for the spreadsheet tip, I think I'll try it next time I go check my logs.

Daniel Jomphe

https://community.logos.com/discussion/comment/823643#Comment_823643

I suspect that the problem was the way the collections were set up. It sounds like you had dozens of similar collections, many of which were very complex (denominational). If you want to create multiple collections all based on the same denominational tagging, you should use compound collections, rather than pasting in the same denominational criteria every time. That way the denominational query only needs to be executed once.

The group who made those collections wrote them all up with completely independent queries so that users may pick and choose which denominational collections to copy into their documents.

In any case, all that denominational information was may too much for my curiosity. I'm still wondering why I ever thought it would be useful to me to use all those collections. Where I live (Quebec, Canada), we're only really aware of something like less than 5 denominations, I would think.

David Ames

https://community.logos.com/discussion/comment/824083#Comment_824083

This makes me just a bit fearful of investing too much in building myself complex custom collections.

Question: How often does one need a 'complex custom collection'? If not needed very often why not save the rules string in the text or doc type file and only have the collection active when needed? Inconvenient when you have to rebuild it when needed but if only needed once a year or so the rest of the time you have faster startups.