Why the Repeated, Sequential Indexing?
Comments
-
DIsciple II said:Dave Hooton said:
The response that is needed has to come from Faithlife
Never has a truer word been spoken on these forums.
For those who are wondering, I *did* get a helpful email response from Tommy at FL with some instructions to trouble-shoot the issue. I very much appreciate the follow-up.
Problem is, they require me to put on a programmer's hat (that I do not own)...and some time (which I will find).
He seems to be concerned that the indexer is crashing (which I don't think it is based on getting finished indexing...just really slowly after repeated indexes) or that I don't have enough free disk drive space (which I clearly do...well over 300GB). When I get time, I'll collect the logs (he wants two iterations over two consecutive days) and we'll see.
I hate collecting logs.
Eating a steady diet of government cheese, and living in a van down by the river.
0 -
-
Doc B said:
For those who are wondering, I *did* get a helpful email response from Tommy at FL with some instructions to trouble-shoot the issue
Great to here.
Doc B said:He seems to be concerned that the indexer is crashing (which I don't think it is based on getting finished indexing...just really slowly after repeated indexes) or that I don't have enough free disk drive space (which I clearly do...well over 300GB).
This is the stock standard solution but to me it says true issue you are raising is not being heard if they believe this will fix the situation … but sometimes you got to go through the steps logically to get people to understand the true issue.
0 -
DIsciple II said:
This is the stock standard solution but to me it says true issue you are raising is not being heard if they believe this will fix the situation … but sometimes you got to go through the steps logically to get people to understand the true issue.
It means that we aren't aware of any situations that would lead to indexing occurring more often than every two weeks (which is how often resource updates get published), with the following expected exceptions:
- When a batch of resources is updated, and some require an application restart (or an application update requires a restart), then indexing may begin before the restart occurs. After the restart, indexing will need to run again for any resources that got updated after the restart. This should only happen once.
- When you download a newly purchased resource, indexing on the new resources must happen.
- When you download a resource that was previously a cloud resource (or a hidden resource), then indexing must happen.
It doesn't sound like Doc B is encountering any of those situations, so we have to investigate to find out why he is experiencing a problem that we can't recreate for ourselves. Doing that requires verifying known situations that can cause the symptoms (indexer crashing or out of disk space) and collecting log files to look for traces of the problem there.
If we're missing some kind of systemic problem with how updating occurs, we'd love for you to help us determine the cause of this problem so that we can fix it.
Andrew Batishko | Logos software developer
0 -
The systematic problem was described earlier in regards to multipart updates requiring user intervention and thus they can‘t go a do something for else productive while the update process does what it needs to do and you can’t rely on overnight updates because of the require user intervention.
Appreciate you taking the time to ask Andrew. Hopefully I explained the issue well enough.
DIsciple II said:This is the stock standard solution but to me it says true issue you are raising is not being heard if they believe this will fix the situation … but sometimes you got to go through the steps logically to get people to understand the true issue.
It means that we aren't aware of any situations that would lead to indexing occurring more often than every two weeks (which is how often resource updates get published), with the following expected exceptions:
- When a batch of resources is updated, and some require an application restart (or an application update requires a restart), then indexing may begin before the restart occurs. After the restart, indexing will need to run again for any resources that got updated after the restart. This should only happen once.
- When you download a newly purchased resource, indexing on the new resources must happen.
- When you download a resource that was previously a cloud resource (or a hidden resource), then indexing must happen.
It doesn't sound like Doc B is encountering any of those situations, so we have to investigate to find out why he is experiencing a problem that we can't recreate for ourselves. Doing that requires verifying known situations that can cause the symptoms (indexer crashing or out of disk space) and collecting log files to look for traces of the problem there.
If we're missing some kind of systemic problem with how updating occurs, we'd love for you to help us determine the cause of this problem so that we can fix it.
0 -
Doc B said:
He seems to be concerned that the indexer is crashing (which I don't think it is based on getting finished indexing...just really slowly after repeated indexes) or that I don't have enough free disk drive space (which I clearly do...well over 300GB). When I get time, I'll collect the logs (he wants two iterations over two consecutive days) and we'll see.
Years ago with a similar problem Bradley diagnosed a dying hard drive ... I would have preferred a software bug [;)]
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
DIsciple II said:
The systematic problem was described earlier in regards to multipart updates requiring user intervention and thus they can‘t go a do something for else productive while the update process does what it needs to do and you can’t rely on overnight updates because of the require user intervention.
That sounds like a nice enough idea, but not one that directly solves Doc B's problem.
Andrew Batishko | Logos software developer
0 -
Rosie Perera said:
I just brought a brand new laptop, asked for one with sizzling fast specs (https://www.msi.com/Laptop/GS65-Stealth-Thin-8RF), and had them upgrade it to 32GB of super-fast RAM (G.SKILL Ripjaws DDR4-2400 SO-DIMM) and a 2TB SSD (WD Black SN750 2TB PCIe Gen3 x4 NVMe M.2 2280 Read:3400Mb/s,Write: 2900MB/s). Installed Logos fresh on it, let it download everything and complete indexing, and was disappointed already at the speed. Ugh. I'm really tired of this.
Thought I now could add my two cents on this. I just got a Dell XPS 15 with an i9 8th Gen CPU 32GB RAM 2666, 4GB Graphics, and a 1 TB Pci-e SSD. (It was on sale--maybe to make room for 9th Gen ) I've only got a bit over 15,000 resources, but it downloaded everything and indexed it all really fast. I was going to wait for the new MacBook Pro 16, but this basically about the same config in a Win 10 box. Logos loads and I can get right to work. I even used it while it was indexing, something that was a real struggle before.
I'd hate to think Logos needs this kind of artillery, but it doesn't matter. It works. I have some other needs for this beast, so it will all get used.
The mind of man is the mill of God, not to grind chaff, but wheat. Thomas Manton | Study hard, for the well is deep, and our brains are shallow. Richard Baxter
0 -
Rosie Perera said:
I just brought a brand new laptop, asked for one with sizzling fast specs (https://www.msi.com/Laptop/GS65-Stealth-Thin-8RF), and had them upgrade it to 32GB of super-fast RAM (G.SKILL Ripjaws DDR4-2400 SO-DIMM) and a 2TB SSD (WD Black SN750 2TB PCIe Gen3 x4 NVMe M.2 2280 Read:3400Mb/s,Write: 2900MB/s). Installed Logos fresh on it, let it download everything and complete indexing, and was disappointed already at the speed. Ugh. I'm really tired of this.
Rosie your machine should be crazy fast. I know when Logos first launches a bible it has to apply all the Visual Filters, and this can be very slow, and can take several minutes if many filters present, or many bibles versions being opened that have the filter applied, as it has to apply it to each version.
However once it completes this process, the cpu should return to normal. Do you have lots of visual filters?NB -
1) If all of that bible version is closed, and reopend, it has to go through this process again.
2) Even if the VF is unticked in the bible, it still has to go through this process. I create a collection called 'VF Applied" and then apply the Visual Filter to the Collection, I can then just add and remove my bible to the collection I want to see or not see the Visual Filters. This means when the usual bible is not in the collection, I do not have to wait for the filters to be applied, if I just unticked them, I would have to wait.
Anyway you may have explored this somewhere in your many other posts, but it may be helpful for others if they have having problems.
0 -
Kevin said:Rosie Perera said:
I just brought a brand new laptop, asked for one with sizzling fast specs (https://www.msi.com/Laptop/GS65-Stealth-Thin-8RF), and had them upgrade it to 32GB of super-fast RAM (G.SKILL Ripjaws DDR4-2400 SO-DIMM) and a 2TB SSD (WD Black SN750 2TB PCIe Gen3 x4 NVMe M.2 2280 Read:3400Mb/s,Write: 2900MB/s). Installed Logos fresh on it, let it download everything and complete indexing, and was disappointed already at the speed. Ugh. I'm really tired of this.
Rosie your machine should be crazy fast. I know when Logos first launches a bible it has to apply all the Visual Filters, and this can be very slow, and can take several minutes if many filters present, or many bibles versions being opened that have the filter applied, as it has to apply it to each version.
However once it completes this process, the cpu should return to normal. Do you have lots of visual filters?
Seems like this is an area that Faithlife should work on speeding up.
Yes, I do have lots of VFs. I've experimented a lot with them, so I probably have lots of leftover test ones that I don't need anymore, and I could delete those. That might help.
When I open the VF menu in my Preferred Bible, I see the following sections, with these numbers of checkboxes in each one:
RESOURCE - 5 (It looks like this is a fixed number and I can't reduce it.)
COMMUNITY NOTES - 11 (I'm guessing this is because of all the FL groups I'm a member of and the only way to reduce this number is to leave the groups?)
MEDIA - 3 (It looks like this is a fixed number.)
NOTES AND HIGHLIGHTS - 8 (I'm guessing this is all the notebooks I have which cover Bible texts; I could probably reduce these)
PASSAGE LISTS - 13 (I could get rid of most of these; most of them were experiments or for short term usage)
READING PLANS - 5 (I don't use reading plans anymore; I could probably get rid of all but one of these)
VISUAL FILTERS - 3 (These were all experiments)
But seriously, if this product is supposed to be able to accommodate power users, it doesn't seem too much to ask that it not slow down appreciably when you have a handful of notes an passage lists and reading plans and VFs and are a member of several FL groups.
Kevin said:1) If all of that bible version is closed, and reopend, it has to go through this process again.
What a pain. Can't it cache that information?
0 -
Rosie Perera said:
But seriously, if this product is supposed to be able to accommodate power users, it doesn't seem too much to ask that it not slow down appreciably when you have a handful of notes an passage lists and reading plans and VFs and are a member of several FL groups.
I agree but the reality is that I converted all my VF to refer to a single resource that I rarely open and it sped up the startup time significantly. When I want to use one, I set it to the resource I'm using. It didn't help that a beta turned everything in the VF icon on and FL left the cleanup to be handled manually. A year later I'm still working on it.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Rosie Perera said:
What a pain. Can't it cache that information?
Agreed It would be nice if they could cache it!
To see what a difference the VF make Rosie, you could just go to your Docs, then find the Visual Filters, then delete them. Then try Logos out to see what difference it makes to startup time.
As described above, it makes no difference if the VF is ticked or not in the menu, it will still be processed if it is applied to the bible in the actual Visual Filter, either directly, or through a collection, slowing Logos dramatically when the Bible is opened.
You can then go to https://documents.logos.com/?offset=0&sortType=Date&sortOrder=Descending&deleted=True&privacy=All&documentKind=visualfilters to restore them afterwards, whereby maybe you can change the filters to be used only occasionally as either MJ has described above, or the way I do using a collection described above that.
0 -
Kevin said:Rosie Perera said:
What a pain. Can't it cache that information?
Agreed It would be nice if they could cache it!
It is partially cached. When there's a visual filter, two things happen:
- The search is run.
- The highlights are applied to the resource.
(1) is cached. (2) isn't. (2) is potentially quite complex because your visual filter could easily cause the resource to re-flow. Re-drawing that takes time.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0
