[resolved] 6.3 B5 Indexer memory hog??
Twice since installing Beta 5 I have noticed my computer virtually unresponsive when the Indexer was working on a book or books that Logos downloaded in the background. I used Task Manager each time to check system usage. Here is what I saw just a few minutes ago:
This is exactly what I saw a few days ago when I had this problem. Note that this is with indexing paused for four hours. (I have never seen as much memory used on my computer before.) As you can see there is little CPU activity (the unseen sections are like the ones showing).
I used Task Manager to terminate the LogosIndexer process and I got this:
This seems to point to a memory management problem with the new indexing system.
Here are my log files:
Pastor, North Park Baptist Church
Bridgeport, CT USA
Comments
-
I experienced the same phenomena.
Fred Greco
Senior Pastor, Christ Church PCA, Katy, TX
Windows 10 64-bit; Logos 7.1 SR-2 (Reformed Platinum)0 -
I'd not had problems with the indexing interfering with my work beyond a minor slowdown until the indexer was "fixed" for others and I started experiencing periods of total nonresponsiveness. But I've not complained because I'm too lazy to do the sampling necessary to help solve the problem.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Can you show the information for LogosIndexer.exe (under "Processes") the next time this happens? Please make sure the "Memory (private working set)" column is showing.
0 -
Will do.
I've closed and relaunched Logos. The Indexer started up and so far everything looks normal.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
It took a little time but the problem repeated itself. The Indexer log shows that the indexer was triggered and ran for a long time. During that time, once again memory use grew to take over the computer. The Processes tab doesn't show that the Indexer is using an extreme amount of memory, but when I force the indexer process to end, memory is suddenly restored.
Here are some screen shots of the problem:
I took a series of screen shots of all the processes running. None of them were using an extreme amount of memory. (I can attach them to a follow-up if you desire.)
Here is what performance looks like right after ending the indexer process:
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
Thanks for the update.
Please download VMMap: https://technet.microsoft.com/en-us/library/dd535533.aspx
Then, the next time this happens, run vmmap, select "LogosIndexer" in the process list, choose File > Save As and save the information. It will create a fairly large file, but it should compress a lot when zipped.
You can post it here, or email it to logosbugs@logos.com.
0 -
Thanks, Bradley. Will do.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
I am now on L6.3 SR-1. The problem persists (as expected).
Here is the VMMap file (it was pretty small): 5758.LogosIndexerVMMap.zip
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
Just to complete the loop: I let Logos continue to run. Indexing did complete and the memory was released:
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
Everything looks fine to me.
Your other applications are only using 2GB of RAM, so Windows is using the remaining 6GB to cache files being read by the Indexer, mostly index.idx. This file is 9.5GB but only 5.9GB of it is in memory, so Windows isn't reading the whole thing and forcing other applications out of memory; instead, it's using otherwise-unused memory to speed up the Indexer.
Strictly speaking, the memory is in use (i.e., not Available); practically speaking, Windows knows that it can instantly free that memory and reuse it for other purposes as soon as it's needed. (It won't need to be paged to disk, because it's an exact copy of the information in the index.idx file. This is reflected in the "Commit (GB): 5 / 15" statistic, where only 5GB of commit charge is used, even though 8GB of RAM is "used"; cf. http://windows.microsoft.com/en-us/windows/see-details-computers-performance-task-manager#1TC=windows-7.) You can force this to happen by making Windows trim the working set of the process, e.g., with https://tekout.wordpress.com/2012/09/20/evilmonkey-part-1-trim-working-set-of-a-running-process/. If you do that, the used memory will instantly drop to basically the level where it was when the Indexer was started. (Or the same thing happens if you kill the Indexer.)
High RAM usage isn't necessarily a bad thing; presumably, RAM is installed so it can be used. [:)] Windows will balance the memory requests of all applications on the system and, when more RAM is requested by other applications, free some of the RAM "in use" by the Indexer.
0 -
Will keep in mind Bradley.
Mission: To serve God as He desires.
0 -
Windows knows that it can instantly free that memory and reuse it for other purposes as soon as it's needed.
If it is supposed to do this, it isn't. As I said I can try to use another open app while this is happening and I get almost no response. It is as if the Indexer has taken over and Windows isn't doing anything to re-allocate memory. I am stuck waiting for the Indexer to stop indexing.
As I said, this is new. I've never had this problem before. Yes stuff has always slowed down when the indexer is running, but as I said, now the computer is basically unresponsive during indexing.
If the new way of running Logos is background downloading and indexing this isn't the way to accomplish the task. All of a sudden something I didn't plan for (a stealth download that triggers indexing) brings my other programs to almost a dead stop.
High RAM usage isn't necessarily a bad thing; presumably, RAM is installed so it can be used.
Windows will balance the memory requests of all applications on the system and, when more RAM is requested by other applications, free some of the RAM "in use" by the Indexer.Ain't happening. That's all I can say. It ain't happening.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
Ordered two free Vyrso books to see how it would respond while having a layout open.
I am using a desktop with 4 Ram, Ssd, and a Pentium Dual Core on Windows 7 Pro.
Computer was still responsive.
Mission: To serve God as He desires.
0 -
Both Internet Explorer and Chrome were open.
Mission: To serve God as He desires.
0 -
The program is a bit sluggish in responding while indexing is taking place.
Mission: To serve God as He desires.
0 -
Just to report a bit more.
Tonight the Indexer started up again and before long almost all of my 8GB of memory was taken up but I needed to use the computer.
I tried to pause indexing for 4 hours. It took a minute or two after clicking on the Logos icon in the task bar to get the option to appear. Finally I was able to pause it.
This screen shot shows what happened. The red arrow points to the time I paused the indexer. As you can see, there was only a small decline in the amount of memory being used. The indexer is stopped but it has released very little memory. I suppose it is keeping its files in memory so it will be ready to go in 4 hours, but until then I have to deal with a very slow computer. As stated yesterday Windows isn't helping the way Bradley suggested. I am not seeming to get any reallocated memory or else I'm so topped out that everything is running very slowly.
If I were to force the indexer to terminate I'd get back a lot of memory. Pausing gains me very little. If Logos is holding its data in memory, I'd suggest that when the indexer is paused it cache it to disc and thereby release the memory.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
Mark Smith said:
Twice since installing Beta 5 I have noticed my computer virtually unresponsive when the Indexer was working on a book or books that Logos downloaded in the background.
This should be improved in 6.3 RC2.
0 -
After too many years, I can actually use my computer while it's indexing. Thank you.
0 -
Mark Smith said:
If I were to force the indexer to terminate I'd get back a lot of memory. Pausing gains me very little. If Logos is holding its data in memory, I'd suggest that when the indexer is paused it cache it to disc and thereby release the memory.
Logos isn't holding that data in memory. Windows itself is caching the indexing file in memory. When the LogosIndexer process terminates, Windows releases the file and hence the memory. I'm not even sure a process can control whether this caching occurs.
You can see what's happening with the RAMMap tool. The screenshot below shows that Windows is caching about 4Gb of my library index, but only 400kb is active (i.e. can't be released because it's in use). There's a further 336Mb that's modified — that can be released, but needs to be written first. The rest of it (almost 3.6Gb) is on standby. That means if LogosIndexer.exe needs the data then it's already in RAM, but if another application needs the RAM, then Windows will just remove index.idx or index-build.idx from the cache and use the freed RAM for whatever's needed.
This is my personal Faithlife account. On 1 March 2022, I started working for Faithlife, and have a new 'official' user account. Posts on this account shouldn't be taken as official Faithlife views!
0 -
Mark, thanks for your response.
What, then accounts for the difference in how much memory is freed up by ending the indexer process in Task Manager versus pausing the indexing?
While I can't explain why it is happening, I can state that when I simply pause indexing my computer is considerably slower (all the while showing about 7 GB of 8GB being used).
This issue also is new. It came along (seemingly) with the new process of downloading and indexing that 6.3 gave us.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
Angela Murashov said:
This should be improved in 6.3 RC2.
Thank you, Angela. I will have to wait until something new downloads to see the improvement.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
Pausing doesn't do anything special to release memory. Instead, the indexer just stops working for a while.
A change unrelated to the new update system was made back in mid march to improve the performance of the indexer. It's possible that this was causing the problems you were seeing, so we reverted that change. I believe it is part of the 6.3 release. Please let us know if indexing seems to be better after the release.
Andrew Batishko | Logos software developer
0 -
Pausing doesn't do anything special to release memory. Instead, the indexer just stops working for a while
This seems to me the design bug behind that. Pausing the indexer should free up the system, that's the reason this feature exists at all. At least as Mark described the issue above, the problem choking his machine seems not to be CPU or disk, but memory usage.
I think, pausing should release memory, after all we told the indexer to not work for four hours.
Have joy in the Lord!
0 -
NB.Mick said:
This seems to me the design bug behind that. Pausing the indexer should free up the system, that's the reason this feature exists at all.
NB.Mick said:I think, pausing should release memory, after all we told the indexer to not work for four hours.
Exactly.
“The trouble is that everyone talks about reforming others and no one thinks about reforming himself.” St. Peter of Alcántara
0 -
SineNomine said:NB.Mick said:
I think, pausing should release memory, after all we told the indexer to not work for four hours.
Exactly.
The indexer doesn't allocate the memory, so pausing can't free it.
From https://community.logos.com/forums/p/104869/726343.aspx#726343:
Mark Barnes said:Logos isn't holding that data in memory. Windows itself is caching the indexing file in memory. When the LogosIndexer process terminates, Windows releases the file and hence the memory. I'm not even sure a process can control whether this caching occurs.
0 -
Mark Smith said:Angela Murashov said:
This should be improved in 6.3 RC2.
Thank you, Angela. I will have to wait until something new downloads to see the improvement.
I am seeing a big improvement. There was a large download this afternoon that triggered indexing. So far total RAM used is under 6GB (closer to 5GB) and I have full (albeit slower) use of my computer. Whatever was done, it has helped, and I appreciate it.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0 -
A change unrelated to the new update system was made back in mid march to improve the performance of the indexer. It's possible that this was causing the problems you were seeing, so we reverted that change. I believe it is part of the 6.3 release. Please let us know if indexing seems to be better after the release.
See my post above to Angela, but I am back to more normal indexing loads. Whatever you 'undid' seems to have helped a lot. Thank you.
Pastor, North Park Baptist Church
Bridgeport, CT USA
0
