[resolved] 6.3 B5 Indexer memory hog??

Twice since installing Beta 5 I have noticed my computer virtually unresponsive when the Indexer was working on a book or books that Logos downloaded in the background. I used Task Manager each time to check system usage. Here is what I saw just a few minutes ago:

This is exactly what I saw a few days ago when I had this problem. Note that this is with indexing paused for four hours. (I have never seen as much memory used on my computer before.) As you can see there is little CPU activity (the unseen sections are like the ones showing).

I used Task Manager to terminate the LogosIndexer process and I got this:

This seems to point to a memory management problem with the new indexing system.

Here are my log files:

3107.LogosIndexer (6).zip

Find more posts tagged with

Comments

Fred Greco

I experienced the same phenomena.

MJ. Smith

I'd not had problems with the indexing interfering with my work beyond a minor slowdown until the indexer was "fixed" for others and I started experiencing periods of total nonresponsiveness. But I've not complained because I'm too lazy to do the sampling necessary to help solve the problem.

Bradley Grainger (Logos)

Can you show the information for LogosIndexer.exe (under "Processes") the next time this happens? Please make sure the "Memory (private working set)" column is showing.

Mark Smith

Will do.

I've closed and relaunched Logos. The Indexer started up and so far everything looks normal.

Mark Smith

It took a little time but the problem repeated itself. The Indexer log shows that the indexer was triggered and ran for a long time. During that time, once again memory use grew to take over the computer. The Processes tab doesn't show that the Indexer is using an extreme amount of memory, but when I force the indexer process to end, memory is suddenly restored.

7077.LogosError (6).zip

Here are some screen shots of the problem:

I took a series of screen shots of all the processes running. None of them were using an extreme amount of memory. (I can attach them to a follow-up if you desire.)

Here is what performance looks like right after ending the indexer process:

Bradley Grainger (Logos)

Thanks for the update.

Please download VMMap: https://technet.microsoft.com/en-us/library/dd535533.aspx

Then, the next time this happens, run vmmap, select "LogosIndexer" in the process list, choose File > Save As and save the information. It will create a fairly large file, but it should compress a lot when zipped.

You can post it here, or email it to logosbugs@logos.com.

Bradley Grainger (Logos)

Everything looks fine to me.

Your other applications are only using 2GB of RAM, so Windows is using the remaining 6GB to cache files being read by the Indexer, mostly index.idx. This file is 9.5GB but only 5.9GB of it is in memory, so Windows isn't reading the whole thing and forcing other applications out of memory; instead, it's using otherwise-unused memory to speed up the Indexer.

Strictly speaking, the memory is in use (i.e., not Available); practically speaking, Windows knows that it can instantly free that memory and reuse it for other purposes as soon as it's needed. (It won't need to be paged to disk, because it's an exact copy of the information in the index.idx file. This is reflected in the "Commit (GB): 5 / 15" statistic, where only 5GB of commit charge is used, even though 8GB of RAM is "used"; cf. http://windows.microsoft.com/en-us/windows/see-details-computers-performance-task-manager#1TC=windows-7.) You can force this to happen by making Windows trim the working set of the process, e.g., with https://tekout.wordpress.com/2012/09/20/evilmonkey-part-1-trim-working-set-of-a-running-process/. If you do that, the used memory will instantly drop to basically the level where it was when the Indexer was started. (Or the same thing happens if you kill the Indexer.)

High RAM usage isn't necessarily a bad thing; presumably, RAM is installed so it can be used. [:)] Windows will balance the memory requests of all applications on the system and, when more RAM is requested by other applications, free some of the RAM "in use" by the Indexer.

Lynden O. Williams

Will keep in mind Bradley.

Mark Smith

Windows knows that it can instantly free that memory and reuse it for other purposes as soon as it's needed.

If it is supposed to do this, it isn't. As I said I can try to use another open app while this is happening and I get almost no response. It is as if the Indexer has taken over and Windows isn't doing anything to re-allocate memory. I am stuck waiting for the Indexer to stop indexing.

As I said, this is new. I've never had this problem before. Yes stuff has always slowed down when the indexer is running, but as I said, now the computer is basically unresponsive during indexing.

If the new way of running Logos is background downloading and indexing this isn't the way to accomplish the task. All of a sudden something I didn't plan for (a stealth download that triggers indexing) brings my other programs to almost a dead stop.

High RAM usage isn't necessarily a bad thing; presumably, RAM is installed so it can be used. Windows will balance the memory requests of all applications on the system and, when more RAM is requested by other applications, free some of the RAM "in use" by the Indexer.

Ain't happening. That's all I can say. It ain't happening.

Lynden O. Williams

Ordered two free Vyrso books to see how it would respond while having a layout open.

I am using a desktop with 4 Ram, Ssd, and a Pentium Dual Core on Windows 7 Pro.

Computer was still responsive.

Lynden O. Williams

Both Internet Explorer and Chrome were open.

Mark Smith

Just to report a bit more.

Tonight the Indexer started up again and before long almost all of my 8GB of memory was taken up but I needed to use the computer.

I tried to pause indexing for 4 hours. It took a minute or two after clicking on the Logos icon in the task bar to get the option to appear. Finally I was able to pause it.

This screen shot shows what happened. The red arrow points to the time I paused the indexer. As you can see, there was only a small decline in the amount of memory being used. The indexer is stopped but it has released very little memory. I suppose it is keeping its files in memory so it will be ready to go in 4 hours, but until then I have to deal with a very slow computer. As stated yesterday Windows isn't helping the way Bradley suggested. I am not seeming to get any reallocated memory or else I'm so topped out that everything is running very slowly.

If I were to force the indexer to terminate I'd get back a lot of memory. Pausing gains me very little. If Logos is holding its data in memory, I'd suggest that when the indexer is paused it cache it to disc and thereby release the memory.

2727.LogosIndexerVMMap2.zip

Angela Murashov

Twice since installing Beta 5 I have noticed my computer virtually unresponsive when the Indexer was working on a book or books that Logos downloaded in the background.

This should be improved in 6.3 RC2.

Jonathan Pitts

After too many years, I can actually use my computer while it's indexing. Thank you.

Mark Barnes

If I were to force the indexer to terminate I'd get back a lot of memory. Pausing gains me very little. If Logos is holding its data in memory, I'd suggest that when the indexer is paused it cache it to disc and thereby release the memory.

Logos isn't holding that data in memory. Windows itself is caching the indexing file in memory. When the LogosIndexer process terminates, Windows releases the file and hence the memory. I'm not even sure a process can control whether this caching occurs.

You can see what's happening with the RAMMap tool. The screenshot below shows that Windows is caching about 4Gb of my library index, but only 400kb is active (i.e. can't be released because it's in use). There's a further 336Mb that's modified — that can be released, but needs to be written first. The rest of it (almost 3.6Gb) is on standby. That means if LogosIndexer.exe needs the data then it's already in RAM, but if another application needs the RAM, then Windows will just remove index.idx or index-build.idx from the cache and use the freed RAM for whatever's needed.

Mark Smith

Mark, thanks for your response.

What, then accounts for the difference in how much memory is freed up by ending the indexer process in Task Manager versus pausing the indexing?

While I can't explain why it is happening, I can state that when I simply pause indexing my computer is considerably slower (all the while showing about 7 GB of 8GB being used).

This issue also is new. It came along (seemingly) with the new process of downloading and indexing that 6.3 gave us.

Mark Smith

This should be improved in 6.3 RC2.

Thank you, Angela. I will have to wait until something new downloads to see the improvement.

Andrew Batishko

Pausing doesn't do anything special to release memory. Instead, the indexer just stops working for a while.

A change unrelated to the new update system was made back in mid march to improve the performance of the indexer. It's possible that this was causing the problems you were seeing, so we reverted that change. I believe it is part of the 6.3 release. Please let us know if indexing seems to be better after the release.

Mark Smith

This should be improved in 6.3 RC2.

Thank you, Angela. I will have to wait until something new downloads to see the improvement.

I am seeing a big improvement. There was a large download this afternoon that triggered indexing. So far total RAM used is under 6GB (closer to 5GB) and I have full (albeit slower) use of my computer. Whatever was done, it has helped, and I appreciate it.