[resolved] 6.3 B5 Indexer memory hog??

Page 1 of 2 (28 items) 1 2 Next >
This post has 27 Replies | 1 Follower

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Posted: Mon, May 11 2015 12:54 PM

Twice since installing Beta 5 I have noticed my computer virtually unresponsive when the Indexer was working on a book or books that Logos downloaded in the background. I used Task Manager each time to check system usage. Here is what I saw just a few minutes ago:

This is exactly what I saw a few days ago when I had this problem. Note that this is with indexing paused for four hours. (I have never seen as much memory used on my computer before.) As you can see there is little CPU activity (the unseen sections are like the ones showing).

I used Task Manager to terminate the LogosIndexer process and I got this:

This seems to point to a memory management problem with the new indexing system.

Here are my log files:

3107.LogosIndexer (6).zip

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 406
Fred Greco | Forum Activity | Replied: Mon, May 11 2015 1:18 PM

I experienced the same phenomena. 

Fred Greco
Senior Pastor, Christ Church PCA, Katy, TX
Windows 10 64-bit; Logos 7.1 SR-2 (Reformed Platinum)

Posts 26999
Forum MVP
MJ. Smith | Forum Activity | Replied: Mon, May 11 2015 1:25 PM

I'd not had problems with the indexing interfering with my work beyond a minor slowdown until the indexer was "fixed" for others and I started experiencing periods of total nonresponsiveness. But I've not complained because I'm too lazy to do the sampling necessary to help solve the problem.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 8076
LogosEmployee

Can you show the information for LogosIndexer.exe (under "Processes") the next time this happens? Please make sure the "Memory (private working set)" column is showing.

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Replied: Mon, May 11 2015 2:12 PM

Will do.

I've closed and relaunched Logos. The Indexer started up and so far everything looks normal.

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Replied: Mon, May 11 2015 2:54 PM

It took a little time but the problem repeated itself. The Indexer log shows that the indexer was triggered and ran for a long time. During that time, once again memory use grew to take over the computer. The Processes tab doesn't show that the Indexer is using an extreme amount of memory, but when I force the indexer process to end, memory is suddenly restored.

7077.LogosError (6).zip

Here are some screen shots of the problem:

I took a series of screen shots of all the processes running. None of them were using an extreme amount of memory. (I can attach them to a follow-up if you desire.)

Here is what performance looks like right after ending the indexer process:

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 8076
LogosEmployee

Thanks for the update.

Please download VMMap: https://technet.microsoft.com/en-us/library/dd535533.aspx

Then, the next time this happens, run vmmap, select "LogosIndexer" in the process list, choose File > Save As and save the information. It will create a fairly large file, but it should compress a lot when zipped.

You can post it here, or email it to logosbugs@logos.com.

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Replied: Mon, May 11 2015 5:54 PM

Thanks, Bradley. Will do.

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Replied: Mon, May 11 2015 7:40 PM

I am now on L6.3 SR-1. The problem persists (as expected).

Here is the VMMap file (it was pretty small): 5758.LogosIndexerVMMap.zip

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Replied: Mon, May 11 2015 8:12 PM

Just to complete the loop: I let Logos continue to run. Indexing did complete and the memory was released:

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 8076
LogosEmployee

Everything looks fine to me.

Your other applications are only using 2GB of RAM, so Windows is using the remaining 6GB to cache files being read by the Indexer, mostly index.idx. This file is 9.5GB but only 5.9GB of it is in memory, so Windows isn't reading the whole thing and forcing other applications out of memory; instead, it's using otherwise-unused memory to speed up the Indexer.

Strictly speaking, the memory is in use (i.e., not Available); practically speaking, Windows knows that it can instantly free that memory and reuse it for other purposes as soon as it's needed. (It won't need to be paged to disk, because it's an exact copy of the information in the index.idx file. This is reflected in the "Commit (GB): 5 / 15" statistic, where only 5GB of commit charge is used, even though 8GB of RAM is "used"; cf. http://windows.microsoft.com/en-us/windows/see-details-computers-performance-task-manager#1TC=windows-7.) You can force this to happen by making Windows trim the working set of the process, e.g., with https://tekout.wordpress.com/2012/09/20/evilmonkey-part-1-trim-working-set-of-a-running-process/. If you do that, the used memory will instantly drop to basically the level where it was when the Indexer was started. (Or the same thing happens if you kill the Indexer.)

High RAM usage isn't necessarily a bad thing; presumably, RAM is installed so it can be used. Smile Windows will balance the memory requests of all applications on the system and, when more RAM is requested by other applications, free some of the RAM "in use" by the Indexer.

Posts 6489
Forum MVP
Lynden Williams | Forum Activity | Replied: Mon, May 11 2015 9:24 PM

Will keep in mind Bradley.

Everything ever written in Religion and Theology formatted for Logos Bible Software.Logos Youtube Channel

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Replied: Mon, May 11 2015 9:35 PM

Bradley Grainger (Faithlife):
Windows knows that it can instantly free that memory and reuse it for other purposes as soon as it's needed.

If it is supposed to do this, it isn't. As I said I can try to use another open app while this is happening and I get almost no response. It is as if the Indexer has taken over and Windows isn't doing anything to re-allocate memory. I am stuck waiting for the Indexer to stop indexing.

As I said, this is new. I've never had this problem before. Yes stuff has always slowed down when the indexer is running, but as I said, now the computer is basically unresponsive during indexing.

If the new way of running Logos is background downloading and indexing this isn't the way to accomplish the task. All of a sudden something I didn't plan for (a stealth download that triggers indexing) brings my other programs to almost a dead stop.

Bradley Grainger (Faithlife):
High RAM usage isn't necessarily a bad thing; presumably, RAM is installed so it can be used. Smile Windows will balance the memory requests of all applications on the system and, when more RAM is requested by other applications, free some of the RAM "in use" by the Indexer.

Ain't happening. That's all I can say. It ain't happening.

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 6489
Forum MVP
Lynden Williams | Forum Activity | Replied: Mon, May 11 2015 9:46 PM

Ordered two free Vyrso books to see how it would respond while having a layout open.

I am using a desktop with 4 Ram, Ssd, and a Pentium Dual Core on Windows 7 Pro.

Computer was still responsive.

Everything ever written in Religion and Theology formatted for Logos Bible Software.Logos Youtube Channel

Posts 6489
Forum MVP
Lynden Williams | Forum Activity | Replied: Mon, May 11 2015 9:48 PM

Both Internet Explorer and Chrome were open.

Everything ever written in Religion and Theology formatted for Logos Bible Software.Logos Youtube Channel

Posts 6489
Forum MVP
Lynden Williams | Forum Activity | Replied: Mon, May 11 2015 9:52 PM

The program is a bit sluggish in responding while indexing is taking place.

Everything ever written in Religion and Theology formatted for Logos Bible Software.Logos Youtube Channel

Posts 9101
Forum MVP
Mark Smith | Forum Activity | Replied: Tue, May 12 2015 8:25 PM

Just to report a bit more.

Tonight the Indexer started up again and before long almost all of my 8GB of memory was taken up but I needed to use the computer.

I tried to pause indexing for 4 hours. It took a minute or two after clicking on the Logos icon in the task bar to get the option to appear. Finally I was able to pause it.

This screen shot shows what happened. The red arrow points to the time I paused the indexer. As you can see, there was only a small decline in the amount of memory being used. The indexer is stopped but it has released very little memory. I suppose it is keeping its files in memory so it will be ready to go in 4 hours, but until then I have to deal with a very slow computer. As stated yesterday Windows isn't helping the way Bradley suggested. I am not seeming to get any reallocated memory or else I'm so topped out that everything is running very slowly.

If I were to force the indexer to terminate I'd get back a lot of memory. Pausing gains me very little. If Logos is holding its data in memory, I'd suggest that when the indexer is paused it cache it to disc and thereby release the memory.

2727.LogosIndexerVMMap2.zip

Pastor, North Park Baptist Church

Bridgeport, CT USA

Posts 1471
Angela Murashov | Forum Activity | Replied: Wed, May 13 2015 4:03 PM

Mark Smith:
Twice since installing Beta 5 I have noticed my computer virtually unresponsive when the Indexer was working on a book or books that Logos downloaded in the background.

This should be improved in 6.3 RC2.

Posts 667
Jonathan Pitts | Forum Activity | Replied: Fri, May 15 2015 12:29 PM

After too many years, I can actually use my computer while it's indexing. Thank you.

Posts 13368
Forum MVP
Mark Barnes | Forum Activity | Replied: Fri, May 15 2015 12:55 PM

Mark Smith:
If I were to force the indexer to terminate I'd get back a lot of memory. Pausing gains me very little. If Logos is holding its data in memory, I'd suggest that when the indexer is paused it cache it to disc and thereby release the memory.

Logos isn't holding that data in memory. Windows itself is caching the indexing file in memory. When the LogosIndexer process terminates, Windows releases the file and hence the memory. I'm not even sure a process can control whether this caching occurs.

You can see what's happening with the RAMMap tool. The screenshot below shows that Windows is caching about 4Gb of my library index, but only 400kb is active (i.e. can't be released because it's in use). There's a further 336Mb that's modified — that can be released, but needs to be written first. The rest of it (almost 3.6Gb) is on standby. That means if LogosIndexer.exe needs the data then it's already in RAM, but if another application needs the RAM, then Windows will just remove index.idx or index-build.idx from the cache and use the freed RAM for whatever's needed.

Page 1 of 2 (28 items) 1 2 Next > | RSS