L8.5 Indexer Crash: throws out of memory exception

The indexer crashed on me. Granted, I had lots of apps open and probably memory usage was high, so not enough memory for indexing to complete. But a more graceful exit with a meaningful error message would be expected rather than a crash.

Log files:

1222.LogosIndexerCrash 2019-06-04.zip

Task Manager Memory Usage screenshot:

Resource Monitor screenshot. Note that Logos is the largest culprit in memory usage.

Find more posts tagged with

Desktop App

Comments

Dave Hooton

Note that Logos is the largest culprit in memory usage

That's not unusual, but usage for LogosIndexer.exe would be more useful. Was it running in either screenshot?

Did indexing complete after this?

Bradley Grainger (Logos)

That phase of indexing is not allocating very much memory, and it "shouldn't" fail. The commit charge being almost at 100% on your system is a worrying sign. Have you disabled/limited the size of your page file?

(Your system has enough free RAM to meet LogosIndexer's needs, but it can't back that allocated RAM by the page file in case it needs to page it out to disk, so the allocation fails. This is a fairly rare scenario on a 64-bit machine, and not one the LogosIndexer currently anticipates.)

Rosie Perera

https://community.logos.com/discussion/comment/1051740#Comment_1051740

Have you disabled/limited the size of your page file?

Nope. Its size should be ample.

Rosie Perera

https://community.logos.com/discussion/comment/1051707#Comment_1051707

That's not unusual, but usage for LogosIndexer.exe would be more useful. Was it running in either screenshot?

I doubt it. It had crashed, and I hadn't yet restarted Logos by the time I took the screenshots. But I neglected to scroll through the processes in Task Manager or Resource Monitor looking for LogosIndexer.exe.

Did indexing complete after this?

Still waiting for it. The first time I restarted Logos and tried to let it index, Windows told me it was out of memory and requested to shut down the Indexer, so I let it. I shut down and restarted Logos and let it try again. Had to shut down a bunch of other apps (see next paragraph), but it looks like it might finish this time.

[Bradley, please note] I suspect there might be a memory leak in the Indexer, because I had about a dozen apps running, and I systematically saved my work and shut each of them down one by one. The amount of free memory would shoot up momentarily (up to 1000 MB at one point), but it would get chewed up again and was pegged at 0 much of the time, until I'd shut down every visible running app. Still had a few behind-the-scenes apps running like my antivirus and my screenshot program and the Indexer was churning along, not looking like it was taking up more than about 100 MB most of the time (though it did shoot up to over 400 MB at one point, but still that shouldn't have been a problem with all the memory I've got).

But here is a troubling pair screenshots, taken within seconds of each other:

The only process taking any CPU time is LogosIndexer.exe. It seems to be choking my poor memory down to 0. It is apparently finishing, though. It's up to 97% finally as I type this.

EDIT: Yup, it finished. Here's the LogosIndexer.log file:

3782.LogosIndexer.log

Once the Indexer finished, my Free Memory was back up to normal amounts:

Mark Barnes

https://community.logos.com/discussion/comment/1051746#Comment_1051746

Something's not right. There's no way that all those processes in your screenshot add up to 94% of 12Gb. I wonder whether clicking Show processes from all users might reveal where the 'missing' RAM went — probably the SYSTEM user.

mab

https://community.logos.com/discussion/comment/1051762#Comment_1051762

My two-cents on this is that the indexer is just going to being a pig. When it runs, the best thing is to turn everything else off and maybe just use another machine. I never have a problem when I do this, it's only when I don't that I become exasperated.I can multitask with almost everything except when the indexer kicks in.

Mark Barnes

https://community.logos.com/discussion/comment/1051805#Comment_1051805

My two-cents on this is that the indexer is just going to being a pig. When it runs, the best thing is to turn everything else off and maybe just use another machine. I never have a problem when I do this, it's only when I don't that I become exasperated.I can multitask with almost everything except when the indexer kicks in.

Personally, I don't even notice the indexer running.

mab

https://community.logos.com/discussion/comment/1051832#Comment_1051832

Personally, I don't even notice the indexer running.

My hardware is old. It's not usually bad so much anymore, but a big pile of new or updated resources will clobber things.

Dave Hooton

https://community.logos.com/discussion/comment/1051742#Comment_1051742

Have you disabled/limited the size of your page file?

Nope. Its size should be ample.

When you click Change..., what is the Recommended amount?

Rosie Perera

https://community.logos.com/discussion/comment/1051832#Comment_1051832

My two-cents on this is that the indexer is just going to being a pig. When it runs, the best thing is to turn everything else off and maybe just use another machine. I never have a problem when I do this, it's only when I don't that I become exasperated.I can multitask with almost everything except when the indexer kicks in.

Personally, I don't even notice the indexer running.

My experience is like mab's. Everything grinds to a halt when the Indexer is running. I have to pause it and let it run overnight. Mind you, I have a ginormous library (~30,000 resources), but I thought indexing was incremental when you got new resources, that it didn't have to go back and reindex everything, so the size of my library shouldn't matter. However I do get new resources downloading just about every day, so I have to index every day.

Rosie Perera

https://community.logos.com/discussion/comment/1051867#Comment_1051867

Something's not right. There's no way that all those processes in your screenshot add up to 94% of 12Gb.

After the Indexer finished, it's back down to 71% now, and I still haven't rebooted my computer, and have restarted some of the dozen apps I had running before I had to shut them all down.

I wonder whether clicking Show processes from all users might reveal where the 'missing' RAM went — probably the SYSTEM user.

There's not much else taking up memory for other users. Here are the bigger items:

CrashPlanService.exe SYSTEM 124 MB -- I could ditch this since I don't use CrashPlan anymore as backup software.

svchost.exe SYSTEM 99 MB -- part of Windows; can't get rid of it

SearchIndexer SYSTEM 43 MB -- part of Windows; can't get rid of it

AVGSvc.exe SYSTEM 39 MB -- my antivirus; can't get rid of it

And a bunch of other smaller stuff.

Have you disabled/limited the size of your page file?

Nope. Its size should be ample.

When you click Change..., what is the Recommended amount?

18418 MB. Should I lower it to that?

EDIT: BTW, the system is automatically managing the paging file size, so blame Windows for setting my it to twice the recommended size. I can still ask should I disable automatic management of it and set it down to the recommended size? Why would Windows recommend a size and then automatically set it to double that size?

Dave Hooton

https://community.logos.com/discussion/comment/1051872#Comment_1051872

18418 MB. Should I lower it to that?

EDIT: BTW, the system is automatically managing the paging file size, so blame Windows for setting my it to twice the recommended size. I can still ask should I disable automatic management of it and set it down to the recommended size? Why would Windows recommend a size and then automatically set it to double that size?

You could set it to 18418 MB but also set a Maximum size of around 32532 MB. Then you can monitor what it is actually allocating.

My setting is half (4096 MB) the physical memory of 8 GB and it hasn't changed. It is double what is recommended, but I rarely use more than 50% of memory.

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/1051876#Comment_1051876

You could set it to 18418 MB but also set a Maximum size of around 32532 MB. Then you can monitor what it is actually allocating.

Use "Automatically manage size"; don't set hard limits.

EDIT: BTW, the system is automatically managing the paging file size, so blame Windows for setting my it to twice the recommended size. I can still ask should I disable automatic management of it and set it down to the recommended size? Why would Windows recommend a size and then automatically set it to double that size?

IIRC, "Recommended" is just 150% of RAM, which is a reasonable default, but not tailored to your system. The actual size will have been driven by applications' demands on virtual memory, and reflects what Windows actually needs to run all the applications you use at once.

Don't set it lower or you may run into "out of memory" conditions earlier with other applications.

Dave Hooton

https://community.logos.com/discussion/comment/1051877#Comment_1051877

You could set it to 18418 MB but also set a Maximum size of around 32532 MB. Then you can monitor what it is actually allocating.

Use "Automatically manage size"; don't set hard limits.

IIRC, "Recommended" is just 150% of RAM,

I've always set limits, and monitor actual usage.

My desktop, also with 8 GB RAM, is set to a minimum 1920 MB and that is what is currently allocated. Recommended is 1898 MB, whilst my Notebook recommends 1904 MB. I've only had one hard problem with out of memory, but it wasn't on my current Notebook; which is probably why the minimum is a generous 4096 MB!

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/1051746#Comment_1051746

I suspect there might be a memory leak in the Indexer, because I had about a dozen apps running, and I systematically saved my work and shut each of them down one by one. The amount of free memory would shoot up momentarily (up to 1000 MB at one point), but it would get chewed up again and was pegged at 0 much of the time, until I'd shut down every visible running app.

There's not a memory leak, but it's very aggressive at using free RAM to cache data. (It seems to me that sometimes there's a myth that programs using RAM is bad. It's not; it's good! If you don't want your RAM to be used, pull it out of the computer and put the chip on your desk. [;)])

Specifically, the indexer asks Windows to cache as much of the index files in RAM as possible (to speed up random access to those files). This doesn't show up as "Memory (Private Working Set)" because it's not "private" to the indexer process; it can be shared with Logos 8 or any other process on the system.

The indexer hasn't "locked" the data from the index file in memory; Windows is free to discard it as soon as it's not being used and other applications need RAM. In practice, it seems that Windows (perhaps particularly Windows 7) doesn't feel much sense of urgency in removing this cached data, and your RAM will appear as 0% free for a long time. Unless lots of page file thrashing is happening, I wouldn't worry too much about it.

We have tweaked the process to not be quite as aggressive in the past; see this thread: https://community.logos.com/forums/t/104869.aspx

Right now, it seems to me to be functioning as designed. However, I am concerned about your system commit limit. Note that in Mark's screenshot (from that other thread), even though 8GB of RAM was in use, system commit was only "5 / 15 GB". On your system, 12 GB RAM is used and system commit is "46 / 47 GB". That's both a) a lot higher, and b) a very high percentage, 98% or more.

If it were me, I would use SysInternals VMMap, RAMMap, or Resource Monitor to investigate which processes have the greatest amounts of committed memory and see if any are unreasonably high.

(In my experience, the JVM is notoriously bad at returning GC'ed memory to the OS (look at the difference between commit and working set), so I would look particularly closely at CrashPlan or any other Java applications you have running...)

Mark Barnes

https://community.logos.com/discussion/comment/1051958#Comment_1051958

CrashPlan is a notorious memory hog. 1.5Gb is not uncommon. It’s almost as bad as Chrome.

Rosie Perera

https://community.logos.com/discussion/comment/1051988#Comment_1051988

CrashPlan is a notorious memory hog. 1.5Gb is not uncommon. It’s almost as bad as Chrome.

Thanks. I've disabled CrashPlan, since I'm no longer using it. I use Acronis True Image now instead. I hadn't even realized CrashPlan was still running on my machine.

Yes, Chrome is a hog. I had switched to it from Firefox a few years back, because I was led to believe it was more secure. But now I'm wondering whether I should switch back.

Dave Hooton

https://community.logos.com/discussion/comment/1052020#Comment_1052020

I had switched to it from Firefox a few years back, because I was led to believe it was more secure. But now I'm wondering whether I should switch back.

You should!

Rosie Perera

https://community.logos.com/discussion/comment/1051958#Comment_1051958

There's not a memory leak, but it's very aggressive at using free RAM to cache data. (It seems to me that sometimes there's a myth that programs using RAM is bad. It's not; it's good! If you don't want your RAM to be used, pull it out of the computer and put the chip on your desk. )

I know the difference between a memory leak and aggressive usage of RAM. I'm a software engineer.

I'm not saying using RAM is bad. I'm just a software engineer from the era when we actually used to write code that was optimized for memory use as well as speed. So that it would run well on machines that didn't have as much RAM.

And so that it wouldn't cause the operating system to do this on a machine with lots of memory (12 GB):

If it were me, I would use SysInternals VMMap, RAMMap, or Resource Monitor to investigate which processes have the greatest amounts of committed memory and see if any are unreasonably high.

Logos is using the most. I'm not sure what to consider "unreasonably high." I'm surprised that AVG (my antivirus software) is using that much. The others don't look like that big a deal.

Also, LogosIndexer is causing 2000 page faults per second. All of that thrashing causes my computer to run extremely slowly when the indexer is running. That's another reason not to be quite so aggressive in using memory. The Indexer is a background process. I want to still be able to get work done when it's running, and would accept that it might take longer to finish so that I can be productive. I know I can pause indexing to get work done, but it ought to be able to adjust its process priority based on whether there's been user input or whether the user is idle.