Strange Disk Usage Stats due to Logos index/scratch files???

I have a large library (roughly 31 gig). My 500 gig hard disk (HD) was showing only 8 gig left (neither Dell or I could calculate more than 230 gig actually being used of a the 464 gig drive (after formatting)). Why the 234 gig discrepancy?

Logos 6 downloaded some more files, ran the indexing routine and suddenly I had 230 gig available on the HD!!! I wonder what the size of the index files run for a large library? Are they deleted when new indices are created? More important, are they hidden from the properties command of file explorer so that their size is not obvious to the casual user (I unhide all files on my Win 7 ultimate computer)? Where are these indices found on the HD? On the other hand, does the OS disk space remaining tally accurately reflect their presence? Dell and I are stumped and a programmer friend of mine at my synagogue strongly suspected that indexing scratch files could be the cause.

I am not accusing Logos indices of being the culprit, rather I am trying to gather information and eliminate possibilities. I fear that if the available space tally drops much less than 6 gig, the OS will be unable to write its scratch files and the system will grind to a halt or stall.

Find more posts tagged with

Comments

JAL

Nick,

I'm sorry that I can only offer a vaguely recollected anecdote of my own experience when I updated to 6 from Verbum Beta 5.3.

When I updated the installation on my Dell M6600 workstation, with a quad-core i7 & 16GiB RAM running Windows 7 Pro (64 bit), the paged memory swap file grew enormously (If memory serves, it was well in excess of 100GiB.). I closed all of the other user-processes and allowed the indexer to run selecting to continue, several times, when the system promted to end the indexing process with a warning that memory was extremely low.

Eventually the process did complete. I have, more than once, subsequently entered the command "Rebuild index" on this same hardware without similar resource usage. This only occurred immediately following the initial installation of Verbum 6.

Nick Bartol

https://community.logos.com/discussion/comment/699633#Comment_699633

JAL,

Thanks. Your computer configuration is almost identical to mine (I only have 8 GB of memory)! I will start looking at the paged memory file. Interestingly, it is 12 gig now which strikes me as high and may explain much of what is going on. Neither Dell (2 techs) or I thought to look at the size of individual file in the root directory in our count of total disk space used. It looks like your problem progressed farther than mine as I have not received the warning messages YET!

Nick

Dave Hooton

I have a large library (roughly 31 gig). My 500 gig hard disk (HD) was showing only 8 gig left (neither Dell or I could calculate more than 230 gig actually being used of a the 464 gig drive (after formatting)). Why the 234 gig discrepancy?

I can't think of a reason why Logos indexing would influence File Explorer to that extent, but then i haven't observed it during indexing. It requires space up to the limit of the size of your index folder which could be 10 GB or even 20 GB. how big is your \LibraryIndex folder in the Logos folder?

Nick Bartol

https://community.logos.com/discussion/comment/699702#Comment_699702

Dave, can you give me the full path for that folder? I can't find it and I would very much like to know where the indices are located!!!! Thanks for your help.

Nick

JAL

https://community.logos.com/discussion/comment/699706#Comment_699706

Hi Nick,

Has your system completed indexing? If so has your free disk space returned to a reasonable amount?

Dave Hooton

https://community.logos.com/discussion/comment/699706#Comment_699706

Dave, can you give me the full path for that folder? I can't find it and I would very much like to know where the indices are located!!!! Thanks for your help.

Nick

Sorry, the default path is C:\Users\username\AppData\Local\Logos\Data\randomId\LibraryIndex

your \Logos folder could be Logos5 or Logos4.

Nick Bartol

https://community.logos.com/discussion/comment/699702#Comment_699702

I do not know of a way to limit the size of a directory (other than running out of HD space). Currently it is 6.54 GB. Who knows what it was before. I wonder whether JAL's idea of a bloated page file is part of the solution as that would explain the HD usage discrepancy (I didn't count individual files, only directories under the root drive). BUT that page file was limited to 12.110 GB and it seems to be observing the limit. Will the index files grow pretty much without limit? If so, how does one pare them back?

Nick Bartol

https://community.logos.com/discussion/comment/699751#Comment_699751

It was around when the indexing stopped Friday that the drive returned to normal.

Dave Hooton

https://community.logos.com/discussion/comment/699861#Comment_699861

I do not know of a way to limit the size of a directory (other than running out of HD space). Currently it is 6.54 GB. Who knows what it was before.

I've never seen an abnormal size in File Explorer outside of indexing. I'm fairly certain that Logos knows the size of your current index so it will not start if there is insufficient free space. Therefore, if File Explorer is only affected during indexing it is temporary and no cause for concern (I have Logos on an 80 GB drive and a 32 GB micro SD card).

Nick Bartol

https://community.logos.com/discussion/comment/700102#Comment_700102

Dave,

These comments are helpful. I was frightened to death when I got down to the last 10 GB of my 500 GB HD, fearing that the system would grind to a halt when it got only a bit smaller. What I understand you to say is

1. Logos 6 knows when it is space is getting critical and it will manage the situation. The runaway disk space problem is self-limiting. It won't go where I fear.

2. The issue of disk space use not being accounted for properly in File Explorer is known to occur on occasion during Logos indexing operations, so that too is not a cause for panic!

Have I got this right? If so, I will stop my work with Dell on this matter.

Also, if I get back in a critical space jam again, is there a way to tell Logos 6 to kill my current bloated index and start all over again with one of reasonable size (as I have now)? This would provide me with an additional level of assurance that I am not headed to my last Blue Screen of Death!

Nick

Dave Hooton

https://community.logos.com/discussion/comment/700199#Comment_700199

1. Logos 6 knows when it is space is getting critical and it will manage the situation. The runaway disk space problem is self-limiting. It won't go where I fear.

Yes

2. The issue of disk space use not being accounted for properly in File Explorer is known to occur on occasion during Logos indexing operations, so that too is not a cause for panic!

I've known one program to negatively affect free disk space, necessitating a reboot, and eventually I gave it the boot! Whilst I've never studied the issue whilst Logos was indexing, there has never been a cause for concern outside of indexing.

Have I got this right? If so, I will stop my work with Dell on this matter.

At least investigate the effect on free space during indexing, and don't be concerned about individual file/folder sizes. Ensure it is close to the previous value after indexing. Use Resource monitor (in Task Manager) to monitor disk activity of LogosIndexer.exe. I'll do the same during a rebuild of my Bible index.

Dave Hooton

https://community.logos.com/discussion/comment/700461#Comment_700461

I'll do the same during a rebuild of my Bible index.

During indexing free space varied as little as I expected, and it ended with the same amount of free space as at the start (in Resource Monitor and File Explorer >> Properties). You need to work with Dell but Logos is not the culprit.

Nick Bartol

https://community.logos.com/discussion/comment/700567#Comment_700567

Dave, this is SO BIZARE! You have been so kind to do all these checks for me but we are coming to differing conclusions because we are observing different events.

I decided to run the same test as you because Logos 6 had decided to reindex (likely a full, not incremental index). Here is what I saw:

Ln	Date	Time	Free HD Space (GB)	Comment
1	02/20/15	01:39p	14.1
2	02/20/15	02:44p	14.1
3	02/20/15	02:54p	11.5	Logos 6 is indexing
4	02/20/15	04:02p	223	total disk size after formatting is 464. Indexing complete
5	02/20/15	05:00p	220	two microsoft updates were downloaded and installed.
6	02/22/15	10:05a	219
7	02/22/15	02:50p	216
8	02/23/15		215
9	02/24/15	03:45p	213
10	02/24/15	06:07p	217	Logos 6 is indexing
11	02/24/15	06:14p	216	Logos 6 is indexing
12	02/24/15	06:38p	214	Logos 6 is indexing
13	02/24/15	06:41p	213	Logos 6 is indexing
14	02/24/15	06:54p	212	Logos 6 is indexing
15	02/24/15	07:04p	211	Logos 6 is indexing; Acronis is doing a local incremental backup likely started at 7:00p
16	02/24/15	07:09p	209	Logos 6 is indexing; Acronis is doing a local incremental backup likely started at 7:00p
17	02/24/15	07:15p	208	Logos 6 is indexing; Acronis is doing a local incremental backup likely started at 7:00p
18	02/24/15	07:32p	209	Acronis stopped (and released 1GB?); Logos 6 indexer 61% complete and still running.
19	02/24/15	07:35p	209	Logos 6 is indexing
20	02/24/15	08:30p	209	Logos 6 is indexing
21	02/24/15	10:15p	210	Logos 6 has completed indexing and likely only a minute or two earlier.
22	02/25/15	09:06a	210	after turning off computer for night and rebooting in the morning.

This table is easier to see, if you print or copy it to word and print it from there.

It seems that between lines 3 and 4 Logos 6 saw that HD space was running low and it deleted its scratch files with the result that I gained roughly 210 GB of HD space.

During the next 4 days I lost 10 GB for no known reason. I don't know whether Logos was indexing during this period. I have a HUGE library (check with Dave Kaplan) so indexing is a common experience for me. Not daily but nearly so. All it takes is one update and the process starts. Maybe twice a week is a good average.

At line 10 it gained 4 GB but during the rest of the indexing period I lost 7 GB.

It appears that index routine writes scratch files that are not removed until things get very low (i.e. less than 10 GB?) Is this cleanup implemented by Logos 6 or the OS? Is your system working differently because of a different build or lower HD space or some other variable. Is there a Logos 6 switch that we can play with to change things?

It could be that the scratch files are valuable (i.e., permit the implementation of differential indexes (vs full indexes)), so one does not want to eliminate them unless there is a strong need.

Today, I added up the total disk usage (forgetting anything less than 1GB in size) and came up with 464-220=244 residual vs displayed free space of 210. I am convinced that my 220 figure is no more than 5 GB understated. Where is the 30 GB of missing space?

Any ideas about any of this?

Nick

Dave Hooton

https://community.logos.com/discussion/comment/700689#Comment_700689

It seems that between lines 3 and 4 Logos 6 saw that HD space was running low and it deleted its scratch files with the result that I gained roughly 210 GB of HD space.

The list starts with low free space at 1 & 2 and then Logos started indexing. If it needed 200+ GB the indexer would never have started at 3 or would have aborted. When it finished the correct free space is showing --> the other instances of Indexing show that it is using around 6 - 7 GB.

You need to ascertain why the HDD got to 14 GB free. So keep monitoring from that last point of 210 GB free.

Today, I added up the total disk usage (forgetting anything less than 1GB in size) and came up with 464-220=244 residual vs displayed free space of 210. I am convinced that my 220 figure is no more than 5 GB understated. Where is the 30 GB of missing space?

Did you use "size" or "size on disk"? Did you include protected system files (e.g. pagefile.sys, system restore points) and hidden files/folders (e.g. the Logos folder)? They can easily account for 30 GB.

In real terms free space went from 223 GB to 210 GB. Most of that was probably lost with the MS update and its restore points. 3-4 GB could be temporary files and log/event files from apps and the OS.

Nick Bartol

https://community.logos.com/discussion/comment/700897#Comment_700897

Dave,

Thanks for your continued help. Yes, how it got to 14 GB is the key and it seems that the HD loses about 10 GB a day (it is now 201 GB). I suspect that Logos scratch files that are not being erased is part of the problem but I am beginning to think that such is not all of it. Are these files marked for reclaiming but the OS never gets around to it?

Size is size on disk, not usable size. Yes, I did include the 12.4 GB page file. I don't know where the system restore points are hidden but presumably by checking each directory I caught them through the properties file size calculations. The first thing I do when I get a new computer is to turn off the hiding of "hidden system files." Nevertheless, I have no idea where the Logos scratch files are so there maybe some that I am missing.

Maybe now you see why I am so confused.

I think I am ready to talk with Dell again as I infer that you are pretty convinced this is not a Logos issue or if so only very tangentially. Something bigger is going wrong.

Thanks for all your help!!

Nick

Dave Hooton

https://community.logos.com/discussion/comment/700905#Comment_700905

I suspect that Logos scratch files that are not being erased is part of the problem but I am beginning to think that such is not all of it. Are these files marked for reclaiming but the OS never gets around to it?

You are still misinterpreting the monitor results. The 210 GB at the end shows it has nothing to do with Logos (nor the MS update) as indexing showed a usage of 6-7 GB which is consistent with the size of your index (6.5 GB).. If anything, Logos garbage collection at 4. is responsible for restoration of the correct figure. Until you reproduce the 14 GB you can't know what process is doing incorrect housekeeping.

Maybe now you see why I am so confused.

Keep away from the file/folder level. Run a chkdsk on the drive as that could reveal damage with incorrect free space being reported.

Nick Bartol

https://community.logos.com/discussion/comment/700961#Comment_700961

Dave,

We ran chkdsk and disk cleanup early in the game. It freed up space (30 GB) but it was eaten up within a day or two.

The system restore file idea was VERY HELPFUL. I missed such because file explorer lists such as 0 files and 0 bytes but currently it is actually 26 GB, close to solving the discrepancy problem. Thanks! Dell missed such too.

We are going to just watch it for a while. I have dropped system restore allocation to 100 GB, instead of the original 240 GB. Dell is baffled too.

Thanks again for all your help!!!!

Nick

Dave Hooton

https://community.logos.com/discussion/comment/701145#Comment_701145

We ran chkdsk and disk cleanup early in the game.

Disk Clean-up will give additional free space by deleting temporary files and System files, but you should continue to run chkdsk as it fixes free space that was not being reported correctly (due to an errant program or unexpected shut-down).