Indexing Woes

https://community.logos.com/discussion/comment/175558#Comment_175558

I deleted the BibleIndex and LibraryIndex folders and reopened Logos. It indexed for about 3 hours and stopped.

It didn't crash, but I have no index, just an indexer log file filled with warnings. When I reopen Logos it starts indexing again.

Here is the Indexer log file.

4137.Logos4Indexer.zip

Fred Chapman

Mark,

Man you are going through it this weekend. I have been following your threads and feel for you.

I am far from an expert reading these logs, but take a look at this thread http://community.logos.com/forums/t/19295.aspx and see if there is something that might help you. One of the posts includes a log that is similar to your crash log. Here is a portion of that log

Error    Program    Unhandled exception: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: nNumber
   at Libronix.Utility.BitWriter.WriteUnary(Int32 nNumber)
   at Libronix.Utility.BitUtility.WriteUnary(BitWriter writer, Int32 nNumber)
   at Libronix.Utility.BitUtility.WriteGamma(BitWriter writer, Int32 nNumber)
   at Libronix.Utility.BitUtility.WriteDelta(BitWriter writer, Int32 nNumber)
   at Libronix.SearchEngine.SortedPostingWriter.Write(Posting posting)
   at Libronix.SearchEngine.PostingsFileCreator.ExternalMergeSortFileBlocks(BlockRunManager mgrIn, BlockRunManager mgrOut, IWorkState state, Action`1 fnReportProgress)
   at Libronix.SearchEngine.PostingsFileCreator.WritePostings(IWorkState state, IIndexerProgress progress)
   at Libronix.SearchEngine.Indexer.WritePostings(IWorkState state)
   at Libronix.DigitalLibrary.LibraryIndex.DoBuildIndex(ILibraryIndexWorkState state, Int32 nResourceCount, String strIndexName, Int32 nIndexableResourceCount, Int32 nMaxAdditionalThreads, Boolean bShouldMerge)
   at Libronix.DigitalLibrary.LibraryIndex.BuildIndex(LibraryIndexWorkState state, String strIndexName, Int32 nIndexableResourceCount, Boolean bShouldMerge)
   at Libronix.DigitalLibrary.LibraryIndex.IndexResources(IPausableWorkState threadOwnerWorkState, LibraryIndexWorkState state)
   at Libronix.Utility.Threading.WorkStateThreadOwner`1.ThreadProc(Object objData)
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)

I hope you get this worked out soon

https://community.logos.com/discussion/comment/175575#Comment_175575

Thanks for the link, Fred. Unfortunately they tried a lot and never finished the thread with the solution.

I have the feeling that something is corrupt in my Resources file. I just deleted the ResourceManager.db file and the Index files again and am trying over. At least I can use my 4.0b SR-2 installation even with some of the bad collections it contains.

Thanks again.

https://community.logos.com/discussion/comment/175579#Comment_175579

I have the feeling that something is corrupt in my Resources file.

There are a lot of warnings from resources I would not expect to see:

Warn NativeLogosResourceIndexer Failed to get word position for xxxx

They don't necessarily cause a critical error/crash later on but may be instrumental in this case, where you are crashing as it is writing the index to disk (all resources were indexed). It only affects the library index, but I would perform a full re-installation if your latest attempt fails.

use a Quick method if your 4.0d resources are OK.
Method 2 should work after you sort the issue of collections
Note: hidden files will be lost because you deleted ResourceManager.db (unless you prevented it from sync'ing).

https://community.logos.com/discussion/comment/175599#Comment_175599

Thanks, Dave.

I am letting it try to reindex. I will leave it overnight and take a look after church in the morning. If that doesn't work reinstalling is all I can think of.

BTW: My hidden files did not get lost in the process. I do have 1 less resource in my Library resource count than I did before I deleted that file. That is strange.

https://community.logos.com/discussion/comment/175621#Comment_175621

BTW: My hidden files did not get lost in the process.

May have had time to sync from server?

I do have 1 less resource in my Library resource count than I did before I deleted that file. That is strange.

It could be an obsolete resource from 4.0 beta testing - I saw an obsolete database in your Indexer log.

Melissa Snyder

https://community.logos.com/discussion/comment/175621#Comment_175621

I am letting it try to reindex. I will leave it overnight and take a look after church in the morning. If that doesn't work reinstalling is all I can think of.

Mark ~ I've submitted your original log to development. ~ Thanks, Melissa

https://community.logos.com/discussion/comment/175621#Comment_175621

I am letting it try to reindex. I will leave it overnight and take a look after church in the morning.

Mark

How is this progressing?

https://community.logos.com/discussion/comment/176779#Comment_176779

How is this progressing?

Thanks for asking.

It hasn't progressed. Everything I attempted by way of deleting files was ineffective, seeming to leave a reinstall as the only option. I decided not to bother with a reinstall since the beta looks to go live in a short while. Since I have 4.0b SR-2 on the same computer I can still use Logos.

https://community.logos.com/discussion/comment/177151#Comment_177151

I do have some new information. Today I let SR-1 download and install. It indexed afterward. There is no Indexer Crash log nor does the Indexer log show that it terminated unusually. However that file is full of the same strange warnings the last files were. I have a Bible index (I had it before the update). I cannot search the Greek NT (haven't tried the Hebrew) so that must not be indexed. I have no ability to search other resources in my Library. While it gives the yellow bar notice about the possibility of missing data while indexing, the fact is no indexing is going on.

Attached are the latest log files.

3426.Logos4 Logs 2.zip

Mark Barnes

Mark,

I think the most likely option is corrupt downloads, though Google suggests only Dave has had this error before. In the absence of better solutions, I'd:

Check previous error logs to see if the "Failed to get word position" errors are in the same resources each time.
If so, I'd delete them and re-download them.

https://community.logos.com/discussion/comment/177160#Comment_177160

Thanks for the suggestion. Unfortunately there are thousands of lines of these errors referring to an uncounted number of resources. I can't imagine working through that one at a time, if that's what you mean.

The other thing is the indexer log just ends when a fraction of my resources are reported as indexed. No crash. I seem to have a LibraryIndex.db file (see below) but as I reported I get no results from a simple search.

For comparison here is the same folder in my stable 4.0b SR-2 installation:

I have thought about just re-downloading all my resources, but that would take a long time and I don't know if that is what the problem is. If the developers could explain the source of the warnings in the log file as stemming from corrupt resources, then I would re-download. But maybe the warnings have nothing to do with the problem, or the problem has nothing to do with corrupt resources. Since I do have 4.0b running fine I am not in a must-have situation, so don't want to spend a bunch of time to fix this without being sure I would fix it. I felt I should report after the update to 4.1 SR-1, however.

Mark Barnes

https://community.logos.com/discussion/comment/177170#Comment_177170

If you run the indexer a couple of times, and it always baulks on the same resource(s), I think it would be worth deleting a couple of those resources and seeing if when they're re-downloaded they cause the same problems or not. At least then you'd know whether it's a file corruption issue or something else.

Of course if the indexer problems are different every time than clearly something else is at work.

https://community.logos.com/discussion/comment/177170#Comment_177170

Unfortunately there are thousands of lines of these errors referring to an uncounted number of resources.

The Warnings I encountered didn't affect the indexing process, and I can access the resources. I feel it is a document integrity issue that should be addressed.

I have thought about just re-downloading all my resources, but that would take a long time and I don't know if that is what the problem is.

I too would like to see a response from the developers rather than a quick fix. It would help to upload a compressed version of your indexer log.

https://community.logos.com/discussion/comment/177243#Comment_177243

I'm currently debugging a crash we have encountered when indexing GNOSTSCRPT or OTPSEUD01. Those resources aren't in your log, so they're not the ones causing the problem for you, but (at the moment) it seems possible that there could be other resources that might be causing this crash (and you have one or more of them). Unfortunately, it's not possible to tell from the log which ones they are.

I'll post an update once we know more about the problem.

https://community.logos.com/discussion/comment/178400#Comment_178400

I'll post an update once we know more about the problem.

Thanks, Bradley.

https://community.logos.com/discussion/comment/178639#Comment_178639

The crash I encountered was caused by having descriptive metadata attached to an image in a footnote. It will be fixed in 4.1 SR-2.

Although you were getting a different exception message (and indexing different resources) than I was, it is possible that the same underlying bug also caused your crash, so I think that SR-2 should solve this problem. (And if it doesn't, I also added a little more logging to try to help identify the cause.)

https://community.logos.com/discussion/comment/178651#Comment_178651

I downloaded and installed SR-2. The indexer crashed once again. The indexer log file seems to be full of errors again but of a different sort than before. The indexer crash log seems to show an error similar to the one dated 9/26/10, submitted earlier.

5808.Logos logs.zip

https://community.logos.com/discussion/comment/178937#Comment_178937

The first problem reported in your log was indexing ANCH.logos4 (AYBD). In the initial Logos 4.0 beta, the indexer used to crash if this happened, but we changed it to keep running (so that one corrupt old LLS resource wouldn't prevent indexing succeeding). However, in your case it seems very likely that indexing AYBD has corrupted the internal state of the indexer, such that it crashes on every following resource. It then completely fails in the second phase of indexing because the data it's trying to process is completely invalid.

I suggest either hiding the AYBD resource (in settings) or just deleting the ANCH.logos4 file and seeing if the indexer is able to run to completion without that resource.

https://community.logos.com/discussion/comment/179018#Comment_179018

I was debugging a crash dump (from Windows Error Reporting) today, and I was actually able to identify it as having come from your computer (under the "Mark's Test" account). The crash dump it recorded was a System.ExecutionEngineException (basically, an unexpected error inside the .NET runtime itself) while the indexer was processing the reverse interlinear data for an undetermined resource).

Unfortunately, this type of exception tends to indicate that something very bad happened an indeterminate time in the past, and it has only just been detected. (As an example, if the Logos indexing code has a buffer overflow error, and it overwrites the memory used by .NET to track memory allocation, the problem won't be detected until the next garbage collection happens, which could be seconds or minutes later.) Thus, the problem may not actually be related to indexing reverse interlinears; it may just be that that's what happened to be getting indexed when the problem was detected. This kind of problem could also be the trigger for the exception when indexing ANCH.logos4 (discussed in my previous post), so that resource may not be the culprit. (In fact, it might be more likely to be one of the ~30 resources indexed before ANCH.)

I'll keep debugging the crash dump for clues, but since it was probably created long after the original error occurred, it's unlikely to be very helpful in identifying the root cause. Without being able to reproduce and debug it here, I'm not sure what to suggest on how to proceed.

Do you have a second machine on which you run the software (under the same account)? If so, has it been able to index all these resources successfully?

https://community.logos.com/discussion/comment/179184#Comment_179184

Bradley,

I did hide ANCH and also (I believe) updated to SR-2 (can't recall when I did that). I have since reindexed (no ANCH) and now have what seems to be a working index. Attached are the log files.

0160.Logos4 (3).zip

Tonight or tomorrow I will try to unhide ANCH and see if I have any problems.

In answer to your question, I do not have the beta installed elsewhere. My non-beta version updated to 4.1 SR-3 just a bit ago and there is still some indexing going on there. The ANCH is hidden there, of course, since everyhting is synched to the same account.

Thanks for persevering. I will get back to you once I unhide the ANCH volume.

https://community.logos.com/discussion/comment/179190#Comment_179190

Just to update you. The 4.1 SR-3 which is now on the stable channel has finished indexing and everything is fine for that installation. I will unhide ANCH but probably not until later tonight.

https://community.logos.com/discussion/comment/179194#Comment_179194

Thanks for the update; it sounds like ANCH.logos4 may have been causing the problem.

In 4.2 Beta 1, I changed indexing to crash earlier (with a better error message) if the indexer encounters an unrecoverable exception. (Previously, it would have kept indexing until it crashed at the end, with an unhelpful (for us) error message.) If you unhide AYBD and it starts crashing again, the new 4.2 Logos4Indexer.log file may give us the information we need to understand and solve the underlying problem.