Why does indexing ONE book take so long?

Page 1 of 1 (14 items)
This post has 13 Replies | 2 Followers

Posts 3770
Francis | Forum Activity | Posted: Wed, Jan 28 2015 5:47 AM

I don't understand this. I just download one average-length book from vyrso and the indexing part takes nearly 10 minutes and is sufficiently taxing on my computer to the fan going, freeze my mouse or get the "no responding" message from Logos. All this for just one book???

Posts 29274
Forum MVP
JT (alabama24) | Forum Activity | Replied: Wed, Jan 28 2015 6:00 AM

I have wondered that too... I think it must depend upon the book. I have had very quick books, but others which took much longer.

Which book was it?

macOS, iOS & iPadOS | Logs |  Install

Posts 8967
RIP
Matthew C Jones | Forum Activity | Replied: Wed, Jan 28 2015 6:03 AM

Francis:
All this for just one book???

I have often wondered this, especially when that Vyrso book has little to do with scripture or other reference material.

Logos 7 Collectors Edition

Posts 3770
Francis | Forum Activity | Replied: Wed, Jan 28 2015 6:16 AM

alabama24:
Which book was it?

"Learning Patience" by Relaxham Press Wink

"Israel: Ancient Kingdom or Late Invention?". Apparently, Logos had to think long on that one.

Posts 29274
Forum MVP
JT (alabama24) | Forum Activity | Replied: Wed, Jan 28 2015 9:12 AM

I picked that one up along with several others, but haven't indexed yet... I will try to remember to watch the clock... but changing diapers gets me side tracked. Smile

macOS, iOS & iPadOS | Logs |  Install

Posts 5573
Forum MVP
Rich DeRuiter | Forum Activity | Replied: Wed, Jan 28 2015 9:55 AM

Francis:

I don't understand this. I just download one average-length book from vyrso and the indexing part takes nearly 10 minutes and is sufficiently taxing on my computer to the fan going, freeze my mouse or get the "no responding" message from Logos. All this for just one book???

I don't know either. I have sometimes passed up free books, because I didn't want to take the time to let it index (I have a slow HDD).

 Help links: WIKI;  Logos 6 FAQ. (Phil. 2:14, NIV)

Posts 733
JH | Forum Activity | Replied: Wed, Jan 28 2015 10:45 AM

Indexing is slow by virtue of what it is doing (pay for performance at the time of the search, or pay for it when a document is added):

http://en.wikipedia.org/wiki/Inverted_index

Posts 29274
Forum MVP
JT (alabama24) | Forum Activity | Replied: Wed, Jan 28 2015 10:51 AM

JH:

Indexing is slow by virtue of what it is doing (pay for performance at the time of the search, or pay for it when a document is added):

http://en.wikipedia.org/wiki/Inverted_index

Yes, but why does ONE book sometimes take 10 minutes, while 10 books might take only 1 minute? The strangeness is that sometimes the book seems to be a small, straightforward monograph. 

macOS, iOS & iPadOS | Logs |  Install

Posts 50
Cale Judd | Forum Activity | Replied: Wed, Jan 28 2015 11:07 AM

It could be that sometimes the Faithlife servers are very busy and at others they're not. It could also be that your Internet speed varies, sometimes quite a lot, as it does for most people who connect via cable.

Posts 2589
Lee | Forum Activity | Replied: Wed, Jan 28 2015 11:12 AM

Those factors should have no effect on indexing, which takes place after the resource has been downloaded.

Posts 50
Cale Judd | Forum Activity | Replied: Wed, Jan 28 2015 11:34 AM

In that case, the only thing that comes to mind is that memory and drive resources on one's computer may be less available at some times than at others.

Posts 26537
Forum MVP
Dave Hooton | Forum Activity | Replied: Thu, Jan 29 2015 4:30 PM

The indexer log shows the process. In this case 3 bibles which have to be indexed twice (Bible Index + Library Index) + 1 bible dictionary.

...

2015-01-29 10:45:30.0796 22 Info BibleIndex Library indexing completed in 48.23s.    ---->  3 resources indexed
2015-01-29 10:45:30.0796 22 Info BibleIndex Merging supplemental index into main index.
...
2015-01-29 10:45:35.6801 22 Info Indexer (Timed) Merging postings...
...
2015-01-29 10:46:17.9540 22 Info BibleIndex Library index merge completed in 00:00:47.8624550.
...
...
2015-01-29 10:47:21.4502 19 Info LibraryIndex Library indexing completed in 1m 3.4s.     ---->  4 resources indexed
2015-01-29 10:47:21.4658 19 Info LibraryIndex Merging supplemental index into main index.
...
2015-01-29 10:47:31.9959 19 Info Indexer (Timed) Merging postings...
...
2015-01-29 10:50:21.5226 19 Info LibraryIndex Library index merge completed in 00:03:00.0659963.---> 3 min
...
015-01-29 10:50:21.6006 12 Info PersonalBookIndex 0 unindexed resources out of 2 indexable resources; supplemental index count is 0.
2015-01-29 10:50:21.6006 12 Info PersonalBookIndex All indexable resources are indexed in a single main index; marking index up-to-date.
2015-01-29 10:50:21.6318 16 Info IndexerProgram (5m 39.8s) Indexing.

The total time the Indexer took was 5m 39.8, of which Indexing the resources took 1m 51.63s and merging took about 3m 47.9s.

The important point is that the Library index merge takes significantly longer  -- about 3.75x, and it is 3.7x larger than the bible index!  My library index is 2.63 GB for about 1500 resources. So if your Library index is twice as big it will take roughly twice as long (6 min) for the merge of  just 1 resource. If 3x bigger the merge will take about 9 min (3x my 3 min).

Of course there other variables like processor speed and SSD (which I have) vs HDD. But does 10m equate to the size of your Library index compared to mine?

Dave
===

Windows 10 & Android 8

Posts 3770
Francis | Forum Activity | Replied: Thu, Jan 29 2015 10:46 PM

Dave Hooton:
The important point is that the Library index merge takes significantly longer

Thanks Dave for the helpful explanation. I still wished it took less time or at least was less of a resource hog. Perhaps one day, we'll get there. I could not help but think the other day as I looked at the blog that essentially said that a bigger library was better (for research) that there were downsides as well to having too many resources.

Posts 667
Jonathan Pitts | Forum Activity | Replied: Thu, Jan 29 2015 10:59 PM

I wouldn't mind how long it took if it truly happened in the background or was responsive to what else I was doing on the computer. However, adding new resources effectively puts my PC out of action for sometimes an hour or more, which is frustrating.

Also, not infrequently the indexing crashes, and it then starts again the next time I open Logos.

Page 1 of 1 (14 items) | RSS