Speaking of Painful Indexing...

Page 1 of 2 (25 items) 1 2 Next >
This post has 24 Replies | 0 Followers

Posts 1344
PL | Forum Activity | Posted: Thu, Oct 3 2013 5:40 PM

Starting a new thread to report my personal experience...

(Ref: http://community.logos.com/forums/t/75989.aspx)

After the recent big resource download, Logos of course needed to index.  It started 3 days ago.  I believe I saw some kind of fatal error (corrupted index or something) at one point, and decided it needed to rebuild the entire index from scratch.

It's been going slow. Today, on Day 3, it finally got to 95%.  Light was at the end of the tunnel.  Then an hour or so later, it couldn't find the microSD card on which Logos is installed.  I rebooted the machine, and voila!  Now Logos is starting to rebuild the index from scratch again... it's at 9% now, 1.5 hours later.

Log files from the last 3 days attached.

3632.LogosLogs.zip

Running: Asus VivoTab Smart tablet, with Atom processor and 2GB of RAM.  (I know this is below the recommended specs -- no need to lecture me about that... as soon as a more powerful, portable, affordable tablet comes out, I will consider upgrading, but the VivoTab Smart is what I have for now.)

The purpose of my post:

- Not to complain or compare Logos to other competitors' indexing / search performance.

- To urge Bob/Logos to consider streamlining not just the indexing performance but also improve its user experience, e.g. the ability to intelligently resume an aborted indexing -- pick up where it left off due to a system failure.

- To urge Bob/Logos to further streamline the indexing performance by exploring ways to reduce/batch the disk reads and writes, which seems to be the slowest part of the process even when using non-mechanical / solid-state media (microSD card class 10 in this case) -- does it really need to be writing to "disk" after every book is indexed?

- To see if any experts on this forum can suggest a way to salvage the previous 3 days of indexing work and not to have to start over again.

Thanks.  Logos is a good tool.  We just keep hoping that we can have the best of both worlds (fast search without paying the price of a long indexing process).

Peter

 

Posts 26857
Forum MVP
MJ. Smith | Forum Activity | Replied: Thu, Oct 3 2013 6:42 PM

Thanks for the tone and useful content of your report. I hope Logos sees it. Perhaps tablets should be treated as mobile devices with their own lighter app.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 2472
Lee | Forum Activity | Replied: Thu, Oct 3 2013 6:44 PM

PL:

- To see if any experts on this forum can suggest a way to salvage the previous 3 days of indexing work and not to have to start over again.

Since your computer has begun re-indexing, I think salvaging the temp files from the previous indexing would be a lost cause.

Posts 8893
fgh | Forum Activity | Replied: Thu, Oct 3 2013 7:30 PM

I should probably stay away from a Windows thread, but...

PL:
Now Logos is starting to rebuild the index from scratch again... it's at 9% now

I believe that means 9% from where it started today, which isn't necessarily the same as 9% from where it started 3 days ago.

The good news -- unless I misread the log completely -- is that the Bible Index is already finished: "BibleIndex No resources need to be indexed".

The bad news is that the Library Index is being completely rebuilt: "LibraryIndex 4,664 resources need to be indexed, and will be merged into an index of 0 resources".

The other bad news is that there are an awful lot of error and warning lines that someone smarter than me will need to look at.

"The Christian way of life isn't so much an assignment to be performed, as a gift to be received."  Wilfrid Stinissen

Mac Pro OS 10.9.

Posts 25050
Forum MVP
Dave Hooton | Forum Activity | Replied: Fri, Oct 4 2013 2:32 AM

PL:
It's been going slow. Today, on Day 3, it finally got to 95%.  Light was at the end of the tunnel.  Then an hour or so later, it couldn't find the microSD card on which Logos is installed.

i thought that would have been fixed as per my previous post. I now run Logos from a microSD card on the VivoTab and have not had this issue. OTOH I don't have a library anywhere near 4664 resources.

In future try copying a working index from a desktop/notebook computer to the tablet as time for installing, copying & resource discovery is the only overhead.

PL:
- To see if any experts on this forum can suggest a way to salvage the previous 3 days of indexing work and not to have to start over again.

It retains the previous temporary index and I'm optimistic that it plays a part in recovery. The indexer log (post-crash) would tell you how many resources are being indexed. If available use Notepad to read it, even if the Indexer is running.

The many warning messages "Sinai.SinaiMessage Unable to read run type 81. Returning NULL pointer." need an explanation as I did not see it the last time I indexed one of the commentaries e.g. Matthew Henry's Concise Commentary (and it took 6s vs. 41s!).

Dave
===

Windows & Android

Posts 1344
PL | Forum Activity | Replied: Fri, Oct 4 2013 7:55 AM

Dave Hooton:
i thought that would have been fixed as per my previous post. I now run Logos from a microSD card on the VivoTab and have not had this issue.

Yes, I thought so too... until last night.

Dave Hooton:
In future try copying a working index from a desktop/notebook computer to the tablet as time for installing, copying & resource discovery is the only overhead.

I did do that to get my initial installation of L5 on the VivoTab.  But since there's been new resources and updates, I need to let it reindex.  My other L5 machine is a netbook and my wife is mostly using that now, so I don't keep the L5 downloads up-to-date on that.

Thanks,

Peter

Posts 251
Rod | Forum Activity | Replied: Fri, Oct 4 2013 8:59 AM

Logos 5 recently crashed and I had to reindex my entire library which took 9+ hours on a new, relatively upscale machine.

Can we get indexing to happen only when the computer is not being used?  There are numerous programs that only access significant CPU cycles when the computer is not being used for some other task.

Posts 178
DavidS | Forum Activity | Replied: Fri, Oct 4 2013 9:15 AM

You might try pausing for four hours using the computer. And then starting the indexing process when you're done

Taxee:

Logos 5 recently crashed and I had to reindex my entire library which took 9+ hours on a new, relatively upscale machine.

Can we get indexing to happen only when the computer is not being used?  There are numerous programs that only access significant CPU cycles when the computer is not being used for some other task.

Posts 13368
Forum MVP
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 10:36 AM

PL:
(microSD card class 10 in this case)

There are enormous differences between SD cards, even those of the same class: http://www.tomshardware.com/reviews/sdxc-sdhc-uhs-i,2940-9.html. Plus, they're very unsuited for database use: "Since these cards were designed to read and write data sequentially, they are not very impressive when it comes to performing random read/write operations at varying block sizes. Most of the SD cards are comparable to hard drives from somewhere around 1995." And look at the bottom chart for the speed of random writes: http://www.tomshardware.com/reviews/sdxc-sdhc-uhs-i,2940-10.html

(I know you know your spec isn't ideal, but I thought I ought to mention that no matter how much code optimisation is done, SD cards are always going to be terrible for indexing.)

Posts 1344
PL | Forum Activity | Replied: Fri, Oct 4 2013 1:22 PM

Good to know that.  Thanks Mark.  Unfortunately the SSD on my tablet doesn't have enough space for Logos.  When I have time I'll try to figure out if I can move some things around and move Logos back to the SSD.

I have a feeling the microSD is one of the causes of the slow indexing. I was trying to look for speed comparison between microSD, SSD, and spinning hard drives but wasn't able to find comparison info that I can decipher.

Thanks,

Peter

Posts 13368
Forum MVP
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 3:04 PM

PL:
I have a feeling the microSD is one of the causes of the slow indexing. I was trying to look for speed comparison between microSD, SSD, and spinning hard drives but wasn't able to find comparison info that I can decipher.

I had a dig around for you, and couldn't find direct comparisons. This quote just about sums it up, though:

There’s one specific scenario in which we’d recommend looking for a fast card, and that’s if you’re using an SD card to expand the storage of an Ultrabook or MacBook Air. SD flash memory is a fraction of the speed of conventional hard disks and SSDs, so if you try to use a slow card for general-purpose data storage, you’re likely to find the experience infuriatingly sluggish.

http://www.pcpro.co.uk/features/380167/does-your-camera-need-a-fast-sd-card/5

Posts 2472
Lee | Forum Activity | Replied: Fri, Oct 4 2013 9:00 PM

Since we're on the topic of micro-SD cards, maybe the OP should take a look at the Sandisk Extreme line, which has gotten good reviews.

OTOH, I'm not sure that the bottleneck in his system is the SD card.

Posts 285
Luigi Sam | Forum Activity | Replied: Fri, Oct 4 2013 10:14 PM

PL:

Starting a new thread to report my personal experience...

(Ref: http://community.logos.com/forums/t/75989.aspx)

After the recent big resource download, Logos of course needed to index.  It started 3 days ago.  I believe I saw some kind of fatal error (corrupted index or something) at one point, and decided it needed to rebuild the entire index from scratch.

It's been going slow. Today, on Day 3, it finally got to 95%.  Light was at the end of the tunnel.  Then an hour or so later, it couldn't find the microSD card on which Logos is installed.  I rebooted the machine, and voila!  Now Logos is starting to rebuild the index from scratch again... it's at 9% now, 1.5 hours later.

Log files from the last 3 days attached.

3632.LogosLogs.zip

Running: Asus VivoTab Smart tablet, with Atom processor and 2GB of RAM.  (I know this is below the recommended specs -- no need to lecture me about that... as soon as a more powerful, portable, affordable tablet comes out, I will consider upgrading, but the VivoTab Smart is what I have for now.)

The purpose of my post:

- Not to complain or compare Logos to other competitors' indexing / search performance.

- To urge Bob/Logos to consider streamlining not just the indexing performance but also improve its user experience, e.g. the ability to intelligently resume an aborted indexing -- pick up where it left off due to a system failure.

- To urge Bob/Logos to further streamline the indexing performance by exploring ways to reduce/batch the disk reads and writes, which seems to be the slowest part of the process even when using non-mechanical / solid-state media (microSD card class 10 in this case) -- does it really need to be writing to "disk" after every book is indexed?

- To see if any experts on this forum can suggest a way to salvage the previous 3 days of indexing work and not to have to start over again.

Thanks.  Logos is a good tool.  We just keep hoping that we can have the best of both worlds (fast search without paying the price of a long indexing process).

Peter

 

resuming indexing after a crash = needing to know the last good write before the crash.

Implementing either some of the sqlite concurrency control measures ( if sqlite has any suitable for this scenario ) or creating a custom implementation of this so that it can find the exact place where it left off.  ( for those who dont know what I mean: like ntfs file system does when it crashes to solve the fat32 problem on an unexpected power down. )

I am just thinking outloud: I think the main problem isn't really due to logos as such: it has to do with problems that come with reliance on a general purpose database software. Personally, i've found that it is more time consuming to learn someone else's code / api, and way of thinking, than it is to just make it exactly what you wanted it to be from scratch.   I bet if logos played with this idea one day a week (fridays for example) that they would really look foward to working that day, and would get big results quite fast.

Posts 762
Patrick S. | Forum Activity | Replied: Sat, Oct 5 2013 3:20 AM

PL:

Running: Asus VivoTab Smart tablet, with Atom processor and 2GB of RAM.

At the risk of saying what you don't want to hear... you are attempting to run Logos 5 on an under-specced and wrong platform. It is never going to work 'smoothly' and/or fast — unfortunately it's as simple as that. Again sure that's not what you want to hear but that's the fact. It's like you have an elephant (Logos 5) and you expect a mouse (your tablet) to pull it.

Firstly the Asus VivoTab runs an Intel Atom processor - the Atom processor is meant for... baby machines. Secondly as Mark Barnes has detailed SD cards are not the same as a SSD drives or HDD drives. To try and build indexes on that is not going to be a pleasant experience.

It is very unlikely the situation will change, Logos is constantly being pushed by Windows & Mac desktop users to make Logos faster, which will mean that they will steadily increase the minimum computer requirements so they can up the performance of Logos. If they change the indexing process it will be to make it consume more, not less, computing resources.

PL:

as soon as a more powerful, portable, affordable tablet comes out

There is one, if you want a Windows platform you can get a Surface Pro 2 (not the Surface, that is a dud) for only $1,799 with 512GB of storage. Or you could get a real machine and buy a similarly specced (but bigger screen) 11" MacBook Air for $1,599 (a keyboard is thrown in for free!). Wink

"I want to know all God's thoughts; the rest are just details." - Albert Einstein

Posts 1344
PL | Forum Activity | Replied: Sat, Oct 5 2013 3:50 AM

Fwiw, this is the microSD card I'm using.  I thought it is a decently good one...

http://www.amazon.com/gp/product/B009QZH6JS/ref=oh_details_o02_s01_i00?ie=UTF8&psc=1

the current crop of Core tablets like the Surface Pro 2 or MBA are outside of my budget and form factor is also not as portable as the vivotab Smart.

Posts 762
Patrick S. | Forum Activity | Replied: Sat, Oct 5 2013 4:13 AM

PL:

Fwiw, this is the microSD card I'm using.  I thought it is a decently good one...

http://www.amazon.com/gp/product/B009QZH6JS/ref=oh_details_o02_s01_i00?ie=UTF8&psc=1

the current crop of Core tablets like the Surface Pro 2 or MBA are outside of my budget and form factor is also not as portable as the vivotab Smart.

I believe it is more the technology design behind SD cards vs. SSD drives that is the issue (but I haven't researched in detail myself directly).

If you are using the VivoTab unit then Dave had a very good suggestion - copy a working, fully indexed, copy of Logos 5 from a desktop machine to the SD card. That is a workable alternative, there are instructions on the wiki I believe on how to copy a full Logos installation from one machine to another. There are even ways you can automate this to keep the installation on the SD card up to date.

Assuming you have a reasonably specced desktop machine:

  • do install of Logos, fully indexed.
  • copy whole folder structure to SD card from desktop
  • on tablet Logos 5 turn off updating from Internet
  • whenever there is update to Logos do it on desktop
  • sync/copy to SD card

There are utilities which can do syncing at file level to the SD card so that only changed files are copied over. NOTE this is not any syncing with Logos 5 it is purely at operating system level.

There are utilities such as Rsync - http://en.wikipedia.org/wiki/Rsync and great tools like Total Commander - http://www.ghisler.com which can do the copy/syncing.

This way you use the desktop to do the 'heavy lifting' and then use tablet to view, read, search.

"I want to know all God's thoughts; the rest are just details." - Albert Einstein

Posts 1344
PL | Forum Activity | Replied: Sat, Oct 5 2013 8:01 AM

Thank you for your detailed suggestions, Patrick.  I'll see what I can do.

Peter

Posts 13368
Forum MVP
Mark Barnes | Forum Activity | Replied: Sat, Oct 5 2013 8:58 AM

PL:
Fwiw, this is the microSD card I'm using.  I thought it is a decently good one

It is a reasonably good SD card, (the product range is Extreme Pro, Extreme, Ultra, Standard), which SanDisk describe as suitable for everyday use

- but it's still an SD card. They're designed for cameras, really, where you need to read/write large files quickly. So they're good at sustained reads/writes. But they're poor for random reads, and exceptionally poor for random writes. This is what you need for indexing.

My own SSD drive gives a speed of 35MB/s for 4kb random reads, and around 100MB/s for random writes.

You can use CrystalDiskMark to test your drives, but I'd be surprised if you get a tenth of that for reads, and probably a fiftieth (or even a hundredth) for writes.

Posts 1344
PL | Forum Activity | Replied: Sun, Oct 6 2013 5:02 PM

Dear all,

I am happy to report that I was able to finish indexing on another laptop and then copy the files over to my tablet.  I followed the wiki instructions to do a scan command.  It took 2 hours to finish the scan (Preparing the Library) and now it works beautifully.  Thanks for all the tips and help from everyone.

I will be more careful going forward about downloading the Logos updates on this tablet, and only attempt indexing if it's small (e.g. downloading a few new/revised resources and generating/merging their indexes is not going to be a straining job for the machine, right?  I believe I've done that a couple times successfully, until this recent major download and the index got corrupted and a complete rebuild of the index was required.)

Bob/Logos -- If you're reading this, may I add one request when you review and re-tweak the indexing process?  Please make it genuinely resumable and recoverable.  In the past week while struggling with this complete re-indexing and watching it fail multiple times due to microSD card failure, I was monitoring the log files and found that everytime it restarts, it started from scratch and redid all the steps over and over again, everytime.  THE INDEXING PROCESS HAS NO MEMORY OF WHAT IT HAD SUCCESSFULLY COMPLETED BEFORE IT WAS FORCED TO ABORT!  Can you please fix this?  Simply having the indexer read the log file as a first step, and try to resume from where it failed, would have drastically reduced my frustration and desperation level in the past week.  Thanks.

Dave Hooton -- by the way, the microSD card disappearing without warning is still a problem that happens to me especially when it is strained by the heavy indexing process.  I'm up-to-date on all the Windows updates.  Unfortunately I simply don't have enough free space on my SSD to move my Logos installation + resources (4,600+) back to the SSD.  Perhaps I should look at what Dell Venue has to offer: http://community.logos.com/forums/p/57269/532335.aspx#532335

At least now I'm happy I have a working version of L5 on my tablet again.

Thanks,
Peter

Posts 25050
Forum MVP
Dave Hooton | Forum Activity | Replied: Sun, Oct 6 2013 9:59 PM

PL:
At least now I'm happy I have a working version of L5 on my tablet again.

I'm pleased for you!

Dave
===

Windows & Android

Page 1 of 2 (25 items) 1 2 Next > | RSS