Indexing Again?

Not sure why, but started up Logos 4 and it had to prepare my library and is indexing all over again. I did an update of v.3 via the update script. Christian history magazine and ESV reverse interlinear OT downloaded. Would this then cause V.4 to reindex my entire library? If so it shouldn't. Why not just index the updated resources?

Find more posts tagged with

Performance

Comments

Kevin A. Purcell

https://community.logos.com/discussion/comment/9057#Comment_9057

Forgive my ignorance; I know what I am about to ask may be like asking why you Detroit doesn't deliver cars with enough gas to run forever, but I'll ask.

Why can't you deliver the program preindexed? Index on your end and then ship so the indexing files are put in place already done. Is there a technical reason for not doing this? Is it because everyone has a different library?

If it is the latter then why can't you just index each book and tell the database to add that book once I have it installed?

I'm obviously not a programmer. I'm sure someone on your end has to be as smart or more likely smarter than me and had to think of this. But wanted to ask.

Bob Pritchett

https://community.logos.com/discussion/comment/9334#Comment_9334

Why can't you deliver the program preindexed?

Yes, it's because everyone has a different library. We intend to deliver a pre-built index for each core collection, but users who have purchased other books will need to build a new index to incorporate those other books.

Dave Hooton

https://community.logos.com/discussion/comment/8917#Comment_8917

I had an update when I started up my laptop today.

Please describe the process and how you kept track of its progress. What was updated?

Jacob Hantla

https://community.logos.com/discussion/comment/9384#Comment_9384

Bob-

I don't think my problem is the indexing per-se, it's that while it is indexing it kills performance of other apps and makes my computer run hot and hurts battery life.

Would it be possible run the reindexing as a lower priority operation in the background? I think we're understanding and appreciative of the initial indexing process. it is the computer slowing reindexing that is the pain.

Terri & Keith

indexing resources never finishes. I shut off automatic hibernate, standby, updates, which appears to stop the function to completion only to start over again at start up. Returning hours later expecting indexing to be finished i see no indication it finished. Then when initiating a search the note indicates the rescources are not indexed. When I open logos 4 indexing begins again. I am looking for a post, 'ABC's of indexing' with a notebook computer. btw, my tower office computer, no problem, all resources are indexed. Please direct me what I should do or look for so indexing will finish on my notebook computer. Thank you.

Terri & Keith

https://community.logos.com/discussion/comment/9831#Comment_9831

while the notebook comuter is endexing the resource file i have been working on an essay watching the resource counter decrease. But the counter just disappeared and then trying to do a search i get an indication the resource files are being indexed. I do not believe the files are being indexed. The computer set running for days and no change with this condition.

Justin Langley

https://community.logos.com/discussion/comment/9904#Comment_9904

Ditto to Keith's post.

I've tried 4 times now, and each time indexing starts over.

And, I haven't added any new resources since I got Logos4.

--Justin Langley

Windows XP Home, running via Parallels on Macbook Pro
2.4 GHz Intel Core 2 Duo
2 GB 667 MHz DDR2 SDRAM

Terri & Keith

https://community.logos.com/discussion/comment/9904#Comment_9904

It just occured to me that maybe, just maybe, the reason why the notebook computer will not allow the indexing to complete is because I am using "slimbrowser" for my internet software. Would this make a difference? I am not a computer kid, just asking. thank you.

J.R. Miller

https://community.logos.com/discussion/comment/10124#Comment_10124

Keith, from what I understand, the v4 has no connection to your browser so that should not matter any more.

John Minter

https://community.logos.com/discussion/comment/8121#Comment_8121

Bob, you guys REALLY need to rethink this indexing scheme. Users just WILL NOT put up with this multi-hour re-index every time a resource is changed. It is a major time waster for people who just want to get work done. I have to tell you that my irritation with indexing eclipses my appreciation of the work you put in to the new features in 4.0.

Charlene

https://community.logos.com/discussion/comment/8697#Comment_8697

I'm sure on some computers things don't grind to a halt...but even on my netbook (1.6 Hz Intel Atom; 2 GB RAM) indexing slows everything down, makes the computer run hot, and noticably hurts battery life.

I have the same problem. Yesterday I downloaded 2 new resources to V3. Then I opened V4. At that point, it started the whole process over again. This was at about 8:30 a.m. I tried to use V4, but it was so sluggish and at times froze, that it was impossible to use the program. When I went to bed last night, it still had 109 resources to index! So basically I lost the use of my laptop for the entire day. I only have 2645 resources. I am assuming there is no way to just index the new resources and have it merge with the previous index?

Charlene

Damian McGrath

https://community.logos.com/discussion/comment/12244#Comment_12244

I'd like a really good heads up about when to expect the next Beta so that I can purchase a couple of resources and add them to my computer before all the necessary indexing begins.

I'd also like my version of Christian History and Biography to ship so that I can add it before the next upgrade.

Juanita

https://community.logos.com/discussion/comment/12245#Comment_12245

At the very least, the current indexing scheme if it becomes the way things are done in v. 4 when it is released, will force me to re-think how, when and what I purchase. Now, once a month I might just purchase a resource from Logos website that is not pre-pub. If I know my cpu has to re-index every resource, I won't just purchase one. I'll wait unti I have several or until my next pre-pub becomes live. I cannot put my cpu through it and I don't want to be without my computer for long stretches of time and I really don't want to leave it at maximum use all through the night. I hope this scenario never happens.

Mark Martin

https://community.logos.com/discussion/comment/12258#Comment_12258

I am having the same issue also. I recieved Ariel's Genesis and updated v.3 on both of my computers. i wasn't going to reindex until the next beta. When I turned off the computers and later fired them up and restarted v.4, they both went into the preparation phase and indexing. I left both of them on over night. They appeared to finish and Ariel's Genesis is now available in v.4. Now, everytime I start v.4 it immediately goes into preparation phase (about 2 minutes or less) and begins reindexing (forever). Somewhere in the vacinity of about 4 to 5 hours it stops indexing and the blue icon disappears from the tray. When I start v.4, it starts the process all over again from the beginning. With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?

Juanita

https://community.logos.com/discussion/comment/12261#Comment_12261

Somewhere in the vacinity of about 4 to 5 hours it stops indexing and the blue icon disappears from the tray. When I start v.4, it starts the process all over again from the beginning. With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?

Mark, you echo my sentiments, exactly, and I am not a computer Geek; it just makes common sense at some point. To make it worse, I bought a new laptop and desktop a year ago in anticipation of Logos next release (v. 4) and thinking I will need better, bigger and faster. Now, I wish I had waited until I knew what Logos would need to work optimally. By optimally, I mean being able to use my computer while v. 4 is indexing and also to be able to use v. 4 with several apps running with it, and without it slowing down.

TCBlack

https://community.logos.com/discussion/comment/12261#Comment_12261

With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?

The answer is: "Shouldn't but might".

It shouldn't shorten the life of your computer because it's designed to compute. Your computer manufacturer is supposed to design and stress test their systems so that they adequately cool under load. The heat generated is, as you suggest a potential killer; that's why computers have noisy fans.

The reality is that not all manufacturers design their systems well nor test them under extreme loads. Most 3-5 year old through modern BIOS (don't worry if you don't know what that is) have features that shut off your computer to prevent damage in case of overheating.

I have a notebook that is ill designed and if the slightest buildup of dust occurs on it, overheating from indexing will shut it down. Thus I periodically fire up my compressor and blow the dust out of it. That's just good maintenance though I suggest a can of compressed air instead of a compressor which can generate water vapor. Do as I say, not as I do. :-)

So the heavy CPU usage Might, but probably Won't damage your system. One of our brothers here has found he must put a room fan on the computer to keep it from shutting down, but the good news is that the shut down feature is protecting his computer.

If you're not getting sudden power downs, don't worry about the overheating. If you are concerned you should blow the dust out of your system. If you're really concerned put a room fan on it during indexing, if you're uber concerned buy a cooling unit and have a geek wire it in for you.

But in all honesty don't worry about it.

Juanita

https://community.logos.com/discussion/comment/12292#Comment_12292

I think Bob P. mentioned in an earlier post that they are working to optimize all of this. So it may be less of an adjustment on our part than we now think.

Bob Pritchett

https://community.logos.com/discussion/comment/12210#Comment_12210

Bob, you guys REALLY need to rethink this indexing scheme.

The good news: We're optimizing it as much as we can, and you'll see improvements in Beta 3.

The bad news: We can't bend time or space. :-)

If you've got 3 gigs of books, you're going to need about 3 gigs of index. If you want to merge a new 2 meg book into your 3 gig index, the computer will have to examine and move much of your 3 gigs, just as you would have to touch or move every book on a large, tightly packed bookshelf to alphabetically insert 50 new books.

There are some tricks: instead of a large, tightly packed bookshelf you could buy two bookshelves and leave lots of space on each shelf as room to insert new books when they arrive. Then you're only sliding over the books on the shelf where you do the insertion, not every shelf that follows. But sometimes you'll get enough new books that shelf 1 is packed, and you have to put some on shelf 2, which packs it, and pushes into shelf 3, etc.

So we could speed most, but not all, additional indexing by merging things in. (So instead of needing one wall of your house for bookshelves, we'd need two walls -- 6 gigs, say -- to store the sparsely packed shelves.) But this would make things a bit slower. (When you retrieve a book, you're walking along twice as many running feet of shelves to get to it.)

There is no magic bullet in indexing. There are optimizations, which we're doing. There are also tricks that make it feel less painful, which we're exploring.

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

We're planning to do A. We can do B, too, though we imagined that many users would want to "get it over with," and that if we offer A, and then put in a feature which prefers indexing at 1 am, we could "power through" and be done by morning. (On most machines, once we've finished optimizing.)

What do you think? Will A make things better? Is the annoyance that the index isn't done, or that the processor and disk are thrashing too much? Do we need a "take longer, but hurt less" option?

(I should also point out that how "heavy" the indexing feels has a lot to do with particular machine configurations. Some machines will see the processor being burdened, and feel other applications be slowed. Machines with multiple cores will not feel the processor load as heavily. And the more memory, the less disk usage.)

Ray Timmermans

https://community.logos.com/discussion/comment/8964#Comment_8964

Bob,

How about using multiple smaller indexes such as one for Scholar's Gold, Silver, etc., already pre-indexed at Logos, then multiple smaller ones for our respective add-ons and some then write a routine to merge them rather than reindex them?

Juanita

https://community.logos.com/discussion/comment/12346#Comment_12346

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

The only downside that I have heard from George about indexing taking longer is that if I turn my computer off while it is indexing, then when I re-start it, indexing has to begin all over again. Can that be remedied? I understand the "pause" option but the requirement for that is for my computer to always be on. Please keep exploring. (My husband thinks the whole thing about bringing a fan into your room is way out of the box for him and he would NEVER have this kind of software.) Thanks, Bob.

Todd Phillips

https://community.logos.com/discussion/comment/12364#Comment_12364

I do like option A, but Joan is right: I think we need a way to temporarily stop indexing so we can shut off the computer and have it pick up where it left off when it starts up. Or, better, just have the indexer save it's state after each resource or after so many minutes, so that even if it gets killed unexpectedly, it can remember where it was.

By the way, what happens to the indexer if we send the computer to Standby or Hibernate while it's running? I haven't been brave enough to try. I normally put my computer to sleep in standby mode instead of shutting it down.

I also like Option B if a shutdown doesn't cause it to start over.

And maybe give the indexer a slow and fast mode that is selectable, so I can have the best of both worlds.

Simon’s Brother

https://community.logos.com/discussion/comment/12346#Comment_12346

Bob, you guys REALLY need to rethink this indexing scheme.

The good news: We're optimizing it as much as we can, and you'll see improvements in Beta 3.

The work Logos is doing to improve this is appreciated... The big issue for some of us seems to be CPU thrashing.....

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

...so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

......

We're planning to do A.

so I think A is the better alround solution

(I should also point out that how "heavy" the indexing feels has a lot to do with particular machine configurations. Some machines will see the processor being burdened, and feel other applications be slowed. Machines with multiple cores will not feel the processor load as heavily. And the more memory, the less disk usage.)

I don't want you designing the software to be optimized for 5 year old technology....but any gains that can be made in indexing for users in that situation would be appreciated....if the issue of having to re-start indexing after a computer shutdown can be overcome then option A followed by B might be a better solution for users in that scenario if they can leave their machines logged into the one profile for that long.... but we will have to see the improvements in b3 and whether the CPU thrashing issue is in any way alleviated before really makiing firm thoughts on some of these things....whatever improvements you are able to make thought I certainly am no looking for you to make a single core machine behave like a dual core....

J.R. Miller

https://community.logos.com/discussion/comment/12410#Comment_12410

I am with Andrew, I like option A... small new index, then integrate over a few days.

Todd Phillips

https://community.logos.com/discussion/comment/12410#Comment_12410

...so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..

I'm not sure I understand this...Indexing doesn't have to be done to use the resource. You just can't search the whole library.

I just purchased Luther's Galatians from Logos a couple hours ago. Of course I had to reindex. I didn't try using it right away, but I still have 1210 resources left to index and I'm reading the new resource now, plus I can search my library and get results for the books that are indexed.

Simon’s Brother

https://community.logos.com/discussion/comment/12414#Comment_12414

...so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..

I'm not sure I understand this...Indexing doesn't have to be done to use the resource. You just can't search the whole library.

I just purchased Luther's Galatians from Logos a couple hours ago. Of course I had to reindex. I didn't try using it right away, but I still have 1210 resources left to index and I'm reading the new resource now, plus I can search my library and get results for the books that are indexed.

For those of us who have older technology and CPU's thrashing at close to 100% you really can't do anything else...even though in theory you can....

Todd Phillips

https://community.logos.com/discussion/comment/12417#Comment_12417

For those of us who have older technology and CPU's thrashing at close to 100% you really can't do anything else...even though in theory you can....

Interestingly, on my 5 yr old single-core Pentium Thinkpad, the indexer is averaging around 85% CPU. The computer is quite usable.

Bobby Terhune

https://community.logos.com/discussion/comment/12346#Comment_12346

Bob,

Thanks for trying fo find a better solution to our concerns. I would prefer ontion A with the option of doing the complete indexing at a time of my choosing.

I had posted earlier that with over 7800 resources it took me approx 30 hours to complete the process, that was with 2 gb of memory. I purchased 8 gb to replace it and this afternooon I removed Logos 4 and started to reinstalle it (with much dread I might add), but I have be blown away by how much faster the whole process is going! I have been able to multitask without the previous pain of losing the use of my computer or having too noticable of a slowdown in using other programs including Logos 3. I'll report back when everything is finished.

That being said I think this will turn out to be like the roll out of first version of Libronix when everyone complained of the slowness of the searches and compaired it's sluggishness to Logos 2.1's snappy performance. Computers got more powerful, the program go optimized and most of us ended up happy. I would rather have a little pain now to tap into the power of tomorrow, than sacrifice that power. I spent 200.00 for the memory, but I have a small fortune in my library.

Bobby

Intel 2.4 gz Dual Core

Nvidia 8800 GTX 768 mb

1.5 TB Hard Drive

8 gb Memory

Vista 64 bit Ultimate Edition

Damian McGrath

https://community.logos.com/discussion/comment/12346#Comment_12346

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

I like this idea if I control when the master indexing is done. I'd be happy to run the master indexing after very major purchases (say NICOT/NT). But if I purchase a series of very small resources, it could well be a few months before I wish to update the major index. A nag screen once a week on startup would be bearable.

To me, the simple selling point is that the computer is not in a continuous state of re-indexing. I can't see any new users making many additional purchases if they have to go through the process of re-indexing every time. You've heard the outcry from "power users" - imagine what the regular person will think.

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

I'd rather do it all in one hit when I want.

I'm still not sure why we also cannot permanently have multiple indexes. Why are bibles indexed together with other resources? It would save me considerable time indexing if bibles were not included every time (it's years since I bought a new bible).

Bob Pritchett

https://community.logos.com/discussion/comment/12437#Comment_12437

Reverse interlinear indexing is the only really slow part of Bible indexing, and that's been dramatically sped up for Beta 3. So indexing Bibles separately doesn't help that much, since they're not the largest part of the library. (Though we may look into that post 4.0 for other reasons.)

The vast majority of Logos users (disappointingly) never add any books to their library -- they stick with the collection they purchased, or upgrade only to other whole collections. (This beta program is full of the minority who buy lots of books -- and we love you!) So most users will install the default index, and then never see indexing happen.

Unfortunately, it's the "best" users who will see the most indexing... but it's that time and space problem. (It's even hard to "pause" the index and restart it. We get much of the speed it does have from using a huge amount of memory. If you've got nearly a gig -- or 4 gigs, on a bigger machine -- of the index in memory, and you want to be able to turn off the machine and restart later, you've got to write code and take the time to write that gig of memory to the hard drive, then reload it when you start, etc.

And if you want to be able to shut down anytime, quickly, we've got to be keeping all that on the hard drive, not in memory, and that just makes it even slower....

Damian McGrath

https://community.logos.com/discussion/comment/12488#Comment_12488

Reverse interlinear indexing is the only really slow part of Bible indexing, and that's been dramatically sped up for Beta 3. So indexing Bibles separately doesn't help that much, since they're not the largest part of the library

Thanks for the response Bob.

I was pretty sure that the Interlinear bibles were amongst the largest files in my library - with only a couple of exceptions. While I await Beta 3 and its promise of a dramatic improvement in indexing these resources, I still wonder whether they wouldn't be better off separate to commentaries and monographs.

(Though we may look into that post 4.0 for other reasons.)

It could improve searches on these titles, couldn't it?

This beta program is full of the minority who buy lots of books -- and we love you!

Thanks Bob. It''s also why we press hard on issues important to us.

(It's even hard to "pause" the index and restart it. We get much of the speed it does have from using a huge amount of memory. If you've got nearly a gig -- or 4 gigs, on a bigger machine -- of the index in memory, and you want to be able to turn off the machine and restart later, you've got to write code and take the time to write that gig of memory to the hard drive, then reload it when you start, etc.

And if you want to be able to shut down anytime, quickly, we've got to be keeping all that on the hard drive, not in memory, and that just makes it even slower....

This is a real concern..... My little netbook doesn't want it to take any longer than it already takes...

Robert

Running an ASUS EeePC 900, indexing turns my computer into a paperweight until it is finished. I can't reasonably run anything. It needs to just sit there (preferably on a cooling pad with another fan blowing directly on it since it is working for 2 straight days). The first time I indexed I was out of down for a day and had other things to do the next so I was able to give it the apx. 48 hours needed to index. I will not be able to do that every time there is a new resource. If it needs to reindex every time there is a resource update or new book I will likely run Libronix when disconnected from the internet to avoid updates and rarely buy new books (not a goal Logos has in mind). Actually, I have a CD with new books to install and am leaving it unopened for now since it isn't worth making my computer unusable for 2 days. I will probably wait a few months until another product I have on pre-pub order is released and install them together (when I am going to be away from my computer for 2 days, which may take another few months).

Somehow Libronix needs to be able to add the new/updated resources without needing a complete reindex.

Thank you for listening to our thoughts. I look forward to seeing the final release version of v4.

Rob

Charlene

https://community.logos.com/discussion/comment/12346#Comment_12346

There is no magic bullet in indexing. There are optimizations, which we're doing. There are also tricks that make it feel less painful, which we're exploring.

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

Bob,

Thanks for all the effort you guys are making in order to speed up the indexing. I am one of the ones whose older laptop is constantly working for 1-2 days to get the titles indexed. I would go for A at this point. I would like to be able to use the laptop, while a painless indexing is taking place. I am sure that there will be better ways in the future and that you will continue to see how to expediate this process. Just wanted you to know that I appreciate all the hard work that Logos is doing.

Charlene

TCBlack

https://community.logos.com/discussion/comment/12346#Comment_12346

1. Joke.

2. A Comment.

3. A Wild Hair Idea

The bad news: We can't bend time or space. :-)

1. Aye Capt'n We're given it all we've got, but I'm afraid Impulse engines won't cut it, we're needed a new warp core.

There is no magic bullet in indexing. There are optimizations, which we're doing. There are also tricks that make it feel less painful, which we're exploring.

A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)

We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

We're planning to do A. We can do B, too

2. It sounds like A is a winner idea. But Don't give up on B. You don't need to do B by slowing the process, just drop the process priority. For instance I've done lots of Distributed computing projects and while they're designed to peg your CPU at 100% they're also designed to let loose of the processor immediately; because they run in a lower priority process. Best of both worlds I think. If you don't need the power, the indexer has it, but if you need the power the indexer let's you have it immediately until you don't need it anymore.

3. Now for the wild hair: Borrow the B.O.I.N.C. Open source code and index via distributed computing? Nah. But I said it was a wild hair.

Robert

https://community.logos.com/discussion/comment/12523#Comment_12523

I am excited to hear these possibilities. The first, to have a separate index for new books until they are put into the full index sounds like a great new "feature".

The second sounds good if it works. I know it will work fine for the powerful new computers, but will it work with the more limited netbooks (such as EeePC)? They have a hard time handling too much at once. It would be great if there was a slow and steady method to get the book indexed without stopping the computer.

I figure most users (including myself) will want instant access to their new book(s) (idea A), but also aren't going to want to have to wait a day or more to use their computer (thus some version of idea . These sound like a win, win, especially if they work with the netbooks.

Personally, I don't mind if it takes a while to be fully indexed as long as I can access it immediately (or at least very soon) and it doesn't take me away from my other computer needs.

Thank you for being so amazingly responsive to our needs. I know it is good for business, but the way you all handle it is fantastic.

Thank you,

Rob

Jacob Hantla

https://community.logos.com/discussion/comment/12346#Comment_12346

Bob, I love A...and so long as B doesn't begin again on restart, it can take days for the reindex to occur and I don't think we'd mind.

-Jacob

Jacob Hantla

https://community.logos.com/discussion/comment/12523#Comment_12523

Borrow the B.O.I.N.C. Open source code and index via distributed computing? Nah. But I said it was a wild hair.

I nominate Thomas' computer to do all my indexing!

Joe K

https://community.logos.com/discussion/comment/12488#Comment_12488

Bob, maybe i am not understanding the whole picture of indexing, but why are we doing on the customer's machines - over and over again. Each customer has to run the index on the same resources. Seems like such a waste. My opinion is that Logos ought to deliver index files with resources and merge them into the customer's larger index. That way it is done once, which i am sure Logos does anyway, and not done again for each customer. Am i not seeing this clearly?

God bless!

JoeK

TCBlack

https://community.logos.com/discussion/comment/12557#Comment_12557

I nominate Thomas' computer to do all my
indexing!

That's funny Jacob, I was getting ready to use yours.
[;)]

Each customer has to run the index on the same resources. Seems like such a waste. My opinion is that Logos ought to deliver index files with resources and merge them into the customer's larger index. That way it is done once, which i am sure Logos does anyway, and not done again for each customer. Am i not seeing this clearly?

Joe,

So far I've gathered that since most people's libraries are slightly different, each library will need a different index, so it has to be custom built on the computer. Earlier, Bob mentioned that they are considering (or will?) ship indexes of collections as they are, so scholars library will ship with a scholars library index. And somewhere else, he mentioned that merging indexes would actually make them much larger than doing it the way their' doing it.

I don't truly understand the indexing process myself but there is a good post from Bob on it at the top of the third page in this thread.

Simon’s Brother

https://community.logos.com/discussion/comment/12579#Comment_12579

I don't truly understand the indexing process myself but there is a good post from Bob on it at the top of the third page in this thread.

I get the concept of why it re-builds each time but not well enough to explain it to the massess since I don't know the particular technology they are using....in short I think of it as the sum of the whole is greater than the sum of the parts...so if you just throw in all the parts (lots of little indexes) it just aint going to give as good as a result in then end as one big index rebuilt each time a new resource is added (sum of the whole).....

That said option A (sum of the parts) till option B could then run (sum of the whole) provided as you said option b can resume rather than restarting from the beginning and assuming we have the hard disk space to hold the data of the partially completed index... the greater the number of books one has licensed the greater amount of free space that will be required it seems....

Option C - index bibles separately - I think for the most part doesn't detract too much from the sum of the whole since often one would want to search bibles on their own or non bible books on their own - at least I would others would have to think about that from their own perspective.

Scott Chambers

The program has been indexing my resources for several hours now since installing beta 3. During this time it seems to have also slowed my computer down quite a bit. What is the purpose of this indexing?

My Computer is an HP with Vista Premium 64 bit with SP2/ 2GB Ram/AMD Turion 64 X2 1.6ghz processor.

Juanita

https://community.logos.com/discussion/comment/13398#Comment_13398

If you want Bob's explanation, you can find it in this thread on pages 3 & 4.

Simon’s Brother

https://community.logos.com/discussion/comment/13398#Comment_13398

What is the purpose of this indexing?

In short....greatly improved search speeds..

Joe K

https://community.logos.com/discussion/comment/13499#Comment_13499

Logos,

I really do not like the indexing that is occuring, and on every single customer's computer. I am not that big of a search through my library type of guy, whereby i could live with the search speed of L3. This constant index rebuilding is a big turn off for me. This really needs to be done better. I cannot imagine folk are going to live with a 12-hour indexing after a new resource is acquired. People will stay away from purchasing in order to avoid the dreadful indexing. Please re-think the process!

Thanks! -JoeK

Mark Smith

https://community.logos.com/discussion/comment/13528#Comment_13528

Please re-think the process!

My feelings exactly. Whether our resources are indexed and when the indexing occurs should we choose it should be up to us. There have been many good points made in support of this. Let Logos ship this with full indexing turned on and indexing at the control of the program, updater, or whatever it is that now controls it, but give us some control over this behavior.

Right now other things in 4.0 (rendering of search results is a big one) slow things down so much that all advantages for the types of searches I routinely do is lost. I take quite a hit in performance for everything else I use my computer for in the six hours or more it takes to index my Library. At least let me schedule doing that after I go to bed!

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/12523#Comment_12523

You don't need to do B by slowing the process, just drop the process priority. For instance I've done lots of Distributed computing projects and while they're designed to peg your CPU at 100% they're also designed to let loose of the processor immediately; because they run in a lower priority process. Best of both worlds I think. If you don't need the power, the indexer has it, but if you need the power the indexer let's you have it immediately until you don't need it anymore.

The indexer already runs its indexing threads at the lowest possible priority. It doesn't drop the process priority during that period, but Windows uses thread priority as the basis of scheduling. (The process priority is just a base value that is applied to the thread.) Still, we could try dropping the process priority during indexing to see if it makes a difference.

My feeling is the biggest problem is that the indexer is doing a huge amount of I/O, which makes the system feel really sluggish. (Programs that are just cracking RC5 keys, for example, do almost zero I/O.) On Vista, we tried using "background processing mode" (http://code.logos.com/blog/2008/10/using_background_processing_mode_from_c.html); while it definitely improved the subjective responsiveness of the app, it made indexing run for 5-10x longer than it normally would; even on a top-of-the-line machine, indexing would now take days. (Basically, every indexer I/O request got pushed behind everything else running on the system: virus scanners, backup, defrag, etc.)

We are profiling I/O usage and trying to bring it to a minimum; this should really help systems with slower HDDs. Unfortunately, we're using a third-party component during the indexing that probably issues 10x more I/O than is strictly necessary; even more unfortunately, it's doubtful that we'll have time to rewrite this before we ship 4.0. But we're doing everything else we can to reduce CPU and disk usage, and we hope to deliver even more improvements in the upcoming betas.

Jacob Hantla

https://community.logos.com/discussion/comment/13593#Comment_13593

it made indexing run for 5-10x longer than it normally would; even on a top-of-the-line machine, indexing would now take days.

I personally don't think that an index time in the week+ range is a problem AFTER the initial index is built...so long as we still have use of the previous index while the next is being built.

Would it be possible to search new resources indexless (or generate a secondary, "quick" index as soon as new resources are downloaded and then display those as "new" results. Then about a week or two or more later, once the new index is all built, the old one can be replaced and the "new" results will no longer appear.

I think we can forgive some processor work upon install (especially if it's pausable and can survive a restart), but I could imagine quite a few people would be hesitant to install (or purchase new resources) if it would consign their machine to the perpetual indexing overload that we're feeling now.

TCBlack

https://community.logos.com/discussion/comment/13593#Comment_13593

Thanks for the explanation Bradley.

It helps to keep the beta testers fed with lots of nubilous information. [:)]

Dave Hooton

https://community.logos.com/discussion/comment/13593#Comment_13593

A) We can index newly installed books in their own tiny index, and
then run searches against two indexes. Your results would be presented
as "Hits in new books:" (a short list) and "Hits in your library:" (the
longer, normal list). Then we could re-index or merge the indexes
later. You'd get immediate access to your new book, but have to wait a
day or so for it to be integrated into the master index. To sell this
as a feature..."For the first day you own it, we highlight everytime it
shows a search hit! See the value of your new book!" :-)

We can index slowly. (This is what Windows and other large
indexers are doing.) You won't notice your CPU getting pounded, and
your drive will spin less often. We'll just take 12 hours to do 6 hours
of work. Advantage: less thrashing. Disadvantage: Takes twice as long.

I somehow missed this thread from 4 days ago, so my answer is that option A is needed and my laptop will love option B!

But we're doing everything else we can to reduce CPU and disk usage, and we hope to deliver even more improvements in the upcoming betas.

I really appreciate the 30% improvement I experienced with indexing for B3 vs B2.