Indexing Again?
Comments
-
Jacob Hantla said:
I'm sure on some computers things don't grind to a halt...but even on my netbook (1.6 Hz Intel Atom; 2 GB RAM) indexing slows everything down, makes the computer run hot, and noticably hurts battery life.
I have the same problem. Yesterday I downloaded 2 new resources to V3. Then I opened V4. At that point, it started the whole process over again. This was at about 8:30 a.m. I tried to use V4, but it was so sluggish and at times froze, that it was impossible to use the program. When I went to bed last night, it still had 109 resources to index! So basically I lost the use of my laptop for the entire day. I only have 2645 resources. I am assuming there is no way to just index the new resources and have it merge with the previous index?
Charlene
Charlene
0 -
I'd like a really good heads up about when to expect the next Beta so that I can purchase a couple of resources and add them to my computer before all the necessary indexing begins.
I'd also like my version of Christian History and Biography to ship so that I can add it before the next upgrade.
0 -
At the very least, the current indexing scheme if it becomes the way things are done in v. 4 when it is released, will force me to re-think how, when and what I purchase. Now, once a month I might just purchase a resource from Logos website that is not pre-pub. If I know my cpu has to re-index every resource, I won't just purchase one. I'll wait unti I have several or until my next pre-pub becomes live. I cannot put my cpu through it and I don't want to be without my computer for long stretches of time and I really don't want to leave it at maximum use all through the night. I hope this scenario never happens.
0 -
I am having the same issue also. I recieved Ariel's Genesis and updated v.3 on both of my computers. i wasn't going to reindex until the next beta. When I turned off the computers and later fired them up and restarted v.4, they both went into the preparation phase and indexing. I left both of them on over night. They appeared to finish and Ariel's Genesis is now available in v.4. Now, everytime I start v.4 it immediately goes into preparation phase (about 2 minutes or less) and begins reindexing (forever). Somewhere in the vacinity of about 4 to 5 hours it stops indexing and the blue icon disappears from the tray. When I start v.4, it starts the process all over again from the beginning. With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?
0 -
Mark, you echo my sentiments, exactly, and I am not a computer Geek; it just makes common sense at some point. To make it worse, I bought a new laptop and desktop a year ago in anticipation of Logos next release (v. 4) and thinking I will need better, bigger and faster. Now, I wish I had waited until I knew what Logos would need to work optimally. By optimally, I mean being able to use my computer while v. 4 is indexing and also to be able to use v. 4 with several apps running with it, and without it slowing down.Mark Martin said:Somewhere in the vacinity of about 4 to 5 hours it stops indexing and the blue icon disappears from the tray. When I start v.4, it starts the process all over again from the beginning. With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?
0 -
Mark Martin said:
With the drain on the CPU and the heat generated, is this shortening the life of the computer with all the extreme use?
The answer is: "Shouldn't but might".
It shouldn't shorten the life of your computer because it's designed to compute. Your computer manufacturer is supposed to design and stress test their systems so that they adequately cool under load. The heat generated is, as you suggest a potential killer; that's why computers have noisy fans.
The reality is that not all manufacturers design their systems well nor test them under extreme loads. Most 3-5 year old through modern BIOS (don't worry if you don't know what that is) have features that shut off your computer to prevent damage in case of overheating.
I have a notebook that is ill designed and if the slightest buildup of dust occurs on it, overheating from indexing will shut it down. Thus I periodically fire up my compressor and blow the dust out of it. That's just good maintenance though I suggest a can of compressed air instead of a compressor which can generate water vapor. Do as I say, not as I do. :-)
So the heavy CPU usage Might, but probably Won't damage your system. One of our brothers here has found he must put a room fan on the computer to keep it from shutting down, but the good news is that the shut down feature is protecting his computer.
If you're not getting sudden power downs, don't worry about the overheating. If you are concerned you should blow the dust out of your system. If you're really concerned put a room fan on it during indexing, if you're uber concerned buy a cooling unit and have a geek wire it in for you.
But in all honesty don't worry about it.
Sarcasm is my love language. Obviously I love you. 0 -
I think Bob P. mentioned in an earlier post that they are working to optimize all of this. So it may be less of an adjustment on our part than we now think.
0 -
JohnMinter said:
Bob, you guys REALLY need to rethink this indexing scheme.
The good news: We're optimizing it as much as we can, and you'll see improvements in Beta 3.
The bad news: We can't bend time or space. :-)
If you've got 3 gigs of books, you're going to need about 3 gigs of index. If you want to merge a new 2 meg book into your 3 gig index, the computer will have to examine and move much of your 3 gigs, just as you would have to touch or move every book on a large, tightly packed bookshelf to alphabetically insert 50 new books.
There are some tricks: instead of a large, tightly packed bookshelf you could buy two bookshelves and leave lots of space on each shelf as room to insert new books when they arrive. Then you're only sliding over the books on the shelf where you do the insertion, not every shelf that follows. But sometimes you'll get enough new books that shelf 1 is packed, and you have to put some on shelf 2, which packs it, and pushes into shelf 3, etc.
So we could speed most, but not all, additional indexing by merging things in. (So instead of needing one wall of your house for bookshelves, we'd need two walls -- 6 gigs, say -- to store the sparsely packed shelves.) But this would make things a bit slower. (When you retrieve a book, you're walking along twice as many running feet of shelves to get to it.)
There is no magic bullet in indexing. There are optimizations, which we're doing. There are also tricks that make it feel less painful, which we're exploring.
A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)
We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.We're planning to do A. We can do B, too, though we imagined that many users would want to "get it over with," and that if we offer A, and then put in a feature which prefers indexing at 1 am, we could "power through" and be done by morning. (On most machines, once we've finished optimizing.)
What do you think? Will A make things better? Is the annoyance that the index isn't done, or that the processor and disk are thrashing too much? Do we need a "take longer, but hurt less" option?
(I should also point out that how "heavy" the indexing feels has a lot to do with particular machine configurations. Some machines will see the processor being burdened, and feel other applications be slowed. Machines with multiple cores will not feel the processor load as heavily. And the more memory, the less disk usage.)
0 -
Bob,
How about using multiple smaller indexes such as one for Scholar's Gold, Silver, etc., already pre-indexed at Logos, then multiple smaller ones for our respective add-ons and some then write a routine to merge them rather than reindex them?
0 -
The only downside that I have heard from George about indexing taking longer is that if I turn my computer off while it is indexing, then when I re-start it, indexing has to begin all over again. Can that be remedied? I understand the "pause" option but the requirement for that is for my computer to always be on. Please keep exploring. (My husband thinks the whole thing about bringing a fan into your room is way out of the box for him and he would NEVER have this kind of software.) Thanks, Bob.Bob Pritchett said: We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.0 -
I do like option A, but Joan is right: I think we need a way to temporarily stop indexing so we can shut off the computer and have it pick up where it left off when it starts up. Or, better, just have the indexer save it's state after each resource or after so many minutes, so that even if it gets killed unexpectedly, it can remember where it was.
By the way, what happens to the indexer if we send the computer to Standby or Hibernate while it's running? I haven't been brave enough to try. I normally put my computer to sleep in standby mode instead of shutting it down.
I also like Option B if a shutdown doesn't cause it to start over.
And maybe give the indexer a slow and fast mode that is selectable, so I can have the best of both worlds.
MacBook Pro (2019), ThinkPad E540
0 -
Bob Pritchett said:JohnMinter said:
Bob, you guys REALLY need to rethink this indexing scheme.
The good news: We're optimizing it as much as we can, and you'll see improvements in Beta 3.
The work Logos is doing to improve this is appreciated... The big issue for some of us seems to be CPU thrashing.....
Bob Pritchett said: We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long....so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..
Bob Pritchett said:A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)
......
We're planning to do A.
so I think A is the better alround solution
Bob Pritchett said:(I should also point out that how "heavy" the indexing feels has a lot to do with particular machine configurations. Some machines will see the processor being burdened, and feel other applications be slowed. Machines with multiple cores will not feel the processor load as heavily. And the more memory, the less disk usage.)
I don't want you designing the software to be optimized for 5 year old technology....but any gains that can be made in indexing for users in that situation would be appreciated....if the issue of having to re-start indexing after a computer shutdown can be overcome then option A followed by B might be a better solution for users in that scenario if they can leave their machines logged into the one profile for that long.... but we will have to see the improvements in b3 and whether the CPU thrashing issue is in any way alleviated before really makiing firm thoughts on some of these things....whatever improvements you are able to make thought I certainly am no looking for you to make a single core machine behave like a dual core....
0 -
-
AndrewMckenzie said:
...so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..
I'm not sure I understand this...Indexing doesn't have to be done to use the resource. You just can't search the whole library.
I just purchased Luther's Galatians from Logos a couple hours ago. Of course I had to reindex. I didn't try using it right away, but I still have 1210 resources left to index and I'm reading the new resource now, plus I can search my library and get results for the books that are indexed.
MacBook Pro (2019), ThinkPad E540
0 -
Todd Phillips said:AndrewMckenzie said:
...so option b sounds appealing till you digest takes twice as long..not good for larger libraries and not good for those who use Libronix as a tool in their vocation I would think either as I would think they would want to get into their newly acquired resources fairly quickly so, and not good as indexer currenty stands if you need to turn off your computer and it starts re-indexing from the beginning..
I'm not sure I understand this...Indexing doesn't have to be done to use the resource. You just can't search the whole library.
I just purchased Luther's Galatians from Logos a couple hours ago. Of course I had to reindex. I didn't try using it right away, but I still have 1210 resources left to index and I'm reading the new resource now, plus I can search my library and get results for the books that are indexed.
For those of us who have older technology and CPU's thrashing at close to 100% you really can't do anything else...even though in theory you can....
0 -
AndrewMckenzie said:
For those of us who have older technology and CPU's thrashing at close to 100% you really can't do anything else...even though in theory you can....
Interestingly, on my 5 yr old single-core Pentium Thinkpad, the indexer is averaging around 85% CPU. The computer is quite usable.
MacBook Pro (2019), ThinkPad E540
0 -
Bob,
Thanks for trying fo find a better solution to our concerns. I would prefer ontion A with the option of doing the complete indexing at a time of my choosing.
I had posted earlier that with over 7800 resources it took me approx 30 hours to complete the process, that was with 2 gb of memory. I purchased 8 gb to replace it and this afternooon I removed Logos 4 and started to reinstalle it (with much dread I might add), but I have be blown away by how much faster the whole process is going! I have been able to multitask without the previous pain of losing the use of my computer or having too noticable of a slowdown in using other programs including Logos 3. I'll report back when everything is finished.
That being said I think this will turn out to be like the roll out of first version of Libronix when everyone complained of the slowness of the searches and compaired it's sluggishness to Logos 2.1's snappy performance. Computers got more powerful, the program go optimized and most of us ended up happy. I would rather have a little pain now to tap into the power of tomorrow, than sacrifice that power. I spent 200.00 for the memory, but I have a small fortune in my library.
Bobby
Intel 2.4 gz Dual Core
Nvidia 8800 GTX 768 mb
1.5 TB Hard Drive
8 gb Memory
Vista 64 bit Ultimate Edition
0 -
Bob Pritchett said:
A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)
I like this idea if I control when the master indexing is done. I'd be happy to run the master indexing after very major purchases (say NICOT/NT). But if I purchase a series of very small resources, it could well be a few months before I wish to update the major index. A nag screen once a week on startup would be bearable.
To me, the simple selling point is that the computer is not in a continuous state of re-indexing. I can't see any new users making many additional purchases if they have to go through the process of re-indexing every time. You've heard the outcry from "power users" - imagine what the regular person will think.
Bob Pritchett said: We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.I'd rather do it all in one hit when I want.
I'm still not sure why we also cannot permanently have multiple indexes. Why are bibles indexed together with other resources? It would save me considerable time indexing if bibles were not included every time (it's years since I bought a new bible).
0 -
Reverse interlinear indexing is the only really slow part of Bible indexing, and that's been dramatically sped up for Beta 3. So indexing Bibles separately doesn't help that much, since they're not the largest part of the library. (Though we may look into that post 4.0 for other reasons.)
The vast majority of Logos users (disappointingly) never add any books to their library -- they stick with the collection they purchased, or upgrade only to other whole collections. (This beta program is full of the minority who buy lots of books -- and we love you!) So most users will install the default index, and then never see indexing happen.
Unfortunately, it's the "best" users who will see the most indexing... but it's that time and space problem. (It's even hard to "pause" the index and restart it. We get much of the speed it does have from using a huge amount of memory. If you've got nearly a gig -- or 4 gigs, on a bigger machine -- of the index in memory, and you want to be able to turn off the machine and restart later, you've got to write code and take the time to write that gig of memory to the hard drive, then reload it when you start, etc.
And if you want to be able to shut down anytime, quickly, we've got to be keeping all that on the hard drive, not in memory, and that just makes it even slower....
0 -
Bob Pritchett said:
Reverse interlinear indexing is the only really slow part of Bible indexing, and that's been dramatically sped up for Beta 3. So indexing Bibles separately doesn't help that much, since they're not the largest part of the library
Thanks for the response Bob.
I was pretty sure that the Interlinear bibles were amongst the largest files in my library - with only a couple of exceptions. While I await Beta 3 and its promise of a dramatic improvement in indexing these resources, I still wonder whether they wouldn't be better off separate to commentaries and monographs.
Bob Pritchett said:(Though we may look into that post 4.0 for other reasons.)
It could improve searches on these titles, couldn't it?
Bob Pritchett said:This beta program is full of the minority who buy lots of books -- and we love you!
Thanks Bob. It''s also why we press hard on issues important to us.
Bob Pritchett said:(It's even hard to "pause" the index and restart it. We get much of the speed it does have from using a huge amount of memory. If you've got nearly a gig -- or 4 gigs, on a bigger machine -- of the index in memory, and you want to be able to turn off the machine and restart later, you've got to write code and take the time to write that gig of memory to the hard drive, then reload it when you start, etc.
And if you want to be able to shut down anytime, quickly, we've got to be keeping all that on the hard drive, not in memory, and that just makes it even slower....
This is a real concern..... My little netbook doesn't want it to take any longer than it already takes...
0 -
Running an ASUS EeePC 900, indexing turns my computer into a paperweight until it is finished. I can't reasonably run anything. It needs to just sit there (preferably on a cooling pad with another fan blowing directly on it since it is working for 2 straight days). The first time I indexed I was out of down for a day and had other things to do the next so I was able to give it the apx. 48 hours needed to index. I will not be able to do that every time there is a new resource. If it needs to reindex every time there is a resource update or new book I will likely run Libronix when disconnected from the internet to avoid updates and rarely buy new books (not a goal Logos has in mind). Actually, I have a CD with new books to install and am leaving it unopened for now since it isn't worth making my computer unusable for 2 days. I will probably wait a few months until another product I have on pre-pub order is released and install them together (when I am going to be away from my computer for 2 days, which may take another few months).
Somehow Libronix needs to be able to add the new/updated resources without needing a complete reindex.
Thank you for listening to our thoughts. I look forward to seeing the final release version of v4.
Rob
www.3rdcultureliving.com - Simple Abundant Legacy
0 -
Bob Pritchett said:
There is no magic bullet in indexing. There are optimizations, which we're doing. There are also tricks that make it feel less painful, which we're exploring.
A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)
We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.Bob,
Thanks for all the effort you guys are making in order to speed up the indexing. I am one of the ones whose older laptop is constantly working for 1-2 days to get the titles indexed. I would go for A at this point. I would like to be able to use the laptop, while a painless indexing is taking place. I am sure that there will be better ways in the future and that you will continue to see how to expediate this process. Just wanted you to know that I appreciate all the hard work that Logos is doing.
Charlene
Charlene
0 -
1. Joke.
2. A Comment.
3. A Wild Hair Idea
1. Aye Capt'n We're given it all we've got, but I'm afraid Impulse engines won't cut it, we're needed a new warp core.Bob Pritchett said:The bad news: We can't bend time or space. :-)
Bob Pritchett said:There is no magic bullet in indexing. There are optimizations, which we're doing. There are also tricks that make it feel less painful, which we're exploring.
A) We can index newly installed books in their own tiny index, and then run searches against two indexes. Your results would be presented as "Hits in new books:" (a short list) and "Hits in your library:" (the longer, normal list). Then we could re-index or merge the indexes later. You'd get immediate access to your new book, but have to wait a day or so for it to be integrated into the master index. To sell this as a feature..."For the first day you own it, we highlight everytime it shows a search hit! See the value of your new book!" :-)
We can index slowly. (This is what Windows and other large indexers are doing.) You won't notice your CPU getting pounded, and your drive will spin less often. We'll just take 12 hours to do 6 hours of work. Advantage: less thrashing. Disadvantage: Takes twice as long.We're planning to do A. We can do B, too
2. It sounds like A is a winner idea. But Don't give up on B. You don't need to do B by slowing the process, just drop the process priority. For instance I've done lots of Distributed computing projects and while they're designed to peg your CPU at 100% they're also designed to let loose of the processor immediately; because they run in a lower priority process. Best of both worlds I think. If you don't need the power, the indexer has it, but if you need the power the indexer let's you have it immediately until you don't need it anymore.
3. Now for the wild hair: Borrow the B.O.I.N.C. Open source code and index via distributed computing? Nah. But I said it was a wild hair.
Sarcasm is my love language. Obviously I love you. 0 -
I am excited to hear these possibilities. The first, to have a separate index for new books until they are put into the full index sounds like a great new "feature".
The second sounds good if it works. I know it will work fine for the powerful new computers, but will it work with the more limited netbooks (such as EeePC)? They have a hard time handling too much at once. It would be great if there was a slow and steady method to get the book indexed without stopping the computer.
I figure most users (including myself) will want instant access to their new book(s) (idea A), but also aren't going to want to have to wait a day or more to use their computer (thus some version of idea
. These sound like a win, win, especially if they work with the netbooks.Personally, I don't mind if it takes a while to be fully indexed as long as I can access it immediately (or at least very soon) and it doesn't take me away from my other computer needs.
Thank you for being so amazingly responsive to our needs. I know it is good for business, but the way you all handle it is fantastic.
Thank you,
Rob
www.3rdcultureliving.com - Simple Abundant Legacy
0 -
Bob, I love A...and so long as B doesn't begin again on restart, it can take days for the reindex to occur and I don't think we'd mind.
-Jacob
Jacob Hantla
Pastor/Elder, Grace Bible Church
gbcaz.org0 -
-
Bob, maybe i am not understanding the whole picture of indexing, but why are we doing on the customer's machines - over and over again. Each customer has to run the index on the same resources. Seems like such a waste. My opinion is that Logos ought to deliver index files with resources and merge them into the customer's larger index. That way it is done once, which i am sure Logos does anyway, and not done again for each customer. Am i not seeing this clearly?
God bless!
JoeK
0 -
That's funny Jacob, I was getting ready to use yours.Jacob Hantla said:I nominate Thomas' computer to do all my
indexing!
[;)]JosephMKreifels said:Each customer has to run the index on the same resources. Seems like such a waste. My opinion is that Logos ought to deliver index files with resources and merge them into the customer's larger index. That way it is done once, which i am sure Logos does anyway, and not done again for each customer. Am i not seeing this clearly?
Joe,
So far I've gathered that since most people's libraries are slightly different, each library will need a different index, so it has to be custom built on the computer. Earlier, Bob mentioned that they are considering (or will?) ship indexes of collections as they are, so scholars library will ship with a scholars library index. And somewhere else, he mentioned that merging indexes would actually make them much larger than doing it the way their' doing it.
I don't truly understand the indexing process myself but there is a good post from Bob on it at the top of the third page in this thread.
Sarcasm is my love language. Obviously I love you. 0 -
tcblack said:
I don't truly understand the indexing process myself but there is a good post from Bob on it at the top of the third page in this thread.
I get the concept of why it re-builds each time but not well enough to explain it to the massess since I don't know the particular technology they are using....in short I think of it as the sum of the whole is greater than the sum of the parts...so if you just throw in all the parts (lots of little indexes) it just aint going to give as good as a result in then end as one big index rebuilt each time a new resource is added (sum of the whole).....
That said option A (sum of the parts) till option B could then run (sum of the whole) provided as you said option b can resume rather than restarting from the beginning and assuming we have the hard disk space to hold the data of the partially completed index... the greater the number of books one has licensed the greater amount of free space that will be required it seems....
Option C - index bibles separately - I think for the most part doesn't detract too much from the sum of the whole since often one would want to search bibles on their own or non bible books on their own - at least I would others would have to think about that from their own perspective.
0 -
The program has been indexing my resources for several hours now since installing beta 3. During this time it seems to have also slowed my computer down quite a bit. What is the purpose of this indexing?
My Computer is an HP with Vista Premium 64 bit with SP2/ 2GB Ram/AMD Turion 64 X2 1.6ghz processor.
0
