Logos 6 Best Feature: NO INDEXING !!!!!!!!!!

Doug Witmer said:

With a B.T. index, each book is individually indexed. The individual index for each book is downloaded with the book, and then it is placed in just the right location in the master index. It's analogous to hanging an ornament on a Christmas tree. And B.T indexes are fast! There is no need to separately look at the individual indexes for each book. Once a book's index is placed in just the right location in the B.T. master index, it becomes seamlessly integrated into the whole.

Are you sure about this?

I ask partly because we're pretty confident Logos is using a B-Tree index [I presume this is what you mean], and using one requires index merges, or separate indexes. If there was no merge, it's possible that WordSearch has one index for your 1,700 books, and another for your seven. Logos used to do that in the early days of 4.0.

It also appears to contradict the information on WordSearch's site:

Indexing is a process that makes the Search feature run super fast.

The indexing process runs automatically in the background so you don't have to do anything to make it index your Library.

You can still use the program while it is indexing. You can open books, create sermons and use all the WORDsearch features except the Search feature will not be able to search a book until it has been indexed.

Finally, the progression bar you see at the bottom of your screen that says Indexing step 2 of 5 or Indexing step 3 of 5 is simply our way of letting you know when it has indexed your new book or entire library. Remember, you can open books and safely use all the WORDsearch program features except the Search feature until the indexing is completed.

NOTE: You will need about 5 GBs of free space on your hard disk to allow Indexing to complete. Also, the more books you have, the longer it takes to finish Indexing.

Mark Barnes said:

Doug Witmer said:

With a B.T. index, each book is individually indexed. The individual index for each book is downloaded with the book, and then it is placed in just the right location in the master index. It's analogous to hanging an ornament on a Christmas tree. And B.T indexes are fast! There is no need to separately look at the individual indexes for each book. Once a book's index is placed in just the right location in the B.T. master index, it becomes seamlessly integrated into the whole.

Are you sure about this?

I ask partly because we're pretty confident Logos is using a B-Tree index [I presume this is what you mean], and using one requires index merges, or separate indexes. If there was no merge, it's possible that WordSearch has one index for your 1,700 books, and another for your seven. Logos used to do that in the early days of 4.0.

It also appears to contradict the information on WordSearch's site:

Indexing is a process that makes the Search feature run super fast.

The indexing process runs automatically in the background so you don't have to do anything to make it index your Library.

You can still use the program while it is indexing. You can open books, create sermons and use all the WORDsearch features except the Search feature will not be able to search a book until it has been indexed.

Finally, the progression bar you see at the bottom of your screen that says Indexing step 2 of 5 or Indexing step 3 of 5 is simply our way of letting you know when it has indexed your new book or entire library. Remember, you can open books and safely use all the WORDsearch program features except the Search feature until the indexing is completed.

NOTE: You will need about 5 GBs of free space on your hard disk to allow Indexing to complete. Also, the more books you have, the longer it takes to finish Indexing.

Doug W, I would be interested in knowing how long that progress bar runs for. I suspect the difference between the two programs is that WS has made the process relatively invisible, but I would also think that it would take a little bit longer to index in that case. How many Logos resources do you have? I would try to find a Logos user with similar library size to your WS library, but I think the specs of the computer indexing matter far more (so it wouldn't be a fair comparison). But if you have a similarly sized Logos library, you could run the test.

I'm sure Accordance has an index too. It's just not feasible to have a search function in any software program without one (okay, that's not strictly true, but I'm sure you want it to finish before Christ returns). I think the fundamental request of this thread is for Logos to figure out how to make indexing fluid enough that it doesn't disrupt computer usage on any reasonable system...like all the other Bible programs out there. 

I personally have not used those other ones, but it's not hard to see ways that Logos can improve the user experience.

Bradley Grainger (Logos) said:

One possibility is to move the index from your computer to our servers (similar to what is done for the mobile apps right now). Would it be an acceptable solution to require an Internet connection for searching to work?

This should only be done if one is given the choice between server indexing or offline indexing.

I haven't created a binary tree in a few years, but there was never any effort required to combine (merge) two indexes. It was simply a matter of connecting the entire new index (a new book's index) at the appropriate place in the master index.

As far as the WordSearch index is concerned. The fact that they've finally discovered the binary tree is a supposition on my part based on the fact that the latest version of WordSearch doesn't force me to re-index every time I take a breath.

The WordSearch documentation you quote is consistent with the previous version of the product.

My Logos library is currently showing 1799 items. My WordSearch library is closer to 1710. Logos can index for several hours. WordSearch used to index for 30 minutes or so until the current version. I don't think I've had to do a full rebuild of the index since WordSearch 10.6 came out. This is the version that just added seven books without any indexing at all. How this happens I really don't know, but I'm assuming it has to do with a proper implementation of a binary tree index.

The amount of indexing you describe goes way beyond anything I need. When I do a search, I usually limit the search to just a few Bibles or other books because I don't want a boat load of hits -- just a bucket full. :-)

Hi David Ames,

Here's my solution to your problem -- for what it's worth.

Download a copy of WinPatrol from http://www.winpatrol.com/download.html.

Fire it up and look in the Startup tab for Logos4Indexer.exe and disable it. (It doesn't do any good to remove it. It just comes back.)

Now go find Logos4Indexer.exe on your hard drive and make a desktop shortcut for it. (Mine is located at C:\Users\dad8\AppData\Local\Logos4\System\Logos4Indexer.exe)

Now you can use the Task Manager (Ctrl-Alt-Del) to cancel the Logos4Indexer process anytime it gets in your way. Later you can re-run it using your desktop shortcut whenever it's convenient.

Hope this helps.

Hi William:

You said, "I have a challenge for you." Well, I admit your challenge is an interesting one, but if WordSearch didn't solve their indexing problem with binary trees, how did they do it?

For what it's worth I brought this problem to the attention of Logos Support only to be referred to this forum. Now I seemed to have stirred up some emotions. Perhaps I should have just suffered in silence.

Doug Witmer said:

Hi William:

You said, "I have a challenge for you." Well, I admit your challenge is an interesting one, but if WordSearch didn't solve their indexing problem with binary trees, how did they do it?

For what it's worth I brought this problem to the attention of Logos Support only to be referred to this forum. Now I seemed to have stirred up some emotions. Perhaps I should have just suffered in silence.

Can't we simply assume that Logos knows what it's doing and let it go at that?

Bob Pritchett said:

Wow... this is a fascinating discussion....

you might feel justified in assuming we're incompetent... [smile]

We have. (I have been a guest speaker to the graduate-level class of one of the authors.) :-)

I feel weird saying "trust me, we're professionals", but in a sense that is the answer. It's hard to explain when your experience is unpleasant or undesirable, but at some level this is like arguing why a flight from Seattle to New York takes 5 hours when it's 2013. "That's crazy! We can fly faster than that!" Yes, by burning insane amounts of fuel. By taking the SR-71 Blackbird instead of a 737. By using rockets. But not really if you want scheduled daily service and a Coke on board.

If you want Logos to do what it does, indexing is going to take A) a fast, dedicated web connection, a massive, massive hard drive with a large file that's updated nightly, or C) some time when new books or new data are released.

We definitely hear your concern, though, and will continue to work on improvements. We can probably re-engineer around certain specific problems (we're looking at a different way of integrating reverse interlinears and the detailed data tables for referent, sense, lemma, morph, etc.), and we can tweak process priority, scheduling, etc. But -- short of adding rocket engines (or SSD hard drives to your computer) -- we can't fly from Seattle to New York in 1 hour. It's math and physics, not engineering competency.

-- Bob

P.S. There are possibly some 'completely re-engineered' solutions that would be faster, but they involve trade-offs. Index merging with ranked results, etc. could lead to some speed improvements, but can bring massive costs in disk space, and remove other options, like the ability to tweak or improving the term breaking, stemming, and at the cost of lower-quality ranking. We've made what we think are the right choices in these areas, but will continue to revisit them in light of user feedback.

Frankly, I dont notice a performance issue... But I'm running an octocore, 32gb of ddr3, and an ssd (with multiple terabytes of mechanical drive space). This is obviously not the typical end user set up.

I apologize if my replies infered you guys didn't know what you're doing. Clearly you do... What Logos does is pretty amazing, and comparable software doesn't really exist in other sectors that I'm aware of, and for its level of complexity it does remarkably well.

Forcing a rebuild of the index (4012 resources) while watching TV on same computer, reading my email, surfing the forums etc. Started at 11:30. Will post back the complete time.

Okay as long as I'm stirring up trouble, let me give a simple, concrete, non-technical example of what I'm talking about. It's still Friday, and every Friday WordSearch gives a free book. So just a few minutes ago I ordered my free book. When I opened WordSearch it automatically downloaded and installed the book. That took about 10 seconds. Then I watched for the indexing. There was none. "So", I thought, "let's try a search." I told the search facility to search for "second blessing", and to search across the entire library. In about three seconds it had retrieved 183 hits, and six of them were from the new free resource, "The Second Crisis in Christian Experience by Christian Wismer Ruth".

You can try this for yourself. Just download a free copy of WordSearch Basic, and then "purchase" the free book and search on it.

mike said:

I love updates & I WANT THEM RIGHT AWAY (I'm sure everyone is on the same boat with me) .. but the indexing is just killing me.

hi Mike,

I was just thinking when I read your post: do the updates get listed for users to see exactly what each update is?  ( ie in windows 7, the updates can list each update, and describe them breifly)

does logos 5 do this? 

it would be funny if we were reindexing just because one book corrected one word (and we never actaully even use that book anyway)

if this happened 10 times a day: we are wearing out our hardware reindexing.

thanks if anyone can answer my query.

Having other questions Bob, what about Multi-thread and x64 support for those computers that can handle it? Most everything has multiple cores these days. Is this something that Logos does already? If not can it be something that is considered for L6?

it took 15 minutes for the first 10%. Was at 25% by 29 minutes.

I was typing out the mile stones above as they happened, but fell asleep before 50%. When I woke up 90 minutes later it was complete.

Jack Caviness said:

It is just too much fun trying to teach Logos basic programming 

...and how to run a business. [:D]

Doug Witmer said:

Actually WordSearch does have reverse interlinears. I own two of them. The difference is that the WordSearch reverse interlinears are fixed. That is, the index for that individual book should not need to be rebuilt unless the book is revised.

Sorry, I didn't realise that. Logos indexes are fixed, too, however. The index won't be rebuilt unless the book is revised.

Doug Witmer said:

The beauty of a binary tree index is that the index for each book can be generated at the source (Logos) and then the individual indexes can simply be downloaded to a user's computer, and properly placed in the master index for that computer. It takes no time at all!

As William Gabriel points out, that's not how b-tree indexes work. By definition, b-tree indexes have to be merged (or re-created). If WordSearch isn't using indexes that require merging, then searches across the library are going to be slower. You either have the slowness when you index, or the slowness when you search, it's as simple as that.

Doug Witmer said:

Now I seemed to have stirred up some emotions. Perhaps I should have just suffered in silence.

Stirred up some passion, yes. That's not a bad thing!

Doug Witmer said:

When I opened WordSearch it automatically downloaded and installed the book. That took about 10 seconds. Then I watched for the indexing. There was none.… You can try this for yourself. Just download a free copy of WordSearch Basic, and then "purchase" the free book and search on it.

OK, I'm now running WordSearch basic. There is indexing when you install books (check your status bar), but it's very quick, and doesn't use much CPU. 12 resources took 60 seconds to index.

An index is stored at C:\ProgramData\WORDsearch\Library\ws10searchindex.dat

Search is also very quick (almost instant for the 13 resources that came by default). But each resource appears to have it's own index file (which is downloaded with the resource).

It's impossible to tell without detailed analysis, but it seems that the WORDsearch index isn't a true index, but more of a word list - listing which words are stored in which resources. If I'm right, that means the index is only used to choose which resources are searched (i.e. those that contain the word being searched for). Then, every individual index file is searched independently.

This is a perfectly reasonable design choice. But it's a basic design choice. It's worth bearing in mind that WORDsearch searching can't do any of the following:

• Do a case-sensitive search.
• Rank searches.
• Search for any datafield at all (i.e. you can't search for Louw-Nida numbers, Bible references, links to the church fathers, etc.)
• Do word-stemming (search for all forms of the word)
• Specify specific proximity (you can use NEAR, but not WITHIN x WORDS)
• Search for words in a specific language
• Search for special fields (e.g. heading text, etc.)

So, as someone pointed out above, this is not a like-for-like comparison. WORDsearch is faster - there's no doubt about that - but there's a lot less going on, and a lot less flexibility. That's good - we can all make a choice as to what type of Bible software we want.

But if Logos indexing became became more like WORDsearch indexing, there are very, very, very few Logos users who would see that as an improvement.

Denise said:

No offense Jack and George. But the last two comments either have to be joking or on their way to meanness. You both know the Bible software companies prevent using purchased books on each others software. People aren't rich, so they're at mercy of profit interests. You both know the bulk of users use Bible software visa viz their future with the Divine. And you both know, absent Dave and the Marks, that the issue is sufficiently complex, that competitor experience is the only evidence one can validly point to.  Especially when those same competitors have customers ALSO studying for their future withe the Divine.  

Don't like it, leave?

So, I'll assume we're all just joking!

No, I'm perfectly serious.  It is acceptable to mention a problem one is having with the software, but it is not acceptable to treat the company and its employees as though they are pure neophytes.  They are in a thriving business so they must be doing something right.  I wouldn't walk into a doctor's office and refer him to a Wikipedia article on appendicitis or inserting a stent.  I would assume that Logos is fully acquainted with b-trees to the point where a Wikipedia article is like reading a child's story book.  I have never had problems with indexing so I must conclude that there is something wrong with the hardware or setup of those who do.  I wonder how much free space they have on their hard drives since it's difficult to reorganize something on a computer if you don't have the room to do it.  Then they talk about how program X or Y does it better in their opinion which makes me wonder whether they really are interested in Logos or whether they are propagandizing for another company.

[quote]

I have a challenge for you. I'm going to give you two binary trees, and you can pretend that they're two indexes. Can you show the simplicity of merging? I do not believe you can do it by simply connecting the entire new index at the appropriate place in the original/master index.

Tree 1 (Master Index):

                               50

                            /       \

                        10          80

                     /      \       /      \

                  1        30  70         90

Tree 2 (New book index):

                              45

                            /     \

                         15       51

                                      \

                                        75

Hello William:

I thought about your challenge over night, and I came up with a solution. You can quickly turn these two indexes into a single index (without merging), and the combined index will be perfectly useable (and fast). But I'm not going to give the solution here. If I did, it would just give rise to more questions and more challenges. It's your challenge. Lets see if you can solve it.

Doug Witmer

Doug Witmer said:

The question is what happens to the index when a new book is added.

Correct. And as I stated in my post, Logos is doing a lot more than WordSearch. That's what matters, because it explains that the 'slow' indexing isn't down to sloppy programming, but to a difference in specification.

Do you really think Logos customers would be happy if Bob or Bradley said, "We'll reduce the time it takes to index to almost nothing, but you won't be able to do a case-sensitive search, rank searches, search for any reference types (e.g. Louw-Nida numbers), do word-stemming, specify specific proximity, search for words in a specific language or search for special fields". There is almost no-one on this forum who would think that was a good idea.

Bradley Grainger (Logos) said:

Logos downloads just the text and creates an index locally. This is typically very fast; for example, most Bibles take under 6s to index on my computer, and indexing almost 2000 Bibles and morph resources took about 24m. This created almost 3GB of indexing data, which is about 2MB/sec. You'd have to have a pretty fast sustained Internet connection (and/or a much slower computer) to make downloading quicker overall.

Twenty-four minutes? What do you have that can index that quickly? I have three different Macs that I run Logos on - a 2006 MacBook with 3GB of RAM, which I expect to index slowly; a 2009 MacBook Pro with an SSD and 8GB of RAM, and a 2011 Mac Mini i5 with 8GB of RAM. None of them approach anything remotely close to 24 minutes to index my library of 2,500 resources. Even the MBP with an SSD requires at least 2 hours.

Yeh, Jack.  I had trouble reading the whole two lines.  Let's see what the second line said:

"If WS does all you want Bible Software to do, then use it. Frankly, getting tired of those who come to the Logos forums to sell WS and Accordance."

Now, I presume the first of these sentences suggests if anyone compares Logos to another software, then they should use the other software.  I assume you're joking.  

The second sentence suggests that comparing Logos functionality to other software means the other software is being advertised. I assume you're joking again.

Really, Jack.  Surely you jest.

Taxee said:

I have two major beefs with the software.  I have a large library on a Windows 7 machine that is a couple of years old.

The first is the "preparing your library" stage which prevents me from using the software at all.  This is just crazy.  I want to do a search and I can't get into the program sometimes for for an hour or more.  I appreciate that the large library that I have may be part of the problem but there is something terribly wrong with the process.  I should be able to use the software within a reasonably short time (60 sec) of starting it up.

Secondly, while indexing, the machine becomes unusable as it uses resources to the extent that everything else slows down so much that I have to shut off indexing entirely.  When I am using the machine for other purposes, indexing should stop and then start up again when I am not using the machine.

Those 2 things would increase my approval rating of the software by a couple of hundred percent.  Currently, while I love your software, I am also VERY frustrated with it.

How large is your HD and how much free space do you have?