Logos 6 Best Feature: NO INDEXING !!!!!!!!!!

Page 6 of 14 (263 items) « First ... < Previous 4 5 6 7 8 Next > ... Last »
This post has 262 Replies | 5 Followers

Posts 1085
William Gabriel | Forum Activity | Replied: Fri, Oct 4 2013 2:56 PM

Michael Childs:

Obviously, most indexing problems would be solved if we just stopped buying any new books.

So, what's the problem?

Ha, if only. Logos is far too gracious and offers us resource updates on what we already own, and that triggers a re-index too!

Wink

Posts 13420
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 3:10 PM

Doug Witmer:

With a B.T. index, each book is individually indexed. The individual index for each book is downloaded with the book, and then it is placed in just the right location in the master index. It's analogous to hanging an ornament on a Christmas tree. And B.T indexes are fast! There is no need to separately look at the individual indexes for each book. Once a book's index is placed in just the right location in the B.T. master index, it becomes seamlessly integrated into the whole.

Are you sure about this?

I ask partly because we're pretty confident Logos is using a B-Tree index [I presume this is what you mean], and using one requires index merges, or separate indexes. If there was no merge, it's possible that WordSearch has one index for your 1,700 books, and another for your seven. Logos used to do that in the early days of 4.0.

It also appears to contradict the information on WordSearch's site:

Indexing is a process that makes the Search feature run super fast.

The indexing process runs automatically in the background so you don't have to do anything to make it index your Library.

You can still use the program while it is indexing. You can open books, create sermons and use all the WORDsearch features except the Search feature will not be able to search a book until it has been indexed.

Finally, the progression bar you see at the bottom of your screen that says Indexing step 2 of 5 or Indexing step 3 of 5 is simply our way of letting you know when it has indexed your new book or entire library. Remember, you can open books and safely use all the WORDsearch program features except the Search feature until the indexing is completed.

NOTE: You will need about 5 GBs of free space on your hard disk to allow Indexing to complete. Also, the more books you have, the longer it takes to finish Indexing.

Posts 9947
George Somsel | Forum Activity | Replied: Fri, Oct 4 2013 3:17 PM

William Gabriel:

Michael Childs:

Obviously, most indexing problems would be solved if we just stopped buying any new books.

So, what's the problem?

Ha, if only. Logos is far too gracious and offers us resource updates on what we already own, and that triggers a re-index too!

Wink

If you truly object to the updates on resources you already own, I'm sure Logos could be prevailed upon to never update your resources.  Wink  I thought not.  Big Smile

george
gfsomsel

יְמֵי־שְׁנוֹתֵינוּ בָהֶם שִׁבְעִים שָׁנָה וְאִם בִּגְבוּרֹת שְׁמוֹנִים שָׁנָה וְרָהְבָּם עָמָל וָאָוֶן

Posts 1085
William Gabriel | Forum Activity | Replied: Fri, Oct 4 2013 4:02 PM

Mark Barnes:

Doug Witmer:

With a B.T. index, each book is individually indexed. The individual index for each book is downloaded with the book, and then it is placed in just the right location in the master index. It's analogous to hanging an ornament on a Christmas tree. And B.T indexes are fast! There is no need to separately look at the individual indexes for each book. Once a book's index is placed in just the right location in the B.T. master index, it becomes seamlessly integrated into the whole.

Are you sure about this?

I ask partly because we're pretty confident Logos is using a B-Tree index [I presume this is what you mean], and using one requires index merges, or separate indexes. If there was no merge, it's possible that WordSearch has one index for your 1,700 books, and another for your seven. Logos used to do that in the early days of 4.0.

It also appears to contradict the information on WordSearch's site:

Indexing is a process that makes the Search feature run super fast.

The indexing process runs automatically in the background so you don't have to do anything to make it index your Library.

You can still use the program while it is indexing. You can open books, create sermons and use all the WORDsearch features except the Search feature will not be able to search a book until it has been indexed.

Finally, the progression bar you see at the bottom of your screen that says Indexing step 2 of 5 or Indexing step 3 of 5 is simply our way of letting you know when it has indexed your new book or entire library. Remember, you can open books and safely use all the WORDsearch program features except the Search feature until the indexing is completed.

NOTE: You will need about 5 GBs of free space on your hard disk to allow Indexing to complete. Also, the more books you have, the longer it takes to finish Indexing.

Doug W, I would be interested in knowing how long that progress bar runs for. I suspect the difference between the two programs is that WS has made the process relatively invisible, but I would also think that it would take a little bit longer to index in that case. How many Logos resources do you have? I would try to find a Logos user with similar library size to your WS library, but I think the specs of the computer indexing matter far more (so it wouldn't be a fair comparison). But if you have a similarly sized Logos library, you could run the test.

I'm sure Accordance has an index too. It's just not feasible to have a search function in any software program without one (okay, that's not strictly true, but I'm sure you want it to finish before Christ returns). I think the fundamental request of this thread is for Logos to figure out how to make indexing fluid enough that it doesn't disrupt computer usage on any reasonable system...like all the other Bible programs out there. 

I personally have not used those other ones, but it's not hard to see ways that Logos can improve the user experience.

Posts 13420
Mark Barnes | Forum Activity | Replied: Fri, Oct 4 2013 4:26 PM

William Gabriel:
I suspect the difference between the two programs is that WS has made the process relatively invisible.

There are other factors, too. For example, Reverse Interlinears take a long time to index*, and WordSearch doesn't have RIs. RI's take a long time because they have a lot of data to index - every word has surface, lemma, for and morphology, and many have senses and other information which may also need indexes; in addition, the surface is indexed in both the main index and the Bible index. So there's at least five times as much data to index in an RI than a standard Bible.

The slowness of RI indexing is what has really kicked off this round of complaints, because lots of RIs were updated, and because it went wrong they were updated twice.

Posts 1721
LogosEmployee
Bob Pritchett | Forum Activity | Replied: Fri, Oct 4 2013 4:28 PM

Wow... this is a fascinating discussion....

For what it's worth, we're pretty familiar with the B-Tree and the fundamentals of databases. I appreciate the desire to be helpful, but we can save everyone some time if you assume we are not completely incompetent in our field of expertise. :-)

(I know that in light of your frustration with indexing, you might feel justified in assuming we're incompetent... [smile] ...but there's another possible explanation, which is that I'm actually telling you the truth about seek time, and that our software does not work exactly like our competition's, so "they do it faster" arguments aren't really relevant, since they don't do what we do.)

If you want to weigh in on the technical topics, I suggest first reading:

We have. (I have been a guest speaker to the graduate-level class of one of the authors.) :-)

Full-text search isn't a database operation. We do ranking, even if some of you don't want it. (Though if we took it away, I think you'd find searching your library near useless.) Logos searching supports more than full-text search, too -- we dynamically integrate reverse interlinear and other markup data into Bibles during indexing. We can do field and sub-field aware searching. We support range searching within full-text searching of exotic and unusual data type references; even if you don't write queries that do intersection of Bible references with full-text queries, our software does -- it's how the cool features of the Passage Guide, Topic Guide, Sermon Starter Guide, etc. work.

It would really, really help if every complaint/issue/report about indexing indicated Mac or Windows. It would also help to know the age (rounded to years) of your system.

On Windows we've been looking at managing process priority, and it looks like the only options are "too much" and "too little." Intense use for a few hours, or low level use that takes days. There doesn't seem to be a good intermediate step. On the Mac, I know less about the situation, but I think it's where we have more issues and where maybe we can explore more options to managing the process, at least.

I'm not trying to be (too) snarky, and hope you'll take my suggested reading list not as an insult or rebuke, but just as a slightly defensive point that we actually are pretty well informed. It's just a difficult problem because of what we chose to have the software do.

I feel weird saying "trust me, we're professionals", but in a sense that is the answer. It's hard to explain when your experience is unpleasant or undesirable, but at some level this is like arguing why a flight from Seattle to New York takes 5 hours when it's 2013. "That's crazy! We can fly faster than that!" Yes, by burning insane amounts of fuel. By taking the SR-71 Blackbird instead of a 737. By using rockets. But not really if you want scheduled daily service and a Coke on board.

If you want Logos to do what it does, indexing is going to take A) a fast, dedicated web connection, B) a massive, massive hard drive with a large file that's updated nightly, or C) some time when new books or new data are released.

If you want it to be as fast as WordSearch or some other product, without A, B, or C, we'll have to give up significant functionality (even some non-obvious things you may think you're willing to give up, but may not understand how they connect things in the system), or stop releasing maintenance updates to books, or stop adding new data to the system.

We definitely hear your concern, though, and will continue to work on improvements. We can probably re-engineer around certain specific problems (we're looking at a different way of integrating reverse interlinears and the detailed data tables for referent, sense, lemma, morph, etc.), and we can tweak process priority, scheduling, etc. But -- short of adding rocket engines (or SSD hard drives to your computer) -- we can't fly from Seattle to New York in 1 hour. It's math and physics, not engineering competency.

-- Bob

P.S. There are possibly some 'completely re-engineered' solutions that would be faster, but they involve trade-offs. Index merging with ranked results, etc. could lead to some speed improvements, but can bring massive costs in disk space, and remove other options, like the ability to tweak or improving the term breaking, stemming, and at the cost of lower-quality ranking. We've made what we think are the right choices in these areas, but will continue to revisit them in light of user feedback.

Posts 1085
William Gabriel | Forum Activity | Replied: Fri, Oct 4 2013 4:45 PM

Bob Pritchett:

If you want to weigh in on the technical topics, I suggest first reading:

I would, but those titles aren't available in Logos! Wink

Thanks for the feedback from your end. You might be surprised how involved your community would be willing to get. Of course we can only make educated guesses since we don't have the source, but a lot of us deal in these trades as well.

By the way, this, in my mind, is one of the top reasons to invest in Logos. I don't know of any other company that interacts with its customers this way.

Thanks, Bill

Posts 164
DivineCordial | Forum Activity | Replied: Fri, Oct 4 2013 5:45 PM

Bradley Grainger (Logos):

One possibility is to move the index from your computer to our servers (similar to what is done for the mobile apps right now). Would it be an acceptable solution to require an Internet connection for searching to work?

This should only be done if one is given the choice between server indexing or offline indexing.

Logos 9 | Faithlife Connect Essentials
27" Win10 (21H2) AMD FX-8350 (4GHz) 16GB DDR3 2x500SSD 1x240SSD 2x3TBHDD
rMBP13" macOS 10.15.7 i7 (2.9GHz) 8GB DDR3 512SSD
iPadPro (2020) iPadOS 15.1 | iPhoneXR iOS 15.1

Posts 2875
David Ames | Forum Activity | Replied: Fri, Oct 4 2013 6:03 PM

EEEEEK!

Got a new resources ready message. Did not want my machine tied up so I ignored the message.  Closed Logos to do the other important thing I had planed.   [Yes, there are other things in life then Logos] And just like magic the indexer started. 

Can we give indexing its own  run during given time option?  

Or ask if we want to delay indexing instead of just giving us the option of pausing it for four hours? 

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 6:31 PM

I haven't created a binary tree in a few years, but there was never any effort required to combine (merge) two indexes. It was simply a matter of connecting the entire new index (a new book's index) at the appropriate place in the master index.

As far as the WordSearch index is concerned. The fact that they've finally discovered the binary tree is a supposition on my part based on the fact that the latest version of WordSearch doesn't force me to re-index every time I take a breath.

The WordSearch documentation you quote is consistent with the previous version of the product.

Posts 9947
George Somsel | Forum Activity | Replied: Fri, Oct 4 2013 6:31 PM

David Ames:

EEEEEK!

Wee, sleekit, cow'rin, tim'rous beastie,
O, what a panic's in thy breastie!
Thou need na start awa sae hasty
Wi bickering brattle!
I wad be laith to rin an' chase thee,
Wi' murdering pattle.

I'm truly sorry man's dominion
Has broken Nature's social union,
An' justifies that ill opinion
Which makes thee startle
At me, thy poor, earth born companion
An' fellow mortal!

I doubt na, whyles, but thou may thieve;
What then? poor beastie, thou maun live!
A daimen icker in a thrave 'S a sma' request;
I'll get a blessin wi' the lave, An' never miss't.

Thy wee-bit housie, too, in ruin!
It's silly wa's the win's are strewin!
An' naething, now, to big a new ane,
O' foggage green! An' bleak December's win's ensuin,
Baith snell an' keen!

Thou saw the fields laid bare an' waste,
An' weary winter comin fast,
An' cozie here, beneath the blast,
Thou thought to dwell,
Till crash! the cruel coulter past
Out thro' thy cell.

That wee bit heap o' leaves an' stibble,
Has cost thee monie a weary nibble!
Now thou's turned out, for a' thy trouble,
But house or hald,
To thole the winter's sleety dribble,
An' cranreuch cauld.

But Mousie, thou art no thy lane,
In proving foresight may be vain:
The best-laid schemes o' mice an' men
Gang aft agley,
An' lea'e us nought but grief an' pain,
For promis'd joy!

Still thou are blest, compared wi' me!
The present only toucheth thee:
But och! I backward cast my e'e,
On prospects drear!
An' forward, tho' I canna see,
I guess an' fear!

george
gfsomsel

יְמֵי־שְׁנוֹתֵינוּ בָהֶם שִׁבְעִים שָׁנָה וְאִם בִּגְבוּרֹת שְׁמוֹנִים שָׁנָה וְרָהְבָּם עָמָל וָאָוֶן

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 6:39 PM

My Logos library is currently showing 1799 items. My WordSearch library is closer to 1710. Logos can index for several hours. WordSearch used to index for 30 minutes or so until the current version. I don't think I've had to do a full rebuild of the index since WordSearch 10.6 came out. This is the version that just added seven books without any indexing at all. How this happens I really don't know, but I'm assuming it has to do with a proper implementation of a binary tree index.

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 6:47 PM

Actually WordSearch does have reverse interlinears. I own two of them. The difference is that the WordSearch reverse interlinears are fixed. That is, the index for that individual book should not need to be rebuilt unless the book is revised.

The beauty of a binary tree index is that the index for each book can be generated at the source (Logos) and then the individual indexes can simply be downloaded to a user's computer, and properly placed in the master index for that computer. It takes no time at all!

Here's a link that explains the concept: http://en.wikipedia.org/wiki/B-tree

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 6:55 PM

The amount of indexing you describe goes way beyond anything I need. When I do a search, I usually limit the search to just a few Bibles or other books because I don't want a boat load of hits -- just a bucket full. :-)

Posts 2875
David Ames | Forum Activity | Replied: Fri, Oct 4 2013 7:04 PM

George Somsel:

David Ames:

EEEEEK!

Wee, sleekit, cow'rin, tim'rous beastie,
O, what a panic's in thy breastie!

Just what I needed - thanks

Doug Witmer:

I don't want a boat load of hits -- just a bucket full.

I, on the other hand like full boats.

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 7:07 PM

Hi David Ames,

Here's my solution to your problem -- for what it's worth.

Download a copy of WinPatrol from http://www.winpatrol.com/download.html.

Fire it up and look in the Startup tab for Logos4Indexer.exe and disable it. (It doesn't do any good to remove it. It just comes back.)

Now go find Logos4Indexer.exe on your hard drive and make a desktop shortcut for it. (Mine is located at C:\Users\dad8\AppData\Local\Logos4\System\Logos4Indexer.exe)

Now you can use the Task Manager (Ctrl-Alt-Del) to cancel the Logos4Indexer process anytime it gets in your way. Later you can re-run it using your desktop shortcut whenever it's convenient.

Hope this helps.

Posts 1085
William Gabriel | Forum Activity | Replied: Fri, Oct 4 2013 7:32 PM

Doug Witmer:

I haven't created a binary tree in a few years, but there was never any effort required to combine (merge) two indexes. It was simply a matter of connecting the entire new index (a new book's index) at the appropriate place in the master index.

As far as the WordSearch index is concerned. The fact that they've finally discovered the binary tree is a supposition on my part based on the fact that the latest version of WordSearch doesn't force me to re-index every time I take a breath.

The WordSearch documentation you quote is consistent with the previous version of the product.

Hmmmm...I should probably mention (based on your other posts too) that a binary tree != b-tree. The worst case for searching a binary tree is O(n) and the worst case for b-tree is O(log(n)), so a tuned data structure is highly important when you're dealing with millions of pieces of data.

Here's the thing on the merge, you can't get around it taking a long time, no matter what program you use and no matter what sophisticated data structure you're using. An insert operation on a b-tree is O(log(n)), just like the search. That's not so bad--for one piece of data--but you have "k" pieces of data. Worst case you're looking at O(k * log(n)). That's not insignificant, and that's why my laptop burns me while indexing.

If WS is not doing this index merge, then they're paying the price on search. You have to put in the work now or later, there's no way around that.

I have a challenge for you. I'm going to give you two binary trees, and you can pretend that they're two indexes. Can you show the simplicity of merging? I do not believe you can do it by simply connecting the entire new index at the appropriate place in the original/master index.

Tree 1 (Master Index):

                               50

                            /       \

                        10          80

                     /      \       /      \

                  1        30  70         90

Tree 2 (New book index):

                              45

                            /     \

                         15       51

                                      \

                                        75

Posts 31
Doug Witmer | Forum Activity | Replied: Fri, Oct 4 2013 7:52 PM

Hi William:

You said, "I have a challenge for you." Well, I admit your challenge is an interesting one, but if WordSearch didn't solve their indexing problem with binary trees, how did they do it?

For what it's worth I brought this problem to the attention of Logos Support only to be referred to this forum. Now I seemed to have stirred up some emotions. Perhaps I should have just suffered in silence.

Posts 2589
Lee | Forum Activity | Replied: Fri, Oct 4 2013 8:14 PM

Bob Pritchett:

On Windows we've been looking at managing process priority, and it looks like the only options are "too much" and "too little." Intense use for a few hours, or low level use that takes days. There doesn't seem to be a good intermediate step.

 

Graduate-level. Pros. Ex-Microsoft? Great. Love them all.

No good intermediate step? How about throttling your indexer to 75% CPU usage max (if unscheduled) and 95% CPU (if scheduled). Big Smile

I get the impression that users are not complaining about the length of time it takes to index per se, but the unresponsiveness of the system during that process.

Posts 9947
George Somsel | Forum Activity | Replied: Fri, Oct 4 2013 8:15 PM

Doug Witmer:

Hi William:

You said, "I have a challenge for you." Well, I admit your challenge is an interesting one, but if WordSearch didn't solve their indexing problem with binary trees, how did they do it?

For what it's worth I brought this problem to the attention of Logos Support only to be referred to this forum. Now I seemed to have stirred up some emotions. Perhaps I should have just suffered in silence.

Can't we simply assume that Logos knows what it's doing and let it go at that?

george
gfsomsel

יְמֵי־שְׁנוֹתֵינוּ בָהֶם שִׁבְעִים שָׁנָה וְאִם בִּגְבוּרֹת שְׁמוֹנִים שָׁנָה וְרָהְבָּם עָמָל וָאָוֶן

Page 6 of 14 (263 items) « First ... < Previous 4 5 6 7 8 Next > ... Last » | RSS