Search Speeds
Good Morning friends,
I found this post in a thread about the DVD's but thought I needed a new topic.
I am very impressed with the power and speed of the search engine. My first search was "Calvin WITHIN 10 words Geneva" and it returned several hundred hits within 12 seconds. As I noted in a previous post, at this point my main worry is about importing my collections over from 3.x to 4. Hopefully, Logos will see fit to offer an import collections feature in the final release. Well, back to my new power tool!
A correction to my previous post. My search results of "Calvin WITHIN 10 words Geneva" returned 2100 results in 0.43 seconds. My original time of 12 seconds was a typo.
This post REALLY bothered me. In fact, I've been seeing a lot of posts out there from a lot of testers saying how wonderful the search engine is for v. 4, and that's not only bothered me, but it was beginning to tick me off. I'm wondering, have you all got a different version than I do? Ok, to be fair, I'll run the exact same search the Anthony ran on my old work horse at the church office. It's my slowest machine (it's a six year old, 3 GHz, hyperthreaded P4, with 2 gb RAM running Win XP/Pro SP3). so I won't expect the same outstanding results that he posted, but jsut for the ducks of it, let's see what happens.
I ran "Calvin WITHIN 10 words Geneva, no quotes, as a basic search, across my entire library. Is that right? Ok, the results are: 2,143 results in 702 articles, in (drum roll please) 1.79 seconds. Ok, so it took me a little longer, but 1.79 seconds is pretty nearly instantaeous. And, I must say, I'm beginning to see what everyone is raving about.
HOWEVER (comma), Whoa, backup, let's not get too excited.
Those figures that Tony and I found are great for the library. But, I do Bible searches at least as often as I do library searches. So, let's see what happens with a Bible search.
How about "Jesus Wept" -- shouldn't be hard, there's only one occurrence, every kid memorizes that verse so they can claim to know at least one verse in the Bible. So, I'll do a Bible search, across All Bibles, searching for "Jesus Wept" in quotes, in the New Testament (no need of even looking in the OT). BTW v. 3f found 23 occurrences in 1 verse in about 13 seconds. And the results for v. 4 are:
- 22 results in 1 verse, in (another drum roll please) 63.51 seconds.
Whew! Not quite as snappy as finding ol' John Calvin in relationship to Geneva is it? Ok, maybe the machine's not awake yet -- you know, I don't work too well when I first wake up, so let's try it again -- maybe something was cached so we can get a better time. Yep, it's quicker. This time it only took 63.48 seconds. Man, let's cut the anchor chain, something's holding us back 
Well, there's another good search we can run -- how about "John the Baptist" (in quotes), a New Testament search, against all Bibles. I ran this same search on the same machine quite recently, with the same library, in v. 3.f. and found 627 hits in about 17 seconds. Surely v. 4 can do better than that; and the results are in:
- 454 results in 50 verses in (one more drum roll please) 149.82.
Wow, version 3 came up with more hits, in about 1/10th of the time.
Maybe we ought to take off the quotation marks -- maybe that's holding us back. Let's see, Nope, that's not the answer! We've now found 518 results in (I don't know, I didn't run a check, so they may be the same) 50 verses, and it still took 144.87 seconds.
So, I'll definitely applaud the Basic search, across the library.
But the Bible search feature is woefully lacking. In fact, I'd have to say it's broke 
. The same Bible search in v. 3f is quicker and finds more results.
Well, back to building Powerpoint presentations for next week's worship and adult confirmation classes
Dale
Blessings,
Dale Durnell
Coming to you from Henryetta Oklahoma (45 miles south of Tulsa, and 85 miles east of OKC)
Comments
-
Dale,
Just an FYI there is a thread about what configuration of searches works bests...it's not immediately apparent is it?
for the record; When searching for English words; I only search my "English bible" collection BUT leave it Old Testament AND New Testament. It seems to slow it down when you limit it to the NT.
Also, it seemed to slow it down when adding "NT only " resources to a collection that you are going to search Old And NT in, such as the Interlinear GNT etc.
I hope this helps...
Robert Pavich
For help go to the Wiki: http://wiki.logos.com/Table_of_Contents__
0 -
across All Bibles, searching for "Jesus Wept" in quotes, in the New Testament (no need of even looking in the OT).
Hard as it is to believe, the slowness comes from limiting your search to the New Testament. Try "All Passages" and see what speed you get for that.
The index stores search hits at character offsets within the articles of the book. When you ask for everything, we can quickly get them all back, and then render them onscreen as necessary. When you ask to limit it to one Bible range, and then use multiple terms, we have to examine every hit, convert its character position to a Bible reference through another index, and then do a (relatively expensive) comparison of Bible references to see if the ranges intersect. (Our Bible range comparison is a bit slow, too, since we designed a flexible system to handle multiple versification schemes and other exotics stuff.)
We're working on optimizing this, but in the meantime I'd be curious to hear how long it takes you if you switch to All Passages.
0 -
We're working on optimizing this, but in the meantime I'd be curious to hear how long it takes you if you switch to All Passages.
Good Morning Bob --
Well, "all passages" in all Bibles certainly helped. I'm now finding the 454 results in 25.05 seconds. Still a bit slower than I timed it v. 3f, but expotentially faster than the 2 1/2 minutes of the previous search.
That said, I appreciate you continuing to optimize this -- by golly, if we can find over 2,000 references to Calvin in Geneva, in over 3000 resources, in under 5 seconds, then surely we ought to be able to find 454 references to John the Baptist in less than 100 specific resources, nearly instantaneously (don't you think?).
Thanks for responding
Dale
Blessings,
Dale Durnell
Coming to you from Henryetta Oklahoma (45 miles south of Tulsa, and 85 miles east of OKC)
0 -
for the record; When searching for English words; I only search my "English bible" collection BUT leave it Old Testament AND New Testament. It seems to slow it down when you limit it to the NT.
Thanks Robert -- that helped tremendously
I just built an English Bible collection such as I had in v.3f, and my search for John the Baptist just found 318 results in 38 verses, in 16.63 seconds. That's 1/10th the time it was taking originally, but it's not any faster than v.3f. In fact, compared to an NT only search in v.3f, against my English Bible collection there -- it takes twice as long in v. 4
We'll see what happens as the test cycle moves along -- Bob's said they're working on optimizing this, so there's still hope
Dale
Blessings,
Dale Durnell
Coming to you from Henryetta Oklahoma (45 miles south of Tulsa, and 85 miles east of OKC)
0 -
Dale,
For reference:
For the following
searches in English I used a collection called “English Bibles”
which are my English bibles minus the Lexhams and a few others that I
don't see a use for. Here is the filter in my collection:type:bible
lang:english ANDNOT title:sept, lexham, Paul's, Jewish, Message,
Pastoral, study, notelessNote: This is a
more realistic search for me; no need to have ALL BIBLES being
searched...Greek , Hebrew, etc.Here are the results:
Search
Syntax: “Jesus Wept”Search
Type: Basic
SearchCollection:
English biblesResults:
12 results, 10 articles, .19
secondsSearch
Syntax: “Jesus Wept”Search
Type: Bible
search10 results .45
seconds in grid view.Search:
“John
the Baptist”Search
type: Basic
SearchCollection: English
Bibles
Results: 361 hits 4.94
secondsSearch:
“John
the Baptist”Search
type: Bible
searchCollection:
English BiblesResults:
178 hits 17 verses 6.63
seconds.Robert Pavich
For help go to the Wiki: http://wiki.logos.com/Table_of_Contents__
0 -
Hard as it is to believe, the slowness comes from limiting your search to the New Testament. Try "All Passages" and see what speed you get for that.
Personally I would rather search the whole Bible, have a summary display of where that passage applies, and then jump to the place I want to go to. for instance, after searching the whole Bible a list of books containing the search would be displayed in line. clicking an a book name would jump me to that place in the list. Or, if you are going for the Iphone feel, list them like the iphone contacts, with the book names along the right side. as your cursor moves over a book name, the list moves to that location. Color code the names (Pent, History, Poetic, etc). I would add this to the existing format since some will want greater control of the passage.
0 -
Just for the record, a Basic Search for "Calvin WITHIN 10 WORDS Geneva" found 15,558 results in 393 articles in 2.84 seconds on my fastest machine.
- A Bible search, all passages, all Bibles for "John the Baptist" found those 454 results in 69.12 sec
- A Bible search, all passages, "English Bibles" collection for "John the Baptist" found 304 results in 38 verses in 16.91 sec
Interestingly, I'm not finding search results all that improved over an older P4 running a 32 bit version of XP/Pro and a new Core 2 Duo running 64 vit Vista
Blessings, Dale
- Dell, Studio 1737, Core 2 Duo T6600, 2.2 GHz, 2200 MHz, 4 gb RAM
- 64 bit, Vista Home Premium, SP2
Blessings,
Dale Durnell
Coming to you from Henryetta Oklahoma (45 miles south of Tulsa, and 85 miles east of OKC)
0 -
That said, I appreciate you continuing to optimize this -- by golly, if we can find over 2,000 references to Calvin in Geneva, in over 3000 resources, in under 5 seconds, then surely we ought to be able to find 454 references to John the Baptist in less than 100 specific resources, nearly instantaneously (don't you think?).
I'm hardly a Logos insider, but, to be honest, the answer this semi-computer geek would give you is no, I would not expect it. The way Logos 4.0 finds 2000 references to Calvin in Geneva so fast is because it is NOT really searching 3000 resources. It is searching one index which took a while to build. When you look for John the Baptist, it looks in the same index, and then has to take more time to cut out the results you don't want (as in the other 2900 resources). There are often times where we would want to cut the results down to a managable number to look at ourselves, but this does take computer time, and the computer can be much faster at this than I would be.
I have not tested this, but searches SHOULD be faster with a dynamically defined collection because it uses keys the computer can check faster than a semi-random list of files like the old ones.
Of course, I suppose it would be theoretically possible for Logos to create indexes on our collections so that they can be searched lightning fast, but do we really want this?
Ken McGuire
The Gospel is not ... a "new law," on the contrary, ... a "new life." - William Julius Mann
L8 Anglican, Lutheran and Orthodox Silver, Reformed Starter, Academic Essentials
L7 Lutheran Gold, Anglican Bronze
0 -
my search for John the Baptist just found 318 results in 38 verses
Hi Dale,
You'll find that Luke 5:33 "John the Baptist's" is missing from v4 results (if you have NLT or Living Bible) as v4 has an "exact phrase search" (I was fond of saying that v3 had an "inexact phrase search"). So a search for "kingdom of God" will exclude the v3 result "Kingdom of our God".
Also v4 counts real words in a proximity search like:
Jesus BEFORE 1 words Lord
v3 will return "Jesus is Lord" (2 words) whilst v4 returns only "Jesus Lord" (1 word)!
Dave
===Windows 11 & Android 13
0 -
Personally I would rather search the whole Bible, have a summary display of where that passage applies, and then jump to the place I want to go to.
It is also a call to allow the deletion of unwanted results. If you want NT, select all results from Genesis to Malachi and delete!
Dave
===Windows 11 & Android 13
0 -
It is also a call to allow the deletion of unwanted results. If you want NT, select all results from Genesis to Malachi and delete!
I could see a lot of usefulness for that. What would be cool is if you could export the results to clippings somehow too after you filter it down to a range.
0 -
Philip,
I WOULD be good to be able to export the results to a list of some sort....I SECOND this!
Robert Pavich
For help go to the Wiki: http://wiki.logos.com/Table_of_Contents__
0 -
The two slowdowns for "John the Baptist" compare to your Calvin search are:
"the" appears a lot, and we have to look at all the hits to see if it's next to both John and Baptist.
A three word exact phrase requires two "are these next to each other? tests. The Calvin query has only one.
John Baptist
should be a faster query, though granted it's not as exact. But it has the advantage of finding phrases like "the Baptist (John)" etc.
Most search engines speed things up by throwing out "stop words" -- the, and, of, a.
We don't because they are important for exact phrases, and more relevant in the types of queries our users do. But just so you know, a "bag of words" will always search faster than an exact phrase, and the ranking should keep the best results at the top in both scenarios. (Though you'll get more false positives -- just like Google delivers. But who looks at the 300th page of search results?)
0 -
Bob,
thanks for the info; I, for one am not a software guy so these types of insights really help.
If it meant changing my search style from "John the baptist" to John baptist to make things faster, then I have no problem changing my style.
for example:
"John the Baptist"
293 results 5.96 seconds
John Baptist
1,700 results 0.25 seconds.
And the results of the second search were virtually identical for the first 50 entries and were just as usable.
thanks,
bob
Robert Pavich
For help go to the Wiki: http://wiki.logos.com/Table_of_Contents__
0 -
We'll see what happens as the test cycle moves along -- Bob's said they're working on optimizing this, so there's still hope
Search speed, L4B2 -- much improved!! Thanks so very much
- Searched for "John the Baptist" - searching all passages in my English Bible collection, 305 results in 38 verse in 13.86 seconds. That's about 1/4 the time I experienced in L4B1
- Searched for "Jesus Wept" - again, all passages, English Bible collection, 17 results, 1 verse, 3.11 seconds. Again, vast improvement in search speed.
My compliments to the chef!!
Dale
Blessings,
Dale Durnell
Coming to you from Henryetta Oklahoma (45 miles south of Tulsa, and 85 miles east of OKC)
0 -
If it meant changing my search style from "John the baptist" to John baptist to make things faster, then I have no problem changing my style.
Ideally, you wouldn't have to change your style: we want people to be able to type in "John the Baptist" and get fast, accurate results. (This is especially important for reviewers or first-time users, who won't know--or expect to have to use--such tricks to make a search run faster.)
These first betas have not been fully optimised for searching yet; until they are, dropping common English words from your query can definitely improve search times. (And some queries can't be run properly without the stop words; for example, the query 'day Lord' is just as likely to find hits for "Lord's Day" as "day of the Lord", even though the two are quite different!)
0 -
Bradley,
I know that it's not imperative that I adapt my style to the search quirks, but you know how we are....trying to suck that last .001 millesecond out of the software!
Thanks for clarifying the optimization issue again...I have a short memory... [:(]
Robert Pavich
For help go to the Wiki: http://wiki.logos.com/Table_of_Contents__
0
