Search Speeds

Good Morning friends,

I found this post in a thread about the DVD's but thought I needed a new topic.

I am very impressed with the power and speed of the search engine. My first search was "Calvin WITHIN 10 words Geneva" and it returned several hundred hits within 12 seconds. As I noted in a previous post, at this point my main worry is about importing my collections over from 3.x to 4. Hopefully, Logos will see fit to offer an import collections feature in the final release. Well, back to my new power tool!

A correction to my previous post. My search results of "Calvin WITHIN 10 words Geneva" returned 2100 results in 0.43 seconds. My original time of 12 seconds was a typo.

This post REALLY bothered me. In fact, I've been seeing a lot of posts out there from a lot of testers saying how wonderful the search engine is for v. 4, and that's not only bothered me, but it was beginning to tick me off. I'm wondering, have you all got a different version than I do? Ok, to be fair, I'll run the exact same search the Anthony ran on my old work horse at the church office. It's my slowest machine (it's a six year old, 3 GHz, hyperthreaded P4, with 2 gb RAM running Win XP/Pro SP3). so I won't expect the same outstanding results that he posted, but jsut for the ducks of it, let's see what happens.

I ran "Calvin WITHIN 10 words Geneva, no quotes, as a basic search, across my entire library. Is that right? Ok, the results are: 2,143 results in 702 articles, in (drum roll please) 1.79 seconds. Ok, so it took me a little longer, but 1.79 seconds is pretty nearly instantaeous. And, I must say, I'm beginning to see what everyone is raving about.

HOWEVER (comma), Whoa, backup, let's not get too excited.

Those figures that Tony and I found are great for the library. But, I do Bible searches at least as often as I do library searches. So, let's see what happens with a Bible search.

How about "Jesus Wept" -- shouldn't be hard, there's only one occurrence, every kid memorizes that verse so they can claim to know at least one verse in the Bible. So, I'll do a Bible search, across All Bibles, searching for "Jesus Wept" in quotes, in the New Testament (no need of even looking in the OT). BTW v. 3f found 23 occurrences in 1 verse in about 13 seconds. And the results for v. 4 are:

22 results in 1 verse, in (another drum roll please) 63.51 seconds.

Whew! Not quite as snappy as finding ol' John Calvin in relationship to Geneva is it? Ok, maybe the machine's not awake yet -- you know, I don't work too well when I first wake up, so let's try it again -- maybe something was cached so we can get a better time. Yep, it's quicker. This time it only took 63.48 seconds. Man, let's cut the anchor chain, something's holding us back Confused

Well, there's another good search we can run -- how about "John the Baptist" (in quotes), a New Testament search, against all Bibles. I ran this same search on the same machine quite recently, with the same library, in v. 3.f. and found 627 hits in about 17 seconds. Surely v. 4 can do better than that; and the results are in:

454 results in 50 verses in (one more drum roll please) 149.82.

Wow, version 3 came up with more hits, in about 1/10th of the time.

Maybe we ought to take off the quotation marks -- maybe that's holding us back. Let's see, Nope, that's not the answer! We've now found 518 results in (I don't know, I didn't run a check, so they may be the same) 50 verses, and it still took 144.87 seconds.

So, I'll definitely applaud the Basic search, across the library.

But the Bible search feature is woefully lacking. In fact, I'd have to say it's broke Crying . The same Bible search in v. 3f is quicker and finds more results.

Well, back to building Powerpoint presentations for next week's worship and adult confirmation classes

Dale

Find more posts tagged with

Performance

Comments

Robert Pavich

Dale,

Just an FYI there is a thread about what configuration of searches works bests...it's not immediately apparent is it?

for the record; When searching for English words; I only search my "English bible" collection BUT leave it Old Testament AND New Testament. It seems to slow it down when you limit it to the NT.

Also, it seemed to slow it down when adding "NT only " resources to a collection that you are going to search Old And NT in, such as the Interlinear GNT etc.

I hope this helps...

Bob Pritchett

across All Bibles, searching for "Jesus Wept" in quotes, in the New Testament (no need of even looking in the OT).

Hard as it is to believe, the slowness comes from limiting your search to the New Testament. Try "All Passages" and see what speed you get for that.

The index stores search hits at character offsets within the articles of the book. When you ask for everything, we can quickly get them all back, and then render them onscreen as necessary. When you ask to limit it to one Bible range, and then use multiple terms, we have to examine every hit, convert its character position to a Bible reference through another index, and then do a (relatively expensive) comparison of Bible references to see if the ranges intersect. (Our Bible range comparison is a bit slow, too, since we designed a flexible system to handle multiple versification schemes and other exotics stuff.)

We're working on optimizing this, but in the meantime I'd be curious to hear how long it takes you if you switch to All Passages.

Dale Durnell, Retired UM Clergy

We're working on optimizing this, but in the meantime I'd be curious to hear how long it takes you if you switch to All Passages.

Good Morning Bob --

Well, "all passages" in all Bibles certainly helped. I'm now finding the 454 results in 25.05 seconds. Still a bit slower than I timed it v. 3f, but expotentially faster than the 2 1/2 minutes of the previous search.

That said, I appreciate you continuing to optimize this -- by golly, if we can find over 2,000 references to Calvin in Geneva, in over 3000 resources, in under 5 seconds, then surely we ought to be able to find 454 references to John the Baptist in less than 100 specific resources, nearly instantaneously (don't you think?).

Thanks for responding

Dale

Dale Durnell, Retired UM Clergy

for the record; When searching for English words; I only search my "English bible" collection BUT leave it Old Testament AND New Testament. It seems to slow it down when you limit it to the NT.

Thanks Robert -- that helped tremendously

I just built an English Bible collection such as I had in v.3f, and my search for John the Baptist just found 318 results in 38 verses, in 16.63 seconds. That's 1/10th the time it was taking originally, but it's not any faster than v.3f. In fact, compared to an NT only search in v.3f, against my English Bible collection there -- it takes twice as long in v. 4

We'll see what happens as the test cycle moves along -- Bob's said they're working on optimizing this, so there's still hope

Dale

Robert Pavich

Dale,

For reference:

For the following
searches in English I used a collection called “English Bibles”
which are my English bibles minus the Lexhams and a few others that I
don't see a use for. Here is the filter in my collection:

type:bible
lang:english ANDNOT title:sept, lexham, Paul's, Jewish, Message,
Pastoral, study, noteless

Note: This is a
more realistic search for me; no need to have ALL BIBLES being
searched...Greek , Hebrew, etc.

Here are the results:

Search
Syntax: “Jesus Wept”

Search
Type: Basic
Search

Collection:
English bibles

Results:
12 results, 10 articles, .19
seconds

Search
Syntax: “Jesus Wept”

Search
Type: Bible
search

10 results .45
seconds in grid view.

Search:
“John
the Baptist”

Search
type: Basic
Search

Collection: English
Bibles

Results: 361 hits 4.94
seconds

Search:
“John
the Baptist”

Search
type: Bible
search

Collection:
English Bibles

Results:
178 hits 17 verses 6.63
seconds.

spitzerpl

Hard as it is to believe, the slowness comes from limiting your search to the New Testament. Try "All Passages" and see what speed you get for that.

Personally I would rather search the whole Bible, have a summary display of where that passage applies, and then jump to the place I want to go to. for instance, after searching the whole Bible a list of books containing the search would be displayed in line. clicking an a book name would jump me to that place in the list. Or, if you are going for the Iphone feel, list them like the iphone contacts, with the book names along the right side. as your cursor moves over a book name, the list moves to that location. Color code the names (Pent, History, Poetic, etc). I would add this to the existing format since some will want greater control of the passage.

Dale Durnell, Retired UM Clergy

Just for the record, a Basic Search for "Calvin WITHIN 10 WORDS Geneva" found 15,558 results in 393 articles in 2.84 seconds on my fastest machine.

A Bible search, all passages, all Bibles for "John the Baptist" found those 454 results in 69.12 sec
A Bible search, all passages, "English Bibles" collection for "John the Baptist" found 304 results in 38 verses in 16.91 sec

Interestingly, I'm not finding search results all that improved over an older P4 running a 32 bit version of XP/Pro and a new Core 2 Duo running 64 vit Vista

Blessings, Dale

Dell, Studio 1737, Core 2 Duo T6600, 2.2 GHz, 2200 MHz, 4 gb RAM
64 bit, Vista Home Premium, SP2

Ken McGuire

That said, I appreciate you continuing to optimize this -- by golly, if we can find over 2,000 references to Calvin in Geneva, in over 3000 resources, in under 5 seconds, then surely we ought to be able to find 454 references to John the Baptist in less than 100 specific resources, nearly instantaneously (don't you think?).

I'm hardly a Logos insider, but, to be honest, the answer this semi-computer geek would give you is no, I would not expect it. The way Logos 4.0 finds 2000 references to Calvin in Geneva so fast is because it is NOT really searching 3000 resources. It is searching one index which took a while to build. When you look for John the Baptist, it looks in the same index, and then has to take more time to cut out the results you don't want (as in the other 2900 resources). There are often times where we would want to cut the results down to a managable number to look at ourselves, but this does take computer time, and the computer can be much faster at this than I would be.

I have not tested this, but searches SHOULD be faster with a dynamically defined collection because it uses keys the computer can check faster than a semi-random list of files like the old ones.

Of course, I suppose it would be theoretically possible for Logos to create indexes on our collections so that they can be searched lightning fast, but do we really want this?

Ken McGuire

Dave Hooton

my search for John the Baptist just found 318 results in 38 verses

Hi Dale,

You'll find that Luke 5:33 "John the Baptist's" is missing from v4 results (if you have NLT or Living Bible) as v4 has an "exact phrase search" (I was fond of saying that v3 had an "inexact phrase search"). So a search for "kingdom of God" will exclude the v3 result "Kingdom of our God".

Also v4 counts real words in a proximity search like:

Jesus BEFORE 1 words Lord

v3 will return "Jesus is Lord" (2 words) whilst v4 returns only "Jesus Lord" (1 word)!

Dave Hooton

Personally I would rather search the whole Bible, have a summary display of where that passage applies, and then jump to the place I want to go to.

It is also a call to allow the deletion of unwanted results. If you want NT, select all results from Genesis to Malachi and delete!

spitzerpl

It is also a call to allow the deletion of unwanted results. If you want NT, select all results from Genesis to Malachi and delete!

I could see a lot of usefulness for that. What would be cool is if you could export the results to clippings somehow too after you filter it down to a range.

Bob Pritchett

The two slowdowns for "John the Baptist" compare to your Calvin search are:

"the" appears a lot, and we have to look at all the hits to see if it's next to both John and Baptist.

A three word exact phrase requires two "are these next to each other? tests. The Calvin query has only one.

John Baptist

should be a faster query, though granted it's not as exact. But it has the advantage of finding phrases like "the Baptist (John)" etc.

Most search engines speed things up by throwing out "stop words" -- the, and, of, a.

We don't because they are important for exact phrases, and more relevant in the types of queries our users do. But just so you know, a "bag of words" will always search faster than an exact phrase, and the ranking should keep the best results at the top in both scenarios. (Though you'll get more false positives -- just like Google delivers. But who looks at the 300th page of search results?)

Dale Durnell, Retired UM Clergy

We'll see what happens as the test cycle moves along -- Bob's said they're working on optimizing this, so there's still hope

Search speed, L4B2 -- much improved!! Thanks so very much

Searched for "John the Baptist" - searching all passages in my English Bible collection, 305 results in 38 verse in 13.86 seconds. That's about 1/4 the time I experienced in L4B1
Searched for "Jesus Wept" - again, all passages, English Bible collection, 17 results, 1 verse, 3.11 seconds. Again, vast improvement in search speed.

My compliments to the chef!!

Dale