Beta 5.3: Search improvement

The comparison comes from different computers (both 8 MB RAM) but the improvement obtained in 5.3 64-bit transcends the hardware differences:-

Basic Search for <Gen-Rev> in Bible Knowledge Commentary

64-bit 5.3 ---> 87304 results in 11s + c. 10s to display results. About 306 MB memory

32-bit 5.2b --> 87304 results in 210s + c. 3m to display results. About 395 MB memory

I don't think there has been any code optimization, and there was no re-indexing with 5.3. So this may well be attributed to the new SQLite Memory-mapped I/O!?

Find more posts tagged with

Comments

Simon

Searching my library is considerably faster here. 72,276 results in 25,468 articles and 1,827 resources in 0.70 seconds (a search for Qumran).

Showing the search results however, is slower. When, for example, I move to the next page in the search results, it takes 2.5 seconds to show the results of the next page.

Bradley Grainger (Logos)

So this may well be attributed to the new SQLite Memory-mapped I/O!?

That may help a little, but there are two major differences in search in 64-bits:

Memory-map the postings file (the index) instead of using regular file I/O.
Use RAM for temporary storage of results, instead of temp files. (We had to use temp files in 32-bit to avoid running out of memory if you were running lots of searches simultaneously.)

abondservant

https://community.logos.com/discussion/comment/621951#Comment_621951

Would be interested to know the system spec differences any way.

Dave - would you mind doing a basic search across all resources for "the" on both? Its a pretty massive search.

HansK

https://community.logos.com/discussion/comment/621968#Comment_621968

FWIW, here's mine. MBP 2010 build, OS 10.9.4, 8 GB 1067 MHz DDR3, SSD 1 TB

Nearly 17.000 books in library.

Dave Hooton

https://community.logos.com/discussion/comment/621968#Comment_621968

Dave - would you mind doing a basic search across all resources for "the" on both? Its a pretty massive search.

It would be meaningless as I don't have anywhere near 5900 resources! That's why I did the test on a single book that many users would have.

I'll do a comparison run after Logos let me know if 5.2b can run on the same machine as 64-bit 5.3.

Dave Hooton

https://community.logos.com/discussion/comment/621968#Comment_621968

Would be interested to know the system spec differences any way.

5.2b on first generation laptop Core i5 2.5 GHz, 180 GB SSD with 8 GB RAM

5.3 on second gen desktop Core i7 2.80 GHz, 250 GB 10,000 rpm HDD with 8 GB RAM.

abondservant

https://community.logos.com/discussion/comment/622139#Comment_622139

One could still generate a per resource average, which would be an interesting baseline.

abondservant

https://community.logos.com/discussion/comment/622141#Comment_622141

Any way, searching simply what you did took nearly as much time as searching my entire library.

Dave Hooton

https://community.logos.com/discussion/comment/622141#Comment_622141

One could still generate a per resource average, which would be an interesting baseline.

In 64-bit 5.3 the <Gen-Rev> search took 19s for 59 commentaries and 377686 results that included BKC, whereas BKC alone took 11s for 87304 results. The 2-volume EBC (also included in the 59 commentaries) took 9s for 76463 results. What is the meaningful average?

'the' on 5.2b took 0.17s for 129726 results in BKC. ---> laptop

'the' on 5.3 took 0.21s in BKC! ----> desktop

UPDATE:

'the' on 5.2b took 0.18s for 129726 results in BKC. ---> desktop!

Essentially the same time as the laptop.

Dave Hooton

The comparison comes from different computers (both 8 MB RAM) but the improvement obtained in 5.3 64-bit transcends the hardware differences:-

Basic Search for <Gen-Rev> in Bible Knowledge Commentary

64-bit 5.3 ---> 87304 results in 11s + c. 10s to display results. About 306 MB memory ==> desktop 2.80 GHz Core i7 with 10000rpm HDD

32-bit 5.2b --> 87304 results in 210s + c. 3m to display results. About 395 MB memory ==> laptop 2.53 GHz Core i5 with SSD

On the same machine as Logos 5.3:-

32-bit 5.2b --> 87304 results in 242s

The laptop with SSD (Solid state Drive) beats the faster Core i7 desktop with a very fast HDD in this test.

abondservant

https://community.logos.com/discussion/comment/622181#Comment_622181

The laptop - also running 5.3 beta 1. was twice as slow.

1st gen i5 chip, 4gb ram, ssd

87304 results, in 59 seconds.

Searching entire library for "the" required 37 seconds.

the <gen-rev> search is a bit more demanding. I ran <gen-rev> on my entire library on my desktop and after 10 minutes or so Logos crashed.