Benchmarking

Some of you know that I'm shortly going to be building myself a new PC, which offers a great time to do some comparative benchmarking to help get to the bottom of the RAM/CPU vs Disk debate. So I'll do the tests on four configurations: (a) Dual Core + HDD, (b) Dual Core + SSD, (c) i7 + HDD, and (d) i7 + SSD.

I doubt I'll have time to do more than 10 tests in each configuration. Are there particular pain points in Logos that it's worth me testing. I'm currently intending the following, but am happy to swap other things in if they'd be appreciated:

Start up time
Search entire library (non-wildcard)
Search entire library (wildcard)
Passage guide with all non-internet sections open
Parallel Resources lookup
Morphological analysis search
Open saved layout with 30-40 resources
Power Lookup
Compile PBB
Scroll through a Bible with all visual filters turned on
Reindex

Any other suggestions? (If you suggest more than one, I insist that you also suggest one to remove!)

I'm still waiting for my SSD to arrive, so I won't have results for at least a week.

Find more posts tagged with

Comments

BKMitchell

Greetings Mark,

Are there particular pain points in Logos that it's worth me testing.

Yes, syntax searches in the Andersen-Forbes database.

NB.Mick

Are there particular pain points in Logos that it's worth me testing. Any other suggestions? (If you suggest more than one, I insist that you also suggest one to remove!)

Great project! My take:

Re-sorting the library from default (by-title) to by-last-updated.

Even though I only offered one: if the cumulative amount of suggestions leads you to reconsider your list, I'd suggest looking critically at #10, for two reasons

a) your benchmark test addresses the effect of processor power versus disk speed, this "pain point" will probably be strongly dependend on the graphics capabilities of a user's machine

b) the other pain points are nearly inevitable for users, whereas the "pain" in #10 may be much reduced by simply turning off or re-designing the most outlandish VFs

Fr. Charles R. Matheny

https://community.logos.com/discussion/comment/459758#Comment_459758

Dear Mark: The benchmarks may well be useful to you and for you testing new things in Logos, but will not mean much out here.

The reason being is simply that no one out here will have your exact same set-up. ( Very few at most).

Unless others have your exact same motherboard, same brand of Ram, Hdd, video card, cooling fan, exact same operating system and it set up exactly the same, same network card, same bios settings and version and, same software in other areas ( Office, games, flash player and setting etc.) then the benchmarks you take will not correspond to others usage, especially with Logos.

Looking at your video's, I feel safe to say you are an excellent builder and tweaker, looks like you do well in keeping your system clean and freed up on conflicts.

Many folks do not have that skill set in place.

One of the things we have seen in Logos is that it is very sensitive to these things and to interactions with other software installed.

Thus Logos may run really well on one machine, but on another similar machine, with different softwares installed, it does not.

Though I use Macs for most everything, I still love building Windows boxes and repairing them, removing malware etc.

I have build "side by side" boxes with the exact same components, exact same install methods, exact same operating system and, with nothing else installed, found the machines did not benchmark the same- the tolerances allowed for in the specs of the components comes into play and could be seen in the testing.

Alas, that was a few years ago and I do believe tolerances are much tighter now as things get smaller and smaller, faster and faster.

Look forward to seeing and hearing about your new box.

Blessings

NB.Mick

https://community.logos.com/discussion/comment/459769#Comment_459769

Dear Mark: The benchmarks may well be useful to you and for you testing new things in Logos, but will not mean much out here.

The reason being is simply that no one out here will have your exact same set-up.

Fr. Charles,

I'm sure Mark will answer this much more thoroughly than I can. However, my preliminary understanding of his project is not to provide any absolute benchmark, like "searching all library for "god" takes XYZ seconds on i7 and 23.8% longer on a Dual Core processor". Rather it will allow a kind of insight into the relative importance of a faster disk versus a faster processor. From lurking in some of the respective threads, this is a practical question that comes up often, especially when users can afford only a limited budget for hardware upgrade. Meaningful results would allow more informed recommendations for users.

Mick

Mitch Davis

https://community.logos.com/discussion/comment/459777#Comment_459777

it will allow a kind of insight into the relative importance of a faster disk versus a faster processor. From lurking in some of the respective threads, this is a practical question that comes up often, especially when users can afford only a limited budget for hardware upgrade. Meaningful results would allow more informed recommendations for users.

[Y]

Fr. Charles R. Matheny

https://community.logos.com/discussion/comment/459780#Comment_459780

ahh, my mistake.

Thanks and , sorry.

Xegesis

https://community.logos.com/discussion/comment/459780#Comment_459780

Great Idea Mark, an idea if you like would be to test the speed of link set combinations.

1) Bible and Commentary

2) Bible, Commentary and Passage Guide

3) Multiple Bibles, Multiple Commentaries

4) Multiple Bibles, Multiple Commentaries and Passage Guide

Randy W. Sims

Hi Mark,

This is probably beyond the scope of what you're looking to do, and I'm not sure what memory configurations you have available to try, but I'm curious how Logos would perform your tests in very low memory configurations with an SSD, like that found in many of the new Windows tablets that are coming out. In particular, I'm wondering how much the usual slowdowns associated with swapping out to disk are negated by the SSDs speed in 4 - 8GB configurations (possibly with a few apps or resource hog browser running with a few tabs open to simulate typical usage).

I understand if that's beyond the scope of what you're looking to test, but I thought I'd throw it out there.

Looking forward to seeing the results of you benchmarks.

Thank you.

Unix

https://community.logos.com/discussion/comment/459852#Comment_459852

Suggestion: the time it takes from clicking the library button until you can type in "Find"

Dave Hooton

Search entire library (wildcard)

This isn't a real-world test and SSD will always win. Save a lot of time and replace it with:

Response - initial time to open a resource panel/tab in a layout
Response - initial time to open a resource dropdown (Search in) in Bible search

Is the i7 a quad core? Are you simulating dual-core on the same i7 or is it a different processor? If different processor, is it on a different machine?

How much RAM?

David P. Moore

Any other suggestions?

My suggestion: on the Bible search tab, see how long it takes for the pull down menu that changes the Bible to populate, a la this discussion:

http://community.logos.com/forums/p/65022/455019.aspx

This needs to be done after first launch of the program, as subsequent attempts to access the menu appears much quicker. With all the Bible fragments I have, it take over a minute to populate on my i7 HDD laptop.

Mark Barnes

https://community.logos.com/discussion/comment/459926#Comment_459926

There's some great ideas here, thanks. I'll certainly try and incorporate at least some of them.

Jon

I would only suggest turning off use Internet to remove background/sync related variability as a confounder...

Mark Barnes

My SSD has now arrived, so I've started the benchmarking. My existing system happens to have 5,400, 7,200 and 10,000rpm drives, so I decided that the initial comparison would be between these four drives. I still need to do some further analysis, but the indications so far are that the benchmarks confirm what we all probably suspected:

That disk speed really does matter for most Logos operations. There's a discernable difference between all four drives for most Logos operations.
That having a decent amount of RAM evens things out considerably because files are cached in RAM.

Here's a quick and dirty chart showing a sample of the differences with no files cached in RAM:

And here's a chart to the same scale that shows the difference after files are cached in RAM:

The file caching is performed automatically by Windows, and will use all available RAM to cache frequently used files. So an SSD will make much less of a difference if you have plenty of RAM and frequently perform similar operations (i.e. you tend to study one book of the Bible at a time, stick to four or five tools). If you're short on RAM, or tend to jump about quite a bit in your studies, then an SSD will make a much bigger difference.

I'll do a better write-up once all the results are in.

Randy W. Sims

https://community.logos.com/discussion/comment/461216#Comment_461216

Nice! Great analysis. Appreciate your work and sharing. Looking forward to final results.

Mark Barnes

https://community.logos.com/discussion/comment/461485#Comment_461485

Putting in re-indexing as one of the tests was not a good idea! It's taking about 15 hours to reindex on the slower disks, so I'm still waiting for that figure. Once I have it, I'll post up the results of the drive tests, and do the RAM/CPU tests later.

Bob Schlessman

https://community.logos.com/discussion/comment/461989#Comment_461989

Putting in re-indexing as one of the tests was not a good idea! It's taking about 15 hours to reindex on the slower disks, so I'm still waiting for that figure. Once I have it, I'll post up the results of the drive tests, and do the RAM/CPU tests later.

Wow! That is a long time Mark, even for a mechanical hard drive. How many resources do you have?

I have 1,375 resources and it took my system 28 minutes to complete a "Rebuild Index" command on my SSD. I know from your posts you have considerably more resources than I do.

System:

CPU - i5-2500K (OC to 4.4 Ghz)

System drive - Mushkin SATA 3, 120 Gbyte SSD

RAM - 16 Gbytes

Video - Saphire AMD - 7750, 1 Gbyte GDDR 5

Donnie Hale

https://community.logos.com/discussion/comment/461216#Comment_461216

My SSD has now arrived, so I've started the benchmarking.

Mark,

Which SSD did you get? And do you or anyone else have recommendations on brand / model / highest priority features?

I'm considering getting one "real soon now." However, having followed SSDs closely since they first came on the market, I've mostly noted the problems that have accompanied their release: data corruption from firmware bugs; premature failure; worse than advertised / expected performance; etc. My suspicion is that most of those challenges (along with price/GB) have been overcome, but I don't know who's doing it best.

Advice appreciated... Thanks (and hopefully this doesn't totally hijack the thread ,

Donnie

Randy W. Sims

https://community.logos.com/discussion/comment/462104#Comment_462104

You might find the following links helpful; these are some of my favorite geek hardware sites:

http://www.tomshardware.com/reviews/ssd-recommendation-benchmark,3269.html

http://www.anandtech.com/tag/storage

http://arstechnica.com/search/?query=ssd

Dave Hooton

https://community.logos.com/discussion/comment/462104#Comment_462104

My suspicion is that most of those challenges (along with price/GB) have been overcome, but I don't know who's doing it best.

OCZ Vertex 4 seems to be doing it best but Samsung 840 Pro and Intel 520 series are also highly regarded. All have a 5 yr warranty. On the budget side I would go for Intel 335 series with 3 yr warranty. I have 180 GB from the superceded 330 series and very pleased with it.

Mark Barnes

https://community.logos.com/discussion/comment/462104#Comment_462104

Which SSD did you get? And do you or anyone else have recommendations on brand / model / highest priority features?

I went for the Samsung 840 Pro, which is pretty much the fastest available - although it's expensive, and for most users the added performance (over a standard 840) may not be worth the extra dollars. But for my desktop PC, which is running MySQL as well as Logos , the extra performance will be very welcome.

In my next post in this thread, I'm going to give the real-world results of my SSD tests. I'll add more test results for other components at a later stage.

Mark Barnes

https://community.logos.com/discussion/comment/462431#Comment_462431

The base specification of the computer used for these tests was:

ASRockP45XEmotherboard with Intel P45 chipset
Intel Core 2 Duo E8300@ 2.83GHz
4GB PC2-5300 RAM
GeForce 8800GTS320MB graphics card
Windows 8 Pro
Logos 5.0b (build 5.0.2.0562)

Testing methodology

During testing, various tasks were performed and timed. Where possible, timings were given by the program, either onscreen or in the log file. Otherwise, a stopwatch was used. During testing, unnecessary background processes were turned off, other applications closed, and Logos' access to the internet was also turned off to ensure syncing didn't interfere with the testing.

The test was performed with a large Logos library of 9,173 books plus an additional 506 hidden resources. All Logos components and datasets were available.

The twelve tasks performed were:

Starting Logos
Perform a basic search of the entire library for (Paul OR "Saul of Tarsus") AND apostle ANDNOT missionary
Populate the Bibles menu on the Bible search dropdown.
Perform a Passage Guide on John 3:16, with all non-internet sections opened.
Perform a parallel resources lookup for Genesis 15:21.
Perform a morphological analysis search across all Greek resources using Logos Greek Morphology, for @V?M
Open a saved layout of around 40 resources and four large user documents.
Close All Windows.
Perform a syntax search for passives with an implied subject in the OpenText.org resource.
Compile a large Personal Book (Index to Spurgeon's Sermons).
Perform a basic search of the entire library for Jesus BEFORE 4 WORDS Christ.
Reindex Logos (this was only performed once, after the second set).

For each round, all the tasks were performed in the same order (set 1), and then repeated (set 2). Before each round, Windows RAM was flushed using RAMMap. This meant that set 1 always began with no files cached in memory, whilst set 2 had files cached. For most tasks, the two sets had significantly different times.

In order to try and provide the most realistic assessment, all of the main charts provide an adjusted average, not raw data. Tasks that are likely to happen frequently (like look-ups and searches) are weighted much more heavily than tasks that happen rarely (like compiling personal books, or reindexing). I've also weighted cached results more heavily than uncached, because rough observations from my own study suggested that about one in four operations seemed to be cached. Of course no two Logos users will have the same library, hardware or usage patterns, so your mileage will vary. But I hope the results will at least give some indication of what speed increases different hardware upgrades might bring.

Hard Disk / Solid State Drives

Four drives were tested:

2TB Samsung SpinPoint F4 (HD204UI) - 5,400rpm HDD.
500GB Seagate Barracuda (ST3500630AS) - 7,200rpm HDD.
150GB Western Digital Velociraptor (WD1500HLFS) - 10,000rpm HDD.
256GB Samsung 840 Pro - SSD (operating at SATAII, not SATAIII).

For each test only Logos was installed on the test drive. Other files were not moved.

As the graph below shows, the performance gain is significantly higher for non-cached results, with only the SSD giving a noticeable speed increase for cached results. This means performance gains will be even more marked for those with large libraries and/or less RAM.

The weighted average below is likely to be a more realistic indicator, as it tries to take into consideration how likely it is that the files needed for a task will already be cached.

A faster drive made a significant difference in some, but not all tasks. The chart below shows the difference between the slowest and fastest drives for each task. Several tasks are very noticeably quicker on the SSD (particularly #1‑2, #4‑7 and 12) when not cached, whilst on others there is little difference. Generally speaking, users who start with searches and guides and open lots of different resources will get a higher performance boost than those who tend to start from a fixed layout with commonly used resources (because the latter method makes better use of the cache, and therefore performance gains are more limited).

In summary, these figures suggest that 10,000rpm drives and particularly SSDs will give a significant performance benefit in Logos 5. 128GB SSDs can be had for just over $100, which is about the same as a 250GB 10,000rpm drive. Both offer excellent value as a relatively cheap way of getting a significant performance gain.

Perhaps the most surprising aspect of this test is that the 7,200rpm drive was slower than the 5,400rpm drive. This is a helpful reminder that rpm is not necessarily an accurate indicator of transfer speed, as the higher data density of larger drives can more than compensate for the slower spin speed. So if you're thinking of upgrading you should check the data transfer rate and seek times before you purchase. (In my system the 5,400rpm has an average read rate of 94MB/s, compared to 63.1MB/s for the 7,200rpm drive.)

Finally, Logos frequently accesses the system drive (for .NET) and occasionally the temporary folder (for saving large search results), so if you have large, fast drive Logos will benefit from having the system files and temporary folder on that drive.

Lynden O. Williams

https://community.logos.com/discussion/comment/462436#Comment_462436

Very informative, I book marked it. Thanks for posting.