Benchmarking

Page 1 of 2 (24 items) 1 2 Next >
This post has 23 Replies | 3 Followers

Posts 13399
Mark Barnes | Forum Activity | Posted: Wed, Feb 6 2013 5:37 AM

Some of you know that I'm shortly going to be building myself a new PC, which offers a great time to do some comparative benchmarking to help get to the bottom of the RAM/CPU vs Disk debate. So I'll do the tests on four configurations: (a) Dual Core + HDD, (b) Dual Core + SSD, (c) i7 + HDD, and (d) i7 + SSD.

I doubt I'll have time to do more than 10 tests in each configuration. Are there particular pain points in Logos that it's worth me testing. I'm currently intending the following, but am happy to swap other things in if they'd be appreciated:

  1. Start up time
  2. Search entire library (non-wildcard)
  3. Search entire library (wildcard)
  4. Passage guide with all non-internet sections open
  5. Parallel Resources lookup
  6. Morphological analysis search
  7. Open saved layout with 30-40 resources
  8. Power Lookup
  9. Compile PBB
  10. Scroll through a Bible with all visual filters turned on
  11. Reindex

Any other suggestions? (If you suggest more than one, I insist that you also suggest one to remove!)

I'm still waiting for my SSD to arrive, so I won't have results for at least a week.

Posts 468
BKMitchell | Forum Activity | Replied: Wed, Feb 6 2013 5:44 AM

Greetings Mark,

Mark Barnes:
Are there particular pain points in Logos that it's worth me testing.

Yes, syntax searches in the Andersen-Forbes database.

חַפְּשׂוּ בַּתּוֹרָה הֵיטֵב וְאַל תִּסְתַּמְּכוּ עַל דְּבָרַי

Posts 10887
Forum MVP
NB.Mick | Forum Activity | Replied: Wed, Feb 6 2013 6:03 AM

Mark Barnes:

Are there particular pain points in Logos that it's worth me testing. Any other suggestions? (If you suggest more than one, I insist that you also suggest one to remove!)

Great project! My take:

  • Re-sorting the library from default (by-title) to by-last-updated.

Even though I only offered one: if the cumulative amount of suggestions leads you to reconsider your list, I'd suggest looking critically at #10, for two reasons

a) your benchmark test addresses the effect of processor power versus disk speed, this "pain point" will probably be strongly dependend on the graphics capabilities of a user's machine

b) the other pain points are nearly inevitable for users, whereas the "pain" in #10 may be much reduced by simply turning off or re-designing the most outlandish VFs

Running Logos 8 latest beta version on Win 10

Posts 757
Fr. Charles R. Matheny | Forum Activity | Replied: Wed, Feb 6 2013 6:58 AM

Dear Mark: The benchmarks may well be useful to you and for you testing new things in Logos, but will not mean much out here.

The reason being is simply that no one out here will have your exact same set-up. ( Very few at most).

Unless others have your exact same motherboard, same brand of Ram, Hdd, video card, cooling fan, exact same operating system and it set up exactly the same, same network card, same bios settings and version and, same software in other areas ( Office, games, flash player and setting etc.)  then the benchmarks you take will not correspond to others usage, especially with Logos.

Looking at your video's, I feel safe to say you are an excellent builder and tweaker, looks like you do well in keeping your system clean and freed up on conflicts.

Many folks do not have that skill set in place.

One of the things we have seen in Logos is that it is very sensitive to these things and to interactions with other software installed.

Thus Logos may run really well on one machine, but on another similar machine, with different softwares installed, it does not.

Though I use Macs for most everything, I still love building Windows boxes and repairing them, removing malware etc.

I have build "side by side" boxes with the exact same components, exact same install methods, exact same operating system and, with nothing else installed, found the machines did not benchmark the same- the tolerances allowed for in the specs of the components comes into play and could be seen in the testing.

Alas, that was a few years ago and I do believe tolerances are much tighter now as things get smaller and smaller, faster and faster.

Look forward to seeing and hearing about your new box.

Blessings

Posts 10887
Forum MVP
NB.Mick | Forum Activity | Replied: Wed, Feb 6 2013 7:12 AM

Fr. Charles R. Matheny:

Dear Mark: The benchmarks may well be useful to you and for you testing new things in Logos, but will not mean much out here.

The reason being is simply that no one out here will have your exact same set-up.

Fr. Charles,

I'm sure Mark will answer this much more thoroughly than I can. However, my preliminary understanding of his project is not to provide any absolute benchmark, like "searching all library for "god" takes XYZ seconds on i7 and 23.8% longer on a Dual Core processor". Rather it will allow a kind of insight into the relative importance of a faster disk versus a faster processor. From lurking in some of the respective threads, this is a practical question that comes up often, especially when users can afford only a limited budget for hardware upgrade. Meaningful results would allow more informed recommendations for users.

Mick 

Running Logos 8 latest beta version on Win 10

Posts 347
Mitch Davis | Forum Activity | Replied: Wed, Feb 6 2013 7:20 AM

NB.Mick:

it will allow a kind of insight into the relative importance of a faster disk versus a faster processor. From lurking in some of the respective threads, this is a practical question that comes up often, especially when users can afford only a limited budget for hardware upgrade. Meaningful results would allow more informed recommendations for users.

Yes

Posts 757
Fr. Charles R. Matheny | Forum Activity | Replied: Wed, Feb 6 2013 7:38 AM

ahh, my mistake.

Thanks and , sorry.

Posts 98
Xegesis | Forum Activity | Replied: Wed, Feb 6 2013 7:47 AM

Great Idea Mark, an idea if you like would be to test the speed of  link set combinations. 

1) Bible and Commentary

2) Bible, Commentary and Passage Guide

3) Multiple Bibles, Multiple Commentaries

4) Multiple Bibles, Multiple Commentaries and Passage Guide

Posts 2081
Randy W. Sims | Forum Activity | Replied: Wed, Feb 6 2013 11:06 AM

Hi Mark,

This is probably beyond the scope of what you're looking to do, and I'm not sure what memory configurations you have available to try, but I'm curious how Logos would perform your tests in very low memory configurations with an SSD, like that found in many of the new Windows tablets that are coming out. In particular, I'm wondering how much the usual slowdowns associated with swapping out to disk are negated by the SSDs speed in 4 - 8GB configurations (possibly with a few apps or resource hog browser running with a few tabs open to simulate typical usage).

I understand if that's beyond the scope of what you're looking to test, but I thought I'd throw it out there.

Looking forward to seeing the results of you benchmarks.

Thank you.

Posts 2038
Unix | Forum Activity | Replied: Wed, Feb 6 2013 11:15 AM

Suggestion: the time it takes from clicking the library button until you can type in "Find"

Aply!
trulyergonomic.com 6,200own
12G A9-9420 V8.6 Acc 11
d:'13Q3 12G

Posts 25862
Forum MVP
Dave Hooton | Forum Activity | Replied: Wed, Feb 6 2013 3:23 PM

Mark Barnes:

  • Search entire library (wildcard)
  • This isn't a real-world test and SSD will always win. Save a lot of time and replace it with:

    • Response - initial time to open a resource panel/tab in a layout
    • Response - initial time to open a resource dropdown (Search in) in Bible search

    Is the i7 a quad core?  Are you simulating dual-core on the same i7 or is it a different processor? If different processor, is it on a different machine?

    How much RAM?

    Dave
    ===

    Windows 10 & Android 8

    Posts 552
    David P. Moore | Forum Activity | Replied: Wed, Feb 6 2013 7:40 PM

    Mark Barnes:
    Any other suggestions?

    My suggestion: on the Bible search tab, see how long it takes for the pull down menu that changes the Bible to populate, a la this discussion:

    http://community.logos.com/forums/p/65022/455019.aspx

    This needs to be done after first launch of the program, as subsequent attempts to access the menu appears much quicker. With all the Bible fragments I have, it take over a minute to populate on my i7 HDD laptop.

    Posts 13399
    Mark Barnes | Forum Activity | Replied: Thu, Feb 7 2013 3:22 AM

    There's some great ideas here, thanks. I'll certainly try and incorporate at least some of them.

    Posts 687
    Jon | Forum Activity | Replied: Thu, Feb 7 2013 4:49 AM

    I would only suggest turning off use Internet to remove background/sync related variability as a confounder...

    Posts 13399
    Mark Barnes | Forum Activity | Replied: Sat, Feb 9 2013 3:42 PM

    My SSD has now arrived, so I've started the benchmarking. My existing system happens to have 5,400, 7,200 and 10,000rpm drives, so I decided that the initial comparison would be between these four drives. I still need to do some further analysis, but the indications so far are that the benchmarks confirm what we all probably suspected:

     

    1. That disk speed really does matter for most Logos operations. There's a discernable difference between all four drives for most Logos operations.
    2. That having a decent amount of RAM evens things out considerably because files are cached in RAM.

     

    Here's a quick and dirty chart showing a sample of the differences with no files cached in RAM:

    And here's a chart to the same scale that shows the difference after files are cached in RAM:

    The file caching is performed automatically by Windows, and will use all available RAM to cache frequently used files. So an SSD will make much less of a difference if you have plenty of RAM and frequently perform similar operations (i.e. you tend to study one book of the Bible at a time, stick to four or five tools). If you're short on RAM, or tend to jump about quite a bit in your studies, then an SSD will make a much bigger difference.

    I'll do a better write-up once all the results are in.

    Posts 2081
    Randy W. Sims | Forum Activity | Replied: Sun, Feb 10 2013 1:30 PM

    Nice! Great analysis. Appreciate your work and sharing. Looking forward to final results.

    Posts 13399
    Mark Barnes | Forum Activity | Replied: Tue, Feb 12 2013 2:34 AM

    Putting in re-indexing as one of the tests was not a good idea! It's taking about 15 hours to reindex on the slower disks, so I'm still waiting for that figure. Once I have it, I'll post up the results of the drive tests, and do the RAM/CPU tests later.

    Posts 291
    Bob Schlessman | Forum Activity | Replied: Tue, Feb 12 2013 7:47 AM

    Mark Barnes:

    Putting in re-indexing as one of the tests was not a good idea! It's taking about 15 hours to reindex on the slower disks, so I'm still waiting for that figure. Once I have it, I'll post up the results of the drive tests, and do the RAM/CPU tests later.

    Wow! That is a long time Mark, even for a mechanical hard drive. How many resources do you have?

    I have 1,375 resources and it took my system 28 minutes to complete a "Rebuild Index" command on my SSD. I know from your posts you have considerably more resources than I do.

    System:

    CPU - i5-2500K (OC to 4.4 Ghz)

    System drive - Mushkin SATA 3, 120 Gbyte SSD

    RAM - 16 Gbytes

    Video - Saphire AMD - 7750, 1 Gbyte GDDR 5

    Posts 1943
    Donnie Hale | Forum Activity | Replied: Tue, Feb 12 2013 10:41 AM

    Mark Barnes:
    My SSD has now arrived, so I've started the benchmarking.

    Mark,

    Which SSD did you get? And do you or anyone else have recommendations on brand / model / highest priority features?

    I'm considering getting one "real soon now." However, having followed SSDs closely since they first came on the market, I've mostly noted the problems that have accompanied their release: data corruption from firmware bugs; premature failure; worse than advertised / expected performance; etc. My suspicion is that most of those challenges (along with price/GB) have been overcome, but I don't know who's doing it best.

    Advice appreciated... Thanks (and hopefully this doesn't totally hijack the thread :),

    Donnie

     

    Page 1 of 2 (24 items) 1 2 Next > | RSS