Storage speed effect on searches

There is a falloff!

I have a 5GB library (not including the 11GB of baggage files.)

Example- on my laptop with its 25MB/sec hard drive searching "jn 3 16" takes 32 seconds to complete.

On my desktop build project with the Core i7-920 and 12GB ram:

On an SSD array that has 584MB/sec read rate, the same search takes 8.4 seconds.

However, making a 5.5GB Ramdisk with a read rate of 4,234 MB/sec and loading the non-baggage resources there, it still takes 7.8 seconds!! So even thoug the read rate on the ramdisk is over 7x that of the SSD array, it only picked up less than 1/2 second of speed!

The raid array was 4 Intel 80GB X25-M Generation 2 SSD in raid 0.

A study of this drive in showed less return for your money on read rate beyond 2 drives (going form 3 to 4 drives not as big an improvement in read rate, but write rate (for windows swap file, etc goes up with each drive.)

So, 2 of these drives would be the best bang for the buck which would be around $450 if you find them fast enough at good ebay pricing.

I should point out that new firmware for the 160GB model X25-M Gen 2 boosts the write rate from 70MB/sec to 100MB/sec (the firmware upgrade did not boost the 80GB mode's write rate from 70MB.) So you could get even better write rates with just 2 160GB models, but they cost more than twice as much and you might do better with 4 80GB raid0

Next I went to processor speed.

Bumping the cpu speed from 2.6GHz to 3.6GHZ lowered the search from 8.4 seconds to 6.6 seconds. Now we see the effects of horsepower on the search engine.

The RAM was running at 1066MHZ. Boosting ram speed to 1600MHZ lowered the search time to 6.2 seconds.

I cant overclock anymore without a lot of voltage tweeking which takes a lot of time.

I wish Libronix had a 64-bit version. A tech said its a ram hog so it would be interesting to see what throwing 12GB at it does.

I might get around to trying to install the program to ramdisk to see what that does.

For the laptop crowd, there are some (but $$$$) laptops that take 2 harddrives (or SSD) where they can be raid0.

Find more posts tagged with

Comments

Stuart A Weber

I treid the ramdisk again now that the ram is at 1600MHZ (it was ta 1066MHZ before).

The read rate is now over 6200MB/sec, and our search time for jn 3 16 in a 5GB library (no baggage files) with the PC set at 3.6GHZ RAM 1600MHZ 7-7-7-21 6.5GB free, 5.5GB in RAMDISK is:

5.9 seconds

I've wondered how Logos would run on a ramdisk. If I understand correctly, you're putting your 5.5Gbs of resources on the ramdisk, but what about the index? For searches it might be more important to have the index file on the ramdisk than the resource folder, though if you've got enough RAM why not toss everything on the ramdisk, right?

I will look forward to hearing more of your experiences playing with this hardware. I'd love to give it a whirl myself but I don't have the budget for it while SSDs are still so expensive.

Stuart A Weber

https://community.logos.com/discussion/comment/36904#Comment_36904

but Dave H. stated in another thread:

"More a cache than an index - stored in \Documents and Settings\All Users\Application Data\Libronix DLS\Library\Cache\Metadata (XP) or \ProgramData\Libronix DLS\Library\Cache\Metadata (Vista/Win7). Created during Resource Discovery it is used more for Reports (Library, Bibliography) than searching."

So, intsalling the program on the Ramdisk wont put that cache there as its not in the Libronix directory under Programs.

However, there are commands out there to create alias directories. so, possibly could create an alias for the cache subdirectory and put the real subdirectory on the ramdisk.

I will play.

Stuart A Weber

https://community.logos.com/discussion/comment/37588#Comment_37588

I installed the program (ver 3) into the Ramdisk.

That cut the Passage search to 4.5 seconds from 5.9! (again, on a 5GB resource directory without baggage files.)

However, when I did an exegetical search, it got an error that t couldnt find a file.

This is under Win7, so ill have to figure out how to get it to let me into the the Documents and Settings section.

J.R. Miller

https://community.logos.com/discussion/comment/38893#Comment_38893

This is under Win7, so ill have to figure out how to get it to let me into the the Documents and Settings section.

FYI, Logos has said that if users mess with the resource folders, it will break the functionality of the program and they will not provide any support in this situation. I am not sure if this is what you are doing, but just thought I would pass the news along as I understand it.

Stuart A Weber

https://community.logos.com/discussion/comment/38934#Comment_38934

"mess with"?

you mean relocating? Its part of the program to put resources wherever you want and tell Logos where they are.

I was referring to one file that is apparanetly missing when you dont install the program to the C drive- which is something that Logos must support becasue it allows you to change that while installing. So I am referring to go look and se if the file it wants is really there or not.

I am NOT hacking.

J.R. Miller

https://community.logos.com/discussion/comment/38958#Comment_38958

"mess with"?

you mean relocating? Its part of the program to put resources wherever you want and tell Logos where they are.

If you are using L3, then yes you are right. However, if you are using L4, it has to be installed on the same drive as the resources and we cannot move the resource directory like in L4 without jeopardizing the integrity of the software to update itself.

Anyway, if you are on L3, good luck to you. :-)

Dave Hooton

https://community.logos.com/discussion/comment/38893#Comment_38893

I installed the program (ver 3) into the Ramdisk.

That cut the Passage search to 4.5 seconds from 5.9! (again, on a 5GB resource directory without baggage files.)

However, when I did an exegetical search, it got an error that t couldnt find a file.

This is under Win7, so ill have to figure out how to get it to let me into the the Documents and Settings section.

Which file?

Documents and Settings does not exist on Win7? If that is a setting it got from Power Tools Options | User files, then copy the "Documents and Settings" folder to your Win 7 Users directory at \Users\<you>\Documents\Libronix DLS and leave the path empty (default).

Stuart A Weber

https://community.logos.com/discussion/comment/41439#Comment_41439

my system is disassembled as I am reconfiguring. Dave, youre correct. I meant the Users directory- not Documents and Settings.

The ram was buggy at 12gb- Corsair's online configurator says this 6GB kit will work with my motherboard-but tech support says you cant use 2 kits at that speed. The configurator knew my motherboard supports 24GB, but it doesnt say you cant expand when using certain highspeed kits. Oh well. I returned with no restock fee becasue of that.

So I am picking up a kit that has all 6 2GB sticks in it (much cheaper, but slower ram.)

I also dumped the 4 80GB SSD because the TRIM feature is not active in RAID, so I would get degredation.

Dave- IS the cache in the Programs directory for V3/V4(which you can change the install drive), or in some settings directory (which cant be moved?) SOMETHING sped up considerably when installing the V3 program to ramdisk.

Dave Hooton

https://community.logos.com/discussion/comment/48801#Comment_48801

Dave- IS the cache in the Programs directory for V3/V4(which you can change the install drive), or in some settings directory (which cant be moved?) SOMETHING sped up considerably when installing the V3 program to ramdisk.

The index is embedded with each resource file in v3. Providing faster access to resources will improve search speed. Install the program to default location on HDD and keep resources in RAM (use Option for Resource Paths). The cache created during resource Discovery cannot be moved.unless you create an NTFS Junction (might be worth experimenting with this in Win 7).

Stuart A Weber

https://community.logos.com/discussion/comment/48989#Comment_48989

so its the cache thats installed in the Program Files\Libronix subdirectory then, yes? See my timings where installing V3 to the Ramdisk shaved 1.5 sc of the search. Maybe not much- but if you see what ive been doing, is seeing what effect each aspect had.

When I get back up and running, I want to see what happens when the program is installed to ramdisk and the resources are not, to see if just speeding the cache is more important than the resource files. I have to wonder that because you see the little improvement from having the resources at about 500mb/s and over 4000mb/s.

Stuart A Weber

https://community.logos.com/discussion/comment/49057#Comment_49057

so regarding V4, where are the cache and indexes in that?

Dave Hooton

https://community.logos.com/discussion/comment/49061#Comment_49061

... the standard installation path is:-

XP -> C:\Documents and Settings\<user>\Local Settings\Application Data\Logos4

Vista/Win7 -> C:\Users\<user>\AppData\Local\Logos4

Resources: \Logos4\Data\{random}\ResourceManager\Resources

Index: \Logos4\Data\{random}\LibraryIndex

Make sure you can view hidden files and folders in Windows Explorer!

Stuart A Weber

https://community.logos.com/discussion/comment/53798#Comment_53798

so yo do have to have the Resources in the program directory now?