Wild card searches very slow

I know that wildcard searches are slow - but they seem to be even slower in 4.2a B7. See screenshot below for an example:

as per the attached log, it seems to spend a lot of time writing temporary files - has it always done this?

8306.Logos4.log

Anyone else seeing a slowdown here?

Graham

Find more posts tagged with

Comments

Fred Chapman

Anyone else seeing a slowdown here?

I had not tried anything until I read this thread. It just so happens a few weeks ago I did a wildcard search for the Words of Christ in the ESV. The search took just under 5 minutes. I did the same search this morning and it took just a little over 17 minutes, so it does appear to have slowed down significantly.

Kevin Becker

https://community.logos.com/discussion/comment/222226#Comment_222226

I wonder if one of the indexing bugs quashed by B7 was the speed of wildcard searches [;)]

Fred Chapman

https://community.logos.com/discussion/comment/222230#Comment_222230

I wonder if one of the indexing bugs quashed by B7 was the speed of wildcard searches

Well thank goodness that bug is gone[:^)]

Graham Criddle

https://community.logos.com/discussion/comment/222234#Comment_222234

Can I vote for it to come back??[:)]

Rosie Perera

I know that wildcard searches are slow - but they seem to be even slower in 4.2a B7.

Anyone else seeing a slowdown here?

I just tried the same search you depicted in your image above and it was excessively slow (169.39 sec) even on my ultra-fast PC. And indeed, most of the time (2 minutes) was spent writing all those temp files.

I can't tell if it's slower than before, as I've not done this kind of wildcard search recently since I got my new computer. I'd been so focused on how happy I was that other wildcard searches (e.g., for * in Highlighted text) have gotten so much faster (relatively speaking, like 12 minutes instead of over an hour) since I got my new
blazingly fast PC.

Melissa Snyder

I know that wildcard searches are slow - but they seem to be even slower in 4.2a B7.
Graham

I've submitted your report and log to development.

Graham Criddle

https://community.logos.com/discussion/comment/222246#Comment_222246

Thanks Melissa

Rosie Perera

https://community.logos.com/discussion/comment/222252#Comment_222252

I just did a Basic Search for "the Lord of *" in my Entire Library on my super blazing fast PC (see benchmarks) and it took 3399.65 seconds (nearly an hour)!!!

Log file attached:6170.Logos4.zip

Writing the temporary files definitely took most of the time. If a user has gobs of RAM available, as I do, Logos should be using memory for that temporary stuff, not writing to the hard disk!

Secondarily, the SyncManager also seems to have taken extraordinarily long (about 20 minutes, from 08:33 to 08:53 at the end of the log file is mostly Syncing).

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/222269#Comment_222269

We reduced the maximum number of file handles that can be open concurrently in order to avoid out of memory exceptions. (Each memory mapped file uses up to 64K of virtual address space (that's the allocation granularity of Windows), so there's a fairly tight limit on the number that can be created at once.) This is probably causing less processing to be done in memory, and more done on disk, which will slow things down.

It would have been too difficult in this release to allow the search engine to push the program to the limits of memory, yet scale back if other panels in the application needed more memory for work you're doing in the foreground, so we erred on the side of caution and not crashing the application. We did not have a goal of optimising searches for "*" in this release. (But non-* searches should have become faster.)

Rosie Perera

https://community.logos.com/discussion/comment/222377#Comment_222377

We did not have a goal of optimising searches for "*" in this release.

But you'd just done such great work optimizing "*" searches in a previous release and now you've essentially undone that work by taking this trade-off. Every programming problem to be solved has constraints and trade-offs. I understand that. I don't presume to understand the overall design of your use of memory, but it seems to me that using hard disk space as work area for data crunching is something you should leave up to the operating system. That's what virtual memory is, after all, isn't it? If the user has enough RAM, those operations will take place in actual memory and Logos shouldn't have to think about it.

I had the Temp folder open while all this was going on and noticed that though the log file lists dozens if not hundreds of temp files that get created in the search process, only about six of them were ever open at one time. I don't care how many are open at once. That isn't the issue. It's the hitting of the hard disk all those times in total that's the issue (and I've got a relatively fast HD at 7200 rpm; but that's still way slower than using memory).

I wish it didn't feel like each release is two steps forward in one area of the program balanced by two steps backward in another. I wish you could maintain the performance improvements of the past while still pushing forward for more.

All-in-all, though, I still appreciate all the work you've been doing on it, and I continue to be impressed with Logos as a product every day. But come on, please. Wildcard searches are so useful and important. Please keep them a priority.

Mark Barnes

https://community.logos.com/discussion/comment/222269#Comment_222269

Secondarily, the SyncManager also seems to have taken extraordinarily long (about 20 minutes, from 08:33 to 08:53 at the end of the log file is mostly Syncing).

This is probably not relevant. Syncing happens in a separate thread, and therefore is in essence a background process.

Rosie Perera

https://community.logos.com/discussion/comment/222384#Comment_222384

Secondarily, the SyncManager also seems to have taken extraordinarily long (about 20 minutes, from 08:33 to 08:53 at the end of the log file is mostly Syncing).

This is probably not relevant. Syncing happens in a separate thread, and therefore is in essence a background process.

But if the timestamps are accurate, this part of it is happening after everything else is done, so it's still slowing down the whole operation.

Mark Barnes

https://community.logos.com/discussion/comment/222386#Comment_222386

It's much more likely that whatever process is taking the time is not being logged properly. If you notice in the logs, not all events are timed. Those that are have two entries - the first is when the procedure starts and contains the word (timed), the second is when the procedure ends and contains the time taken in milli-seconds. But not all procedures are timed, and therefore it's not possible to see how long they're taking (nor are all procedures logged, of course).

Graham Criddle

https://community.logos.com/discussion/comment/222377#Comment_222377

We did not have a goal of optimising searches for "*" in this release

Hi Bradley

I understand it wasn't a goal to optimise "*" searches in this release and appreciate the desire/need to reduce out-of-memory crashes but share Rosie's concern (above) about losing the improvements that you had delivered in this area.

My quoted example shows 434 seconds to find every word in John 11:35 which has to be much too slow!

If there is anything which can be done to reintroduce some of the performance improvements it would be appreciated.

Thanks again for all your hard work

Graham

PS: Anyone know why there are 5 results in the verse which just has two words?

Rosie Perera

https://community.logos.com/discussion/comment/222456#Comment_222456

PS: Anyone know why there are 5 results in the verse which just has two words?

Hmm, maybe the * is matching some internal stuff that doesn't show on screen? Perhaps the Greek interlinear text (there are 3 words in Greek in that verse which would add up to a total of 5). I bet that's it. If I search for just ἐδάκρυσεν, it finds it and highlights the corresponding English word:

Graham Criddle

https://community.logos.com/discussion/comment/222461#Comment_222461

Perhaps the Greek interlinear text (there are 3 words in Greek in that verse which would add up to a total of 5)

Thanks Rosie

I think you're right - if I restrict the search to "Surface Text" I just get the two results.

Graham

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/222238#Comment_222238

Can I vote for it to come back??

I have some ideas for minimising memory usage like 4.2a while still retaining the performance of 4.2; it's probably a little too late in this beta cycle to implement them without running the risk of destabilsing the build, though.

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/222383#Comment_222383

Wildcard searches are so useful and important.

Can you give some examples of useful and important searches so I can better understand the various use cases of searching for "*"?

Graham Criddle

https://community.logos.com/discussion/comment/223215#Comment_223215

I have some ideas for minimising memory usage like 4.2a while still retaining the performance of 4.2; it's probably a little too late in this beta cycle to implement them without running the risk of destabilsing the build, though.

Hi Bradley

Excellent - thanks for that.

Graham

Kevin Becker

https://community.logos.com/discussion/comment/223216#Comment_223216

Wildcard searches are so useful and important.

Can you give some examples of useful and important searches so I can better understand the various use cases of searching for "*"?

finding all examples of a highlighting style
Similarly it would be nice be able to use a wildcard search to recolor links to a particular datatype
Finding all the examples of verses in a passage list to make a visual filter that highlights those verses with a visual filter
Using the Analysis to construct an exhaustive word list i.e. lemma:*

Dave Hooton

https://community.logos.com/discussion/comment/223226#Comment_223226

Can you give some examples of useful and important searches so I can better understand the various use cases of searching for "*"?

Finding occurrences of a word root eg. xyz* or lemma:xyz* (usually English or Greek words).

Mark Barnes

https://community.logos.com/discussion/comment/223216#Comment_223216

Can you give some examples of useful and important searches so I can better understand the various use cases of searching for "*"?

Here are some examples that have come up in the forums just in the last week or so.

Returning a list of all text highlighted in a particular style.
Returning a list/count of all the Greek words in a particular verse range.
Returning a list of all occurrences of a particular field, e.g.:
- wordsofchrist:*
- largettext:*

Rosie Perera

https://community.logos.com/discussion/comment/223226#Comment_223226

Wildcard searches are so useful and important.

Can you give some examples of useful and important searches so I can better understand the various use cases of searching for "*"?

This is the main one I want to be able to use it for:

finding all examples of a highlighting style

But yes, all those others ones that Kevin, Dave, and Mark mentioned.

Robert Pavich

https://community.logos.com/discussion/comment/223403#Comment_223403

I'm not a brainiac like you all are but I'd also vote to optimize these particular searches.

Todd Phillips

https://community.logos.com/discussion/comment/223216#Comment_223216

Wildcard searches are so useful and important.

Can you give some examples of useful and important searches so I can better understand the various use cases of searching for "*"?

Most of these searches don't need to be a "wildcard" search (when I think wildcard, I think pattern matching and regular expressions). If we had a search that returned all matching words and/or verses that met the other search conditions, then wildcard-based pattern matching wouldn't need in most cases.

Mark Barnes

https://community.logos.com/discussion/comment/223923#Comment_223923

Most of these searches don't need to be a "wildcard" search (when I think wildcard, I think pattern matching and regular expressions). If we had a search that returned all matching words and/or verses that met the other search conditions, then wildcard-based pattern matching wouldn't need in most cases.

I agree, and have wondered if there could be a new keyword that returned all words/verses.

Rosie Perera

https://community.logos.com/discussion/comment/223938#Comment_223938

Most of these searches don't need to be a "wildcard" search (when I think wildcard, I think pattern matching and regular expressions). If we had a search that returned all matching words and/or verses that met the other search conditions, then wildcard-based pattern matching wouldn't need in most cases.

I agree, and have wondered if there could be a new keyword that returned all words/verses.

Couldn't they simply detect that the wildcard is the only thing in the search string and return all words/verses that are highlighted or are in Word of Christ or whatever other criteria are selected?

Dave Hooton

https://community.logos.com/discussion/comment/223923#Comment_223923

If we had a search that returned all matching words and/or verses that met the other search conditions, then wildcard-based pattern matching wouldn't need in most cases.

That would assume that wildcard searching (pattern matching) doesn't work that way? It would seem that it lacks optimisation rather than needing a special operator.

David Sloan

https://community.logos.com/discussion/comment/224016#Comment_224016

I very often do wildcard searches when I am looking for parallels between two texts. For example, I think Luke 14:1-25 deliberately parallels Luke 13:10-35 with the two Sabbath healings leading into two discussions of the spread of the kingdom to outsiders while insiders miss the eschatological banquet followed by calls to take up the cross. But I want to see at a lexical level what connections there are, so I do a wildcard analysis search on the passages of interest and then sort by lemma. There seems to be a chiastic structure to the Lukan travel narrative but it is a little difficult to quantify. I do these kind of searches all the time to see where parallels can really be seen at a lexical level. But each search takes an hour or so on my computer (and with a lengthy chiasm this becomes very tedious!). I could almost sort through all lemmas with a paper and pencil in that amount of time (though I'm glad I don't have to be looking at my screen for the entire hour). This is just one example of how I use wildcard searches; there are others. So to me this is an important issue that is WAY to slow in Logos 4.

Dave Hooton

https://community.logos.com/discussion/comment/323449#Comment_323449

So to me this is an important issue that is WAY to slow in Logos 4.

I wonder if Uservoice has a suggestion for wildcard searches which you can vote for --> enter "wildcard" at Logos User Voice

Keep Smiling 4 Jesus :)

https://community.logos.com/discussion/comment/323449#Comment_323449

I very often do wildcard searches when I am looking for parallels between two texts. For example, I think Luke 14:1-25 deliberately parallels Luke 13:10-35 ... But I want to see at a lexical level what connections there are, so I do a wildcard analysis search on the passages of interest and then sort by lemma.

Wonder about Morph Search analysis for parallel passages grouped by lemma ?

Morph Search for all morphologically tagged words takes a few seconds vis many minutes for wildcard search:

@B OR @C OR @D OR @I OR @J OR @N OR @P OR @R OR @T OR @V OR @X

Keep Smiling [:)]

Keep Smiling 4 Jesus :)

https://community.logos.com/discussion/comment/323537#Comment_323537

So to me this is an important issue that is WAY to slow in Logos 4.

I wonder if Uservoice has a suggestion for wildcard searches which you can vote for --> enter "wildcard" at Logos User Voice

Logos User Voice suggestion => implement the ability to search by colors not just within colors has 1 vote, suspect search by color would need fast wildcard search capabilities.

Searching Logos User Voice suggestions for "search" finds many results, including => Stem/Cognate Search that includes: construct a search with a wildcard.

Keep Smiling [:)]