Future of Search
I do find myself wondering about the future of Search with Logos. Our libraries will continue to grow HUGE, and while doing searches based on algebraic combinations of words etc. is useful, the number of results will become so large no matter what the UI, it won't be that useful unless you know specifically what you are looking for. I wonder if Logos has any data on the library size of some typical user, and how it has grown over the years (or maybe the top 1000 users, top 5000 users, etc.)? That would be an interesting piece of data to watch.
Here are some thoughts I have, I wonder if Bob could give us the 5 year vision on search? Some big questions are out there to be pondered!
The future of search needs to provide better contextual search capability. We have limited capability to do this now via Collections, but there could for example be commentary type comments in a devotional that are of value to me and I don't know it's there.
Computers are getting fast enough that speed is not necessarily the #1 issue in the future, "relevancy" of the search results is. It does me no good today to get results in 3.68 seconds, when there are 5000 of them and I can't find any gems. Would I rather have it take 30 seconds and have it return the 6 results first that I am looking for? Absolutely!
If you read about the big guys' search engines, a lot of work is being put into this area, to analyze the text and create context for it. All the artificial categorizations via Collections should become part of the search engine, so it figures out what resources are best suited for my search. Even things like my most recent searches give an indication of what context to look for, as I am likely searching to write a sermon, write a paper, answer a specific question, etc.
Reindexing everything will become obsolete. Think about if Google had to reindex all the pages it has in its databases. I suspect we will see more effort put into some sort of updated indexing scheme. Maybe the need for merge index goes away, and there are lots of indexes that are searched and merged based on results. Maybe new resources get their indexes merged right in. But again if the number of resources really grows, how Logos does indexing now probably strains and breaks.
How we ask questions will get a lot better. Again, Searching in context will need a different way of asking questions. Getting better at asking quesitons will enable us to answer more questions, making ou investment in the resources more beneficial!
One interesting quote I found from one of the search gurus: "User-dependence on single-word search queries present a "huge barrier to advanced technologies" and called on developers and researchers to avoid the trap of giving up relevancy at the altar of increased speed."
"What are some good devotional thoughts about grieving?"; "Why do bad things happen to good people"; "What does St. Augustine say about grace" are some simplistic examples. Of course there are ways to look for these now, but there is so much stuff that gets returned that is not relevant, it's hard to find the gems. It's not unusual to have these kinds of searches returning 5000 results, which means I go to specific resources or specific collections. So why do I own all these resources then?
The user experience will improve in other ways too. Can I give an example of what I am looking for, and the search engine finds other examples of it? Can it get more sophisticated in how I drill down into the data with successive searches? Can it be blended into using other external search engines, for example could I use Google to do a search, and feed its results into Logos to create context for what I am looking for? Can I more easily create my own tags (I know we have tagging now, but can you really use it pervasively today if you have 4000 or 5000 resources? What if when users have 10,000 resources?)
Search needs to incorporate content other than text better. Technology exists today where software can scan pictures and identify people if you give it a reference photo. Video and pictures should be tagged a lot more than they are.
Just some thoughts here, and there are more issues, but you get the idea. This stuff is a ways away (5 years? 10?), but progress moves to this vision every year. Does Logos have the horsepower to do this kind of search software? Do they partner with someone, i.e. is their vision that they could partner with someone, or do they buy this technology, or do they recruit top technical minds to design this?
I failed to mention I think Logos is the leader in Bible software today, they and all of their competiitors face these issues. Logos is probably best equipped to solve these issues IMHO.
Any thoughts from Logos on the future?
Comments
-
I think your observations are correct. We are working hard to track the latest in this field, and to build new algorithms, new tagging, and new data sets that will enhance the usefulness of searching in Logos.
Search for "magnificat" in Logos 4 and you'll see several things beyond just "ranked hits."
We're also adding tagging to images, video, and even text to enhance our ability to find what you're looking for. You can even think of the Passage Guide (and other Guides) as specialized search tools -- they really just take a passage and search your library for organized, relevant resources.
In the realm of answering questions, dealing with topical inquiries, etc. we've got some very cool things coming.
To enable this, we've even been expanding our team capabilities. We don't just have great programmers, we have added great information architects and computational linguists, too. (See http://www.logos.com/press/releases/oracle-senior-application-developer-moves-to-logos-bible-software)
(As for reindexing... Google keeps the entire Internet in memory. We need to store -- and use -- our index on users' machines, so we need a tighter, disk-bound index. If we move completely to the cloud -- or if you're willing to get 64 gigs of memory and never turn off your machine! -- we can eliminate these indexing hassles. <smile>)
0 -
Thanks Bob, that is what I was hoping to hear!
And FYI, I have 32GB memory and 8 cores right now, so we are not so far away ;-). But I know what you mean, that's not mainstream. Some of my thoughts are 10 years out, maybe more. But your reply indicates a vision and hiring the right kind of people to get us there eventually.
If "the cloud": makes indexing go away from a users perspective, I think you'll win over some of the doubters. Functionality, feature, performance, and quality will be the reason that things successfully move to the cloud. Until then the debates will continue.
Thanks and keep the long term perspective! Don't let us suck you into today's bug 99% of your time ;-)
0 -
I welcome what might be possible in time, but I'd still always want Logos Bible Software to run offline for a long time yet.
I think New Zealand no longer has any ISP offering unlimited traffic plans, unless you want to accept dial-up speed caps, or $20 per GB traffic charges.
(One of the largest ISP just withdrew its last unlimited data plan.)
For mobile devices, much worse. So an iPhone etc is a nice unit maybe, but the data plans make it unworkable here due to running costs.
My point: the application should continue to operate offline, even if some features are missing unless connected.
0 -
Bob I am wondering. In the future will Logos be making changes to its software so that typos in a book, can be corrected, without downloading the entire book? A patch can be issued to address a bug without downloading the entire engine?
Mission: To serve God as He desires.
0 -
JimT said:
My point: the application should continue to operate offline, even if some features are missing unless connected.
Jim, are you speaking about the future ala Bob's post or now? (this is the current way that L4 works)
Didn't know if you knew brother....no offense.
Robert Pavich
For help go to the Wiki: http://wiki.logos.com/Table_of_Contents__
0 -
Dominick:
I don't have a 5-year plan, and you'd be advised to disbelieve it if I did: the technology changes way too fast for that. But you're absolutely correct that as the library grows, the challenge changes from "more books" to "better access". Here are a few things we're thinking about:
- One of our highest priorities is incorporating more dictionaries
into the Logos Controlled Vocabulary (that's what produces the Topics
section for basic Search). We know people want more out of 'topical search', and there's a lot of activity here. - We're working on better ways to organize search results that should give users more ability to select, filter, and sort
- Search by Example ("retrieve more like this", given a longer query than 1-2 words) is a high priority for me: i think this is the most likely way in the short term to help cope with too many results.
- We're actively working on how to put richer semantics into our book markup so that you can search in more flexible ways
Overall, our biggest limitation isn't computational horsepower (we have plenty of that ) or "top technical minds" (I hope to be one of those someday [:)]). It's just a big challenge to balance innovation and new features with all the day-to-day work (and the fact that we're still catching up on some basic features for Logos 4).
But be assured "we have top men working on it right now" (http://www.imdb.com/title/tt0082971/quotes).
0 - One of our highest priorities is incorporating more dictionaries
-
Thanks Sean, good stuff! And yes, this is not immediate need compared to other issues. Nice to hear you guys are working hard on it.
0 -
Robert Pavich said:
Jim, are you speaking about the future ala Bob's post or now? (this is the current way that L4 works)
It was a general comment, given the direction Logos appears to be moving and hinting, as I read it.
- Already our content is online, and reduced functionally if we don't have a recent sync with the cloud (thinking loss of data / backups here)
- A hint that PBB may have a significant online aspect, including maybe the processing.
- The thin clients already need online for some content and some of the power-tools (Passage Guides etc).
- Aspects of the above seem to hint towards online for power searching and high level tools.
- The "Topic" content draws from online resources / sites.
My ISP has just withdrawn its only remaining unlimited-data plan, except for dial-up speed limits, so data charges and speed is a big deal in New Zealand. I wish and need for Logos to keep its application to run in local mode and to be the best Bible Software offline, not just with a T1 line hanging off my PC.
0 -
JimT said:
It was a general comment, given the direction Logos appears to be moving and hinting, as I read it.
Jim,
that's what I was wondering ....thanks for clarifying.
Robert Pavich
For help go to the Wiki: http://wiki.logos.com/Table_of_Contents__
0 -
Disclaimer: I do work for Logos, but this is my personal view, ok!
JimT said:My ISP has just withdrawn its only remaining unlimited-data plan
I'm following the whole bandwidth issue with much interest. I'm a born & bred African. We've been struggling with really poor internet speed for ever. Lately though, things have been changing: http://www.cnn.com/2009/TECH/07/22/seacom.on/index.html
Recently, one of the biggest ISP's started offering unlimited bandwidth, at real speeds(1Mb+)... without a price increase!
I'm sad that New Zealand is going the way of the Dodo in terms of the Internet, but, if Africa can take this kind of leap(Africa has finally realized the power of it), there is hope.
0 -
Jannie van Niekerk said:
Recently, one of the biggest ISP's started offering unlimited bandwidth, at real speeds(1Mb+)... without a price increase!
I'm sad that New Zealand is going the way of the Dodo in terms of the Internet, but, if Africa can take this kind of leap(Africa has finally realized the power of it), there is hope.
Unlimited bandwidth is easy to give away when the user base is small and the speeds are too slow for voice and video. So the Africans can enjoy it now until the user base gets too big or the endpoint bandwidth speeds allow users to drain the trunk lines. Then they will have usage caps too (until the infrastructure is upgraded again).
MacBook Pro (2019), ThinkPad E540
0 -
JimT said:
It was a general comment, given the direction Logos appears to be moving and hinting, as I read it.
- Already our content is online, and reduced functionally if we don't have a recent sync with the cloud (thinking loss of data / backups here)
- A hint that PBB may have a significant online aspect, including maybe the processing.
- The thin clients already need online for some content and some of the power-tools (Passage Guides etc).
- Aspects of the above seem to hint towards online for power searching and high level tools.
- The "Topic" content draws from online resources / sites.
I don't know about PBB, and nobody knows the future. But in terms of today, you don't need to be online once your installation is established and secure (backups and updates are a different matter), except for resources that actually only live on-line. Those will continue to live on-line and will probably grow: the economics of doing so are too compelling. Furthermore, it doesn't make sense for us to try and replicate web content on every user's local computer.
But it's very unlikely that Logos 4 will ever absolutely require the Internet, and i often use it disconnected with no major loss of functionality. So i expect it will continue to be the best Bible Software offline.
0 -
JimT said:
A hint that PBB may have a significant online aspect, including maybe the processing.
Just for clarification. Logos has implied a online distribution process for PBB's which would not be a change in internet usage. The processing/compiling being potentially online was speculation from a forum user as I recall.
JimT said:The thin clients already need online for some content and some of the power-tools (Passage Guides etc).
Are you referring to the links to online resources by 3rd party vendors? The only Logos sponsored internet links I can think of are the sermons which were in L3 and the topics/reading lists which are new.
The only significant increases in internet load, as I see it are:
- software downloads (for people previously obtaining disks)
- resource downloads (for people previously obtaining disks)
- data sync'ing for which the way an individual uses Logos will strongly impact the overall internet usage - I personally will be on the high end here.
I agree completely that Logos needs to have an alternative for software and resource downloads for individuals in areas where sufficient band-width is not available. I would have to see the actual figures for "average" internet Logos 4 for all other activities to determine whether their responsibility goes beyond that.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
MJ. Smith said:JimT said:
A hint that PBB may have a significant online aspect, including maybe the processing.
Just for clarification. Logos has implied a online distribution process for PBB's which would not be a change in internet usage. The processing/compiling being potentially online was speculation from a forum user as I recall.
MJ, Please note I used "may" and "maybe". I think it was Bob that suggested PBB processing may be online.
MJ. Smith said:JimT said:The thin clients already need online for some content and some of the power-tools (Passage Guides etc).
Are you referring ...?
"Thin Clients" are iPhone, iPad, Web Browser etc. NOT the current PC or Mac "Thick" Client.
Regarding Internet access: Many customer have commented that they have limited speed, bandwidth data caps, massive surcharges for excess usage, or coverage issues with wireless connections etc. This is a real concern for some. My ISP has just withdrawn its only unlimited data plan. I think the only plans that are unlimited are dialup speeds, or fall back to such after about 20 GB per month.
0
