AI search is a joke – Exhibit 2
Comments
-
Don't call it AI search, when it fails some basic criteria for what it should return.
When it was first released in beta at least, it was promoted as an AI-enhanced search not as an AI-search.
Is there some objective criteria that can be employed to judge the quality of AI search output?
There exist such criteria on several levels but I both lack access and equipment. First, there are standard tests for AI in general intended to judge if "intelligence" has been reached. Industry insiders say "yes" or "very nearly"; outside researchers are much more conservative. Second, there are standard tests built for specific implementation to check that a system is working within the accuracy required by the specifications. Where you went "wrong" from the perspective of the Logos search is assuming that the title of the book has a heavier weighing than the content of the book. I suspect, but do not know, that the title is not given extraordinary weight and that the search believed it had collected sufficient data to answer the question before it reached the book. Did the book never appear in the list or was it merely not in the top 50? If the former, you have discovered a bug; if the latter, that is simply how the search works.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
I understand. To be 'frank', Logos really can't 'not' do what is called 'AI'. Large libraries and excited customers. And chances are good they were careful. The problem (as with many Logos features) is that they've not explained in sufficient usability detail.
ICo's questions have merit. I'm still confused as to exactly what an AI search is (in Logos). Is it a legacy search with AI icing (selecting the best)? Or something else? Where are the summaries coming from (eg what's being used as a formation basis … writings or neural summarizing?). Almost a white paper is needed.
But doubtful. Even Bob in the early days had trouble explaining the translation graphs as to what exactly was being measured.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
I'm still confused as to exactly what an AI search is (in Logos).
That surprises me - what more than this do you need?
Mark Barnes described the workings of the smart search as “In Smart Search, we perform a very loose search without AI. That finds 50 articles that roughly match your query. We then use AI to identify a snippet from each article that most closely matches your query. We don't rewrite any text; we only identify the most relevant existing text in the book. Once we have these snippets, we use AI to re-order them to put the most relevant ones at the top.”1
1New feature: Smart Search - Logos Forums accessed 17 September 2024
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
2 -
Indeed, I tried only to underline the different writing in the quotation marks.
The first was exactly this:
“I Am” statements
The second was this:
“I AM” statements
0 -
I hope you're not serious! Smiling. "We perform a very loose search … ". Buried in a post months ago.
Knowing your background, (1) that's VERY sloppy language for 'documentation', and (2) that's not documentation. But an excellent example of my point. I'd really hate to be a Logos 'user'. You're probably more comfortable from beta'ing it, and exploring the process.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
What exactly does "loose search" mean? @DMB would also like to know this, I think, based on his comment above. How is it related to "Precise Search" of Logos 10 and prior? Is is possible to give an example so layman can understand.
I believe in a Win-Win-Win God.
0 -
@MJ. Smith said: "people's expectations are way off base in a variety of directions. It is similar to comparing Google to Logos and finding Logos wanting because it is less capable that Google for some questions."
Maybe true for others, but not for me. I am asking: what is the MINIMUM expectation we should have for Logos AI search? I am not asking for the MAXIMUM expectation (which is what Google/ChatGPT will provide but with hallucination). When Logos isn't even picking a book where the title matches my search phrase, you know it is not satisfying the minimum.
I believe in a Win-Win-Win God.
0 -
@MJ. Smith said: I suspect, but do not know, that the title is not given extraordinary weight and that the search believed it had collected sufficient data to answer the question before it reached the book. Did the book never appear in the list or was it merely not in the top 50? If the former, you have discovered a bug; if the latter, that is simply how the search works.
I went and redid the search. The book didn't appear in the top 100. I didn't want to spend more time scrolling. THe output includes stuff from Christianity magazine, which has ZERO chance of having more content than the Warren Wiersbe's book. See Table of Contents for this book below:
Preface
- Moses Asks a Question
- The Apostle John Provides Some Answers
- The Bread of Life
- The Light of the World
- The Door
- The Good Shepherd
- The Resurrection and the Life
- The Way, the Truth, and the Life
- The True Vine
- The Neglected I AM
- “I Am Jesus” (Acts 9:5; 22:8; 26:15)
- Living and Serving in the Present Tense
I believe in a Win-Win-Win God.
0 -
Thank you. Yes, I tried it. The synopsis is different, but the first 15 references were the same. Some small consolation.
I believe in a Win-Win-Win God.
0 -
You searched, "'I AM' statement" (or something equivalent).
Based on what @MJ. Smith quoted, "In Smart Search, we perform a very loose search without AI. That finds 50 articles that roughly match your query" (I missed this quote, so thank you for posting it to help explain how it works)Logos software found books that related to that query without the assistance of AI. This is a 'loose' search… I would take this to mean, that it is a general search that captures resources containing the phrases 'I AM' and 'statement' without much sophistication in understanding their theological significance - in other words just looking at the words. It would be nice for Logos to clarify this process so we can understand what we are using.
Once those books are found, continuing with the quote, "We then use AI to identify a snippet from each article that most closely matches your query." If this is true, your issue is not with AI, but with how Logos is performing the search by attaining the resources from your library. If this is just a general (loose) search, it is likely not taking anything into consideration except for the words that you are searching. Therefore, it is not going to know what is meaningful or most helpful, it is just going to know that those resources contain those words. Again, be nice for clarification from Logos.
If this is true, then how we word our questions matters greatly, and sometimes this means we will need to ask it in multiple ways to get the most benefit from the smart search.1 -
When Logos isn't even picking a book where the title matches my search phrase, you know it is not satisfying the minimum.
Unfortunately, this is an excellent example of "people's expectations are way off base in a variety of directions.". If you knew more about the inner workings of AI, you would not expect a title match to have much effect on the selection process - a little yes, but not a lot. If you want the title to have a great deal of effect on the selection process, you would use a precise search or a library search. AI is all about statistics, especially Bayesian statistics, and chains of if-then statements (computational logic). It has nothing to do with understanding the meaning.
The output includes stuff from Christianity magazine, which has ZERO chance of having more content than the Warren Wiersbe's book.
"More" is not a operative word here. Terms like TF-IDF (Term Frequency-Inverse Document Frequency) or Cosine Similarity or Latent Dirichlet Allocation (LDA) are representative of the terms used to identify topics. The fact that Warren Wiersbe is known more as a pastor than as an academic and the Baptists are only 5-7% of Christians world wide do not help the likelihood that he will appear near the top. Question: does he appear in the top 100 of a smart book search?
To me, this topic is a piece of the fundamental problem I have with Logos - it encourages people to use data they don't understand. My prime examples are usually treating case: nominative, subject: and agent: as if they are equivalent. Or using case without considering syntactic force …
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Warren Wiersbe's book is not in the top 100.
All the fancy statistical techniques seem useless in some circumstances, I guess, to be able to identify the right book.
I believe in a Win-Win-Win God.
0 -
What we seem to be concluding is to not expect much (or in my case, forego a financial fee). It's an interesting 'discovery' type tool.
I am surprised they can't do data weighting. That's pretty basic to neurals for years.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
Yes, as with all statistical techniques, they are useful in particular cases not all cases.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
1 -
Who said or implied they can't do data weighting - you're right it is so basic to neurals/Bayesian statistics that I didn't consider it worth mentioning.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0
