AI search is a joke – Exhibit 2

https://community.logos.com/discussion/245371/ai-search-is-a-joke-exhibit-2

True. It would be nice if we were able to utilize a bit more of the typical chatbot functionality in telling Logos AI how to search through our library to find results. For example, asking, "I know I have a book dealing with the "I AM" statements of Christ. Can you search through the titles of the books in my library to help me find it?" Instead, the how seems to be more or less predetermined regardless of the question. So if the question doesn't fit in well with how Logos AI is programmed to answer, you may well receive unsatisfactory results.

Not the answer you want, and I do understand your justified frustration, but to present the flip side: I know too much about AI and about Logos/Verbum to be surprised. I consistently get results that are sensible for what not-AI-but-popular-pseudo-AI built with if-then logic and no evaluative judgement of quality (okay, I slightly overstate) and Logos/Verbum general prioritization independent of the user and without fore-knowledge of the answer decides are the most relevant terms/results. I am slowly getting better at guessing the right hint(s) to head both halves in better directions.

It did, however, fail to note my favorite Orthodox resource on the topic which I attribute to it not being carried by Logos. 🙄

Christopher Randall

You appear to be coming at this with a preconceived answer and wanting that answer without considering how AI is coming at it. If this is true, you are going to be disappointed quite often because it will never meet your expectations. I would agree that I wish the AI would search based on our prioritization of resources, but that being said it still provides an answer based on its data. If you know you have a resource that you want considered, you can include that in your question.

Chill my friend @Christopher Randall. Just go back and re-read my post carefully before replying.

It is exactly because of answers like this that I am not on any social media whatsoever in my life. I don't want to waste my time pointing out obvious problems with replies. Unfortunately, since I am not not smart enough, I have to be on this one as I need to ask questions on this Forum to learn to use the software.

Now, I have not prioritized any resources. I don't know where you got this idea.

I also didn't go searching for when AI is screwing up. I wanted to see what books are there in my library that can help me with I AM statement. And I happened to know one book. For other searches, I may not have any idea what book is there in my library that will have the answers I am looking for.

I have no idea how AI search works. All I know is what any search should do at the minimum. When a book has "I AM statement" in the title and my query has "I AM statement", I would hope it should return this book at the top of the search. Instead, it doesn't even list it in the top 50.

I am no Logos hater. In fact, I love Logos just as much as others in this Forum, but it is perfectly fine to point out where it is bad. In fact, AI search is pretty bad.

When a book has "I AM statement" in the title and my query has "I AM statement", I would hope it should return this book at the top of the search.

I would agree with you. But it's possible the AI is searching through the content of the books moreso than the titles.

In fact, AI search is pretty bad.

There are indeed questions that the AI search fails to answer well. There are other questions that it succeeds in answering well (or at least in a way that is helpful to me). I look forward to Logos AI being improved over time.

EDUARDO JIMENEZ

In a movie I heard Benicio del Toro say this line: "Don't ask me how the clock works, ask me the time." Honestly, that's exactly how I feel when I can't clearly express what I'm looking for in Logos, and worse, when the AI responds with something irrelevant or completely ignores certain resources - even when they are explicitly mentioned in a query.

I've started to rely more on ChatGPT for some searches, which I admit has been more efficient in some cases. To "help" Logos a bit, I've also been filtering my searches by collections. While it improves the results slightly, it feels like I'm compensating for the system's shortcomings.

Meanwhile, we humbly come here to listen to a patient and kind team… and occasionally, to endure them!

DAL

I’ve noticed responses vary from mobile to desktop searches also. And yes, some results are not really using books we own.

https://community.logos.com/discussion/245371/ai-search-is-a-joke-exhibit-2

@EDUARDO JIMENEZ

In a movie I heard Benicio del Toro say this line:
"Don't ask me how the clock works, ask me the time."

Whereas my cattleman father in his 60's became interested in having a desktop computer (DOS and floppies day) ordered the components for a computer so he could build his own and therefore, understand what to expect. He went on to become the county computer nerd, retiring from going into the schools to resolve the student problems in his late 80's.

Bob Venem

Perhaps another way to look at it: Precise Search seeks what you asked for, while Smart Search explains what you asked for. (FWIW, I prefer Precise searches; Smart search results are too bland and shallow for my taste.)

Mark Allison

I'm curious about the summary that it provided. If it was accurate and informative, then perhaps the AI felt the contents of the book were redundant. I don't believe the AI is claiming to give you a comprehensive list of the resources in your library that have application to your search question. Or maybe the AI just doesn't like Warren Wiersbe.

https://community.logos.com/discussion/comment/1353740#Comment_1353740

@Bob Venem. Nice nuance to explain difference between Precise and AI search.

In my situation, I couldn't have asked for Precise search because I was looking for all resources in my library that had something to say about I AM statements. It just so happened that I knew of one book in my library whose entire (100%) content was related to I AM statements. And AI search didn't even pick it up. That is the annoying part. Imagine, if I didn't know about Warren Wiersbe's I AM book in my library. Since AI didn't direct me to this book, I would have been wasting my time on other books. I thought they hyped up AI search as one that will help users maximize the content in their library. It is just hype sometimes.

https://community.logos.com/discussion/comment/1353741#Comment_1353741

Here you go my dear: @Mark Allison

The concept of "I AM" statements in religious and philosophical contexts carries significant weight. In biblical texts, such statements are often associated with divine declarations, as exemplified by Jesus' assertion "Before Abraham was, I am". (1) This type of statement is seen as a claim to divinity or absolute truth. In Christian theology, "I AM" statements are also used to affirm a believer's identity in Christ, with some sources providing lists of such affirmations for Christians to embrace. (2) However, the use and interpretation of "I AM" statements can be contentious. Some argue that claiming absolute knowledge or truth through such statements is problematic, as it implies that those who disagree are wrong. (3) In broader religious history, particularly in the context of ancient Israel, similar absolute statements about God's nature and uniqueness appear, such as "I am the first and I am the last; besides me there is no God" found in Isaiah. (4) These declarations often served to assert the supremacy of one deity over others, especially during times of cultural and religious conflict. (4)

1Fred G. Zaspel, The Theology of B. B. Warfield: A Systematic Summary (Wheaton, IL: Crossway, 2010), 263.

2Stan Toler, Rethink Your Life: A Unique Diet to Renew Your Mind (Indianapolis, IN: Wesleyan Publishing House, 2008), 78.

3Cornelius Van Til, The Protestant Doctrine of Scripture (The Presbyterian and Reformed Publishing Company: Phillipsburg, NJ, 1967), 82.

4Carsten Colpe et al., “God,” in The Encyclopedia of Christianity (Grand Rapids, MI; Leiden, Netherlands: Wm. B. Eerdmans; Brill, 1999–2003) 427.

https://community.logos.com/discussion/comment/1353730#Comment_1353730

I'd think (judging from the experiences so far), that the AI is a 'discovery' type feature, while precise is find-it-or-give-up. Meaning, AI's not-find isn't to be relied on. 'However', (hasn't been explained yet otherwise), Mark (Logos) said the actual search is traditional; the AI is picking and choosing the 'best' results.

Not fully related, though similar, but I notice on Logos.com, if you don't put 'title:xxx', the results returned may start out with an exact title match ('may'), but then devolve into who-knows. The 'older' Logos.com search was more 'precise'. I also notice, even with a title search, Logos.com dumps 2025 libraries in, that don't have the title. It's tiresome.

EDUARDO JIMENEZ

Dear MJ,

Thank you for for sharing the inspiring story about your father. I deeply admire that level of curiosity and determination.

I'm practical... but let's not get carried away!😉 I just wanted to point out that tools like Logos can sometimes be more complicated than necessary. While I absolutely value understanding how things work, I also believe that tools should strive to be intuitive so that we can spend more time learning and less time overcoming technical hurdles.

Christopher Randall

https://community.logos.com/discussion/comment/1353716#Comment_1353716

Sorry, I was not trying to come across negatively. Personally, I think you need to chill. You are the one stating AI is not taking into consideration what you see as meaningful and that it is a "joke." I'm just trying to state that we cannot expect it to give us the results from resources we want. Like what Aaron said, "it's possible the AI is searching through the content of the books moreso than the titles."

https://community.logos.com/discussion/comment/1353721#Comment_1353721

Or it is like what MJ is implying with the statement: "I consistently get results that are sensible for what not-AI-but-popular-pseudo-AI built with if-then logic and no evaluative judgement of quality (okay, I slightly overstate) and Logos/Verbum general prioritization independent of the user and without fore-knowledge of the answer decides are the most relevant terms/results."

https://community.logos.com/discussion/comment/1353700#Comment_1353700

All I am saying is, you may know that you have a resource that has the information you want. AI is not going to know what you are looking for without you being more specific in your query. It is going to answer based on its own prioritization of resources or its own algorithm or whatever its processes are that determine which resource it considers bests, whatever that may be. That "prioritization" or whatever we want to call it is not going to be what we necessarily want at any given moment.

When the answer comes back not the way we expect it, we have to remember it is just a tool that is still being developed. But we also have to realize sometimes we need to ask the question differently.

When the answer comes back not the way we expect it, we have to remember it is just a tool that is still being developed. But we also have to realize sometimes we need to ask the question differently.

@Christopher Randall asking the question differently is definitely the way to go (especially short-term). Even as AI is improved, we will have to learn and adapt to maximize its usefulness. That said, it would be great long-term if the way questions were asked would more effectively affect the AI's prioritization choices.

I get the feeling the answer to AI on Logos is 'ok, whatever':

It's in development for whatever.
It didn't like the question' for whatever.
It does something else we don't know about; whatever.

Sounds like worth subscribing to.

https://community.logos.com/discussion/comment/1353789#Comment_1353789

whatever.

NichtnurBibelleser

I just have been curious.

Typing the bold in the search field:

“I Am” statements

yields a more reasonable answer than this text:

“I AM” statements

Mark Allison

https://community.logos.com/discussion/comment/1353789#Comment_1353789

I use the AI features in Logos multiple times a day and it's brilliantly useful for me. When I'm forced to use the competitor's software, it feels a bit crippled now.

https://community.logos.com/discussion/comment/1353793#Comment_1353793

Brilliantly is in the eye of the (AI) beholder! Maybe they designed it for you!!

Yasmin Stephen

https://community.logos.com/discussion/comment/1353807#Comment_1353807

True. Just like not everybody gets the joke.

https://community.logos.com/discussion/comment/1353781#Comment_1353781

Sorry @Christopher Randall , I am not wasting my time responding to you. It will take too much of my time and it will go nowhere, so no one gains and I lose, so I'll pass.

@Aaron Hamilton. I understand it is a a tool, but if you want to call a tool a hammer, it better be able to hammer in a nail. Otherwise, call it something else. Don't call it AI search, when it fails some basic criteria for what it should return.

Dear techie friends! Is there some objective criteria that can be employed to judge the quality of AI search output? That will settle some dust.

How do you bold the search phrase?

Are you saying I should try "I Am" instead of "I AM"? Sorry, could you please clarify?

Good for you. Your experience reminded me of a situation when our friends from Washington DC visited us, they found home prices in our neighborhood to be cheap compared to Washington DC. But that is because they are making a relative comparison. So I understand that, as a Accordance user, you are happy. A functioning Logos software should have made you happy. But I believe in an objective absolute criteria, not some arbitrary relative criteria to judge whether a product is good or not.

https://community.logos.com/discussion/comment/1353823#Comment_1353823

Please understand that the comments regarding Logos AI being a tool were not my own.

Mark Allison

Yes, I'm finding AI search less than helpful. I was hoping for some clarification here.

https://community.logos.com/discussion/comment/1353838#Comment_1353838

I think we may have here a case of not brilliant. Scholars may demur.

EastTN

https://community.logos.com/discussion/comment/1353789#Comment_1353789

I might frame it slightly differently. FaithLife is pitching the new AI-based features as a great leap forward that's going to transform users' Bible study. But at least some users are finding that they aren't as intuitive and easy to use as the sales materials may have led them to believe. That just reaffirms my belief that they've prematurely doubled down on the genrative AI technology.

https://community.logos.com/discussion/comment/1353807#Comment_1353807

Knowing you are non-subscription (until you tell me otherwise), I assure you that Logos went for a very useful, very conservative implementation that ,were it not for all the misleading hype about LLM/Chatbots at this time, would have Logosians very excited. The problem is, and it is a problem that I take seriously, that people's expectations are way off base in a variety of directions. It is similar to comparing Google to Logos and finding Logos wanting because it is less capable that Google for some questions - while ignoring all the questions Logos handles easily that Google can't handle. I think of it as similar to learning when to use an encyclopedia, dictionary, atlas, or almanac in the dead tree era. But managing expectations is a skill I admire rather than have.

@1Cor10 31

Don't call it AI search, when it fails some basic criteria for what it should return.

When it was first released in beta at least, it was promoted as an AI-enhanced search not as an AI-search.

Is there some objective criteria that can be employed to judge the quality of AI search output?

There exist such criteria on several levels but I both lack access and equipment. First, there are standard tests for AI in general intended to judge if "intelligence" has been reached. Industry insiders say "yes" or "very nearly"; outside researchers are much more conservative. Second, there are standard tests built for specific implementation to check that a system is working within the accuracy required by the specifications. Where you went "wrong" from the perspective of the Logos search is assuming that the title of the book has a heavier weighing than the content of the book. I suspect, but do not know, that the title is not given extraordinary weight and that the search believed it had collected sufficient data to answer the question before it reached the book. Did the book never appear in the list or was it merely not in the top 50? If the former, you have discovered a bug; if the latter, that is simply how the search works.

https://community.logos.com/discussion/comment/1353879#Comment_1353879

I understand. To be 'frank', Logos really can't 'not' do what is called 'AI'. Large libraries and excited customers. And chances are good they were careful. The problem (as with many Logos features) is that they've not explained in sufficient usability detail.

ICo's questions have merit. I'm still confused as to exactly what an AI search is (in Logos). Is it a legacy search with AI icing (selecting the best)? Or something else? Where are the summaries coming from (eg what's being used as a formation basis … writings or neural summarizing?). Almost a white paper is needed.

But doubtful. Even Bob in the early days had trouble explaining the translation graphs as to what exactly was being measured.

https://community.logos.com/discussion/comment/1353825#Comment_1353825

@DMB

I'm still confused as to exactly what an AI search is (in Logos).

That surprises me - what more than this do you need?

Mark Barnes described the workings of the smart search as “In Smart Search, we perform a very loose search without AI. That finds 50 articles that roughly match your query. We then use AI to identify a snippet from each article that most closely matches your query. We don't rewrite any text; we only identify the most relevant existing text in the book. Once we have these snippets, we use AI to re-order them to put the most relevant ones at the top.”1
1New feature: Smart Search - Logos Forums accessed 17 September 2024

NichtnurBibelleser

Indeed, I tried only to underline the different writing in the quotation marks.

The first was exactly this:

“I Am” statements

The second was this:

“I AM” statements

https://community.logos.com/discussion/comment/1353969#Comment_1353969

I hope you're not serious! Smiling. "We perform a very loose search … ". Buried in a post months ago.

Knowing your background, (1) that's VERY sloppy language for 'documentation', and (2) that's not documentation. But an excellent example of my point. I'd really hate to be a Logos 'user'. You're probably more comfortable from beta'ing it, and exploring the process.

https://community.logos.com/discussion/comment/1353969#Comment_1353969

What exactly does "loose search" mean? @DMB would also like to know this, I think, based on his comment above. How is it related to "Precise Search" of Logos 10 and prior? Is is possible to give an example so layman can understand.

https://community.logos.com/discussion/comment/1353879#Comment_1353879

@MJ. Smith said: "people's expectations are way off base in a variety of directions. It is similar to comparing Google to Logos and finding Logos wanting because it is less capable that Google for some questions."

Maybe true for others, but not for me. I am asking: what is the MINIMUM expectation we should have for Logos AI search? I am not asking for the MAXIMUM expectation (which is what Google/ChatGPT will provide but with hallucination). When Logos isn't even picking a book where the title matches my search phrase, you know it is not satisfying the minimum.

https://community.logos.com/discussion/comment/1353887#Comment_1353887

@MJ. Smith said: I suspect, but do not know, that the title is not given extraordinary weight and that the search believed it had collected sufficient data to answer the question before it reached the book. Did the book never appear in the list or was it merely not in the top 50? If the former, you have discovered a bug; if the latter, that is simply how the search works.

I went and redid the search. The book didn't appear in the top 100. I didn't want to spend more time scrolling. THe output includes stuff from Christianity magazine, which has ZERO chance of having more content than the Warren Wiersbe's book. See Table of Contents for this book below:

Preface

Moses Asks a Question
The Apostle John Provides Some Answers
The Bread of Life
The Light of the World
The Door
The Good Shepherd
The Resurrection and the Life
The Way, the Truth, and the Life
The True Vine
The Neglected I AM
“I Am Jesus” (Acts 9:5; 22:8; 26:15)
Living and Serving in the Present Tense

https://community.logos.com/discussion/comment/1353983#Comment_1353983

Thank you. Yes, I tried it. The synopsis is different, but the first 15 references were the same. Some small consolation.

Christopher Randall

You searched, "'I AM' statement" (or something equivalent).

Based on what @MJ. Smith quoted, "In Smart Search, we perform a very loose search without AI. That finds 50 articles that roughly match your query" (I missed this quote, so thank you for posting it to help explain how it works)

Logos software found books that related to that query without the assistance of AI. This is a 'loose' search… I would take this to mean, that it is a general search that captures resources containing the phrases 'I AM' and 'statement' without much sophistication in understanding their theological significance - in other words just looking at the words. It would be nice for Logos to clarify this process so we can understand what we are using.

Once those books are found, continuing with the quote, "We then use AI to identify a snippet from each article that most closely matches your query." If this is true, your issue is not with AI, but with how Logos is performing the search by attaining the resources from your library. If this is just a general (loose) search, it is likely not taking anything into consideration except for the words that you are searching. Therefore, it is not going to know what is meaningful or most helpful, it is just going to know that those resources contain those words. Again, be nice for clarification from Logos.

If this is true, then how we word our questions matters greatly, and sometimes this means we will need to ask it in multiple ways to get the most benefit from the smart search.

@1Cor10 31

When Logos isn't even picking a book where the title matches my search phrase, you know it is not satisfying the minimum.

Unfortunately, this is an excellent example of "people's expectations are way off base in a variety of directions.". If you knew more about the inner workings of AI, you would not expect a title match to have much effect on the selection process - a little yes, but not a lot. If you want the title to have a great deal of effect on the selection process, you would use a precise search or a library search. AI is all about statistics, especially Bayesian statistics, and chains of if-then statements (computational logic). It has nothing to do with understanding the meaning.

The output includes stuff from Christianity magazine, which has ZERO chance of having more content than the Warren Wiersbe's book.

"More" is not a operative word here. Terms like TF-IDF (Term Frequency-Inverse Document Frequency) or Cosine Similarity or Latent Dirichlet Allocation (LDA) are representative of the terms used to identify topics. The fact that Warren Wiersbe is known more as a pastor than as an academic and the Baptists are only 5-7% of Christians world wide do not help the likelihood that he will appear near the top. Question: does he appear in the top 100 of a smart book search?

To me, this topic is a piece of the fundamental problem I have with Logos - it encourages people to use data they don't understand. My prime examples are usually treating case: nominative, subject: and agent: as if they are equivalent. Or using case without considering syntactic force …

Warren Wiersbe's book is not in the top 100.

All the fancy statistical techniques seem useless in some circumstances, I guess, to be able to identify the right book.

What we seem to be concluding is to not expect much (or in my case, forego a financial fee). It's an interesting 'discovery' type tool.

I am surprised they can't do data weighting. That's pretty basic to neurals for years.

https://community.logos.com/discussion/comment/1354125#Comment_1354125

Yes, as with all statistical techniques, they are useful in particular cases not all cases.