AI search is a joke – Exhibit 3

1Cor10 31
1Cor10 31 Member Posts: 791 ✭✭✭

All you Logos fanboys and fangirls, please take a break…this AI search is a helluva joke played on its customers. They have the gall to charge us for this.

I had this quote from Charles Spurgeon: "Little things are always swift in anger; great things are not so." I wanted to get the context. I know I have lots and lots of Spurgeon material, so I asked Logos AI search. Plase see below for the query and the output.

Next I went to Perplexity (thank you MJ for introducing this to me) . I asked a slightly different query because I wasn't so lazy with my Logos search query. See below the query and the output:

Perplexity tells me that the sentence is found in a sermon on Nahum 1:3. I know I have lots of sermons, so I type in Nahum 1:3 in the Passage guide and sure enough, I see the sermon titled "Mercy, Omnipotence, and Justice" in my Library.

So I went back to Logos AI search and gave the exact same query that I gave to Perplexity. See the terrible output it gives.

This is a joke.

Please…please don't call this AI search, please call this A-unI search (stands for artificially unIntelligent search), or, even better, call it AS search (i'll allow you the liberty of choosing a word starting in "S" that is reflective of this product)

I am a big fan of Logos but I am not a big fan of being scammed. I invest in stocks and now every firm has to mention that they are incorporating AI into their product. Proof is in the pudding. Where is the pudding?

yes, I am beyond frustrated.

I believe in a Win-Win-Win God.

Tagged:
«1

Comments

  • MJ. Smith
    MJ. Smith MVP Posts: 54,480
    edited February 13

    When you are looking for a precise sequence words, AI is not the appropriate tool. A precise search works very nicely. AI doesn't know texts - it knows networks. Wikipedia unfortunately has a technical article not real world - is https://vitalflux.com/deep-neural-network-examples-from-real-life/ understandable to you or is there too much mathematical jargon?

    or does this answer from perplexity add clarity?

    The difference between searching texts and searching a neural network built from texts lies in the fundamental approach to information retrieval and processing. These two methods represent distinct paradigms in natural language processing and information retrieval.

    Text Search

    Text search, also known as full-text search, involves directly querying a collection of documents or texts to find specific words, phrases, or patterns. This method typically uses techniques such as:

    Inverted indexing: Creating an index that maps words to their locations in the documents

    Boolean operators: Using AND, OR, NOT to combine search terms

    Relevance ranking: Scoring documents based on term frequency and other factors

    Text search is generally faster for exact matches and can be more precise when looking for specific information. However, it has limitations in understanding context, semantics, and relationships between concepts.

    Neural Network Search

    Searching a neural network built from texts, on the other hand, involves querying a model that has been trained on a large corpus of text data. This approach leverages the power of machine learning and natural language processing to understand and generate human-like responses. Key aspects include:

    Semantic understanding: The ability to grasp the meaning and context of words and phrases

    Latent space representation: Encoding text into high-dimensional vectors that capture semantic relationships

    Transfer learning: Applying knowledge gained from training on a large corpus to specific tasks or domains

    Neural network search can handle more complex queries, understand context, and even generate new text based on the input. It excels at tasks like question answering, summarization, and inferring relationships between concepts.

    Key Differences

    Representation: Text search works with raw text, while neural networks transform text into complex numerical representations.

    Flexibility: Neural networks can handle more varied and nuanced queries, while text search is more rigid and literal.

    Contextual understanding: Neural networks can grasp context and semantics, whereas text search is limited to pattern matching.

    Computational requirements: Text search is generally faster and requires less computational power, while neural networks are more resource-intensive.

    Accuracy vs. Generalization: Text search is more accurate for finding exact matches, while neural networks excel at generalizing and understanding implicit relationships.

    Updating: Text search databases can be easily updated by adding or removing documents, while neural networks typically require retraining to incorporate new information.

    In practice, many modern systems combine both approaches, leveraging the strengths of each method to provide more comprehensive and intelligent search capabilities.

    Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."

  • Christopher Randall
    Christopher Randall Member Posts: 92 ✭✭✭

    I will admit, I did a search on a topic that I knew which book answered. I did a search under “books” with all books being searched and the AI synopsis said none of the articles talked about it. I narrowed the search to that specific the book, and it had the answer.

    All I was wanting was to pull up resources on that topic. It was disappointing. The sources for the original source really had nothing related to the query.

  • MJ. Smith
    MJ. Smith MVP Posts: 54,480

    I agree that misunderstandings as to what AI does and how to use it are very common among users - myself included. The results are that people are very disappointed with the results - not understanding that AI was not the correct tool for what they wanted. There are also people who are overly enthusiastic, because they discovered new, interesting information. Unfortunately, they don't realize that some of this new information is either false or unverified. Until most users understand that the AI does not deal with texts but with mathematical constructs based on many texts and the AI does not attempt exhaustive searches but only searches to the point of a reasonable answer, most users will abuse AI. My hope is that many of the kids currently in junior colleges, high school, grade schools will learn the appropriate use of AI … and when they become the majority of average users . . .

    Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."

  • Graham Criddle
    Graham Criddle MVP Posts: 33,005

    I had this quote from Charles Spurgeon: "Little things are always swift in anger; great things are not so." I wanted to get the context.

    If you have an exact quote, it is probably best to use a Precise search

  • ASUNDER
    ASUNDER Member Posts: 128 ✭✭

    I'm no fan of secular AI, but I like this Logos AI being used with Christian books.

    Even if the Logos AI model is half as good as the latest secular tech, Logos is still worth the money without any AI.

  • DMB
    DMB Member Posts: 14,057 ✭✭✭✭

    Asunder, you're comments (like many of mine) are hard to graph out. A BUT B. EVEN IF THEN C.

    "If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.

  • P Or
    P Or Member Posts: 9 ✭✭

    I like Logos AI because it can access my library.

    For sure, there is room to improve for Logos AI. I need to often rewrite my prompt to get the answer that I need.

  • DAL
    DAL Member Posts: 10,811 ✭✭✭

    Isn’t “Helluva” an inappropriate expression?

  • DMB
    DMB Member Posts: 14,057 ✭✭✭✭

    Not out west. Besides, hell is supposed to be bad, bad.

    "If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.

  • DAL
    DAL Member Posts: 10,811 ✭✭✭

    So it would be ok to use it from the pulpit? 😂 I’m trying to expand my vocabulary 😂

  • Richard Dowdy
    Richard Dowdy Member Posts: 8 ✭✭

    One area of AI systems that's ripe for improvement is recognizing when a user's query needs something other than a LLM to answer.
    Ask an AI about a specific chess position, and it will tell you something that sounds plausible until you set it up on a board and realize it's nonsense.
    Ask an AI a math question, and it will give you a mathy-sounding answer that's probably all wrong.
    In the long run, the answer isn't that users need to learn the strengths and weaknesses of different machine learning systems so they know which one to ask which questions. It's that AI systems need to recognize questions that call for a more specific tool, and make use of that in answering the question. ChatGPT, for instance, is good at this when it comes to programming questions. It doesn't just spout programmer-sounding answers, it has specific capability for analyzing computer code.
    In OP's case, the AI should recognize that it's really a search question, run it through that, and provided the right answer.

  • DMB
    DMB Member Posts: 14,057 ✭✭✭✭

    It wouldn't surprise. Even in prayer. Members (even some ladies) wear western hats and boots. Cultural language.

    "If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.

  • DMB
    DMB Member Posts: 14,057 ✭✭✭✭
    edited February 13

    @richard dowdy (app didn't activate; error) Agree. On the forum, I kind of blanche at AI use 'you need to reword the query'. It means the AI is iffy, and users can't be sure what they're even getting.

    And yes, I realize Logos AI isn't intended as real AI.

    "If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.

  • DMB
    DMB Member Posts: 14,057 ✭✭✭✭
    edited February 13

    BTW you have 'only' 3 asterisks (stars). I have 4! I'd think 10k+ posts should earn you that 4th asterisk!

    "If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.

  • Francis
    Francis Member Posts: 3,830 ✭✭✭

    The main problem I see in the search represented in picture 3 is that it goes into territory that AI may have trouble identifying as smart of precise search. The Logos implementation will in certain cases switch to precise mode if the query is, well, too precise. One would argue that searching to locate a precise quote should be recognized as a precise query and switch to that mode. However, there may cases where someone may use a quote as part of a broader query that aims for a smart type of search. It looks like the current implementation is not yet capable of identifying one versus the other.

  • Lew Worthington
    Lew Worthington Member Posts: 1,608 ✭✭✭

    I remember reading where Spurgeon used its root word in a sermon. (Sorry; it's a little off topic.)

  • MJ. Smith
    MJ. Smith MVP Posts: 54,480

    You need to reword questions in all the AI bot's I've tried. The pattern of requiring rewording implies that chat-bot Reinforcement Learning from Human Feedback (RLHF) still has trouble identifying the key words … it goes off on a tangent related to a secondary word. It is usually easy to identify the mistake and reword the question in a way that is clearer to the bot … but that is not a guarantee that further down in the conversation the bot will not revert to the erroneous understanding. In a conversation on Jewish divisions of Torah text in the 1 year and 3 year cycle, I had to give up. I could not find a bot that didn't slip into confusing the two. I also have trouble keeping any bot on track when I don't want it to limit itself to the Protestant canon.

    Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."

  • DMB
    DMB Member Posts: 14,057 ✭✭✭✭
    edited February 13

    Well, that pretty much speaks to consumer-level AI, I'd assume.

    I might also point out Logos use of AI has theological assumptions. Essentially in the learning process; not simply the sources. All of my neurals are purposely fragmented, pretty much violating theology approx 170 ce and thereafter.

    "If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.

  • DAL
    DAL Member Posts: 10,811 ✭✭✭

    Maybe @Jason Stone (Logos) can clarify the number of posts needed for the 4th star ⭐️ Maybe it’s 12K instead of 10K?

    DAL

  • Jason Stone (Logos)
    Jason Stone (Logos) Administrator, Logos Employee Posts: 863

    Ranks and/or points are gifted by resolved questions (when your comment is selected as a question's "Accepted Answer") and reactions from others (particularly reactions to how helpful a member's contributions are, like "Insightful"). A general activity like comments or new posts does not gift users posts or ranks. I hope that answers your question, @DAL and @DMB.

    Sr. Community Manager at Logos.

  • DAL
    DAL Member Posts: 10,811 ✭✭✭

    Thanks @Jason Stone (Logos) That makes more sense.

    DAL

  • Randall Cue
    Randall Cue Member Posts: 680 ✭✭

    I know this is a little off topic, but how do you get a name with the @ symbol in blue hyper link in a comment?

  • DMB
    DMB Member Posts: 14,057 ✭✭✭✭

    Just start typing '@' and then the name … it'll list possibles … choose, when you see the correct name.

    "If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.

  • Jason Stone (Logos)
    Jason Stone (Logos) Administrator, Logos Employee Posts: 863

    Not that we are there, but I want to share a gentle reminder to keep from theological debate and remain focused on the use of our products. Our community rules are linked in the footer of the community webpages. Thank you all!

    Sr. Community Manager at Logos.

  • Randall Cue
    Randall Cue Member Posts: 680 ✭✭

    Thanks @DMB .

  • 1Cor10 31
    1Cor10 31 Member Posts: 791 ✭✭✭

    Thanks @MJ. Smith. Finally, I found a block of time to read the article. It went over my head. But importantly, I sort of understood the difference between AI search and precise search.

    @Graham Criddle : Thank you for showing Precise search.

    Takeaways according to me:

    (1) AI search is not Precise search

    (2) and Precise Search is better than AI search when you have a precise search string you're looking for.

    But now my beef…To the best of my knowledge, Logos has been marketing AI search as the equivalent of Google search for your library. They said, and I'm paraphrasing, "oh, you don't have to worry about search syntax anymore…just type in whatever you want and you'll get the answers." That is simply wrong. Google search seems to be a mix of AI search and Precise search if I am not mistaken. So please roll back your marketing.

    I believe in a Win-Win-Win God.

  • 1Cor10 31
    1Cor10 31 Member Posts: 791 ✭✭✭

    Tom Brady is a helluva football player (if you're an American). Just like @MJ. Smith and @Graham Criddle are helluva Forumites, who are super helpful😋

    @DAL , i would encourage you to expand your vocabulary! Paul asks us to use gentleness, but he also called out the "foolish" Galatians. If I were preaching (which will never happen), I am likely to call people who have small views of God as "foolish" people! When I hear the small views of God in my church small group, I tell them that I would rather go to Hell than spend eternity with a God who has a character that they think our God has!!

    I believe in a Win-Win-Win God.

  • Graham Criddle
    Graham Criddle MVP Posts: 33,005

    But now my beef…To the best of my knowledge, Logos has been marketing AI search as the equivalent of Google search for your library. They said, and I'm paraphrasing, "oh, you don't have to worry about search syntax anymore…just type in whatever you want and you'll get the answers." That is simply wrong. Google search seems to be a mix of AI search and Precise search if I am not mistaken. So please roll back your marketing.

    It might be relevant that if you enclose the entire string in quotation marks, Logos will automatically switch to doing a Precise search.

  • 1Cor10 31
    1Cor10 31 Member Posts: 791 ✭✭✭

    Thank you @Graham Criddle . you are right…if I put in "Little things are always swift in anger; great things are not so." in the search box, it switches to "precise" search automatically and shows just the single hit correctly,

    But…if I put in "Little things are always swift in anger; great things are not so." spurgeon, then adding that spurgeon at the end turns it into an AI search and we get garbage. So giving it more information is actually yielding worse results! Again, this is not google, right? While googling, we just put a bunch of key words and it turns up the right article. but not Logos AI.

    I believe in a Win-Win-Win God.

  • Graham Criddle
    Graham Criddle MVP Posts: 33,005

     So giving it more information is actually yielding worse results! Again, this is not google, right? 

    I agree

    I think the AI capability in Logos is really useful - but it does require some understanding of how to best use it to get the best results.

    A differently formed search does produce good results - here I am using the search string

    where did Spurgeon say "Little things are always swift in anger; great things are not so."

    So you can get the results you are looking for with a fairly intuitive search - but I'm not suggesting it is as "free form" as the corresponding Google search. It does have the benefit of linking directly to the relevant article in my library.

    I do, however, find it surprising that the synopsis does not recognise the quote - and that is something that I hope Logos can look at further.