Serious limitations in Logos’ AI Search: It won't consult full collections unless explicitly invoked

EDUARDO JIMENEZ · 2025-07-24T02:34:32+00:00

There was an error rendering this rich post.

Watched the latest live with Mark Barnes, but this question wasn’t addressed.

After thoroughly testing Logos' AI search capabilities, I’ve come across a structural limitation that deserves open discussion:

The AI assistant does not consult all resources within a collection unless those resources are explicitly mentioned.

Here’s what I mean: if I ask a theological, exegetical, or topical question and expect the assistant to draw from all my commentaries on, say, the Gospel of Matthew, the results are inconsistent. Even when I have a clearly defined collection, the AI does not seem to access the whole group unless I name individual resources in my prompt.

This is problematic for several reasons:

It breaks the basic expectation of semantic or "intelligent" search. One would assume that if a collection is referenced (or implicitly relevant to the question), the AI would automatically draw from all the resources it includes.
It undermines trust in the completeness of the results. It’s not reasonable to require users to write, “Comment on this according to the Word Biblical Commentary, NICNT, Dale Bruner’s Matthew, etc.” just to get substantial output. If we have collections, they should matter.
There’s no clear indication of what the AI is or isn't consulting. We’re left guessing which books were accessed, and there’s no transparent logging or citation system confirming sources unless they’re mentioned explicitly.

If the assistant only works with specific titles named in the prompt, this is not true AI-powered search — it’s a glorified indexer. I believe this needs serious improvement, especially for those of us with large libraries who rely on collection-based workflows.

Have others noticed this? Any workarounds you’ve found? Or is this just a current limitation by design?

Find more posts tagged with

Performance

Features

Comments

Graham Criddle

I'm afraid I don't understand the issue you are describing.

Specifying a series in a Smart search will constrain the search to that series and we should get back the most relevant results from that series. And we will get more results the further we go down the search results.

Please provide a bit more details on what you are seeing and the concerns it raises.

MJ. Smith

What you are describing is exactly the behavior I expect of AI - AI has no intention of being comprehensive - it considers as many resources as it believes it needs to answer your question, document its answer, and provide you the number of references it believes you expect. If you want comprehensive answers, you must use a precise search. Precise is programmed for completeness; AI is emphatically not. A precise search will reproduce its results when you run it a second time; AI may or may not as it is tracking

EDUARDO JIMENEZ

Thank you for your perspective. I agree that AI is not meant to be exhaustive in the same way a precise search is. However, if AI is marketed as a tool that “understands your library” and “draws from your resources,” then users are right to expect a meaningful level of comprehensiveness — especially when working within defined collections.

The issue is not that AI isn't precise, but that it ignores entire resource sets unless prompted by name, even when they are thematically relevant. That’s not intelligent curation — that’s a blind spot. If AI selectively draws from a subset without transparency, it undermines trust in its usefulness for serious study.

Defining a collection is precisely a way to give structure and boundaries to AI-driven queries. If the assistant fails to recognize that collection unless each title is named, then the collection mechanism becomes irrelevant within the AI context — which contradicts its intended purpose.

In short: not being exhaustive is one thing; being unaware of relevant material is another.

EDUARDO JIMENEZ

I must admit that since my last experience there have been some improvements. I just hope that soon the results can be categorized by resources as well.

Graham Criddle

Defining a collection is precisely a way to give structure and boundaries to AI-driven queries. If the assistant fails to recognize that collection unless each title is named, then the collection mechanism becomes irrelevant within the AI context — which contradicts its intended purpose.

I'm sorry but I'm still not understanding the issue you are seeing - could you clarify please?

MJ. Smith

@EDUARDO JIMENEZ

In short:
not being exhaustive is one thing; being unaware of relevant material is another
.

It is aware only of such material as it needs to believe it has a satisfactory answer. It is always unaware of relevant material - in fact it does not reliably identify relevant material.

However, if AI is marketed as a tool that “understands your library” and “draws from your resources,” then users are right to expect a meaningful level of comprehensiveness

I don't even bother to read such claims. Instead, I read the logic driving the AI including how to trims branches to save time by deciding it has explored as far as is fruitful. AI understands nothing - it is very sophisticated pattern recognition and probability calculations. And AI may draw from your resources without quotation or reference … your resources fed into th process which produced the results - effecting the paths taken and therefore the results.

Bradley Grainger (Logos)

We’re left guessing which books were accessed

It's not shown in the UI, but it's covered in the documentation: https://support.logos.com/hc/en-us/articles/35181728416397-How-Logos-uses-AI

For Smart Synopsis, Logos does something more sophisticated: it takes your search query and top five search results, sends those results to the AI model along with your original query, and asks the model to extract and combine the most relevant portions into a coherent synopsis.

The synopsis isn't drawn exhaustively from the entire collection you specified; it's from the top five search results. You can set a collection to limit what might appear in the top five results.

The AI assistant does not consult all resources within a collection unless those resources are explicitly mentioned.

It's not an AI assistant. It's AI-enhanced Smart Search and Smart Synopsis.

It doesn't take instructions. You can't tell it what books to search in in the prompt. (If that appears to work, it's a coincidence.)

It's a search engine. Use the "Search" textbox to type a Google-style query, not a ChatGPT-style prompt. It's not a chatbot.

Bradley Grainger (Logos)

Having AI summarize a collection of resources (on a topic) would be a really useful feature and one we're considering investigating for a future release.

But Smart Search isn't currently that feature (and trying to use it as though it were will give disappointing results).

EDUARDO JIMENEZ

MJ. Smith: "AI understands nothing…" (Sorry, I couldn’t find the magic button everyone else seems to have for quoting)😁

——-

I see. So now AI is Cinderella. It is misunderstood. It is overworked. It is expected to do just enough before the clock runs out. 😉

And speaking of misunderstandings (remember I rely on some tools to help with my English), maybe things got sidetracked at some point. All I’m really saying is this: a collection is simply a filter. It’s a way of saying “look only in these resources.” But instead of that, I kept getting the classic: “Not enough resources were provided for a synopsis...”

BTW, I already mentioned this: in recent searches, I’ve noticed some improvements in Smart Search.

That's why I moved away from smart searches and started using ChatGPT or Perplexity as a complement.

EDUARDO JIMENEZ

https://community.logos.com/discussion/comment/1375034#Comment_1375034

Thanks, Graham. I’ll try to put together a clearer example later — I might not have explained it well. Appreciate your patience.

EDUARDO JIMENEZ

Hi Bradley!

Thanks for clarifying, That last note is helpful. I’ll stop treating it like something it’s not. Looking forward to that future feature if it moves forward!

Jack Caviness

https://community.logos.com/discussion/comment/1375076#Comment_1375076

That would be a very useful feature. Too bad I completed the survey before I read this 🙄