Why is the Study Assistant not able to answer this question?

Recently I've been thinking a lot about how to ask good questions and, as a result, wanted to create an exhaustive list of every question Jesus asked. I thought that this would be a great use of the Study Assistant but the response from the AI search was as follows:

The provided search results don’t directly address a comprehensive summary of all questions Jesus asked. However, the results do offer significant insight into Jesus’s questioning method and include numerous specific examples.

I was disappointed that the Study Assistant was unable to do such a search. Perhaps I was not doing it correctly. Is there a way to search for a list like this using Study Assistant? Are there plans in the future for it to be able to answer this sort of question and not just provide a summary?

Find more posts tagged with

Features

Desktop App

Bible Search

Accepted answers

All comments

Don Awalt

Not sure what question you asked, but out of curiosity I took a shot at it - and I got a somewhat strange answer:

Screenshot 2025-12-23 at 12.43.04 PM.png

Yasmin Stephen

I would've done a Smart Bible search for this one.

NichtnurBibelleser

@Bruce Dunning Did you see this example for the use of Study Assistant?

https://community.logos.com/discussion/comment/1389379#Comment_1389379

Marconi Fabio Vieira

Hy @NichtnurBibelleser, thank you for highlighting this and for bringing others into the conversation, @Bruce Dunning. The idea has always been to share practical examples that help the community better explore the potential of the Study Assistant—especially for those who want to go a bit deeper in their study. I’m glad if this can serve as a starting point for others to test, adapt, and discover new paths.

Bruce Dunning

https://community.logos.com/discussion/comment/1389471#Comment_1389471

I realize that I could do the search another way. My observation and question was around the Study Assistant. I just thought it would be able to handle a question like this.

Bruce Dunning

https://community.logos.com/discussion/comment/1389472#Comment_1389472

No, I have not been following this thread but am going to do so now. Thank you.

Bruce Dunning

I just asked the following question to Claude "Create a comprehensive list of every question Jesus asked in all four Gospels in list format." and it did an okay job, but it was not a complete list either. So I asked it again for a complete numbered list and it provided a longer list but still not complete.

I do wish that the Study Assistant would be able to do this.

John Simpson

It would help if Logos AI could translate natural language searches into Logos search syntax. This precise search answers the question "create an exhaustive list of every question Jesus asked" giving 494 results in 316 verses for the CSB.

speaker:Jesus INTERSECTS question:*

Chris G. Hutchings

Is the success of Study Assistant results really dependent on “tip-toeing” around question phrasing? In these early days, simple wording might help, but I'd bet Faithlife's LLMs are quickly evolving, like other leading LLMS, to handle imperfect or complex queries with excellent results.

MJ. Smith

The basic way AI works makes it very poor at providing exhaustive information. Neither the way the data is stored nor the way it determines a search is complete support exhaustive lists naturally. The best way to come close to exhaustive results is to ask it to provide 10, then ask for another 10, and another …when the answers stop being reasonable you likely have a complete list.

NichtnurBibelleser

https://community.logos.com/discussion/comment/1389486#Comment_1389486

The German Einheitsübersetzung yields 547 Treffer in 316 Versen

Nestle-Aland 28: Novum Testamentum Graece 402 Treffer in 316 Versen

The Greek New Testament: SBL Edition 396 Treffer in 316 Versen

Chris G. Hutchings

I posed the question to Grok-4, and it quickly spit out the questions in the attached document.

Jesus Gospel Questions.pdf

Bradley Grainger (Logos)

Why is the Study Assistant not able to answer this question?

This itself is a great question and there are multiple levels at which we can approach it.

The Study Assistant uses LLM technology. As MJ has already stated (and as you found out when asking Claude), the current state of this technology doesn't perform well at generating exhaustive lists of several hundred items. The first issue is that the output is probabilistic and may be likely to include most of the items (but not all), and the second issue is that these models have a maximum number of "output tokens" (which is sometimes as small as 8,192 tokens although newer models have much higher limits). The model will stop generating text when it's reached the maximum number of output tokens, which may be before it reached the end of the list.

A workaround for this is for the code using the model (i.e., Study Assistant) to recognize this limitation and effectively tell the model "ok, give me the next chunk", "ok, the next chunk", etc. However, this runs the risk of pushing the model to generate too much text, so it might start repeating itself or creating items that shouldn't be in the list. (That's also just code that hasn't been written in Study Assistant yet.)

Or we could take the perspective that LLMs aren't good at creating these exhaustive lists, but Logos datasets are, so instead we should "teach" SA how to generate Logos search syntax and open a search panel with the results. This is clearly a useful idea and a direction we'll probably take SA in the future. If you ask "why" it doesn't do this right now, the answer is that SA just launched, so it doesn't have every feature we could possibly imagine (because they haven't been built yet).

The second "why" is that actually building this will be complex. Logos has hundreds of different biblical datasets and the LLMs have little-to-no intrinsic knowledge of them. So, we have to provide the LLM with information about each user's unlocked datasets and enough information to generate a valid search query. This will fill up the model's "context window" which will (a) cost more, and (b) possibly degrade the quality of results for queries that don't need this information. We want to keep the costs reasonable (so asking a simple question doesn't use up too many AI credits) and not make other queries perform worse, so we'll need to be careful how we approach this.

So maybe SA should just recognize when you're asking a question directly answerable by a book in our catalogue and offer to sell you the title 😀 https://www.logos.com/product/1171/all-the-questions-in-the-bible .

The final "why" might be that this question is open to interpretation and there might not be one right answer. (And this is perhaps a place where an LLM could hypothetically generate a better answer than querying a dataset because it could note this nuance in the answer?) For example, compare https://biblia.com/bible/niv/john/1/50 and https://biblia.com/bible/nasb95/john/1/50 , or https://biblia.com/bible/nkjv/matthew/26/50 and https://biblia.com/bible/ncv/matthew/26/50 . This is not a problem that would prevent us from delivering such a feature but just a reminder that no matter where you get a list of results, it's always good to go back to the primary texts and validate the assumptions that were used in building any such list.

Bradley Grainger (Logos)

https://community.logos.com/discussion/comment/1389498#Comment_1389498

It seems to have missed most of the questions in Matthew 7-8: Matt 7:4, Matt 7:9-10, Matt 7:16, Matt 8:26.

I'm sure it missed many more.

Chris G. Hutchings

https://community.logos.com/discussion/comment/1389502#Comment_1389502

Absolutely! It just reconfirms my longstanding AI position: “Trust, but Verify!” I've challenged AI responses too many times to count, only for AI to apologize. Thanks for the informative response!

Lew Worthington

Thank you, @Bradley Grainger (Logos) for the detailed explanation!!

Dr. Nathan Parker

The Merged Gospels would be another good source book that could be used to train Study Assistant. I recently finished reading it.

DAL

I just typed in the Smart Search Questions Jesus asked and it gave me a bunch of verses with some not being questions Jesus asked.