Beta feedback wanted: Smart Search Synopsis
Smart Synopsis is a new feature in early development that provides an AI-generated synopsis of the top several results from Smart Search. While it is very much a work in progress, we’d love to get your early feedback on the quality of results.
How to test
- Acquire a license. You can use an active Logos Pro subscription, or receive a free temporary license by joining the group Logos Pro (early access) Beta Features.
- Go to app.logos.com.
- Open the Search panel, either to All Search (which includes your books and unlicensed books), or Books Search with Smart Search enabled.
Some questions to consider (in order of increasing subjectivity):
- Does the synopsis include citations?
- Does each cited article seem relevant or appropriate to the query?
- Is the Synopsis's paraphrase a good summary of the cited article?
- Does the Synopsis answer the user query?
- How might a human familiar with the subject have answered this question differently?
Please use the thumbs up/ thumbs down buttons on any particularly good or bad responses. And if you do offer comments on this thread as feedback, it would also be helpful to include the query and response.
As usual, we’d also be interested in bug reports (slow response times, bad formatting, unaligned UI, etc.), but our main goal at this point to gather feedback on the quality of the generated responses.
Thank you very much!
Comments
-
In a rush. I'm a roman catholic Verbum (german) user.
Question in german: "Was ist "Dispensionalismus"?" generates a little helpful answer:
Dispensationalism is a theological belief system that interprets biblical prophecy in the light of premillennial dispensationalism. It gained prominence through the work of Cyrus Scofield, who was known for defending interpretations of biblical prophecy within this framework 1.1Larry K. Eskridge, „Scofield, Cyrus Ingerson“, Religion in Geschichte und Gegenwart (Tübingen: Mohr Siebeck, 2004) 1080–1081.Same question in english: "What is "dispensionalism"?" generates:Chapter 26 of the second article discusses the different types of human character, including the way of humanity as animal in nature, the way of humanity in the U.S. version, and the way of what humanity might have been as a true social being 1.1Jon Huer, Labor Avoidance: The Origins of Inhumanity (Lanham, MD: Hamilton Books, 2015).I still don't understand what this topic includes.0 -
Hi Adam
This looks very interesting - so I tried it with the search string is the creation account to be taken literally?
And I tried it in both an All and a Books Search (see below)
In both cases:
- the synopsis did include citations
- in most cases the cited article seemed relevant (although the creation account: a review article cited - 2nd in the Books search - is slightly strange as it is not the article itself but a review of it)
- And the synopsis does seem a reasonable summary of the article - and it seems fine to present conflicting views (see again the books result)
- The synopsis does make an attempt at answering the query but it is more about listing short summaries than critiquing conflicting positions
- I think a human might have been more likely to present a range of different views than was shown here
I was surprised by a couple of things:
- Why - in both cases - are search results 1,2 and 5 used but not 3 & 4?
- Why, in the Books Search, is the search result from Conspectus above that for the Genealogical Adam & Eve yet doesn't appear in the All Search results?
Thanks, Graham
0 -
I tried "emergent church" as a term I genuinely had only a very vague idea of. What I got was an evaluation that assumes a theological stance that is not mine. I had to go the wikipedia to determine what the term actually means and how off-the-rails this synopsis is. This answer is a good way to insult many of your users. However, I had gotten reasonable synopses on several searches before I hit this one.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Question in german: "Was ist "Dispensionalismus"?" generates a little helpful answer:
Thank you! I should have mentioned, we have not localized the AI prompt; we are happy for you to test in the meantime, but it's likely to always give back English results for now, even in German UI.
0 -
I tried anaphora thinking liturgy, forgetting that it also has a rhetorical meaning. When I specified "in liturgy" it gave me a good synopsis. However, without that qualification, despite the fact that the first 2 search results cover the 2 meanings, only the rhetorical meaning is covered in the synopsis. The ambiguity should be noted.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Adam, I think it is interesting as well. I did notice though that when I tried an All Search for “how many wives did Jacob have” without a question mark, I had no citations in the synopsis; when I tried an All Search for “how many wives did Jacob have?” with a question mark, I did have citations in the synopsis.
Above all these things, walk in love, which is the bond of perfection. - Colossians 3:14
0 -
I was surprised by a couple of things:
- Why - in both cases - are search results 1,2 and 5 used but not 3 & 4?
There is always a degree of speculation when it comes to"Why" questions for AI responses, but these might be helpful:
- When there are multiple search results from the same resource, we are only sending it the first one. That is, the AI receives the search result (plus some context) for the first five resources, not always the first five articles. (That doesn't appear to affect your queries, but may apply to others.)
- The AI may not have thought that results 3 and 4 were relevant to the question (if the query is a question, as in this case).
Why, in the Books Search, is the search result from Conspectus above that for the Genealogical Adam & Eve yet doesn't appear in the All Search results?
Hm... you've got me there.
0 -
when I tried an All Search for “how many wives did Jacob have” without a question mark, I had no citations in the synopsis
Very interesting! We've noticed that some queries don't get a citation, and we have yet to establish a pattern. I tested with a few other with/without question mark combinations, and I found one other example of missing citations. Good find!
0 -
I tried "emergent church" as a term I genuinely had only a very vague idea of. What I got was an evaluation that assumes a theological stance that is not mine. I had to go the wikipedia to determine what the term actually means and how off-the-rails this synopsis is.
Thanks, MJ. This is a good example of a couple of possible shortcomings.
1. Short, one- or two-word queries are still a bit of a wildcard. Most of our testing has been around "questions," but the vast majority of user searches are one word.
2. The synopsis can only be as good as the top-ranked search results from our catalogue. This, of course, would be both the strength and the weakness of a feature like this.
However, I had gotten reasonable synopses on several searches before I hit this one.
The question is, that good enough to be useful? What would have to change for Synopsis to delightful?
0 -
Most of our testing has been around "questions,"
Unfortunately, I find when I use questions I get poor results - the search picks up the wrong elements to focus on.
The question is, that good enough to be useful? What would have to change for Synopsis to delightful?
I really don't know enough to give an answer. It appears that the term has two meanings - one among evangelicals and another among other denominations.
Wikipedia says:
[quote]The emerging church, sometimes wrongly equated with the "emergent movement" or "emergent conversation", is a Christian movement of the late 20th and early 21st century. Emerging churches can be found around the globe, predominantly in North America, Western Europe, Australia, New Zealand, and Africa. Members come from a number of Christian traditions. Some attend local independent churches or house churches while others worship in traditional Christian denominations. The emerging church favors the use of simple story and narrative. Members of the movement often place a high value on good works or social activism, including missional living.[4] Proponents of the movement believe it transcends labels such as "conservative" and "liberal"; it is sometimes called a "conversation" to emphasize its developing and decentralized nature, its range of standpoints, and commitment to dialogue. Participants seek to live their faith in what they believe to be a "postmodern" society. Disillusionment with the organized and institutional church has led participants to support the deconstruction of modern Christian worship and evangelism, and the nature of modern Christian community.
Which seems the same as:
[quote]
emergent church movement A loosely connected movement of the late 20th century in the United States that reflected disaffection with traditional Protestanet ecclesiastical practices. Emphasis is on following the way of Jesus and Christian community, on dialogue and generosity, and on engaging postmodern culture.
Donald K. McKim, The Westminster Dictionary of Theological Terms, Second Edition, Revised and Expanded (Louisville, KY: Westminster John Knox Press, 2014), 101.
While there is more of an evangelical emphasis in
[quote]
As advocates of a modern Reformation, these elders of a movement sometimes called “the emergent church” see the hope of evangelical Christianity resting in a deconstruction of modern, Enlightenment methods that they see both radical liberalism and fundamentalism. Traditional evangelicals (1950–1975), they say, embraced the modern worldview with its affinity towards evidentialism, foundationalism, and rationalism. Meanwhile “pragmatic evangelicals” who followed (1975–2000) overreacted by abandoning much of the historic substance of faith for an ahistorical, consumer-friendly, market-driven and therapeutic form. Led by mega-church pastors, popular authors, and a band of well-intentioned Christian brothers seeking to re-invent Christianity in a new, relevant form the result was a de-historicized Christianity lacking substance, appealing to its consumers by imitating culture rather than creating it.17The new, emerging church (2000–) led by the younger evangelicals involves a re-commitment to the depth and substance of historic Christianity, without the fundamentalism or liberalism that characterized much of the church of the twentieth century. Within these churches one is apt to see a focus on the Trinity and God’s transcendence (“it’s not about me”), a fondness of mystery (“don’t pretend you have all the answers”), a desire for honesty (“don’t make church a show”), space for lament (“I need to know God hears my cries”), the language of story (“don’t give me more principles … show me how I fit into God’s story”), and a craving for community (“size isn’t proof of God’s presence; love is”).
17 Webber, The Younger Evangelicals, 23–42.Michael P. V. Barrett, The Beauty of Holiness: A Guide to Biblical Worship (Greenville, South Carolina; Belfast, Northern Ireland: Ambassador International, 2006), 237–238.
While historical roots seem important here:
[quote]
We must be careful at this point to describe the Emergent Church accurately. They are not suggesting that Christianity has no positive historical roots, or that the form of Christianity needed to address the postmodern culture must be constructed de novo. In fact certain leaders are returning to the ‘ancient faith and practices’ for Christian examples which pre-date modernity. Thus, some writers are deeply interested in the history of the Church Fathers and Medieval Christianity.14 It is difficult to argue against the premise that the study of history has played a crucial role in the intellectual formation of the Emergent Church.
14 Robert Webber, Ancient-Future Faith: Rethinking Evangelicalism for a Postmodern World (Grand Rapids: Baker Books, 1999)
Jeffrey K. Jue, “What’s Emerging in the Church? Postmodernity, the Emergent Church, and the Reformation,” Themelios 31, no. 2 (2006): 23.
And this makes it sound evangelical ... which, in general, lacks deep historical roots:
[quote]
The term “emergent church” refers to a new movement among evangelical Christians that builds on a deep engagement with postmodern culture through appreciation of key components in a postmodern outlook. The emergent church is usually identified with Brian McLaren, author of A Generous Orthodoxy and identified as one of the top twenty-five leading evangelicals by Time magazine. Other emergent church leaders include Leonard Sweet, Robert Webber, Sally Morgenthaler, and Thomas Hohstadt. The movement receives its best intellectual support from Dallas Willard and the late Stan Grenz.
Robert Webber has pictured the emergent movement as a third alternative to traditional evangelicals and pragmatic evangelicals. The former term refers to an older evangelical paradigm that places emphasis on apologetics and sound doctrine. By “pragmatic,” Webber means the seeker-sensitive paradigm, connected most famously with Bill Hybels and Rick Warren. Emergent leaders believe that the older models of evangelical Christianity fail either because of an outdated epistemology or because of a capitulation to a consumer view of religion.
James A. Beverley, Religions A-Z (Nashville, TN: T. Nelson, 2005), 76.What I can say is that:
- No reference book entry that uses evaluative language should be allowed in the top references
- A reference book entry that provides some historical references is a candidate for top references
- A reference book entry that distinguishes the concept from closely related references is a candidate for top reference.
- A reference book should get a boost for a cross-section of theological positions for authors and editors.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
This reaches a new level of non-information. Call stories of prophets went well
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
No citations; most of the results ignore the noncanonical requirement.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
no citations
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Odd as a single citation appears 3 times in a row. BTW Michael E. Stone is the major name in Armenian apocryphal literature.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
no citations - I can force one every 4 or 5 tries on "obscure" data but have not quite found the pattern.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
I am not sure a quote from a book review #3, is the best source for this search. The book review is used as support for most of the conclusions arrived at by the synopsis.
0 -
There are no footnotes in this synopsis showing the references used.
I am only posting the problem searches. Many of my test searches proved very successful.
0 -
In short, I am quite favorably impressed. Adding a ? does help get citations.
I really like what this is doing, but don't quite trust the results. I find they are sometimes weighted by iffy Resources. Could we rank those better? Maybe according to our personal library rankings?
Not a synopsis issue, but I do wish that I had the option to sort unlocked hits to be at the top.0 -
It seems to have a hard time understanding my question
0 -
Adam, thanks so much for all the work you are putting into this upgrade and for keeping us informed about it.
After upgrading to Logos Max (early access), it looks like I may have been knocked out of the Smart Synopsis Beta. Not a major problem, but I’m curious if that might be the case.
Above all these things, walk in love, which is the bond of perfection. - Colossians 3:14
0 -
title wrap around issues with references; could the data issue of all caps for the Schaff title be repaired by the data people?
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
no references
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Adam, thanks so much for all the work you are putting into this upgrade and for keeping us informed about it.
After upgrading to Logos Max (early access), it looks like I may have been knocked out of the Smart Synopsis Beta. Not a major problem, but I’m curious if that might be the case.
An oversight on my part! Thanks for letting me know; you and other Max subscribers should now have access.
0 -
title wrap around issues with references ...
Thanks, we noticed this today as well.
could the data issue of all caps for the Schaff title be repaired by the data people?
Seems reasonable; I'll check with them.
0 -
We've made some behind-the-scenes changes to this feature, which we hope will improve (sometimes significantly) the quality of these synopses.
Compared with the earlier answer MJ posted:
0 -
It's also designed to cope better when the search results aren't great (e.g., see the second example below).
0 -
We've made some behind-the-scenes changes to this feature, which we hope will improve (sometimes significantly) the quality of these synopses.
Yes, there is a distinct improvement in several of my test cases. In one, however, there is a statement that is flat-out wrong (highlighted below). However, it was true at the time the source book was written (1878). Unfortunately, there is no reasonable way to warn the user.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Yes, there is a distinct improvement in several of my test cases. In one, however, there is a statement that is flat-out wrong (highlighted below). However, it was true at the time the source book was written (1878). Unfortunately, there is no reasonable way to warn the user.
That's a really helpful example, thanks.
We could tune the synopsis to give its own answers, irrespective of the published books, which might cause it to give a more up-to-date answer here. But 99% of the time, I think users would prefer information sourced from their own books – even if that is out-of-date or even wrong.
0 -
I have a Logos Pro subscription but do not see this section as I perform a mirror search as Adam.
0 -
Can I just say as someone who is a Logos user first and an employee second (and a lifelong Bible software geek) that this impresses me no end. Nearly every source for St. Maximus the Confessor has his birthplace in Constantinople. It is a small minority of later Syriac sources that place his birthplace in Palestine. I'm very impressed that the Smart Search Synopsis could get both of these.
Senior Publisher Relations Specialist
Logos Bible Software0
