Video Comparison and test: Logos AI x free AI

Professor Robert Plummer, on his Daily Dose of Greek channel, posted a video from a missionary doing language study using AIs. You can watch the video for his review: https://www.youtube.com/watch?v=MrDJV77Btow

I used the exact same question he uses (on the Perplexity AI and others, as you can see on the video) on Logos (Max subscription). Since Logos is a huge library of books, courses and language resources; and having myself a huge library for language research (my research focus) together with commentaries, encyclopedias etc (thousands of books, manuscripts, studies etc), I expected Logos to provide an even better answer, or at least similar to free AIs Claude and Perplexity. To call the result disappointing is a monumental understatement!

Logos' 'synopsis' is a short answer (1 paragraph) that only points to a generic grammar point. The rest is a big list of suggestions for that grammar point on my resources. Nothing really more helpful than a simple search without an AI.

Frankly, I expected Logos to be much better in answering that, considering it has more resources. One of the AIs on the video even said that it does not have the full NT memorized, nor grammars. It just searches for it and is able to formulate the complex answer seen on the video.

I get that Logos AI is probably in its infancy, but with all the resources it supposedly has memorized (that's what AIs are about, memorizing your resources and providing answers); and with all the teaching tools etc, it should be better. And yes, you can probably get better results 'refining' the question, but that simple question "Why in the New Testament is the word for Jerusalem sometimes a feminine singular, and other times a neuter plural noun?" was enough to get a complex biblical language answer from a non-dedicated AI with no NT library memorized.

And that is really the point. In theory, Logos' AI is exactly that: a dedicated AI for biblical answers, including biblical languages and complex research tools and libraries. That was the big sales pitch for subscriptions, anyway.

When push comes to shove, you have to wonder if the "big thing" about subscriptions, the AI (being that less effective than free AIs) is worth it for research. Or anything other than simple 'preaching tools'. I know it's good for producing simple study questions, and maybe its 'vision' about being the 'lazy pastor's to-go system' is what matters for the company, but it certainly has to improve a lot for more than a regular search with a ridiculously thin 'synopsis' as the big AI thing.

Subscriptions have yet to prove more useful. Add to that worry: no more bundles (as it seems to be the case going forward), language research abandonment feelings, horrible tablet experience, and it's starting to feel like Logos is aggressively focusing on 'preaching tools with simple AI twists' and destining it to become a glorified reader for the rest of us.

Right now, BibleArc, Accordance and a free AI seem to be the future of research, while Logos is feeling like I have a gigantic library destined to be just as good as Logos 10 was (which is very good), but no more innovation for research ahead. Since subscriptions are just that, but the side effect is a downgrade in bundles and attention to research tools, it's not looking good ahead.

While I doubt an aggressive (maybe solely) focus on the 'busy pastor' will payoff on the long run, good luck on that. Honestly hope it pays off. But I'm becoming skeptical about Logos' future for research and anything else besides 'preaching tools'.

All in all, at this point, it seems like the subscription side of AI is, at best, questionable. And if it's all that is to it, then...

P.S. - I've seen people getting better results (with much better questions) on this, but that simple question is a good comparison. And even better results on Logos were much thinner than free AIs', and certainly less organized and clean looking.

Find more posts tagged with

Comments

John

Frankly, I expected Logos to be much better in answering that, considering it has more resources. One of the AIs on the video even said that it does not have the full NT memorized, nor grammars. It just searches for it and is able to formulate the complex answer seen on the video.

Logos spent many years Developing a system of advanced indexing and tagging, along with tools, which utilized those things to deliver Precise results.

Now logos is Focusing on AI systems developed by other companies to operate just on the surface Text of the resources. The advantages that logos has over its competition is the indexing in the database searching. With the focus into AI they aren’t really utilizing that except possibly in using a basic search to determine what to feed the AI engine. This eliminates logos advantage over its competition because anyone can feed surface text to an AI engine and then format the results for the end user.

Logos still has the advantage Of having the largest library. But that advantage only benefits the users who have purchased a large library. the new focus on AI Will probably encourage new users not to purchase a large library because they will get better results from a much cheaper and smaller library. In fact Logos is including what they believe is the most cost-effective minimal library with the cost of subscription. This will have the effect for the average user to have less of a need to buy more advanced resources.

Whether the subscription strategy will work to deliver big profits in the long run remains to be seen. I think Logos is smart to be the first to offer AI. But by no means should they abandon the development of the main advantage that they had up to and including Logos version 10.

Right now, BibleArc, Accordance and a free AI seem to be the future of research,

I am unable to figure out why you would believe that accordance is the future of anything. I own accordance and it crashes every time I try to use it. development at the company appears to have come to a complete stop. They have been promising to deliver on release 14 features for two years now. And they have not shown any progress whatsoever.

even if accordance was a viable company for the long-term right now, they only offer a fraction of the huge library of titles that Logos does. And the rate which they offer a new titles is very slow, to the point of being nonexistent. Their forum has hundreds of request for new or updated titles, which are many years old. if you’re not happy with what they currently offer You may as well buy it somewhere else because it will not be available anytime soon. Accordance appears to be in a death spiral.

I do Believe that you are correct that AI will be free. Logos for their subscription fee is essentially making it easy for their users to access AI By clicking on something, without having to cut and paste and then send search results to some other AI engine.

By making it more convenient for their users to use AI, many of them will be happy to use the built-in ability. And it will certainly improve over time. But logos is certainly taking a huge risk by betting the Entire farm on AI.

Mateus de Castro

https://community.logos.com/discussion/comment/1324700#Comment_1324700

Thanks for the reply. About accordance, you are right. It was an example of research focused tool, but it's probably gone. And that is a problem as far as research platforms go.

About Logos' AI, that "basic usage" is troublesome. With all the rest being relegated to "basic use", like research, that AI doesn't seem enough to justify subscription. And with less bundles and focus on other tools, I'm having a hard time justifying it.

And yes, I think free AI will be a problem for Logos' model sooner rather than later. Seen that free AI can develop tools, and even interactive apps that can be used with other systems, I'm not sure Logos' decisions to drop most everything other than "basic AI usage for cataloguing" is viable. I'm sure someone who understands systems and AI will come up with tools to interact with Logos soon. And with Logos' AI being so "basic", not sure what they were thinking.

Being hiper-specialized is the way to survive AI, not charging for "basic" AI search.

James McAdams

I can get behind Logos providing a very brief summary of other people's answers and pointing me to their work for further reading.

There is a difficult balancing act to work out how much info it should provide before that info actively takes away from my reading the most valuable works on a matter. I never want the info I get from AI to replace the info I take from authors who put the work into writing it, but I'm happy for it to prevent me from reading info that isn't relevant to me. I don't want the knowledge I obtain from my searches to be dehumanised. I worry that the more substantive answers you want from the AI search could risk doing that. There are much lesser concerns about accuracy too, but that's going to keep improving naturally as the tools develop. But actually taking work from others and removing their writing style, their personality etc and making their humanity less relevant to the work they produce seems especially troubling to me.

MJ. Smith

The use of AI in Logos and the use of AI on the free chatbots are NOT intended to be the same thing nor to serve the same purpose. The chatbot AI is still basically dumb AI - reading everyone else's works and pulling it together into a coherent whole. However, it is poor at judging the quality of the works it assimilates and it good at making incorrect assumptions as it generalizes what it has read. It both gives brilliant results and total boners. It is NOT doing an in-depth analysis of the Biblical text ... it is doing a deep dive into what others have said about their analysis. And because it can read more commentators and assimilate more data than a human could ever do, AI appears brilliant and is a tremendous helper.

Logos, however, can be used as a belief-confirmation tool - show me how all the authors I agree with, agree with me and how everyone else is wrong. However, it is intended to be a tool for doing an in-depth analysis of the Biblical text - a platform in which a person wrestles with scripture with the intent of personal transformation. If you consider the Biblical text to be a series of propositions in the modern sense of the term, without ambiguity or artistry, and with a single meaning across space and time, AI could theoretically do what Logos intends to do. If you consider the Biblical text to be God's love letter to his children, it is out of AI's league.

Logos is using AI as a simple read and spit back in a different form tool - outline, summary, questions, applications (the latter I'm not yet impressed by) i.e. using what AI is good at and minimizing the opportunities for egregious errors. It is also using it to refine the prioritization of the results of searches - something I am very pleased with except when it focused on the wrong elements of the search query. But I am also glad that I'm not trying the steer Logos at this time. What gives Logos the edge is its tagging and it is becoming more and more obvious where that tagging needs to be expanded - and that AI can continue to help Logos expand it. What is less clear is the market AI chatbots will want to delve into (think of academic.edu and scholar.google's effect on the manual indexing of scholarly journals). Or what denominational specific sites (Magisterium AI) will take of the theological market.

There are a number of innovations that Logos has yet to take advantage of, especially in the data visualization and comparison side. The question in my mind is whether those currently leading Logos have broad enough interests to take advantage of those opportunities.

David S

https://community.logos.com/discussion/comment/1324752#Comment_1324752

I tried AI, but cancelled my subscription. The results (for me) were generic and disappointing, especially given the library it has to work with. Search was by far the most powerful tool using AI, but in the end I decided that its results weren't better than standard search, just different.

ChatGPT is helpful for inspiration and organisation, with good use of prompts, but of course needs a great deal of discernment. (As does LogosAI - as has been already stated it can simply confirm ones own biases.) ChatGPT has far more flexibility - for example I was able to enter my thoughts and constraints for an all-age talk and have it turn it into an engaging, short narrative story. Not the final product, but a pretty good basis to work from, and I was in control throughout. The ideas for children's illustrations in Logos are often simply inappropriate in tone and very high level.

All this is, of course, my own use case and my own findings. I appreciate others are finding it very helpful and I look forward to seeing it evolve. Right now I'm more likely to invest in something like Perlego which has hundreds of thousands of books as part of the subscription, academic but going beyond theology, and is developing AI search to uncover potential resources. Again - in its infancy. I may also pay to create my own ChatGPTs, using my own reference documents.

Mateus de Castro

https://community.logos.com/discussion/comment/1324752#Comment_1324752

Thanks for the reply.

There are a number of innovations that Logos has yet to take advantage of, especially in the data visualization and comparison side. The question in my mind is whether those currently leading Logos have broad enough interests to take advantage of those opportunities.

That is the big question, right? If you're paying for a subscription and AI is the sales pitch for a "bible based research tool", is it enough to be a "slightly better search" for simple comparison?

Let's face it. It could be good enough:

1 - for a better price (right now, it's ridiculously overpriced for just that)

2 - if we were not watching the dismantling of the research aspect of Logos; less bundles; abandoning of tablet and cellphone capabilities; and an exaggerated focus on the 'preaching helper' side in detriment of the rest.

I mean, if I'm losing on what matters to me; and Logos is becoming a reader with 'preaching helping' capabilities; why would I pay for an expensive AI?

I hope they embrace true AI capabilities with all Logos' potential. It could be focused, faith-based, but grand for research and giving answers based on its material. Nothing less would be worth it.

Have a blessed sunday!

Mateus de Castro

https://community.logos.com/discussion/comment/1324755#Comment_1324755

I tried AI, but cancelled my subscription. The results (for me) were generic and disappointing, especially given the library it has to work with. Search was by far the most powerful tool using AI, but in the end I decided that its results weren't better than standard search, just different.

That's exactly what bothers me. With that library, how can Logos' AI be so bad and generic? On the other hand, it's quite good on 'preaching help'. And that sole focus really worries me with such a powerful tool. It seems like they are throwing away the Ferrari to stay with the Beetle when they could have both (and much more). Especially for the price of the subscription.

All this is, of course, my own use case and my own findings. I appreciate others are finding it very helpful and I look forward to seeing it evolve. Right now I'm more likely to invest in something like Perlego which has hundreds of thousands of books as part of the subscription, academic but going beyond theology, and is developing AI search to uncover potential resources. Again - in its infancy. I may also pay to create my own ChatGPTs, using my own reference documents.

Keep us posted about your developments.

Lew Worthington

https://community.logos.com/discussion/comment/1324771#Comment_1324771

The mixed quality of results I've gotten is teaching me to understand the limitations of the Logos AI and how best to leverage its virtues. As a test, I asked how proponents of the new perspective on Paul interpret Romans 7, and I got thoroughly reasonable responses all nicely citing my own resources.

Dave Hooton

I used the exact same question he uses (on the Perplexity AI and others, as you can see on the video) on Logos (Max subscription). Since Logos is a huge library of books, courses and language resources; and having myself a huge library for language research (my research focus) together with commentaries, encyclopedias etc (thousands of books, manuscripts, studies etc), I expected Logos to provide an even better answer, or at least similar to free AIs Claude and Perplexity. To call the result disappointing is a monumental understatement!

Logos' 'synopsis' is a short answer (1 paragraph) that only points to a generic grammar point. The rest is a big list of suggestions for that grammar point on my resources. Nothing really more helpful than a simple search without an AI.

The first (only?) query in the video was "why is the English word Jerusalem sometimes expressed by the plural neuter Greek word Ἱεροσόλυμα and other times by the indeclinable feminine singular Ἰερουσαλήμ?" It provided this Synopsis in an ALL Search:

---> The provided articles do not contain any information directly related to the user's query about the Greek words for Jerusalem (Ἱεροσόλυμα and Ἰερουσαλήμ). The articles mainly discuss Greek grammar, declensions of demonstrative pronouns, and unrelated topics such as the Book of Baruch and Greek adjectives. There is insufficient relevant information to address the specific question about the usage of different Greek forms for Jerusalem. The user may want to try a different query or consult sources specifically related to biblical Greek or the etymology of Jerusalem's name in Greek texts.

I was disappointed with that paragraph (your short answer?).

On searching for Ἱεροσόλυμα it can be plural neuter or singular feminine so I doubt the veracity of querying for one form of the word. But Ἰερουσαλὴμ is a singular feminine or "Indeclinable".

So I chose a simpler ALL query of "why is Jerusalem expressed as both Ἱεροσόλυμα and Ἰερουσαλὴμ in Greek" and it provided this Synopsis:

---> The Greek New Testament uses two different forms to refer to Jerusalem: Ἱεροσόλυμα (Hierosolyma) and Ἰερουσαλήμ (Ierousalēm). According to one of the articles, both forms are listed as valid Greek renderings of the city's name1 2. The first form, Ἱεροσόλυμα, is neuter plural in Greek, while Ἰερουσαλήμ is feminine singular1 2. However, the articles provided do not offer an explanation for why these two different forms are used or the reasons behind their coexistence in Greek texts.

A different query "why is jerusalem Ἱεροσόλυμα and Ἰερουσαλὴμ in Greek" yielded this Synopsis:

---> The Greek names for Jerusalem, Ἱεροσόλυμα (Hierosolyma) and Ἰερουσαλήμ (Ierousalēm), are both used in ancient Greek texts. According to the provided articles, these two forms are listed together as equivalent names for the city of Jerusalem1 2 3. The articles do not provide an explanation for why there are two different Greek forms or their specific usage contexts. It's worth noting that the information given is limited to lexical entries and does not offer historical or linguistic insights into the reasons behind these dual forms.

Was that the exact query you commented on above? I found it yielded a Synopsis of:

---> In the New Testament, the word for Jerusalem is sometimes treated as a feminine singular and other times as a neuter plural noun due to linguistic variations in Greek. One article notes that names of cities are typically feminine in Hebrew, with Jerusalem (יְרוּשָׁלַםִ) being an example1. However, in Greek, the situation is more complex. Matthew, for instance, uses two different spellings for Jerusalem: Ἱεροσόλυμα and Ἱερουσαλήμ. When using Ἱεροσόλυμα, Matthew sometimes treats it as feminine singular and other times as neuter plural2. This variation in usage occurs in both Alexandrian and Majority text-based critical editions. The inconsistency might reflect the flexibility in Greek grammar, where neuter plural nouns can take either singular or plural verbs, with singular verbs being more common (about 60% of the time)3. This grammatical feature, combined with the different spellings and treatments of the city's name, contributes to the varied representation of Jerusalem in the New Testament texts.

Which was fairly comprehensive. But it came from an online query and not from "AI memorization".

Whilst "why is jerusalem expressed by the plural greek neuter Ἱεροσόλυμα?" yielded:

---> The Greek word Ἱεροσόλυμα (Hierosolyma) is indeed a plural neuter form used to refer to Jerusalem. According to the provided articles, Ἱεροσόλυμα is listed as a plural neuter noun (-ων, τά) meaning "Jerusalem"1 2 3. The articles do not provide a specific explanation for why this plural form is used. It's worth noting that the articles also mention an alternative form, Ἰερουσαλήμ (Ierousalēm), which is feminine and indeclinable2 3 4. However, the given information does not offer a reason for the use of the plural neuter form Ἱεροσόλυμα over the singular feminine Ἰερουσαλήμ.

Which is an interesting pick up of the alternative form. But note the ambiguous assertion of "indeed".

So yes, you can get better and alternative results from refining your query. And I would deduce that there is no better reason(s) available for the use of the two forms from the available resources in Logos.

Mateus de Castro

https://community.logos.com/discussion/comment/1324809#Comment_1324809

The first (only?) query in the video was "why is the English word Jerusalem sometimes expressed by the plural neuter Greek word Ἱεροσόλυμα and other times by the indeclinable feminine singular Ἰερουσαλήμ?"

Thanks for the reply. Actually, that was not the question he used on the AIs. The one you quoted was a more complete question formulated to explain the problem for the video audience, but the question asked for the AI was simpler, as seen in the video (picture below of actual question around 0:43s of the video). The same I asked Logos to make a comparison in equal terms.

Anyway, even the better formulated questions you asked still generated answers that are not good enough for such a great library as Logos', IMO.

Again, thanks and God bless.

Justin Gatlin

https://community.logos.com/discussion/comment/1324810#Comment_1324810

I asked Perplexity what it costs to run, pushed back on the initial answer and got this:

[quote]

You're right to be skeptical of overly low estimates for the operational costs of running a large language model (LLM) like Perplexity. Let's take a closer look at some more realistic figures based on the search results:

1. Energy Consumption

The energy costs for running LLMs can be substantial:

- Training a large model like GPT-3 was estimated to consume about 1,287 MWh of electricity[1]. This is equivalent to the annual energy consumption of about 120 American households.

- For inference (actual usage), the energy consumption varies based on the task:
- Simple text generation tasks: around 0.00037 kWh for 250 words[1]
- Image generation: about 2.907 kWh per 1,000 inferences[3]

2. Operational Costs

While exact figures for Perplexity aren't available, we can look at estimates for similar services:

- ChatGPT 3.5 is estimated to cost about $700,000 per day in energy costs alone[5]. This is significantly higher than the previous estimate we discussed.

- Deploying a 7B parameter model (which is smaller than what Perplexity likely uses) to serve 1 million users is estimated to consume about 55.1 MWh[2].

3. Hardware Costs

LLMs require powerful hardware:

- High-performance GPUs like NVIDIA's A100, commonly used for AI tasks, consume up to 400 watts each[2].
- Training a large model might use 1,000 or more of these GPUs, potentially consuming 400 kilowatts per hour just for computation[2].

4. Infrastructure Costs

Beyond direct computation, there are additional infrastructure costs:

- Data center cooling and overhead can add 12% to 100% on top of the computational energy costs, depending on efficiency[2].
- Networking, storage, and other supporting systems also contribute to overall costs.

5. Scale Considerations

Perplexity's rapid growth suggests significant operational costs:

- Estimated $20 million annual recurring revenue as of April 2024, up from $7 million at the end of 2023[2].
- This growth likely necessitates continuous expansion of infrastructure and computational resources.

Given these factors, it's reasonable to conclude that the daily operational costs for Perplexity are likely in the hundreds of thousands of dollars range, possibly approaching or exceeding $1 million per day when accounting for all expenses (energy, hardware, infrastructure, personnel, etc.). This aligns more closely with the $700,000 per day energy cost estimate for ChatGPT 3.5, considering Perplexity's scale and growth.

The previous lower estimates we discussed were likely too conservative, not fully accounting for the scale and complexity of running a service like Perplexity. The actual costs are indeed much higher, as you correctly suspected.

Citations:
[1] https://cacm.acm.org/blogcacm/the-energy-footprint-of-humans-and-large-language-models/
[2] https://adasci.org/how-much-energy-do-llms-consume-unveiling-the-power-behind-ai/
[3] https://www.theverge.com/24066646/ai-electricity-energy-watts-generative-consumption
[4] https://arxiv.org/pdf/2310.03003.pdf
[5] https://news.ucsc.edu/2024/06/matmul-free-llm.html
[6] https://www.reddit.com/r/OpenAI/comments/1ezh2q9/whats_the_energy_cost_per_token_for_openais_llm/
[7] https://www.statista.com/statistics/1384401/energy-use-when-training-llm-models/
[8] https://sacra.com/c/perplexity/

Mateus de Castro

https://community.logos.com/discussion/comment/1324822#Comment_1324822

Phew! Perplexing indeed. Good thing nobody wants Logos to be some kind of 'open AI' answering all types of questions and doing crazy calculations etc, just to be better integrated with its own resources and tools.

Thanks, Justin!