ChatGPT INTEGRATION PLEASE🔥🔥🔥

MJ. Smith said:

Adam said:

how to integrate ChatGPT into the logos bible software

Step 1: Get the Logos users to agree on a very large corpus of books that are deemed to be accurate.

Step 2: Train ChatGPT on that corpus

Step 3: Get all Logos users to agree that answers from ChatGPT to questions such as "What does the Bible say about the necessity of baptism" are correct.

Step 4: Realize that ain't gonna happen. Limit ChatGPT to questions such as "What does Ephrem the Syrian say about the necessity of baptism?"

Step 5: Realize much of the agreed upon corpus is not discursive. Limit ChatGPT to questions such as "What does a 5 point Calvinist say about the necessity of baptism?" and "What does a 4 point Calvinist say about the necessity of baptism?" and "What does a 3 point Calvinist say about the necessity of baptism?" and "What does a 2 point Calvinist say about the necessity of baptism?" and "What does a 1 point Calvinist say about the necessity of baptism?" 

Step 6: Realize that the core issue the users want is access to raw data, information that confirms their current beliefs, and material based on the logic of belief revision to convince others that their current beliefs are correct. Yes, there is also a subset of users who are interested in the array of positions taken on issues (from grammar to theology), who promoted them, when they were prevalent, and available written records of those positions. Ask oneself two questions: 1. What kind of AI fits best with what the users actually want? 2. What is it about ChatGPT that makes users think it is the solution?

Step 7: Recognize that it is best to seek out the best solution for the problems that exist rather than seeking the best problem for a preselected solution. I suspect but have no intention of doing the survey and statistical analysis to prove it, that people are impressed by Chat GPT's conversational tone and memory of past questions in the conversation. Put those elements in the specifications for providing users with the data they want. Then go out and research what flavor of AI best suits the problem. Given the differences in the appropriate logics used, I suspect the answer is not ChatGPT.

Step 8: Recognize has Faithlife already applied artificial intelligence and training to develop some of the tools that are already available to us. Look for the key wording "train" in the documentation as opposed to "curate". Recognize that the surveys re:L9 before the specs for L10 were developed, did not put (re)development of Faithlife Assistant (conversational access to data) high on the list. If you think it should have been, votes for it, push for it but trust FL professionals to select the appropriate tools for the issue and within their budget.

I'm a thinking that's a tall, tall glass of water!  [8-|]

MJ. Smith said:

The "proposed fabricated quote" ... I am unconvinced it is anything other than a post aging out of the web.

Sorry I wasn't clear. I was referring to the quote from Swete. As for the NYT example, you may be right but I think it is fabricated primarily because you can search the NYT online archive and nothing is there either (though I may have missed it). It's possible that it was also scrubbed from the archive, but that would be unlikely for what appears to be a very mundane report.

If it is fabricated I wouldn't find it surprising or even reflecting all that negatively upon it in light of the basic logic of what an LLM is doing (though I may be misinformed here too). It seems like just another example of asking it which weighs more: a pound of feathers or two pounds of bricks. It's predicting what sort of "text" (or tokens) would follow the prompt in light of its training data. Because the sequence of tokens "which weighs more: a pound of feathers or one pounds of bricks?" occurs most frequently in the context of other tokens that carry the meaning "they weigh the same", it shouldn't surprise anyone that asking it "which weighs more: a pound of feathers or two pounds of bricks?" will trick it into giving an incorrect and possibly even incoherent answer. (The cosine similarity between the two sentences when we embed them using OpenAI's embeddings endpoint is 0.99).

In a sense, you could see that sort of phenomenon as a feature (rather than a bug) of the algorithms in play (though again, I may be off base here). Of course there is also training on top of that. With training, some of these mistakes can be corrected but probably not entirely eliminated. Aside from the knowledge problem, they aren't deterministic. For example, when I ran the NYT question with the suggested "engineering prompt" to prevent hallucination, I didn't get the incorrect answer. Taking away the prompt I got the fabricated paragraphs, but if you compare them with the example provided in the other thread you'll notice that it is slightly different (so at the very least we know it isn't quoting).

In brief, I think if we pay attention to the factuality warnings provided by these companies and try to understand how they work then we can find them more useful--even if only because we might have a better idea of what sort of prompts we would expect to return hallucinations and find a better way of asking the question.

MJ. Smith said:

Adam said:

how to integrate ChatGPT into the logos bible software

Welcome to the forums, Adam. Integrating ChatGPT and training ChatGPT are things that any IT specialist who understands AI systems can do. That it not where the "uniqueness" of ChatGPT lies. I think many Logos users would appreciate having a more conversational interface to our resources which is what ChatGPT provides. But it is not designed to be customized at the user level, meaning what we have in our library would not determine the answers we see. We couldn't hide books we think are garbage - if they were included in the training, their view point would still be represented. So to integrate:

Step 1: Get the Logos users to agree on a very large corpus of books that are deemed to be accurate.

Step 2: Train ChatGPT on that corpus

Step 3: Get all Logos users to agree that answers from ChatGPT to questions such as "What does the Bible say about the necessity of baptism" are correct.

Step 4: Realize that ain't gonna happen. Limit ChatGPT to questions such as "What does Ephrem the Syrian say about the necessity of baptism?"

Step 5: Realize much of the agreed upon corpus is not discursive. Limit ChatGPT to questions such as "What does a 5 point Calvinist say about the necessity of baptism?" and "What does a 4 point Calvinist say about the necessity of baptism?" and "What does a 3 point Calvinist say about the necessity of baptism?" and "What does a 2 point Calvinist say about the necessity of baptism?" and "What does a 1 point Calvinist say about the necessity of baptism?" 

Step 6: Realize that the core issue the users want is access to raw data, information that confirms their current beliefs, and material based on the logic of belief revision to convince others that their current beliefs are correct. Yes, there is also a subset of users who are interested in the array of positions taken on issues (from grammar to theology), who promoted them, when they were prevalent, and available written records of those positions. Ask oneself two questions: 1. What kind of AI fits best with what the users actually want? 2. What is it about ChatGPT that makes users think it is the solution?

Step 7: Recognize that it is best to seek out the best solution for the problems that exist rather than seeking the best problem for a preselected solution. I suspect but have no intention of doing the survey and statistical analysis to prove it, that people are impressed by Chat GPT's conversational tone and memory of past questions in the conversation. Put those elements in the specifications for providing users with the data they want. Then go out and research what flavor of AI best suits the problem. Given the differences in the appropriate logics used, I suspect the answer is not ChatGPT.

Step 8: Recognize has Faithlife already applied artificial intelligence and training to develop some of the tools that are already available to us. Look for the key wording "train" in the documentation as opposed to "curate". Recognize that the surveys re:L9 before the specs for L10 were developed, did not put (re)development of Faithlife Assistant (conversational access to data) high on the list. If you think it should have been, vote for it, push for it but trust FL professionals to select the appropriate tools for the issue and within their budget.

I must admit I didn't think about that complication - the issue of interpretation. My mind was more geared towards objective research practicality which GPT functions well for even outside of Logos or Verbum. So I guess i will concede that that is something that does need to be considered as being a hurdle that needs to be figured out. However, there does seem to be a fix for it by using the program as it is right now. For instance, for a Catholic, I can restrict the sources that Chat GPT is using to provide the answers to my questions. Meaning that GPT can only use the Bible, the Catechism, Papal encyclicals, etc. and i can also set the persona of ChatGPT to be a Catholic Theologian. So i would imagine that this can also be done for various protestant users that wish to have their sources tailored to their particular religion so they dont receive skewed results. 

In terms of implementation into Verbum or Logos, after going through some classes on GPT it seems as though individually implementing it into Verbum/logos may really not be as complicated as we think. If a company can use the API of ChatGPT for their own personal chatbots on their site with tailored company information being used as its knowledge base, then its really not far fetched to have it on a personal level tailored to whats in our libraries.

Quite frankly we all know that this is the future. We can argue against it, but thats not going to make this AI breakthrough technology go away. This is massive and its progressing very fast. The question really is not if but when will logos and verbum provide this technology for their customer base. If they move too slow another company may beat them to the race.  

MJ. Smith said:

Adam said:

HOWEVER, if we can have it strictly use the content of our library as the basis for its responses -as its intelligence per say - then THAT would be an amazing development and unparalleled game changer for study and education!

Because of the way Faithlife structures its packages, I have a great deal of garbage in my library. I mean "garbage" not in the sense that I disagree with it but in the sense that it is very poorly thought out or if it was thought out clearly, they've hidden it well. ChatGPT has not made any major breakthroughs in terms of evaluating the quality of/ or even recognition of conflicting information. And because I appreciate "both-and" religious logic and apophatic religious knowledge, I reject the idea that the logic/patterns applied by ChatGPT necessarily apply to religious data. Just label me a skeptic/cynical when it comes to AI for faith formation at this point. Others have opinions that are different than mine. That's no problem.

Yea i hear what you mean; I have a ton of random books that have no value to me in my library as well. When i went through the training for Verbum there is a way to take those books and basically discard them. With an integrated AI, I would imagine that there would be a way to not include those books in the AI's fine-tuning as well.

Also did i mention that if your skeptical about the answers ChatGPT is giving you, you can tell it to provide footnotes including page numbers for each of the responses that it gives you to your questions? This would allow you to double check its work. Moreover, if the answer that GPT gives you to one of your questions is filled with academic jargon that your not familiar with, you can ask it to simplify it or restate it as if it was speaking to an average adult, or an 18 year old, or a 10 year old, or better yet a 5 year old, or even in terms fitting for a professional in the field of study. The possibilities seem endless with this new technology.  

Adam said:

Also did i mention that if your skeptical about the answers ChatGPT is giving you, you can tell it to provide footnotes including page numbers for each of the responses that it gives you to your questions? Moreover, if the answer that GPT gives you to one of your questions is filled with academic jargon that your not familiar with, you can ask it simplify it or restate it as if it was speaking to an average adult - or of course vice versa if you want it to be more precise and detailed. The possibilities seem endless with this new technology.  

I suspect that if you aren't careful with this sort of query that you are actually priming it for hallucination, unless at the same time you're providing it with the resource. An LLM in itself isn't consulting a database of resources from which it answers questions. If you ask it to give some quote or page number it isn't searching in a database. In fact in a lot of scenarios I doubt you want keep all that data around in the first place since it's expensive to store and may have copy-right issues. Once the model is built, all that exists (in terms of it functioning) are the algorithms and vectors. It's possible to add that sort of functionality, like Microsoft has attempted with Bing, and maybe OpenAI has it poll a database in certain cases, but I wouldn't assume that this in fact occuring.

So you can ask it to give you the third letter of the 7th document that it was trained on or you can ask it to give you a citation of a quote, but I'm pretty certain that the response you get is going to be based on some sort of token probability and not based on it consulting a database.

Very good, thanks for sharing this! Clearly GPT needs some further work. Hopefully this will get ironed out in the months to come.

https://feedback.faithlife.com/boards/logos-desktop-app/posts/chatbotgpt4-pluggin

Vote up this suggestion to get the company's attention.

A GPT-4 plugin for Verbum/Logos

And here's how Logos integration might improve accuracy. Here, I set the system message as usual and also give it a chunk of context that contains the some of the user-visible metadata that one can get with copying (page numbers citation). This isn't a great example since Logos wouldn't be feeding it such specific context, but it illustrates how GPT can identify the way Logos signifies page boundaries for users, without being specifically told that information.

The image highlights pick out the nudge, the "metadata", the quote, and the correct answer.

(truncated image)

Justin Gatlin said:

The NYTimes article in the previous thread was entirely made up by me.

Right, and I think everyone who's used things like BingChat, Bard, or OpenAI would consider this a well-known phenomenon or a the very least a known phenomenon since it's advertized at the points of entry. I suggested in another response that it was more a feature than a bug. I would further suggest that it's a feature that people working on these models do not want to remove since it has its own utility.

For example, more deterministic answers move it closer to a plain old database or search engine, it's ability to "extrapolate the gaps" can provide genuine insight for the user (although it's lack of coding ability is pretty notorious on some some programming social media sites and message boards (I think I heard about it being banned by stackoverflow, but never looked into it), it's actually surprisingly good at many things and there are plenty of examples of it finding and fixing bugs in code. Though, to my knowledge, it hasn't taken steps forward, but more like bridging gaps in the user's knowledge based on specific domain "knowledge".

And of course as all of the companies behind these models have advertised, you can use the "gap extrapolation for creative purposes (much like stable diffusion is doing in art). You can ask it to play the role of, say, Socrates and have a Socratic dialogue. It works quite well on some occasions--it accurately explained why a state's population should be 5,040 (from the Republic) in the character of Plato--and not so well in others.

I think ultimately what is desirable is not an elimination of "gap extrapolation", but more control over when it makes the leap and how far it is willing to leap. I'm sure it's a difficult problem in its own right, but I would imagine even more so when we consider control in the hands of users (and OpenAI hosts a paper on their website detailing these issues during GPT4's development). Some progress on this has been made between GPT3.5 and GPT4 but I readily admit that it needs to make a lot more. I just don't think where it currently stands is as problematic as some people do.

To put it differently, the fact that we can engineer a prompt such that these LLMs will give us wrong answers isn't at all concerning. That's like the "Guy shot guy" internet meme.¹ What is concerning are instances where it makes these hallucinations when that's not what the user wants or what the prompt was engineered for.

--

1. BingChat helped me identify what meme I was thinking of here. I couldn't quite remember what it was called and I asked it "What is the meme with the actor sitting on the ground and trying to blame someone for his situation when he caused the situation?" and it responded "I think you are referring to the Guy shot guy meme..." with a correct link to an instance of the meme online. And if you look up the meme you'll see that I was misremembering it, but Bing was still able to extract the intended answer.

MJ. Smith said:

Remember that all human statements regarding truth are of the form if these premises are true, then this deduced statement is true. Most religious knowledge is of the form, because I believe these premises to be true, I also believe these statements which are consistent with those premises to be true. For this reason, I'm still a fan of the old tool Prolog for testing consistency as a verification tool.

I don't know about that.... In the college I attended, in Logic 101, (a tough course for me) I remember well the professor teaching, "Truth can always be proven." And he would say, "You have to get past yourself to find objective truth". And he illustrated that by saying "all things depend on objective truth, math, science, medicine, even the way we treat each other."  I had to write a paper and the title he gave us was "What is objective truth". 

So, if we start with if these premises are true ... then we have already made assumptions and have already entered biases because of those assumptions. Objective truth cannot be found with assumptions or biases. I think that is why we have 38,000 different "churches" today, simply because each is approaching the bible with if these premises are true, whatever those premises are. We cannot approach the bible with biases or assumptions... the bible stands alone as being objective truth and we are the ones that have to find the truth of it. And I believe the bible was written by only one God (Holy Spirit) and as such it does not have 38,000 different ways to view it. Objective truth is singular...not plural. 

It is man with all his biases and presumptions that really foul things up.  So, if AI is written with the algorithms that are based on man's thinking, then AI is flawed to being nothing more than an extension of man's preconceived thoughts. But if and I say if, AI can really be turned loose to actually develop a thought process and able to change that...well... maybe we have something. And that is what the world of AI would have us believe is coming. 

imho...

[H] YES