Quality control in Logos 6 - a new perspective
Recent discussions have highlighted yet how promising Logos 6 is in terms of biblical studies. I think "promise" is a very appropriate word because it has that double-edge: a promise holds hope yet is not always fulfilled.
For instance, searches of cultural concepts or other tagged entities shows a lot of gaps and inconsistencies in tagging. This is a quality control problem.
I think that quality control is the leading issue with the ongoing development of Logos. I don't only think so, I know so. I know that because I participated in a development project for several months and had an insider view of this problem. I became increasingly uncomfortable with my participation in the project because I could see how it was being pushed steadily forward for earlier than later release at the cost of cutting many corners and releasing a product that will be flawed from the start. Those who led the project knew that but they are pressing on. I am sad when I think about the marketing that is being done in anticipation of release, having the awareness that there will be some very disappointed buyers out there.
And yet, it does not need to be so! This post, if anything, is yet another plea to reverse what has become a consistent blotch on a product that is otherwise exciting and holds so much promise.
As it is, I could use some of the tagging "to save hours" (this is a key marketing promise). This can be done by automating what would otherwise need to be done manually. Combing large sections for data over and over is extremely time-consuming. BUT unfortunately, at this point, I cannot rely on results. So, I still have to do the manual work to ascertain what I find. And this is not just compensating for the occasional human error. This is more systemic.
Jesus told the parable of two sons confronted by the request to do right. One said "I will" but did not do it. The other said "no" but then ended up doing it. Logos, what kind of "son" will you prove to be?
This being said, if I and others simply don't understand that it ain't so, perhaps some explaining would help. Is Logos so broke it must release stuff asap or die and thus considers it better overall to release incomplete or error-filled, bug-ridden products? I do not mean to slander those who may be hard-working and trying as hard as they realistically can. But I am perplexed (and in the case of the experience I allude to above, dismayed) and above all, desire a good outcome.
Comments
-
Luke 14:28-30 also comes to mind.
0 -
-
Clearly having a monopoly in a field does not assure an equally dominant product from that field.
0 -
Rene Atchley said:
Clearly having a monopoly in a field does not assure an equally dominant product from that field.
I don't think there's a monopoly. I've sat all week in a classroom where Bibleworks was used the whole week. And one of the podcasts I regularly watch uses Accordance for nearly all of its biblical language work. What those both have in common is that they are better (mostly: faster) than Logos at those features.
Donnie
0 -
I find it incredibly frustrating to post on this site. Every generalization seems to be opposed by someone somehow by injecting that "wow" that statement isn't true because it isn't true for me. The point of the thread and post is a critique of the quality control issue for a product from a company that sells a very expensive bible library product. So yes they may not be a monopoly but that doesn't detract from the ogopolistic nature of the bible software industry on PC/Mac that often (but not all the time under all situations in all time periods) quality controlled problems (but not for everyone all the time under all situations in all time periods of history known to humanity).
0 -
Francis said:
As it is, I could use some of the tagging "to save hours" (this is a key marketing promise). This can be done by automating what would otherwise need to be done manually. Combing large sections for data over and over is extremely time-consuming. BUT unfortunately, at this point, I cannot rely on results. So, I still have to do the manual work to ascertain what I find. And this is not just compensating for the occasional human error. This is more systemic.
I would find it difficult to rely on results from Faithlife-created datasets and tagging for the same reason. With much of Faithlife's future developmental effort going into creating theses new datasets, I hope they aren't shooting themselves in the foot by creating a reputation of delivering unreliable and missing data.
Francis said:Is Logos so broke it must release stuff asap or die and thus considers it better overall to release incomplete or error-filled, bug-ridden products?
I do understand the need to expand and broaden when you see that you may be about to saturate your market. They may not be broke, but they do need to keep finding ways to generate income. (But it probably needs to be done without alienating your customers.)
MacBook Pro (2019), ThinkPad E540
0 -
Having not worked at Logos, one can only surmise. But it sure looks like 'deja vu'. The boss and team put together the next dreams. Everybody gets to work, with varying degrees of success. The boss notices a competitor is releasing the next major version. Asks the teams 'can we do it??'. Of course the teams can (though they really can't). And didn't.
The sad part is the misplaced dreams. I was really hoping the Logos databases, the Proclaim platform, the Faithlife community product, the mobiles, and oh yes, the dating game ... the ecosystem (as Logos calls it) ... had great promise. But each piece is not well designed and always incomplete.
Yesterday and this morning I felt really sorry for the Reverend that just wanted to delete his faithlife group, after giving up. No one seemed to know how. No manuals. I would have like to have helped. The Logos guy answered they'd try to do better ... but no answer for the Reverend.
The Logos6 release I hope won't be repeated in Logos7; too much product that was not complete at sell-time. Pressure now to produce it. Requests for quality? I haven't even bothered to see what I bought (since I can't find it!).
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
If the situation described by the OP is true, then it is sad indeed.
The quality of the tagging, linguistic analysis etc. can be evaluated objectively by folks trained in those areas. If there are major errors, there's nowhere to hide.
0 -
Todd Phillips said:
I hope they aren't shooting themselves in the foot by creating a reputation of delivering unreliable and missing data.
Is not this already happening? There is also a reputation of greed floating around.
Todd Phillips said:without alienating your customers
Here is a sad fact of my experience. I have used Logos since version 2. Over the years, I have "sold" Logos right, left and center. But in the last while, this has changed. I use Logos because I must, not because I want to. But I am no longer an enthusiastic advocate of Logos. There is a language version coming out on which I did translation. Considering what I know of it, I have not and will not recommend it to anyone. In fact, this was part of what I alluded to earlier: I was at first proud of the idea of helping it be available to more people (and bringing better Bible study to them) but now I would be ashamed to be associated with that version.
I am not saying this to bash Logos needlessly. I feel that this is a bad turn but it can be reversed. I sure hope so.
But I do wonder, in the present estate and in light of all that has been going on, what user polls would indicate if the simple question was asked where only yes or no was allowed as answer "Do you [still] trust Logos?" I hope Logos is listening and more importantly, hearing.
0 -
Francis said:
what user polls would indicate if the simple question was asked where only yes or no was allowed as answer "Do you [still] trust Logos?"
YES!
Logos 7 Collectors Edition
0 -
The question was not what individual answers might be but what the overall picture would look like. If the individual answer is yes for you, I'm glad for you. We may not have the same needs, experiences or simply responses.
0 -
Francis said:
The question was not what individual answers might be but what the overall picture would look like.
I am inviting as many individuals as want to to participate in your one word survey. Polls are composed of idividual responses so don't devalue my response.
So how about it Logos users? Do you still trust Logos? Yes or No.
Logos 7 Collectors Edition
0 -
To be fair the poll question is "do you [still] trust Logos?"
I wrote it in square bracket because it may or may not be significant to all users depending on the duration of their experience.
0 -
Depends!
0 -
That's not the most helpful question. The issue is the balance of work on new features vs work on fixing bugs and making things work well.
0 -
As I'm reading this, Roosevelt's line, "the only thing we have to fear is fear itself" comes to mind -- not necessarily for anyone else, but for me. I've always trusted not only in the company and the product, but in the customer base as well. My overly large investment was partially justified by the fact that Logos will be around long enough that it'll run on computers long after I'm gone: Windows 9, 10, ... 45, etc.
But what if the fear/distrust becomes widespread enough that even donating my library to a church in, say, 2027 would be like donating an Apple II to a church: Why would we want that old thing?
I hope that doesn't happen.
0 -
Rene Atchley said:
I find it incredibly frustrating to post on this site. Every generalization seems to be opposed by someone somehow by injecting that "wow" that statement isn't true because it isn't true for me.
This reminds me of the time my son, frustrated with his mother and me, exclaimed, "You always exaggerate everything I say!" (He immediately realized what he had done and started laughing.) Is not "every generalization" a generalization?
In all seriousness, I did not mean to upset you. If you read to the end of my post, you'll see I tried to make the point even stronger. A true monopoly has little incentive to improve. Logos has competition, at least at some level, and yet it still is challenged in producing equivalent core functionality (at least for me, it's core - working with the biblical languages).
My apologies...
Donnie
0 -
0
-
Lew Worthington said:
what if the fear/distrust becomes widespread enough
I agree with your sentiment. I made my sentiment known earlier that there was, in my view a regrettable state of affair, but that I see as reversible and for which I was making a plea. My rhetorical question (about the poll) has been turned into a pointless confrontation...
0 -
Todd Phillips said:
I do understand the need to expand and broaden when you see that you may be about to saturate your market.
I have gotten the impression that they are a company that often reacts and operates out of fear. I am probably wrong.
Looking at my context, there is absolutely no saturation. Out of six pastors in my local group, I am the only one who uses Logos. Another chose Accordance the other do not use anything. And of the community I serve, no one uses bible software. There is plenty of room for growth in my context.
However, as I have said before, I cannot recommend Logos or any FaithLife products because of the quality issues. I would love to be able to "sell" Logos to the people I serve. But, I can see it coming back to embarrass me and having to spend too much time supporting them. (Not to mention that it is not user friendly and has no reference manual. I have been endeavoring to understand the new Milestone search feature. Thank goodness for the community here for helping. But surely there has to be a way to make that into a menu item that is more accessible to someone without mastering search syntax strings. Just my experience.)
Do I trust Logos? As a reader, yes. But, I have gotten into the habit of verifying its result otherwise, or doing them the hard way because it is just as easy. So, maybe the answer is, no. I buy a lot of resources, but I have been contemplating their value since the release of 6.
Like Francis, I want Logos to succeed and grow.
EDIT: On further reflection, I would like to make an amendment/confession. I think I was quick to answer due to many of my resent frustrations with the software, particularly since the 6 release. I think it was unfair of me to insinuate that I do not trust Logos (or to outright say it). I do not know enough about its search perimeters, abilities, etc. on the datasets to even begin to form an opinion, which is what the OP was referring to, I think. My concern for QC centers more around the reliability of the program itself to run, to do its job timely, the UI etc., which I hope came out a bit.
Thank you John Fidel (below) for your perspective and my apologies to the community. I am trying to be more positive towards FaithLife in the forums, yet maintain the truth of what I see that they may improve. I particularly want to be sensitive to those that do the hard work at FaithLife in producing the software and resources. I know they want to do a great job and it is a complex, never-ending job. My apologies to you as well if I have seemed overly harsh. As a pastor, most days we hear more criticisms than anything. I forget that others deal with that as well. Thanks also to Sean for his words.
So, in terms of the quality of the "programmed" application and services, I think there is much room for improvement (hence my difficulty to recommend it. Though maybe I will give it a shot). I know it will get better. Though, I think I would have rather waited for a better product.
I do love Logos. I am in it many hours a day. Even after all these years it is mostly for reading. I think one of the frustrating things is the steep learning curve to really get at the power.
Michael
0 -
Francis said:
My rhetorical question (about the poll) has been turned into a pointless confrontation...
I guess if everyone answered "No" there would be no confrontation. Declaring a poll pointless is one way of retreating from your rhetoric.
Logos 7 Collectors Edition
0 -
Content deleted
0 -
I lead the Content Innovation (CI) team: we're primarily responsible for the data sets and linguistic tagging on the biblical text that ships with Logos. So it matters a great deal to me that we produce the best value we can for our users in these areas, and it gets my attention when people raise questions about quality control. I can only speak to the areas I'm responsible for (which don't include the translation project the OP worked on).
Francis said:For instance, searches of cultural concepts or other tagged entities shows a lot of gaps and inconsistencies in tagging. This is a quality control problem.
I don't know how to respond to this without more specifics. But let me provide some context on the Cultural Concepts project from CI's side, since ultimately I'm the one who's responsible for it. I'm extremely proud of the work our team did here:
- We created a hierarchical categorization of more than 1100 cultural concepts that are relevant to the biblical text (the Lexham Cultural Ontology, or LCO).
- We annotated both the Bible and about 15 other ancient text resources with these concepts: I estimate about 5 million words. Note there's no automated process: our team had to read each text and analyze it from the perspective of the LCO.
- We linked concepts to key dictionary articles explaining their background.
- We coordinated the LCO concepts with our Topic Guide and Bible Sense Lexicon data to "connect the dots" with other Logos data sets
Every data set project we undertake has to find the right middle ground between benefit to users (things like search, organization, and ultimately insight) and cost to Logos. The primary benefits we imagined for the Cultural Concepts are:
- Providing easier access to cultural background information for the biblical text
- Making it possible to find secondary literature on the same concepts (If you browse the Cultural Concepts section in Factbook for Covenant, you'll find a lot of material you'd never find by searching for the word "covenant")
- Organizing the cultural concepts themselves (Covenant is a sub-concept under Contracts and browsing the glossary may help with exploring the cultural background of the Bible)
Our users are ultimately the ones who have to judge whether we delivered these benefits or not: from my perspective, this is not only a successful feature in our product, but a novel and significant contribution to the field of biblical studies as a whole.
As for cost, we've invested roughly 20 person months of effort in building this data set and annotating texts (we even hired three interns this past summer to help get all the annotation done for Logos 6). All of the people doing this work had graduate level training and consulted existing academic resources in their work.
I've gone into the details on this one project (we have many, of course) to try and provide some perspective on our process. Is this data set perfect? No, and it can't be: there's no objective standard to compare it to, and nobody's ever done anything quite like this before. Is the annotation "done"? Again, no: we prioritized our annotation for the resources we thought would be useful to the most Logos users, but there are dozens more resources we could annotate. Deciding how many more we ultimately do will be determined by trading off benefit and cost, for this project and others. Is the ontology complete or comprehensive? Not really: while we did our best to cover the most important biblical cultural concepts, any model can always be enriched and extended.
Instead, the criteria we typically use to assess data projects like the LCO include
- Utility: does it help users accomplish their goals?
- Consistency: have we been consistent throughout in our annotation? Are similar passages annotated the same way?
- Validity: would external observers who understood the issues be in substantial agreement with our judgments?
- Sufficiency: have we provided enough additional data to make the whole useful?
These are all significant challenges, and (like the original analysis itself) none of these criteria can be automatically checked (though we do check as many things as we reasonably can). But I stand by my claims that we did our best to meet these criteria given the effort we invested, and overall, we've provided a great deal of value with this data set.
If you find errors in this or other CI data sets, please email data@logos.com or post on the forums with the specifics, and we'll do our best to fix it.
PS: David Witthoff, who was the project lead for the LCO, has a series of helpful posts on his personal blog about using the LCO for sermon preparation, starting with Logos 6 and Sermon Method.
0 -
Thanks Sean - Enjoyed your post and learned something in the process
0 -
Hello Sean,
Thank you for your reply. I appreciate the information it provides and the time you took to write it.
I understand your request for specifics, but I cannot be too specific here without taking too much time to go back in history and trace back all the various issues that have arisen. But it has been an accumulation. I must say, and I understand that this is not your department, that I was particularly disillusioned by the other project I alluded to. However, I do not want to say more about it here because I don't want to start mentioning specific people or give enough information that it would point to specific people. This is not my goal. My goal is to call for improvement, not to burn someone at the stake.
As an instance, a couple days ago there was this thread about looking for prophecies and/or feasts in the NT, when avenues were found that were thought to be able to produce the expected results, hits were missing because they were tagged differently. For instance, Passover in John was not tagged as a religious feast, but as Passover. Someone who would look for mentions of feasts in the NT and used that cultural concept search would miss a number of hits. This may be a matter of choice, though the choices and the underlying logic is often not obvious and this is the kind of thing for which it is difficult to find documentation.
In another thread -- we're leaving cultural concepts now -- we were talking about searching labels and tags. Again, lots of problems there.
These are the two most recent examples on my mind, but there are others who even in this thread have echoed my experience and perhaps can remember other specific examples. I would think that beyond that, there are very many expressions of frustration about features or data not being complete. Perhaps I am venturing too far (and I am willing to be corrected) but I think that those I hear from who do work that require accuracy generally know that they cannot fully rely on results.
Features are also delivered incomplete, for instance label searches in annotations or user document searches, or narrative outlines. Resources come out untagged, etc.
I appreciate your answer then and certainly I do not mean this as an attack on you and your team, but I really don't know why there are so many errors all over the software, such a lack of proper documentation and so much incompleteness. I spend way too much time looking for solutions to fix or circumvent these problems, instead of just getting the work done!
This is not to say that I do not value what does work. When it works and is accurate/dependance or complete, it's great.
0 -
First I owe Sean a HUGE thanks. You interviewed me a couple years ago when I applied to work with Faithlife (then Logos) as a Lexical Data Curator. You did not hire me, and it is for that I offer a heartfelt thanks. I was called a couple weeks later as a pastor in a wonderful church! The Lord is good!
Now to the topic at hand. I think that perhaps the question of whether one "trusts" Logos/Faithlife is a bit of a misdirection. The Logos Library system has always been just that: a way of collecting and accessing published material in a digital format. Over the last 15 years I have seen huge changes in how that information is accessed, but that has not changed the basic fact that as a library system Logos is in my opinion, unmatched.
The extent to which one chooses to accept/believe/trust the content stored and accessed in the Logos library should always be a matter of wisdom and discretion. As I read different authors, some I develop a respect for and have an increased receptivity to what they have to teach, others I remain more skeptical of if I find their methods or conclusion faulty or lacking.
When it comes to the content that Faithlife has included in their library system I think we need to use the same wisdom and discretion when considering the quality of information they have gathered and published that we do with other content producers/authors. However I think it is important to maintain the distinction between the library system we use to access the library and the production of content itself.
Yet here is where the content from Faithlife is different from any one of the thousands of books and journals I have: it is dynamic. Where there are errors, they can edit and update it where those errors are pointed out or as they become more experienced and identify their own shortcomings. With electronic versions of a "normal" print library, we have to wait for years as the authors themselves grow and change and then write the second edition, third edition, etc. and then be published. With Faithlife as a content producer, they have the ability to make that change and push it out (publish it) tomorrow! What a blessing!
If you are wary or disagree with the content that Faithlife has included in their library system, certainly don't use it. Make use of the resources you are confident in and do trust. However, does disliking Faithlife's content justify abandoning the whole library system? I personally do not think so.
With regards to the library system (how we access the data) certainly tagging is an important part of how data is found and accessed. It seems though that new features provide additional access to existing data so that we have different ways to dig deeper into what we already had. I haven't found anything yet that keeps me from studying and finding the information in the ways I have grown accustomed to. We can hope that Faithlife will continue to expand on what they have already done so that datasets are more complete, but again this does not cripple study, it simply limits the usefulness and viability of the new features until those features are more fully developed.
0 -
Tim said:
If you are wary or disagree with the content that Faithlife has included in their library system, certainly don't use it. Make use of the resources you are confident in and do trust. However, does disliking Faithlife's content justify abandoning the whole library system? I personally do not think so.
With regards to the library system (how we access the data) certainly tagging is an important part of how data is found and accessed. It seems though that new features provide additional access to existing data so that we have different ways to dig deeper into what we already had. I haven't found anything yet that keeps me from studying and finding the information in the ways I have grown accustomed to. We can hope that Faithlife will continue to expand on what they have already done so that datasets are more complete, but again this does not cripple study, it simply limits the usefulness and viability of the new features until those features are more fully developed.
Thank you for your thoughts Tim. I appreciate your desire to be gracious and provide perspective, but I must say I totally disagree with you here.
That your study needs or habits may be such that you can function the way you describe, I can envision. But Logos is now also a product for academics. We do a lot of detailed research and whatever we claim must be accurate. Keeping that in mind, new releases of Logos have each come with a set of promises. What I mean by promises is that features are marketed as reasons to upgrade and they tell you the beauty of all you are going to be able to do. You buy into it (literally) and little by little discover that features are incomplete. You think it's just bugs at the beginning, but then users who have been here since Logos 4 can tell you that while many problems have been addressed, many have also dragged on a long time. Tagging is a sore point. But in addition to this, you have to keep in mind that long-term users have invested a lot in Logos. There is the cost of resources, of all the upgrades but also the investment represented by trusting it with your study and data (all your annotations, all your markings, etc). Over the years, it's huge. The longer you have stuck with the company and the more this dynamic of incompleteness, promises lagging pattern continues, the greater the feeling of being betrayed with regards to the expectations that were fed by the promises and marketing messages and which you have paid for.
So yes, trust is an issue because I have trusted Logos during all these years, but I do feel that there has been a worsening of that dynamic (more stuff coming out more quickly yet incomplete). I was actually a bit shocked by the timing of the "what do you want to see in Logos 7?" question. I don't think I would be the only who, if things do not change, would take whatever marketing messages and promises are made when Logos 7 comes out with a degree of circumspection and if features of Logos 6 are not fixed by then, with a strong dose of cynicism.
0 -
Well, Sean, I took you up on your PS offer. I thought I'd get some free training from the horse's mouth: David Witthof
I did what he said to do, and nothing happened. He warned me not to type in 1 Samuel or 1st Samuel, so I didn't. I carefully copied what he said to type in: First Samuel. My Factbook refused to budge from Moriah (an earlier choice).
Then I looked at what his example actually delivered: 'Book of First Samuel'
Well, there you go! The trick is to not type the identification of the book ... Samuel, Chronicles, and so forth. I felt truly trained. So for my 'finals', I typed 'Peter' (for 1st Peter). I got another 'F'.
Yes, it didn't work and yes, I'm happy you guys worked so hard on WBC.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
Francis said:
As an instance, a couple days ago there was this thread about looking for prophecies and/or feasts in the NT, when avenues were found that were thought to be able to produce the expected results, hits were missing because they were tagged differently. For instance, Passover in John was not tagged as a religious feast, but as Passover. Someone who would look for mentions of feasts in the NT and used that cultural concept search would miss a number of hits. This may be a matter of choice, though the choices and the underlying logic is often not obvious and this is the kind of thing for which it is difficult to find documentation.
Looks like this is the post you mean? I agree there's a data issue here with the ontology: we've categorized Passover as a kind of Organized ceremonial (which is true), but it's also a kind of Religious feast, and our data doesn't capture that. I'll add a case to correct that.
Examples like these are very helpful because they illustrate ways people use our data that we couldn't easily anticipate. As Mark Barnes noted in that thread (for a different case), errors and definitional problems aren't necessarily the same thing. Any time we make some analysis, we're necessarily joining or splitting things that others might not join or split the same way. But (using the criteria from my previous post in this thread) having Passover as a Religious feast would be at least more useful and more valid.
I agree we need to do more to explain the choices and logic behind some of our datasets: creating those introductions is on my list for this year.
Francis said:Features are also delivered incomplete, for instance label searches in annotations or user document searches, or narrative outlines. Resources come out untagged, etc.
We've made a fundamental business decision: not all data sets need to be complete to be useful. Your example of Narrative Character Maps is apt: we could withhold this data until we have character maps for every narrative book in the Bible. But in the meantime, if you're studying the first part of Acts, you might find this character map useful today (and we're actively working on the rest of Acts). We may decide some books aren't worth the effort for this kind of visualization (by contrast, we deliberately decided that we wanted our Biblical People Diagrams to cover every named individual in the Bible, even one of the 30 Zechariah's that's only mentioned once in the text). Some resources get a dozen or more different kinds of markup in our production process: more markup means more production cost, so others get less. These are business and editorial decisions: they're not quality control issues.
0 -
Denise said:
Well, Sean, I took you up on your PS offer. I thought I'd get some free training from the horse's mouth: David Witthof
I did what he said to do, and nothing happened. He warned me not to type in 1 Samuel or 1st Samuel, so I didn't. I carefully copied what he said to type in: First Samuel. My Factbook refused to budge from Moriah (an earlier choice).
Then I looked at what his example actually delivered: 'Book of First Samuel'
Well, there you go! The trick is to not type the identification of the book ... Samuel, Chronicles, and so forth. I felt truly trained. So for my 'finals', I typed 'Peter' (for 1st Peter). I got another 'F'.
Yes, it didn't work and yes, I'm happy you guys worked so hard on WBC.
We have a known problem with looking up Bible Book in Factbook (previously noted here, here. here and elsewhere, no doubt). We've got a much longer list of candidate articles for Factbook than previous guides, so we've still got some work to do in sorting and ranking the alternatives. Clearly Bible books shouldn't be buried down at the end of the list.
The best current workaround I know of is to use the Topic type to start searches for Bible books. So the results for "<Topic peter" includes First Epistle of Peter, though it's still much farther down the list than I'd like.
0
