Quality control in Logos 6 - a new perspective
Recent discussions have highlighted yet how promising Logos 6 is in terms of biblical studies. I think "promise" is a very appropriate word because it has that double-edge: a promise holds hope yet is not always fulfilled.
For instance, searches of cultural concepts or other tagged entities shows a lot of gaps and inconsistencies in tagging. This is a quality control problem.
I think that quality control is the leading issue with the ongoing development of Logos. I don't only think so, I know so. I know that because I participated in a development project for several months and had an insider view of this problem. I became increasingly uncomfortable with my participation in the project because I could see how it was being pushed steadily forward for earlier than later release at the cost of cutting many corners and releasing a product that will be flawed from the start. Those who led the project knew that but they are pressing on. I am sad when I think about the marketing that is being done in anticipation of release, having the awareness that there will be some very disappointed buyers out there.
And yet, it does not need to be so! This post, if anything, is yet another plea to reverse what has become a consistent blotch on a product that is otherwise exciting and holds so much promise.
As it is, I could use some of the tagging "to save hours" (this is a key marketing promise). This can be done by automating what would otherwise need to be done manually. Combing large sections for data over and over is extremely time-consuming. BUT unfortunately, at this point, I cannot rely on results. So, I still have to do the manual work to ascertain what I find. And this is not just compensating for the occasional human error. This is more systemic.
Jesus told the parable of two sons confronted by the request to do right. One said "I will" but did not do it. The other said "no" but then ended up doing it. Logos, what kind of "son" will you prove to be?
This being said, if I and others simply don't understand that it ain't so, perhaps some explaining would help. Is Logos so broke it must release stuff asap or die and thus considers it better overall to release incomplete or error-filled, bug-ridden products? I do not mean to slander those who may be hard-working and trying as hard as they realistically can. But I am perplexed (and in the case of the experience I allude to above, dismayed) and above all, desire a good outcome.
Comments
-
Luke 14:28-30 also comes to mind.
0 -
-
Clearly having a monopoly in a field does not assure an equally dominant product from that field.
0 -
Rene Atchley said:
Clearly having a monopoly in a field does not assure an equally dominant product from that field.
I don't think there's a monopoly. I've sat all week in a classroom where Bibleworks was used the whole week. And one of the podcasts I regularly watch uses Accordance for nearly all of its biblical language work. What those both have in common is that they are better (mostly: faster) than Logos at those features.
Donnie
0 -
I find it incredibly frustrating to post on this site. Every generalization seems to be opposed by someone somehow by injecting that "wow" that statement isn't true because it isn't true for me. The point of the thread and post is a critique of the quality control issue for a product from a company that sells a very expensive bible library product. So yes they may not be a monopoly but that doesn't detract from the ogopolistic nature of the bible software industry on PC/Mac that often (but not all the time under all situations in all time periods) quality controlled problems (but not for everyone all the time under all situations in all time periods of history known to humanity).
0 -
Francis said:
As it is, I could use some of the tagging "to save hours" (this is a key marketing promise). This can be done by automating what would otherwise need to be done manually. Combing large sections for data over and over is extremely time-consuming. BUT unfortunately, at this point, I cannot rely on results. So, I still have to do the manual work to ascertain what I find. And this is not just compensating for the occasional human error. This is more systemic.
I would find it difficult to rely on results from Faithlife-created datasets and tagging for the same reason. With much of Faithlife's future developmental effort going into creating theses new datasets, I hope they aren't shooting themselves in the foot by creating a reputation of delivering unreliable and missing data.
Francis said:Is Logos so broke it must release stuff asap or die and thus considers it better overall to release incomplete or error-filled, bug-ridden products?
I do understand the need to expand and broaden when you see that you may be about to saturate your market. They may not be broke, but they do need to keep finding ways to generate income. (But it probably needs to be done without alienating your customers.)
MacBook Pro (2019), ThinkPad E540
0 -
Having not worked at Logos, one can only surmise. But it sure looks like 'deja vu'. The boss and team put together the next dreams. Everybody gets to work, with varying degrees of success. The boss notices a competitor is releasing the next major version. Asks the teams 'can we do it??'. Of course the teams can (though they really can't). And didn't.
The sad part is the misplaced dreams. I was really hoping the Logos databases, the Proclaim platform, the Faithlife community product, the mobiles, and oh yes, the dating game ... the ecosystem (as Logos calls it) ... had great promise. But each piece is not well designed and always incomplete.
Yesterday and this morning I felt really sorry for the Reverend that just wanted to delete his faithlife group, after giving up. No one seemed to know how. No manuals. I would have like to have helped. The Logos guy answered they'd try to do better ... but no answer for the Reverend.
The Logos6 release I hope won't be repeated in Logos7; too much product that was not complete at sell-time. Pressure now to produce it. Requests for quality? I haven't even bothered to see what I bought (since I can't find it!).
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
If the situation described by the OP is true, then it is sad indeed.
The quality of the tagging, linguistic analysis etc. can be evaluated objectively by folks trained in those areas. If there are major errors, there's nowhere to hide.
0 -
Todd Phillips said:
I hope they aren't shooting themselves in the foot by creating a reputation of delivering unreliable and missing data.
Is not this already happening? There is also a reputation of greed floating around.
Todd Phillips said:without alienating your customers
Here is a sad fact of my experience. I have used Logos since version 2. Over the years, I have "sold" Logos right, left and center. But in the last while, this has changed. I use Logos because I must, not because I want to. But I am no longer an enthusiastic advocate of Logos. There is a language version coming out on which I did translation. Considering what I know of it, I have not and will not recommend it to anyone. In fact, this was part of what I alluded to earlier: I was at first proud of the idea of helping it be available to more people (and bringing better Bible study to them) but now I would be ashamed to be associated with that version.
I am not saying this to bash Logos needlessly. I feel that this is a bad turn but it can be reversed. I sure hope so.
But I do wonder, in the present estate and in light of all that has been going on, what user polls would indicate if the simple question was asked where only yes or no was allowed as answer "Do you [still] trust Logos?" I hope Logos is listening and more importantly, hearing.
0 -
Francis said:
what user polls would indicate if the simple question was asked where only yes or no was allowed as answer "Do you [still] trust Logos?"
YES!
Logos 7 Collectors Edition
0 -
The question was not what individual answers might be but what the overall picture would look like. If the individual answer is yes for you, I'm glad for you. We may not have the same needs, experiences or simply responses.
0 -
Francis said:
The question was not what individual answers might be but what the overall picture would look like.
I am inviting as many individuals as want to to participate in your one word survey. Polls are composed of idividual responses so don't devalue my response.
So how about it Logos users? Do you still trust Logos? Yes or No.
Logos 7 Collectors Edition
0 -
To be fair the poll question is "do you [still] trust Logos?"
I wrote it in square bracket because it may or may not be significant to all users depending on the duration of their experience.
0 -
Depends!
0 -
That's not the most helpful question. The issue is the balance of work on new features vs work on fixing bugs and making things work well.
0 -
As I'm reading this, Roosevelt's line, "the only thing we have to fear is fear itself" comes to mind -- not necessarily for anyone else, but for me. I've always trusted not only in the company and the product, but in the customer base as well. My overly large investment was partially justified by the fact that Logos will be around long enough that it'll run on computers long after I'm gone: Windows 9, 10, ... 45, etc.
But what if the fear/distrust becomes widespread enough that even donating my library to a church in, say, 2027 would be like donating an Apple II to a church: Why would we want that old thing?
I hope that doesn't happen.
0 -
Rene Atchley said:
I find it incredibly frustrating to post on this site. Every generalization seems to be opposed by someone somehow by injecting that "wow" that statement isn't true because it isn't true for me.
This reminds me of the time my son, frustrated with his mother and me, exclaimed, "You always exaggerate everything I say!" (He immediately realized what he had done and started laughing.) Is not "every generalization" a generalization?
In all seriousness, I did not mean to upset you. If you read to the end of my post, you'll see I tried to make the point even stronger. A true monopoly has little incentive to improve. Logos has competition, at least at some level, and yet it still is challenged in producing equivalent core functionality (at least for me, it's core - working with the biblical languages).
My apologies...
Donnie
0 -
0
-
Lew Worthington said:
what if the fear/distrust becomes widespread enough
I agree with your sentiment. I made my sentiment known earlier that there was, in my view a regrettable state of affair, but that I see as reversible and for which I was making a plea. My rhetorical question (about the poll) has been turned into a pointless confrontation...
0 -
Todd Phillips said:
I do understand the need to expand and broaden when you see that you may be about to saturate your market.
I have gotten the impression that they are a company that often reacts and operates out of fear. I am probably wrong.
Looking at my context, there is absolutely no saturation. Out of six pastors in my local group, I am the only one who uses Logos. Another chose Accordance the other do not use anything. And of the community I serve, no one uses bible software. There is plenty of room for growth in my context.
However, as I have said before, I cannot recommend Logos or any FaithLife products because of the quality issues. I would love to be able to "sell" Logos to the people I serve. But, I can see it coming back to embarrass me and having to spend too much time supporting them. (Not to mention that it is not user friendly and has no reference manual. I have been endeavoring to understand the new Milestone search feature. Thank goodness for the community here for helping. But surely there has to be a way to make that into a menu item that is more accessible to someone without mastering search syntax strings. Just my experience.)
Do I trust Logos? As a reader, yes. But, I have gotten into the habit of verifying its result otherwise, or doing them the hard way because it is just as easy. So, maybe the answer is, no. I buy a lot of resources, but I have been contemplating their value since the release of 6.
Like Francis, I want Logos to succeed and grow.
EDIT: On further reflection, I would like to make an amendment/confession. I think I was quick to answer due to many of my resent frustrations with the software, particularly since the 6 release. I think it was unfair of me to insinuate that I do not trust Logos (or to outright say it). I do not know enough about its search perimeters, abilities, etc. on the datasets to even begin to form an opinion, which is what the OP was referring to, I think. My concern for QC centers more around the reliability of the program itself to run, to do its job timely, the UI etc., which I hope came out a bit.
Thank you John Fidel (below) for your perspective and my apologies to the community. I am trying to be more positive towards FaithLife in the forums, yet maintain the truth of what I see that they may improve. I particularly want to be sensitive to those that do the hard work at FaithLife in producing the software and resources. I know they want to do a great job and it is a complex, never-ending job. My apologies to you as well if I have seemed overly harsh. As a pastor, most days we hear more criticisms than anything. I forget that others deal with that as well. Thanks also to Sean for his words.
So, in terms of the quality of the "programmed" application and services, I think there is much room for improvement (hence my difficulty to recommend it. Though maybe I will give it a shot). I know it will get better. Though, I think I would have rather waited for a better product.
I do love Logos. I am in it many hours a day. Even after all these years it is mostly for reading. I think one of the frustrating things is the steep learning curve to really get at the power.
Michael
0 -
Francis said:
My rhetorical question (about the poll) has been turned into a pointless confrontation...
I guess if everyone answered "No" there would be no confrontation. Declaring a poll pointless is one way of retreating from your rhetoric.
Logos 7 Collectors Edition
0 -
Content deleted
0 -
I lead the Content Innovation (CI) team: we're primarily responsible for the data sets and linguistic tagging on the biblical text that ships with Logos. So it matters a great deal to me that we produce the best value we can for our users in these areas, and it gets my attention when people raise questions about quality control. I can only speak to the areas I'm responsible for (which don't include the translation project the OP worked on).
Francis said:For instance, searches of cultural concepts or other tagged entities shows a lot of gaps and inconsistencies in tagging. This is a quality control problem.
I don't know how to respond to this without more specifics. But let me provide some context on the Cultural Concepts project from CI's side, since ultimately I'm the one who's responsible for it. I'm extremely proud of the work our team did here:
- We created a hierarchical categorization of more than 1100 cultural concepts that are relevant to the biblical text (the Lexham Cultural Ontology, or LCO).
- We annotated both the Bible and about 15 other ancient text resources with these concepts: I estimate about 5 million words. Note there's no automated process: our team had to read each text and analyze it from the perspective of the LCO.
- We linked concepts to key dictionary articles explaining their background.
- We coordinated the LCO concepts with our Topic Guide and Bible Sense Lexicon data to "connect the dots" with other Logos data sets
Every data set project we undertake has to find the right middle ground between benefit to users (things like search, organization, and ultimately insight) and cost to Logos. The primary benefits we imagined for the Cultural Concepts are:
- Providing easier access to cultural background information for the biblical text
- Making it possible to find secondary literature on the same concepts (If you browse the Cultural Concepts section in Factbook for Covenant, you'll find a lot of material you'd never find by searching for the word "covenant")
- Organizing the cultural concepts themselves (Covenant is a sub-concept under Contracts and browsing the glossary may help with exploring the cultural background of the Bible)
Our users are ultimately the ones who have to judge whether we delivered these benefits or not: from my perspective, this is not only a successful feature in our product, but a novel and significant contribution to the field of biblical studies as a whole.
As for cost, we've invested roughly 20 person months of effort in building this data set and annotating texts (we even hired three interns this past summer to help get all the annotation done for Logos 6). All of the people doing this work had graduate level training and consulted existing academic resources in their work.
I've gone into the details on this one project (we have many, of course) to try and provide some perspective on our process. Is this data set perfect? No, and it can't be: there's no objective standard to compare it to, and nobody's ever done anything quite like this before. Is the annotation "done"? Again, no: we prioritized our annotation for the resources we thought would be useful to the most Logos users, but there are dozens more resources we could annotate. Deciding how many more we ultimately do will be determined by trading off benefit and cost, for this project and others. Is the ontology complete or comprehensive? Not really: while we did our best to cover the most important biblical cultural concepts, any model can always be enriched and extended.
Instead, the criteria we typically use to assess data projects like the LCO include
- Utility: does it help users accomplish their goals?
- Consistency: have we been consistent throughout in our annotation? Are similar passages annotated the same way?
- Validity: would external observers who understood the issues be in substantial agreement with our judgments?
- Sufficiency: have we provided enough additional data to make the whole useful?
These are all significant challenges, and (like the original analysis itself) none of these criteria can be automatically checked (though we do check as many things as we reasonably can). But I stand by my claims that we did our best to meet these criteria given the effort we invested, and overall, we've provided a great deal of value with this data set.
If you find errors in this or other CI data sets, please email data@logos.com or post on the forums with the specifics, and we'll do our best to fix it.
PS: David Witthoff, who was the project lead for the LCO, has a series of helpful posts on his personal blog about using the LCO for sermon preparation, starting with Logos 6 and Sermon Method.
0 -
Thanks Sean - Enjoyed your post and learned something in the process
0 -
Hello Sean,
Thank you for your reply. I appreciate the information it provides and the time you took to write it.
I understand your request for specifics, but I cannot be too specific here without taking too much time to go back in history and trace back all the various issues that have arisen. But it has been an accumulation. I must say, and I understand that this is not your department, that I was particularly disillusioned by the other project I alluded to. However, I do not want to say more about it here because I don't want to start mentioning specific people or give enough information that it would point to specific people. This is not my goal. My goal is to call for improvement, not to burn someone at the stake.
As an instance, a couple days ago there was this thread about looking for prophecies and/or feasts in the NT, when avenues were found that were thought to be able to produce the expected results, hits were missing because they were tagged differently. For instance, Passover in John was not tagged as a religious feast, but as Passover. Someone who would look for mentions of feasts in the NT and used that cultural concept search would miss a number of hits. This may be a matter of choice, though the choices and the underlying logic is often not obvious and this is the kind of thing for which it is difficult to find documentation.
In another thread -- we're leaving cultural concepts now -- we were talking about searching labels and tags. Again, lots of problems there.
These are the two most recent examples on my mind, but there are others who even in this thread have echoed my experience and perhaps can remember other specific examples. I would think that beyond that, there are very many expressions of frustration about features or data not being complete. Perhaps I am venturing too far (and I am willing to be corrected) but I think that those I hear from who do work that require accuracy generally know that they cannot fully rely on results.
Features are also delivered incomplete, for instance label searches in annotations or user document searches, or narrative outlines. Resources come out untagged, etc.
I appreciate your answer then and certainly I do not mean this as an attack on you and your team, but I really don't know why there are so many errors all over the software, such a lack of proper documentation and so much incompleteness. I spend way too much time looking for solutions to fix or circumvent these problems, instead of just getting the work done!
This is not to say that I do not value what does work. When it works and is accurate/dependance or complete, it's great.
0 -
First I owe Sean a HUGE thanks. You interviewed me a couple years ago when I applied to work with Faithlife (then Logos) as a Lexical Data Curator. You did not hire me, and it is for that I offer a heartfelt thanks. I was called a couple weeks later as a pastor in a wonderful church! The Lord is good!
Now to the topic at hand. I think that perhaps the question of whether one "trusts" Logos/Faithlife is a bit of a misdirection. The Logos Library system has always been just that: a way of collecting and accessing published material in a digital format. Over the last 15 years I have seen huge changes in how that information is accessed, but that has not changed the basic fact that as a library system Logos is in my opinion, unmatched.
The extent to which one chooses to accept/believe/trust the content stored and accessed in the Logos library should always be a matter of wisdom and discretion. As I read different authors, some I develop a respect for and have an increased receptivity to what they have to teach, others I remain more skeptical of if I find their methods or conclusion faulty or lacking.
When it comes to the content that Faithlife has included in their library system I think we need to use the same wisdom and discretion when considering the quality of information they have gathered and published that we do with other content producers/authors. However I think it is important to maintain the distinction between the library system we use to access the library and the production of content itself.
Yet here is where the content from Faithlife is different from any one of the thousands of books and journals I have: it is dynamic. Where there are errors, they can edit and update it where those errors are pointed out or as they become more experienced and identify their own shortcomings. With electronic versions of a "normal" print library, we have to wait for years as the authors themselves grow and change and then write the second edition, third edition, etc. and then be published. With Faithlife as a content producer, they have the ability to make that change and push it out (publish it) tomorrow! What a blessing!
If you are wary or disagree with the content that Faithlife has included in their library system, certainly don't use it. Make use of the resources you are confident in and do trust. However, does disliking Faithlife's content justify abandoning the whole library system? I personally do not think so.
With regards to the library system (how we access the data) certainly tagging is an important part of how data is found and accessed. It seems though that new features provide additional access to existing data so that we have different ways to dig deeper into what we already had. I haven't found anything yet that keeps me from studying and finding the information in the ways I have grown accustomed to. We can hope that Faithlife will continue to expand on what they have already done so that datasets are more complete, but again this does not cripple study, it simply limits the usefulness and viability of the new features until those features are more fully developed.
0 -
Tim said:
If you are wary or disagree with the content that Faithlife has included in their library system, certainly don't use it. Make use of the resources you are confident in and do trust. However, does disliking Faithlife's content justify abandoning the whole library system? I personally do not think so.
With regards to the library system (how we access the data) certainly tagging is an important part of how data is found and accessed. It seems though that new features provide additional access to existing data so that we have different ways to dig deeper into what we already had. I haven't found anything yet that keeps me from studying and finding the information in the ways I have grown accustomed to. We can hope that Faithlife will continue to expand on what they have already done so that datasets are more complete, but again this does not cripple study, it simply limits the usefulness and viability of the new features until those features are more fully developed.
Thank you for your thoughts Tim. I appreciate your desire to be gracious and provide perspective, but I must say I totally disagree with you here.
That your study needs or habits may be such that you can function the way you describe, I can envision. But Logos is now also a product for academics. We do a lot of detailed research and whatever we claim must be accurate. Keeping that in mind, new releases of Logos have each come with a set of promises. What I mean by promises is that features are marketed as reasons to upgrade and they tell you the beauty of all you are going to be able to do. You buy into it (literally) and little by little discover that features are incomplete. You think it's just bugs at the beginning, but then users who have been here since Logos 4 can tell you that while many problems have been addressed, many have also dragged on a long time. Tagging is a sore point. But in addition to this, you have to keep in mind that long-term users have invested a lot in Logos. There is the cost of resources, of all the upgrades but also the investment represented by trusting it with your study and data (all your annotations, all your markings, etc). Over the years, it's huge. The longer you have stuck with the company and the more this dynamic of incompleteness, promises lagging pattern continues, the greater the feeling of being betrayed with regards to the expectations that were fed by the promises and marketing messages and which you have paid for.
So yes, trust is an issue because I have trusted Logos during all these years, but I do feel that there has been a worsening of that dynamic (more stuff coming out more quickly yet incomplete). I was actually a bit shocked by the timing of the "what do you want to see in Logos 7?" question. I don't think I would be the only who, if things do not change, would take whatever marketing messages and promises are made when Logos 7 comes out with a degree of circumspection and if features of Logos 6 are not fixed by then, with a strong dose of cynicism.
0 -
Well, Sean, I took you up on your PS offer. I thought I'd get some free training from the horse's mouth: David Witthof
I did what he said to do, and nothing happened. He warned me not to type in 1 Samuel or 1st Samuel, so I didn't. I carefully copied what he said to type in: First Samuel. My Factbook refused to budge from Moriah (an earlier choice).
Then I looked at what his example actually delivered: 'Book of First Samuel'
Well, there you go! The trick is to not type the identification of the book ... Samuel, Chronicles, and so forth. I felt truly trained. So for my 'finals', I typed 'Peter' (for 1st Peter). I got another 'F'.
Yes, it didn't work and yes, I'm happy you guys worked so hard on WBC.
"If myth is ideology in narrative form, then scholarship is myth with footnotes." B. Lincolm 1999.
0 -
Francis said:
As an instance, a couple days ago there was this thread about looking for prophecies and/or feasts in the NT, when avenues were found that were thought to be able to produce the expected results, hits were missing because they were tagged differently. For instance, Passover in John was not tagged as a religious feast, but as Passover. Someone who would look for mentions of feasts in the NT and used that cultural concept search would miss a number of hits. This may be a matter of choice, though the choices and the underlying logic is often not obvious and this is the kind of thing for which it is difficult to find documentation.
Looks like this is the post you mean? I agree there's a data issue here with the ontology: we've categorized Passover as a kind of Organized ceremonial (which is true), but it's also a kind of Religious feast, and our data doesn't capture that. I'll add a case to correct that.
Examples like these are very helpful because they illustrate ways people use our data that we couldn't easily anticipate. As Mark Barnes noted in that thread (for a different case), errors and definitional problems aren't necessarily the same thing. Any time we make some analysis, we're necessarily joining or splitting things that others might not join or split the same way. But (using the criteria from my previous post in this thread) having Passover as a Religious feast would be at least more useful and more valid.
I agree we need to do more to explain the choices and logic behind some of our datasets: creating those introductions is on my list for this year.
Francis said:Features are also delivered incomplete, for instance label searches in annotations or user document searches, or narrative outlines. Resources come out untagged, etc.
We've made a fundamental business decision: not all data sets need to be complete to be useful. Your example of Narrative Character Maps is apt: we could withhold this data until we have character maps for every narrative book in the Bible. But in the meantime, if you're studying the first part of Acts, you might find this character map useful today (and we're actively working on the rest of Acts). We may decide some books aren't worth the effort for this kind of visualization (by contrast, we deliberately decided that we wanted our Biblical People Diagrams to cover every named individual in the Bible, even one of the 30 Zechariah's that's only mentioned once in the text). Some resources get a dozen or more different kinds of markup in our production process: more markup means more production cost, so others get less. These are business and editorial decisions: they're not quality control issues.
0 -
Denise said:
Well, Sean, I took you up on your PS offer. I thought I'd get some free training from the horse's mouth: David Witthof
I did what he said to do, and nothing happened. He warned me not to type in 1 Samuel or 1st Samuel, so I didn't. I carefully copied what he said to type in: First Samuel. My Factbook refused to budge from Moriah (an earlier choice).
Then I looked at what his example actually delivered: 'Book of First Samuel'
Well, there you go! The trick is to not type the identification of the book ... Samuel, Chronicles, and so forth. I felt truly trained. So for my 'finals', I typed 'Peter' (for 1st Peter). I got another 'F'.
Yes, it didn't work and yes, I'm happy you guys worked so hard on WBC.
We have a known problem with looking up Bible Book in Factbook (previously noted here, here. here and elsewhere, no doubt). We've got a much longer list of candidate articles for Factbook than previous guides, so we've still got some work to do in sorting and ranking the alternatives. Clearly Bible books shouldn't be buried down at the end of the list.
The best current workaround I know of is to use the Topic type to start searches for Bible books. So the results for "<Topic peter" includes First Epistle of Peter, though it's still much farther down the list than I'd like.
0 -
Like the OP I have a bit of experience with working with Faithlife on a specific product -- much smaller and much less important in the overall scheme of things. However, I also experienced a pre-determined timeline resulting in a release with embarrassing errors. Unlike him, I also had a basic timeframe in which corrections would be made and knew that the time frame was short partially by a redesign to get it right ... and the redesign was brilliant even if the data in some cases was flawed.
I think there are three separate issues in the OP's original concern:
- First, there is the tagging that primarily assists the average user find things in their library and is unlikely to be used in an academic sense that requires 100% accuracy. Here, waiting for additional tagging and a certain margin of error is permissible. What is missing is an easy way to tell what one needs to check manually because it is not tagged - this drives me crazy.
- Second, there is the tagging that is apt to be used in an academic setting - primarily the tagging behind the Bible, clause, morphological and syntax searches. Here the tagging needs to be complete and very accurate - and the definitions of how things are tagged must be explicit and accurate. Here I see many of the complaints in the forums based on an incorrect assumption that linguistics is a precise science so they fail to understand why different resources give different results. Here I am often frustrated that Faithlife has not gone with a specific linguistic theory but invented their own (fine to the point) without documenting precisely what their theory is (that's the rub). The release of glossaries with L6 was a step in the right direction but ...
- Third, there is the tagging where Faithlife is truly breaking new ground e.g. the Bible Sense Lexicon. Here I think we need to cut them some slack because it really takes thousands of knowledgeable users using it and providing feedback to "get it right". If, over the next decade we don't see gradual improvement in the usefulness, then we have reason to complain.
We also have to remember that with items like the Factbook, Faithlife runs into a serious problem with differences in vocabulary across different schools of theology. In topics.logos.com they had created the opportunity for users to supply synonyms but in general they got little feedback. We, as users, need to hold Faithlife's "feet to the fire" in terms of the order in which tagging is done.
- No matter how much your library is limited to one tradition, we need to demand that a wide swath of denominations be included in the early tagging e.g. Orthodox and Jewish terminology is sadly missing from the LCV. I would not be surprised is other groups such as Anabaptists, Restorationists, Quakers ... also find their "native" terminology missing.
- If your work requires completeness or if your personal favorites are not tagged (think outlines, journals, sermons ...) you need to push for the ability to have a collection of "not yet tagged" that require different search arguments or manual searching.
- When you find errors in tagging, missing tagging et. al. you have an obligation to report it. Faithlife cannot identify and correct the errors that annoy us if we don't report them.
- If you don't understand the purpose or meaning of Logos tagging (recent examples: BSL, propositional outlines) keep asking questions on the forum. Eventually enough people understand to be able to answer the questions or Faithlife discovers they need to write an introduction or blogs (think of the Discourse Analysis material we now have).
- If you feel the tagging is "too academic" and you need "pastoral/teaching" tools, be precise in what you want: For me it would be tagging of rhetorical devices, microgenres and named relationships between verses (cross-reference and the myriad of other verse lists we have in Logos/Verbum).
Remember that software development is a two way street - the Faithlife software environment will continue to improve only if the users let Faithlife know what does not work, what can be improved and what is needed. That end is our responsibility. Implementing it and broadening our vision is Faithlife's responsibility. Whining is simply poor stewardship of time.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
Sean Boisen said:
We have a known problem with looking up Bible Book in Factbook
... and it should be fixed in 6.0b (which is shipping soon).
0 -
Francis said:
I think that quality control is the leading issue with the ongoing development of Logos.
That has been a concern of mine for some years and particularly with software for Logos 5.1+, Logos 6.0 Beta and 6.0a, 6.0b betas. Complex datasets have always been a concern starting (for me) with OpenText on Logos 3, then Clause Search, BSL, People/Place/Thing tagging, and Referent tagging in Logos 5 followed by the new datasets in Logos 6. OpenText was so buggy it probably drove Logos down the path of doing its own datasets and being able to manage both development and maintenance (parallel to managing their own bibles). LCV (topics) in Logos 4 was the signal for Faithlife wanting to manage data for a better experience in searching. Now we see LCO (Cultural Concepts), Literary Typing, Semantic Roles and Case Frames which are quite complex in the way they are applied to resource text.
The question of trust arises in the design and intended application of datasets. Whilst quality control (QC) is important in their management, most errors are due to human error/interpretation in applying the (bible-based) data. Do I trust Clause Search -- No! Do I use it - Yes. Would I recommend it - No.
The QC of the software and the direction of development showed a marked down turn in latter part of Logos 5. The Logos 6.0 beta showed that Faithlife could not keep up with the bug reports, or wanted developers to interpret what was important to them. The lack of response in the forums and the silly errors in the 6.0a/6.0b betas is indicative.
Dave
===Windows 11 & Android 13
0 -
MJ. Smith said:
- 3. When you find errors in tagging, missing tagging et. al. you have an obligation to report it. Faithlife cannot identify and correct the errors that annoy us if we don't report them.
- 4. If you don't understand the purpose or meaning of Logos tagging (recent examples: BSL, propositional outlines) keep asking questions on the forum. Eventually enough people understand to be able to answer the questions or Faithlife discovers they need to write an introduction or blogs (think of the Discourse Analysis material we now have).
Been there, done that, MJ. Some users will keep quiet after a while; some users will make a personal spending decision. Expecting regular users to take up the slack is just too much.
0 -
Lee said:
Been there, done that, MJ. Some users will keep quiet after a while; some users will make a personal spending decision. Expecting regular users to take up the slack is just too much.
Having spent a few decades teaching the "average departmental secretary/reception/payroll clerk/etc." that the only way for me to know how they used the system (vs. how central offices assumed they worked) was for them to tell me, I know the message needs to be hammered home consistently. But until a min-reader function is available for programmers/analysts ... the end user must recognize their role. The alternative is a few thousand well trained beta-users and a year to two or three of more of time (consider Microsoft). Or as in some software tools, only the top few $$ users have their bugs fixed and improvements considered.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
What I meant to say was: to expect regular users to constantly give feedback about obvious QA issues, is a dangerous game for any business to play.
0 -
0 -
Francis, I agree that the QA is inadequate. The number of resources that have been released then sent back for basic coding is not acceptable. Examples" the BCP without the Psalm links (Latin names rather than Psalm number), Faith of the Fathers without the link to the index that provides the primary value, the number of liturgy books not tagged to appear in the liturgy section ... those are just ones I complained about. I also agree that the Faithlife fall back on an overall error rate and their lack of time to manually check each resource is a poor excuse. At the start of each resource's development a check list of required flags for processing, indexing noting special features etc. etc. ought to be made and ought to follow the resource and manage the workflow. But I do think we have to have realistic expectations and recognize our own position in the process.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
I think MJ's posts are a pretty good summary of how I see the situation.
I would also suggest that it's important to make specific critiques. You would never really hear someone in Academia asking the question "Do you trust the Anchor Yale Bible Commentary Series"? The immediate answer to such a generic question would be, "Which volume?" Same goes with these datasets, as there are varying levels of quality and completeness.
Maybe the simplest change that Faithlife could do to help on a global scale throughout their products would be a company-wide commitment to document all datasets, past, present, and future. All datasets should have authors who sign off on them publicly, ideally including all names, even of interns. All should clearly explain their methodology, presuppositions, and (just as importantly) their limitations!
For example, some of the datasets Faithlife has produced are incomplete - that should be documented. It's not something marketing needs to scream out from the rooftops, but in the documentation that should come with the dataset it needs to be clear. And, anyone who has read much academic work knows that explaining openly a works own limitations is standard fare. Doing so gives credibility, rather than taking it away... at least in the academic world, and that is the world such documentation needs to be written for.
If a dataset has no clear place to put the documentation (e.g. Morph, Clause, etc.), Faithlife should create resource(s) dedicated specifically to documenting those datasets.
When one knows the presuppositions, methodology, and limitations of any work, it can be used profitably. As well, knowing that everything that is produced, and its authors and limitations, needs to be documented, can only be of help to both Faithlife and the specific authors to make sure quality is top-notch.
0 -
Dave Hooton said:
LCV (topics) in Logos 4 was the signal for Faithlife wanting to manage data for a better experience in searching. Now we see LCO (Cultural Concepts), Literary Typing, Semantic Roles and Case Frames which are quite complex in the way they are applied to resource text.
The question of trust arises in the design and intended application of datasets. Whilst quality control (QC) is important in their management, most errors are due to human error/interpretation in applying the (bible-based) data. Do I trust Clause Search -- No! Do I use it - Yes. Would I recommend it - No.
The QC of the software and the direction of development showed a marked down turn in latter part of Logos 5. The Logos 6.0 beta showed that Faithlife could not keep up with the bug reports, or wanted developers to interpret what was important to them. The lack of response in the forums and the silly errors in the 6.0a/6.0b betas is indicative.
There's another piece that needs to be considered in your history --> the move towards giving as set data the results of what had been simply user techniques prior to computers. Many of these datasets, even morphology, hide the ambiguity that was why a scholar used the technique. Add to this the Faithlife attempting to be all things to all people - when we don't agree on a base text, don't agree on the "literal" meaning, don't use the same terminology -- I'm not sure there's a gold standard to measure the Logos data against. Which means Faithlife must be very explicit about what their coding means and how ambiguous cases are resolved.
Edit: I see Fr. Devin Rosa made my point better than I while I was writing this.
Orthodox Bishop Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."; Orthodox proverb: "We know where the Church is, we do not know where it is not."
0 -
This hardly addresses all of the concerns that have been mentioned, but perhaps we need something like the Bible Sense Lexicon to help us find how something may have been tagged (ideally, it would it not make a lot of sense if BSL and tags were consistent or shared the very same base?). I did a quick check to see whether it could be used this way (using the feast search in John as a test case). I searched "feast" and got "feast (meal)" proposed to me (with Passover as sub-category). I right-clicked on it to see if I could search it and I got this:
Apparently, there is only 1 feast-meal (not to mention, Passover) mentioned in John!
0 -
Fr Devin Roza said:
I would also suggest that it's important to make specific critiques. You would never really hear someone in Academia asking the question "Do you trust the Anchor Yale Bible Commentary Series"? The immediate answer to such a generic question would be, "Which volume?" Same goes with these datasets, as there are varying levels of quality and completeness.
Devin, I hear what you are saying, but I am tired of always having to mop around. It is an important part of my complaint that there is way too much of that needed to be able to use the software as it is supposed to (advertised, promised). I am investing time in this thread and in other discussions because I hope it can make a positive difference, but I am also appalled at all the time cost of dealing with constant problems, reporting, searching for answers, etc (which sometimes yield solutions, other times just impasses). In my view, there has been sufficient documentation of all kinds of specific problems to show that there is a broader problem at play. Unless the root causes are addressed, we will always be spending way too much time in an endless process of putting out small fires that keep popping up everywhere. I don't think this is a situation we want to settle for as normality.
0 -
I hear what you're saying, I just guess I don't see the situation as being quite like you paint it. I remember, for example, that long thread about the reliability of the Morph info in Logos. The only really concrete example in the thread ended up being one where Logos was following the standards set by the academic Lexicons, while Accordance and Bibleworks were apparently following some sort of in-house classification, that is used in the popular entry-level Grammars but not in the academic Lexicons. Had they documented that somewhere? I kind of doubt it.
Most of the problems listed in this thread here seem to me to be more related to documentation or debatable opinion / frameworks than real quality problems.
That doesn't mean there aren't datasets that are poor. The LiteraryTypes dataset is very poor as a dataset. But that is one that wasn't developed in house, but just imported from someone else's work that was never designed in the first place to serve as a dataset. Should this really be a dataset? Maybe not, as it currently stands. Would I like to see Faithlife redo this dataset from scratch, using a methodology optimized for dataset use? Absolutely. But even as it is currently, I've found that as long as I understand what it was actually designed for, its limitations, etc., it can and has been of use to me.
When writing Fulfilled in Christ, I used a multitude of Biblical Concordances / indexes to prepare the sections on OT and NT Background. The best I've found is Collin's / Rogets. But one thing that struck me was how none of them were really "complete". They were obviously all the result of years of meticulous work, and some of them are truly wonderful works of scholarship, but even so the only way I could get results I really felt confident with was by using multiple concordances together, plus manual searches. But that doesn't mean those are bad concordances. It's just the nature of concordances / indexes, for many reasons.
I consider datasets along the same line (and, well, actually I consider commentaries, and lexicons, and doctoral dissertations, and... on the same line as well, and even Morph datasets, etc. All require subjective human judgment that is necessarily limited, as MJ has been pointing out).
Dont' get me wrong - I fully concord with you that there needs to be improvement, and was really glad to see this thread started. I have also been pushing for improvements in this area. But Faithlife is a very large company, with lots of different internal groups working on different things. I think that many of those groups do excellent work. Some don't. And there are some things (like documentation) that all the groups need to work on, or that need to be implemented from management.
In summary, I just would suggest that identifying specific problems (even when on a general level) is the best way to push for real world solutions.
0 -
Fr Devin Roza said:
I remember, for example, that long thread about the reliability of the Morph info in Logos.
Yes, if it is the one I think, I had started it. This was a case were I made a mistake. This indeed can happen: criticizing out of ignorance. Sometimes it is the user's fault (it was mine in that case) and many times, the user is ignorant because of the lack of proper documentation. But for all that, it should not cast doubt on whether there is really not all that much errata in Logos and that most of this consists in philosophical disagreement or the vissicitudes of resources/datasets inescapable limitations. I think that there is sufficient, specific evidence of this to be found all over the forums.
Fr Devin Roza said:Most of the problems listed in this thread here seem to me to be more related to documentation or debatable opinion / frameworks than real quality problems.
I find this characterization too limiting and in that sense, not true. I produced two concretes examples above. One was of advertisement vs actual experience (the Journal section). I did not add much comment there: everyone knows how this has panned out. The other just occurred even as I was exploring a possible avenue of betterment (using the BSL in relation to tagging). Surely, this is not an inescapable type of resource limitation. How complicated is it to tag Passover occurrences in the Bible? Either you tag them or you don't, but if you do, it must be more reliable than this! What is even more telling about this specific example is how I encountered it: one does not need to go very far to bump into something like this. It happens all the time. That's the problem. Same thing when I sought to other day to explore with Dave Hooton ways to make user tagging workable by using Labels. Bump, bump, bump. The problem started with something that does not work right and as alternative solutions were explored, they too were found not to work right! Right does not mean my definition but doing what it is advertised to do.
So, while I accept your description of the limitations of resources which can never be exhaustive, I don't think it is quite the proper analogy for the problems I seek to address here.
0 -
MJ. Smith said:
There's another piece that needs to be considered in your history --> the move towards giving as set data the results of what had been simply user techniques prior to computers.
Yes, I had to give a potted history in order to meet an important (family) deadline! The debacle of OpenText in Logos 3 illustrated how a potentially good design (by OpenText.org) can be ruined by a lack of attention to detail (the morphology was improved in Logos 4 but I can't fully trust the method). The ambiguities of Case Frames e.g. 'love' as an Agent, leads to a lack of trust, but I don't believe that the method could help me to "better understand the verb in its context"! It's far too mechanical.
The ultimate frustration is that Faithlife have been doing resource metadata for over a decade and it still isn't consistent and right! I don't look forward to this as an example for correcting the newer datasets!
Dave
===Windows 11 & Android 13
0 -
Francis said:
But for all that, it should not cast doubt on whether there is really not all that much errata in Logos and that most of this consists in philosophical disagreement or the vissicitudes of resources/datasets inescapable limitations. I think that there is sufficient, specific evidence of this to be found all over the forums.
I'll believe it when I see it.
I guess I just don't remember seeing all that many posts regarding morph data, and it was kind of strange to have such a long post with so many specific posters without any specific examples arising. I certainly haven't encountered many problems myself and have been using these databases for years daily.
Remember as well that Logos has over a dozen different morph databases. If there are a couple of posts per morph database on the forums, that might look like a large number, but in fact would indicate things are probably sailing pretty smoothly, or at least are about like in Accordance or Bibleworks, where they occasionally ship updates and minor corrections to their Morph datasets as well.
Francis said:
The other just occurred even as I was exploring a possible avenue of betterment (using the BSL in relation to tagging). Surely, this is not an inescapable type of resource limitation. How complicated is it to tag Passover occurrences in the Bible? Either you tag them or you don't, but if you do, it must be more reliable than this!
I was kind of wondering what you were getting at with that post. I looked through the tagging of Passover in John, and as best I can tell it is all correct. Run a BWS on πάσχα if you want to see for yourself the ways the senses this word is classified with. The key is to open up the Senses section and then look at the actual definition each sense is given. The verses in John seem with pascha seem to be correctly classified to me.
Now, one area where the Senses dataset needs to be improved is documentation. For example, many abstract nouns are not classified in the Sense Lexicon because Faithlife is still figuring out what they want to do with abstract nouns. OK, great, but that needs to be documented in some resource that accompanies the Sense dataset.
The same is true for the Journals section, if I understand correctly. It works with manual tagging of Journals. The problem, I believe, isn't so much the tagging that has been done is incorrect as as the lack of tagging of many journals. The list of tagged journals should be documented, so people know what to expect. That being said, that Journals ad should be reworded to remove words like "all" that make it imprecise.
0 -
My friend, there are different issues that been have mentioned here as part of an overall frustration with the number of errors, the incompleteness of features when delivered and features that don't work. I did not focus on morph errors in this thread: you brought that up. My response to you was not to say that morphologies are rife with errors, but that Logos is rife with errors.
With that in mind, I really do not understand the way you are answering what I wrote. How does the BWS route showing what it does invalidate the fact that 1) there is a BSL in Logos, thus it is a legitimate avenue of searching; 2) I checked whether it would perchance relate "feasts" as a sense to occurrences (that is, make them searchable); 3) I type "feasts" (a natural language query) and do find an entry: feasts (meal) of which "Passover" is a sub-category. 4) I check whether it is searchable and it is. 5) I search for it in John and find only one instance. So, in what way is finding the right results using BWS somehow invalidating the problem that I have encountered using BSL? I don't get your approach.
Same concerning Journals. I did not say that Journal tagging was incorrect. I produced the Journal Section example to show that there is a mismatch between advertisement and actual experience, an all-too common problem in my view. It has improved a bit as of late, but at first, many users found that their journal library produced no hits even when searching vast collections of journals and broad sections of the Bible. This clearly did not correspond to the messaging that essentially said: buy Logos 6 and you get this! It did not say, you might get this in 2015 or later. I did mention tagging errors with regard to other examples. But please don't blend my examples! They are different illustrations of different issues that constitute overall what I think is a more systemic issue ( = the overall number of lack of documentation, errors, incomplete features, broken features = a greater emphasis on production/promising at the expense of quality/delivery).
0 -
Totally agreed about Journals!
Agreed as well I brought up Morph search - but I did so as an example of how it's important to distinguish between different datasets, and specific problems, that's all. [{]
Regarding the BWS, I was just pointing out that the detailed information about the search you ran is found there, in the BWS (or in the Bible Sense Lexicon). So, if the results of a "Sense" search don't make sense, that is the place to go to ask if there really is a problem with the data, or if I need to approach the search differently. When one does that for the word pascha, it turns out that the tagging in the Gospel of John was correct, the problem was expecting something the information wasn't designed to give. i.e. the search you ran wasn't a search for "all feasts" but specifically for "the meal eaten at the Passover celebration", and that as distinguished from the sense of pascha as "a seven day festival..." or as "the lamb killed and eaten during the Passover feast". Knowing this, the results given in the query are clear and correct, even if they weren't what you were hoping for. That just means you need to search for what you were looking for in a different way.
And, finally, thanks for bringing these issues up and continuing to push for them! I think all in all we basically agree on this stuff. I'm sorry if my pushback didn't come across the right way - it really was thought out to make your push more effective, by trying to identify better those specific areas that are in need of work.
0 -
Thanks Devin. Good points about BSL too!
Here again it shows how more training/documentation could help a great deal (where this is the problem). It is not immediately clear that 1) it is not the Passover itself that should be categorized as a feast-meal; 2) that in an instance like Jn 6:4 this connotation should not be tagged considering what follows; 3) that other feasts (most involving eating) should not included just because the activity of eating is not specifically mentioned there. To me, it feels like you can't know what you will get, you can only retrace the meaning of what you get once you get the results. By then you discover that what you thought you searched is not what you were allowed to search and that you must try something else. It makes it difficult to predict accurately what would be the right search terms and tool for one's query. There was the same problem with searching for feasts in Jn as a cultural concept. In the end, one has to search FIRST for how what one is looking for is tagged as in the datasets BEFORE one can actually search for it. So I can use BWS and look at what senses are attached to it. But looking for all feasts originally, I have no guarantee that the datasets findings related to pascha will help me locate all the feasts in John. The same kind of deal happens when one has to use factbook, topic search and lookup to find an entry. It's complicated, somewhat (not completely) opaque and quite time-consuming.
Yes, I keep replying because I want to be a badger here, not drop the ball. I feel that this is hardly an isolated expression of such sentiments, but that the response from Logos has not been commensurate yet. I appreciate the posts by Sean and Bradley (yes, I do!), but they only speak to specific issues isolated from the overall situation. And not rarely, remedies are often left in an indeterminate (indefinite it seems at time) future. "It's on our long list" (question: why is the list so long in the first place?). Will we get proper documentation in 2015 or should we expect Logos 7 to come out first?
I also want to acknowledge a hierarchy of problems. I understand that BSL is innovative (in a very good way) and so tricky. Fine. But it is the accumulation of all the problems that has become too much. But even among those of us who are fed up and irrate, I think Logos should realize that we want the company to succeed! It's a win-win prospect!!!
0
