Quality control in Logos 6 - a new perspective

Like the OP I have a bit of experience with working with Faithlife on a specific product -- much smaller and much less important in the overall scheme of things. However, I also experienced a pre-determined timeline resulting in a release with embarrassing errors. Unlike him, I also had a basic timeframe in which corrections would be made and knew that the time frame was short partially by a redesign to get it right ... and the redesign was brilliant even if the data in some cases was flawed.

I think there are three separate issues in the OP's original concern:

1. First, there is the tagging that primarily assists the average user find things in their library and is unlikely to be used in an academic sense that requires 100% accuracy. Here, waiting for additional tagging and a certain margin of error is permissible. What is missing is an easy way to tell what one needs to check manually because it is not tagged - this drives me crazy.
2. Second, there is the tagging that is apt to be used in an academic setting - primarily the tagging behind the Bible, clause, morphological and syntax searches. Here the tagging needs to be complete and very accurate - and the definitions of how things are tagged must be explicit and accurate. Here I see many of the complaints in the forums based on an incorrect assumption that linguistics is a precise science so they fail to understand why different resources give different results. Here I am often frustrated that Faithlife has not gone with a specific linguistic theory but invented their own (fine to the point) without documenting precisely what their theory is (that's the rub). The release of glossaries with L6 was a step in the right direction but ...
3. Third, there is the tagging where Faithlife is truly breaking new ground e.g. the Bible Sense Lexicon. Here I think we need to cut them some slack because it really takes thousands of knowledgeable users using it and providing feedback to "get it right". If, over the next decade we don't see gradual improvement in the usefulness, then we have reason to complain.

We also have to remember that with items like the Factbook, Faithlife runs into a serious problem with differences in vocabulary across different schools of theology. In topics.logos.com they had created the opportunity for users to supply synonyms but in general they got little feedback. We, as users, need to hold Faithlife's "feet to the fire" in terms of the order in which tagging is done.

1. No matter how much your library is limited to one tradition, we need to demand that a wide swath of denominations be included in the early tagging e.g. Orthodox and Jewish terminology is sadly missing from the LCV. I would not be surprised is other groups such as Anabaptists, Restorationists, Quakers ... also find their "native" terminology missing.
2. If your work requires completeness or if your personal favorites are not tagged (think outlines, journals, sermons ...) you need to push for the ability to have a collection of "not yet tagged" that require different search arguments or manual searching.
3. When you find errors in tagging, missing tagging et. al. you have an obligation to report it. Faithlife cannot identify and correct the errors that annoy us if we don't report them.
4. If you don't understand the purpose or meaning of Logos tagging (recent examples: BSL, propositional outlines) keep asking questions on the forum. Eventually enough people understand to be able to answer the questions or Faithlife discovers they need to write an introduction or blogs (think of the Discourse Analysis material we now have).
5. If you feel the tagging is "too academic" and you need "pastoral/teaching" tools, be precise in what you want: For me it would be tagging of rhetorical devices, microgenres and named relationships between verses (cross-reference and the myriad of other verse lists we have in Logos/Verbum).

Remember that software development is a two way street - the Faithlife software environment will continue to improve only if the users let Faithlife know what does not work, what can be improved and what is needed. That end is our responsibility. Implementing it and broadening our vision is Faithlife's responsibility. Whining is simply poor stewardship of time.

Francis, I agree that the QA is inadequate. The number of resources that have been released then sent back for basic coding is not acceptable. Examples" the BCP without the Psalm links (Latin names rather than Psalm number), Faith of the Fathers without the link to the index that provides the primary value, the number of liturgy books not tagged to appear in the liturgy section ... those are just ones I complained about. I also agree that the Faithlife fall back on an overall error rate and their lack of time to manually check each resource is a poor excuse. At the start of each resource's development a check list of required flags for processing, indexing noting special features etc. etc. ought to be made and ought to follow the resource and manage the workflow. But I do think we have to have realistic expectations and recognize our own position in the process.

LCV (topics) in Logos 4 was the signal for Faithlife wanting to manage data for a better experience in searching. Now we see LCO (Cultural Concepts), Literary Typing, Semantic Roles and Case Frames which are quite complex in the way they are applied to resource text.

The question of trust arises in the design and intended application of datasets. Whilst quality control (QC) is important in their management, most errors are due to human error/interpretation in applying the (bible-based) data. Do I trust Clause Search -- No! Do I use it - Yes. Would I recommend it - No.

The QC of the software and the direction of development showed a marked down turn in latter part of Logos 5. The Logos 6.0 beta showed that Faithlife could not keep up with the bug reports, or wanted developers to interpret what was important to them. The lack of response in the forums and the silly errors in the 6.0a/6.0b betas is indicative.

There's another piece that needs to be considered in your history --> the move towards giving as set data the results of what had been simply user techniques prior to computers. Many of these datasets, even morphology, hide the ambiguity that was why a scholar used the technique. Add to this the Faithlife attempting to be all things to all people - when we don't agree on a base text, don't agree on the "literal" meaning, don't use the same terminology -- I'm not sure there's a gold standard to measure the Logos data against. Which means Faithlife must be very explicit about what their coding means and how ambiguous cases are resolved.

Edit: I see Fr. Devin Rosa made my point better than I while I was writing this.

But for all that, it should not cast doubt on whether there is really not all that much errata in Logos and that most of this consists in philosophical disagreement or the vissicitudes of resources/datasets inescapable limitations. I think that there is sufficient, specific evidence of this to be found all over the forums. 

I'll believe it when I see it. 

I guess I just don't remember seeing all that many posts regarding morph data, and it was kind of strange to have such a long post with so many specific posters without any specific examples arising. I certainly haven't encountered many problems myself and have been using these databases for years daily.

Remember as well that Logos has over a dozen different morph databases. If there are a couple of posts per morph database on the forums, that might look like a large number, but in fact would indicate things are probably sailing pretty smoothly, or at least are about like in Accordance or Bibleworks, where they occasionally ship updates and minor corrections to their Morph datasets as well.

The other just occurred even as I was exploring a possible avenue of betterment (using the BSL in relation to tagging). Surely, this is not an inescapable type of resource limitation. How complicated is it to tag Passover occurrences in the Bible? Either you tag them or you don't, but if you do, it must be more reliable than this! 

I was kind of wondering what you were getting at with that post. I looked through the tagging of Passover in John, and as best I can tell it is all correct. Run a BWS on πάσχα if you want to see for yourself the ways the senses this word is classified with. The key is to open up the Senses section and then look at the actual definition each sense is given. The verses in John seem with pascha seem to be correctly classified to me.

Now, one area where the Senses dataset needs to be improved is documentation. For example, many abstract nouns are not classified in the Sense Lexicon because Faithlife is still figuring out what they want to do with abstract nouns. OK, great, but that needs to be documented in some resource that accompanies the Sense dataset.

The same is true for the Journals section, if I understand correctly. It works with manual tagging of Journals. The problem, I believe, isn't so much the tagging that has been done is incorrect as as the lack of tagging of many journals. The list of tagged journals should be documented, so people know what to expect. That being said, that Journals ad should be reworded to remove words like "all" that make it imprecise.

Totally agreed about Journals!

Agreed as well I brought up Morph search - but I did so as an example of how it's important to distinguish between different datasets, and specific problems, that's all. [{] 

Regarding the BWS, I was just pointing out that the detailed information about the search you ran is found there, in the BWS (or in the Bible Sense Lexicon). So, if the results of a "Sense" search don't make sense, that is the place to go to ask if there really is a problem with the data, or if I need to approach the search differently. When one does that for the word pascha, it turns out that the tagging in the Gospel of John was correct, the problem was expecting something the information wasn't designed to give. i.e. the search you ran wasn't a search for "all feasts" but specifically for "the meal eaten at the Passover celebration", and that as distinguished from the sense of pascha as "a seven day festival..." or as "the lamb killed and eaten during the Passover feast". Knowing this, the results given in the query are clear and correct, even if they weren't what you were hoping for. That just means you need to search for what you were looking for in a different way.

And, finally, thanks for bringing these issues up and continuing to push for them! I think all in all we basically agree on this stuff. I'm sorry if my pushback didn't come across the right way - it really was thought out to make your push more effective, by trying to identify better those specific areas that are in need of work.