Quality control in Logos 6 - a new perspective

Page 3 of 9 (175 items) < Previous 1 2 3 4 5 Next > ... Last »
This post has 174 Replies | 12 Followers

Posts 1518
Forum MVP
Fr Devin Roza | Forum Activity | Replied: Thu, Jan 8 2015 11:54 PM

I think MJ's posts are a pretty good summary of how I see the situation. 

I would also suggest that it's important to make specific critiques. You would never really hear someone in Academia asking the question "Do you trust the Anchor Yale Bible Commentary Series"? The immediate answer to such a generic question would be, "Which volume?" Same goes with these datasets, as there are varying levels of quality and completeness. 

Maybe the simplest change that Faithlife could do to help on a global scale throughout their products would be a company-wide commitment to document all datasets, past, present, and future. All datasets should have authors who sign off on them publicly, ideally including all names, even of interns. All should clearly explain their methodology, presuppositions, and (just as importantly) their limitations!

For example, some of the datasets Faithlife has produced are incomplete - that should be documented. It's not something marketing needs to scream out from the rooftops, but in the documentation that should come with the dataset it needs to be clear. And, anyone who has read much academic work knows that explaining openly a works own limitations is standard fare. Doing so gives credibility, rather than taking it away... at least in the academic world, and that is the world such documentation needs to be written for.

If a dataset has no clear place to put the documentation (e.g. Morph, Clause, etc.), Faithlife should create resource(s) dedicated specifically to documenting those datasets.

When one knows the presuppositions, methodology, and limitations of any work, it can be used profitably. As well, knowing that everything that is produced, and its authors and limitations, needs to be documented, can only be of help to both Faithlife and the specific authors to make sure quality is top-notch.

Posts 29375
Forum MVP
MJ. Smith | Forum Activity | Replied: Fri, Jan 9 2015 12:01 AM

Dave Hooton:

LCV (topics) in Logos 4 was the signal for Faithlife wanting to manage data for a better experience in searching. Now we see LCO (Cultural Concepts), Literary Typing, Semantic Roles and Case Frames which are quite complex in the way they are applied to resource text.

The question of trust arises in the design and intended application of datasets. Whilst quality control (QC) is important in their management, most errors are due to human error/interpretation in applying the (bible-based) data. Do I trust Clause Search -- No! Do I use it - Yes. Would I recommend it - No.

The QC of the software and the direction of development showed a marked down turn in latter part of Logos 5. The Logos 6.0 beta showed that Faithlife could not keep up with the bug reports, or wanted developers to interpret what was important to them. The lack of response in the forums and the silly errors in the 6.0a/6.0b betas is indicative.

There's another piece that needs to be considered in your history --> the move towards giving as set data the results of what had been simply user techniques prior to computers. Many of these datasets, even morphology, hide the ambiguity that was why a scholar used the technique. Add to this the Faithlife attempting to be all things to all people - when we don't agree on a base text, don't agree on the "literal" meaning, don't use the same terminology -- I'm not sure there's a gold standard to measure the Logos data against. Which means Faithlife must be very explicit about what their coding means and how ambiguous cases are resolved.

Edit: I see Fr. Devin Rosa made my point better than I while I was writing this.

Orthodox Bishop Hilarion Alfeyev: "To be a theologian means to have experience of a personal encounter with God through prayer and worship."

Posts 3770
Francis | Forum Activity | Replied: Fri, Jan 9 2015 12:07 AM

This hardly addresses all of the concerns that have been mentioned, but perhaps we need something like the Bible Sense Lexicon to help us find how something may have been tagged (ideally, it would it not make a lot of sense if BSL and tags were consistent or shared the very same base?). I did a quick check to see whether it could be used this way (using the feast search in John as a test case). I searched "feast" and got "feast (meal)" proposed to me (with Passover as sub-category). I right-clicked on it to see if I could search it and I got this:

Apparently, there is only 1 feast-meal (not to mention, Passover) mentioned in John! 

Posts 3770
Francis | Forum Activity | Replied: Fri, Jan 9 2015 12:22 AM

Fr Devin Roza:
I would also suggest that it's important to make specific critiques. You would never really hear someone in Academia asking the question "Do you trust the Anchor Yale Bible Commentary Series"? The immediate answer to such a generic question would be, "Which volume?" Same goes with these datasets, as there are varying levels of quality and completeness. 

Devin, I hear what you are saying, but I am tired of always having to mop around. It is an important part of my complaint that there is way too much of that needed to be able to use the software as it is supposed to (advertised, promised). I am investing time in this thread and in other discussions because I hope it can make a positive difference, but I am also appalled at all the time cost of dealing with constant problems, reporting, searching for answers, etc (which sometimes yield solutions, other times just impasses). In my view, there has been sufficient documentation of all kinds of specific problems to show that there is a broader problem at play. Unless the root causes are addressed, we will always be spending way too much time in an endless process of putting out small fires that keep popping up everywhere. I don't think this is a situation we want to settle for as normality. 

Posts 1518
Forum MVP
Fr Devin Roza | Forum Activity | Replied: Fri, Jan 9 2015 1:58 AM

I hear what you're saying, I just guess I don't see the situation as being quite like you paint it. I remember, for example, that long thread about the reliability of the Morph info in Logos. The only really concrete example in the thread ended up being one where Logos was following the standards set by the academic Lexicons, while Accordance and Bibleworks were apparently following some sort of in-house classification, that is used in the popular entry-level Grammars but not in the academic Lexicons. Had they documented that somewhere? I kind of doubt it. 

Most of the problems listed in this thread here seem to me to be more related to documentation or debatable opinion / frameworks than real quality problems.

That doesn't mean there aren't datasets that are poor. The LiteraryTypes dataset is very poor as a dataset. But that is one that wasn't developed in house, but just imported from someone else's work that was never designed in the first place to serve as a dataset. Should this really be a dataset? Maybe not, as it currently stands. Would I like to see Faithlife redo this dataset from scratch, using a methodology optimized for dataset use? Absolutely. But even as it is currently, I've found that as long as I understand what it was actually designed for, its limitations, etc., it can and has been of use to me. 

When writing Fulfilled in Christ, I used a multitude of Biblical Concordances / indexes to prepare the sections on OT and NT Background. The best I've found is Collin's / Rogets. But one thing that struck me was how none of them were really "complete". They were obviously all the result of years of meticulous work, and some of them are truly wonderful works of scholarship, but even so the only way I could get results I really felt confident with was by using multiple concordances together, plus manual searches. But that doesn't mean those are bad concordances. It's just the nature of concordances / indexes, for many reasons. 

I consider datasets along the same line (and, well, actually I consider commentaries, and lexicons, and doctoral dissertations, and... on the same line as well, and even Morph datasets, etc. All require subjective human judgment that is necessarily limited, as MJ has been pointing out). 

Dont' get me wrong - I fully concord with you that there needs to be improvement, and was really glad to see this thread started. I have also been pushing for improvements in this area. But Faithlife is a very large company, with lots of different internal groups working on different things. I think that many of those groups do excellent work. Some don't. And there are some things (like documentation) that all the groups need to work on, or that need to be implemented from management.

In summary, I just would suggest that identifying specific problems (even when on a general level) is the best way to push for real world solutions.

Posts 3770
Francis | Forum Activity | Replied: Fri, Jan 9 2015 2:27 AM

Fr Devin Roza:
I remember, for example, that long thread about the reliability of the Morph info in Logos.

Yes, if it is the one I think, I had started it. This was a case were I made a mistake. This indeed can happen: criticizing out of ignorance. Sometimes it is the user's fault (it was mine in that case) and many times, the user is ignorant because of the lack of proper documentation. But for all that, it should not cast doubt on whether there is really not all that much errata in Logos and that most of this consists in philosophical disagreement or the vissicitudes of resources/datasets inescapable limitations. I think that there is sufficient, specific evidence of this to be found all over the forums. 

Fr Devin Roza:
Most of the problems listed in this thread here seem to me to be more related to documentation or debatable opinion / frameworks than real quality problems.

I find this characterization too limiting and in that sense, not true. I produced two concretes examples above. One was of advertisement vs actual experience (the Journal section). I did not add much comment there: everyone knows how this has panned out. The other just occurred even as I was exploring a possible avenue of betterment (using the BSL in relation to tagging). Surely, this is not an inescapable type of resource limitation. How complicated is it to tag Passover occurrences in the Bible? Either you tag them or you don't, but if you do, it must be more reliable than this! What is even more telling about this specific example is how I encountered it: one does not need to go very far to bump into something like this. It happens all the time. That's the problem. Same thing when I sought to other day to explore with Dave Hooton ways to make user tagging workable by using Labels. Bump, bump, bump. The problem started with something that does not work right and as alternative solutions were explored, they too were found not to work right! Right does not mean my definition but doing what it is advertised to do.

So, while I accept your description of the limitations of resources which can never be exhaustive, I don't think it is quite the proper analogy for the problems I seek to address here. 

Posts 25825
Forum MVP
Dave Hooton | Forum Activity | Replied: Fri, Jan 9 2015 2:41 AM

MJ. Smith:
There's another piece that needs to be considered in your history --> the move towards giving as set data the results of what had been simply user techniques prior to computers.

Yes, I had to give a potted history in order to meet an important (family) deadline! The debacle of OpenText in Logos 3 illustrated how a potentially good design (by OpenText.org) can be ruined by a lack of attention to detail (the morphology was improved in Logos 4 but I can't fully trust the method). The ambiguities of Case Frames e.g. 'love' as an Agent, leads to a lack of trust, but I don't believe that the method could help me to "better understand the verb in its context"! It's far too mechanical.

The ultimate frustration is that Faithlife have been doing resource metadata for over a decade and it still isn't consistent and right! I don't look forward to this as an example for correcting the newer datasets!

Dave
===

Windows 10 & Android 8

Posts 1518
Forum MVP
Fr Devin Roza | Forum Activity | Replied: Fri, Jan 9 2015 2:57 AM

Francis:

But for all that, it should not cast doubt on whether there is really not all that much errata in Logos and that most of this consists in philosophical disagreement or the vissicitudes of resources/datasets inescapable limitations. I think that there is sufficient, specific evidence of this to be found all over the forums. 

I'll believe it when I see it. 

I guess I just don't remember seeing all that many posts regarding morph data, and it was kind of strange to have such a long post with so many specific posters without any specific examples arising. I certainly haven't encountered many problems myself and have been using these databases for years daily.

Remember as well that Logos has over a dozen different morph databases. If there are a couple of posts per morph database on the forums, that might look like a large number, but in fact would indicate things are probably sailing pretty smoothly, or at least are about like in Accordance or Bibleworks, where they occasionally ship updates and minor corrections to their Morph datasets as well.

Francis:

The other just occurred even as I was exploring a possible avenue of betterment (using the BSL in relation to tagging). Surely, this is not an inescapable type of resource limitation. How complicated is it to tag Passover occurrences in the Bible? Either you tag them or you don't, but if you do, it must be more reliable than this! 

I was kind of wondering what you were getting at with that post. I looked through the tagging of Passover in John, and as best I can tell it is all correct. Run a BWS on πάσχα if you want to see for yourself the ways the senses this word is classified with. The key is to open up the Senses section and then look at the actual definition each sense is given. The verses in John seem with pascha seem to be correctly classified to me.

Now, one area where the Senses dataset needs to be improved is documentation. For example, many abstract nouns are not classified in the Sense Lexicon because Faithlife is still figuring out what they want to do with abstract nouns. OK, great, but that needs to be documented in some resource that accompanies the Sense dataset.

The same is true for the Journals section, if I understand correctly. It works with manual tagging of Journals. The problem, I believe, isn't so much the tagging that has been done is incorrect as as the lack of tagging of many journals. The list of tagged journals should be documented, so people know what to expect. That being said, that Journals ad should be reworded to remove words like "all" that make it imprecise.

Posts 3770
Francis | Forum Activity | Replied: Fri, Jan 9 2015 3:14 AM

My friend, there are different issues that been have mentioned here as part of an overall frustration with the number of errors, the incompleteness of features when delivered and features that don't work. I did not focus on morph errors in this thread: you brought that up. My response to you was not to say that morphologies are rife with errors, but that Logos is rife with errors. 

With that in mind, I really do not understand the way you are answering what I wrote. How does the BWS route showing what it does invalidate the fact that 1) there is a BSL in Logos, thus it is a legitimate avenue of searching; 2) I checked whether it would perchance relate "feasts" as a sense to occurrences (that is, make them searchable); 3) I type "feasts" (a natural language query) and do find an entry: feasts (meal) of which "Passover" is a sub-category. 4) I check whether it is searchable and it is. 5) I search for it in John and find only one instance. So, in what way is finding the right results using BWS somehow invalidating the problem that I have encountered using BSL? I don't get your approach.

Same concerning Journals. I did not say that Journal tagging was incorrect. I produced the Journal Section example to show that there is a mismatch between advertisement and actual experience, an all-too common problem in my view. It has improved a bit as of late, but at first, many users found that their journal library produced no hits even when searching vast collections of journals and broad sections of the Bible. This clearly did not correspond to the messaging that essentially said: buy Logos 6 and you get this! It did not say, you might get this in 2015 or later. I did mention tagging errors with regard to other examples. But please don't blend my examples! They are different illustrations of different issues that constitute overall what I think is a more systemic issue ( = the overall number of lack of documentation, errors, incomplete features, broken features = a greater emphasis on production/promising at the expense of quality/delivery). 

Posts 1518
Forum MVP
Fr Devin Roza | Forum Activity | Replied: Fri, Jan 9 2015 3:48 AM

Totally agreed about Journals! :)

Agreed as well I brought up Morph search - but I did so as an example of how it's important to distinguish between different datasets, and specific problems, that's all. Left Hug 

Regarding the BWS, I was just pointing out that the detailed information about the search you ran is found there, in the BWS (or in the Bible Sense Lexicon). So, if the results of a "Sense" search don't make sense, that is the place to go to ask if there really is a problem with the data, or if I need to approach the search differently. When one does that for the word pascha, it turns out that the tagging in the Gospel of John was correct, the problem was expecting something the information wasn't designed to give. i.e. the search you ran wasn't a search for "all feasts" but specifically for "the meal eaten at the Passover celebration", and that as distinguished from the sense of pascha as "a seven day festival..." or as "the lamb killed and eaten during the Passover feast". Knowing this, the results given in the query are clear and correct, even if they weren't what you were hoping for. That just means you need to search for what you were looking for in a different way.

And, finally, thanks for bringing these issues up and continuing to push for them! I think all in all we basically agree on this stuff. I'm sorry if my pushback didn't come across the right way - it really was thought out to make your push more effective, by trying to identify better those specific areas that are in need of work.

Posts 3770
Francis | Forum Activity | Replied: Fri, Jan 9 2015 4:32 AM

Thanks Devin. Good points about BSL too!

Here again it shows how more training/documentation could help a great deal (where this is the problem). It is not immediately clear that 1) it is not the Passover itself that should be categorized as a feast-meal; 2) that in an instance like Jn 6:4 this connotation should not be tagged considering what follows; 3) that other feasts (most involving eating) should not included just because the activity of eating is not specifically mentioned there. To me, it feels like you can't know what you will get, you can only retrace the meaning of what you get once you get the results. By then you discover that what you thought you searched is not what you were allowed to search and that you must try something else. It makes it difficult to predict accurately what would be the right search terms and tool for one's query. There was the same problem with searching for feasts in Jn as a cultural concept. In the end, one has to search FIRST for how what one is looking for is tagged as in the datasets BEFORE one can actually search for it. So I can use BWS and look at what senses are attached to it. But looking for all feasts originally, I have no guarantee that the datasets findings related to pascha will help me locate all the feasts in John. The same kind of deal happens when one has to use factbook, topic search and lookup to find an entry. It's complicated, somewhat (not completely) opaque and quite time-consuming.

Yes, I keep replying because I want to be a badger here, not drop the ball. I feel that this is hardly an isolated expression of such sentiments, but that the response from Logos has not been commensurate yet. I appreciate the posts by Sean and Bradley (yes, I do!), but they only speak to specific issues isolated from the overall situation. And not rarely, remedies are often left in an indeterminate (indefinite it seems at time) future. "It's on our long list" (question: why is the list so long in the first place?). Will we get proper documentation in 2015 or should we expect Logos 7 to come out first?

I also want to acknowledge a hierarchy of problems. I understand that BSL is innovative (in a very good way) and so tricky. Fine. But it is the accumulation of all the problems that has become too much. But even among those of us who are fed up and irrate, I think Logos should realize that we want the company to succeed! It's a win-win prospect!!!

Posts 35
John McKenna | Forum Activity | Replied: Fri, Jan 9 2015 7:49 AM

I'm a bit bothered by a couple of posters and this thread in general. Why? Because there are actually two issues going on here.

The first issue, that of the thread writer, is about content and the second is bug and coding issues.

On the first issue, if one is relying on Logos and Faithlife to study for them and rely on their interpretations (or references) as being the end all answer then that one will always find difficulty. I love the helps these people put their countless hours into, but they are neither error free nor perfect. The whole reason for have a vast library of biblical data is to help in an individual's personal study and quest for the truth. Because a famous author, church father or even Logos says something it is not the finale court of arbitration.

I appreciate the work of these men and women and I also understand that to "wait" for perfection in the product before distribution would be ludicrous. I also know nothing every would have been printed or distributed though out history if such thought were held.

As far as program bugs go; yes there are unacceptable levels of bugs which do prevent general distribution of all software. However, the beta process is an accepted method of finding critical bugs and coding failures prior to that distribution. Both Apple and Microsoft use the beta programs in similar way to Logos and, like Logos, distribute essential OSs which still include bugs and associated problems. What I admire about Logos is their continued updates and distribution of fixes for such issues.

I vote YES!

Logos 5 (almost ready to update to 6)

2012 Mac Mini, 2014 11" MBA, iPad, iPhone

Posts 8967
RIP
Matthew C Jones | Forum Activity | Replied: Fri, Jan 9 2015 8:15 AM

MJ. Smith:
and their lack of time to manually check each resource is a poor excuse.

Do you think it might be time to have a moratorium on releasing all new resources until Faithlife can catch up on fixing already released resources?  <I don't>

Otherwise we may never see perfection.

Logos 7 Collectors Edition

Posts 715
Kevin A Lewis | Forum Activity | Replied: Fri, Jan 9 2015 8:24 AM

John McKenna:

Because there are actually two issues going on here.

Actually I think there is a third issue - something that lies somewhere between the two you mention.

That is the attention to the construction of the metadata - most notably the "links" and crosslinking between datasets and third party publications.

I don't think anyone is suggesting that as human being we can be error free. Just that in the area of the much vaunted Logos "added value" elements (such as "searching" and "linking" etc.) much is promised and not always delivered. Logos is a premium product, sells publications at a premium digital price, so should be reliable regarding the premium aspects.

Do I trust Faithlife as people, an ethical company etc etc. = Absolutely.

Are there distinct areas for improvement in the areas originally mentioned = Very Probably.

Also there are two levels at which these issue can be addressed. On a case by case bases - and yes logging errors is a good approach to this. But it takes effort away from what we as customers should really be doing - using and enjoying the product - and being empowered better to be about service.

The other level is drawing attention to a "general" need for Faithlife to sharpen and refocus their activities if they wish the retain and promote an enthusiastic user base.

--- good criticism - is constructive criticism - listened to, absorbed and appropriately acted upon!

Shalom all!

Posts 11070
Denise | Forum Activity | Replied: Fri, Jan 9 2015 8:28 AM

Well, in the Francis poll, I'd vote 'no'.  I've never trusted Logos for all the reasons Fr Devin mentions ... who did it?  what were the assumptions?  when was the last error-pass (I keep hearing Logos mentioning Libronix data sets)? etc, etc.   But the same goes for the other sources as well, whether software companies, or books.

I also agree with Francis, though maybe in a different way.  When people say 'it's a work in progress', that just means it's unusable.  I don't do searches for 60% of the results.  Now it's true ... most sources deliver 60% of the results ... you have to comb multiple sources to get a complete picture.  But in the case of Logos, I don't ever know 'where they are'.  I used to pick at the 'Biblical senses' as being eminently naive.  MJ says they've much improved.  I guess ... I don't know where they're at in their 'work in progress'.  So, I don't use it.

I appreciate Francis being brave enough to introduce the issue.  I really think there's great folks at Logos that do great work.  The problem is that some deliver pretty complete results and then tell you where the weaknesses are (Rick comes to mind).  And others are somewhat anonymous and grey as to exactly what is being delivered.  Need some documentation, as Fr Devin mentions.

I also don't agree with MJ on a single point.  I very much doubt there's a whole lot of expertise among forumites.  Yes, MJ, Mark, Dave, Fr Devin, and some other names my brain has forgotten.  And that's the end of the road.  It's a QA issue that belongs to Logos.

"I didn't know God made honky tonk angels."

Posts 819
Lew Worthington | Forum Activity | Replied: Fri, Jan 9 2015 8:32 AM

I posted a few days ago that my biggest fear was that the Faithlife user community would lose faith in the company in large enough numbers that this would be the downfall of everything Logos related. What I didn't add is that I see this as a very unlikely situation for a long, long time. (I also think about the super-volcano in Yellowstone that is 40,000 years overdue: http://en.chessbase.com/post/will-the-world-end-on-kasparov-s-66th-birthday-)

And then I tripped across this post that resonated well with me: I'm a satisfied customer.

I should add that my methods don't require much subjective tagging. I've never been in the habit of accepting someone else's interpretation unless I can make it my own.

Posts 8967
RIP
Matthew C Jones | Forum Activity | Replied: Fri, Jan 9 2015 8:53 AM

Denise:
I don't know where they're at in their 'work in progress'.  So, I don't use it.

Funny how the people complaining about Logos do not use it.

Logos 7 Collectors Edition

Posts 8967
RIP
Matthew C Jones | Forum Activity | Replied: Fri, Jan 9 2015 8:59 AM

Francis:
Yes, I keep replying because I want to be a badger here, not drop the ball.

OK, you are a badger among badgers.

Logos 7 Collectors Edition

Posts 11070
Denise | Forum Activity | Replied: Fri, Jan 9 2015 9:00 AM

Not too funny if you paid for uselessness, ergo Francis' thread.   Now, to be fair, I've bought quite a number of resources that similarly qualify.  But I slowly learn my lesson.  Ergo Francis' concern.

"I didn't know God made honky tonk angels."

Posts 2588
Lee | Forum Activity | Replied: Fri, Jan 9 2015 9:05 AM

Funny how some people are erecting perfection as a straw man.

Page 3 of 9 (175 items) < Previous 1 2 3 4 5 Next > ... Last » | RSS