Quality control in Logos 6 - a new perspective

Here's the problem.

- When you (not really you) curated the people diagrams and also the maps (a major earlier complaint), your algorythm or dataset has no ability to show the user the most likely to be needed (in this case 'Reuben'). Curated = amateurish upon delivery.

My take on Reuben ... when I go to the Factbook and enter Reuben, I don't divulge anything that implies whether I am more interested in his ancestors or his descendants or a chart for a given Bible passage. What I get is:

A mouse over shows me who key individual of the chart and what passage, if any, the chart applies to.

The Search media link informs me that the results shown are incomplete. If I don't see what I want I should run a search.

Ah, I see I have many additional charts to chose between ... I chose two that are not tied to specific passages ... one with Reuben's ancestor Abraham as the focal point, one with Reuben himself as the focal point. But wait, as I do a mouse-over I find others that have Reuben as the focal point that are specific to a particular Bible passage.

Hey look at the symbols... two refer to the tribe of Reuben ... a collection of people referred to as "Reuben". I am a bit surprised that there was not a distinction between Reuben himself and Reuben's descendants ... I may have found a bug. 

Sorry Denise, I can't agree that it is hard to find what you want for genealogies - the focal point and the size of the chart are pretty good give aways. As for tied to a passage vs. not tied to a passage, if you want "not tied to a specific passage" added to the titles, it would do no harm but I am perfectly capable of recognizing there is a passage specified/there is not a passage specified.

I do not know how Logos/Verbum determined which items to show in the Factbook preview ... I will not defend it but will say that the most likely wanted hits is a difficult target shown by how often Google is nowhere in the ballpark and Bing hasn't even found the park.

So you are dissatisfied. Precisely how would you like the behavior changed to meet your needs? What information do you need to select the appropriate genealogies is not available? How would you like it displayed?

If I needed to use Faithlife products for full-time ministry, academic, or scholarly pursuits I think I'd be troubled.

What puzzles me is the lack of specifics.

Note this is written after a nearly 3 day period of having neither my beta or my stable versions running which put an important project behind schedule very near the turnover to another person i.e. very bad time.

?

You want me to embarrass myself by admitting how long it took me to figure out that there was nothing I could do to get Bradley more data when indexing eventually went to lots of memory but no processing? And that I had a big hint in the logs telling me they found a active index thread to kill despite Task Manager showing no task? Funny how restarting the machine rather than just the app cleared the problem ... the indexing completed AND let the main program take over. Still don't know how the problem from running the beta sync'ed to the production but all is well and I know running a beta leaves me out of a call to support so if I want to trust a project to a beta version, I have to take responsibility when it didn't exactly work out. And if I dedicated time to resolving it rather than multi-multi-tasking I hope I'd find it faster.[:$]

However, the point I was trying to make was that people express distrust as a whole rather than distrust of a particular feature or dataset where Faithlife can see if the data is actually error prone, or the algorithm is actually flawed or the user doesn't understand how to use the feature (also a Faithlife concern for an "intuitive" UI)

Denise - I was showing that the divergence was to be expected not a "mystery" without any reference to Libby. If you are looking for the verbal explanation why limit the search to media? You have fun being a quick-witted naysayer; I have fun being the logical alternative finder. Luckily, the forum benefits from the contrast.

Thanks to Francis, Sean, Bradley, and Todd.

I'm going to try to offer some concrete suggestions and feedback about specific areas I've seen could be improved. Some of these are quite specific, but maybe even those can help to identify broader problems, or at least areas where it might be worthwhile to do a review. 

1. Clause Search data. I think Dave had mentioned earlier in this thread he doesn't trust the Clause Search too much, and I have reason to not always trust it either, although it has certainly worked correctly for me most of the time. Here is an email I sent in on the 9th of October, 2014 to data@logos.com. Are these in fact errors, or am I missing something about how this tool is supposed to work?

[quote]

Hello,

I ran a Clause Search today for subject-lemma:אֱלֹהִים verb-morph:V????P and was surprised by the large number of hits that I got where אֱלֹהִים was not a subject at all. It made me worried about the quality of the markup in the Clause Search database.

Some examples of what I would consider false hits (i.e., where Elohim is not in any way a subject in the phrase, not even in a broad sense of being included in a prepositional phrase which modifies the subject) are:

Judg 20:2

1 Sam 8:8

2 Kings 17:7

2 Chr 11:16; 36:16

And I could continue…

Please respond to this email to acknowledge it. In the past I have written to this address but never received a response. If I don’t receive a response I’ll post to the forums. Thanks.

Fr Devin

No response to the email to the date. Apart from the response, I wonder if these errors (supposing they are errors) are indicative of broader problems regarding the classification of subjects of clauses (or of other types of classification?). 

2. The lack of response from data@logos.com. The second is related, and that is the type of response that is given from data@logos.com. That is, none. I think that an email should be sent thanking people for sending in corrections. I'm still not even sure if you ever received this email. 

3. A specific problem with the Bible Sense Lexicon. I reported this problem to data@logos.com on the 1st of July, 2013. I have never received a response, and the problem persists:

[quote]

In Genesis 49:24, 2 Samuel 23:3 and Isaiah 30:29, the word “stone” or “rock” in the phrase “Rock of Israel” or “Stone of Israel” is currently classified under the sense “stone”. It should be classified as “God ⇔ rock”.

My first report could be excused because you were busy releasing Logos and Verbum 6, but given that you have had 18 months to fix this report, and it is easy to fix, it might indicate a serious problem with the way these emails are handled. The sense “God ⇔ rock” does already exist, so I am pretty sure it's not just that you aren't yet doing this type of classification.

4. Morph tagging on the Biblical Dead Sea Scrolls. As I have said before, I am quite happy with the morph tagging of the Greek and Hebrew Bible, and have never had trouble with the Septuagint tagging either. I have not been as happy with the tagging of the Biblical Dead Sea Scrolls, although I can't say it's bad either, just not quite up to par to what I would hope for. From October 2012 to January 2013 I worked quite a bit with these texts at the Hebrew University, and remember running across a surprising number of errors (maybe one or two per chapter? I'm not sure). At the time, I sent in a bunch of corrections through the "Report a Typo" mechanism. If you have those Typo reports still registered somewhere or another, I would recommend trying to pull them up and look through them to see if they are accurate, and then based on what you find there get a feel for how few or how many errors there may be by extrapolation and how worth it or not a revision would be. 

5. Recommendations for Documentation. I am very happy to see the commitment to document tools that are of use for academics. I hope that by the time Logos 7 comes out, all the tools of most interest to academics have been documented. I think that it is important that there be a place to go to that is specifically for academics. I know that Logos has a huge and broad market, but as this thread has made clear, there are certain requirements of academics that are specific. 

Here are the tools that I think should be highest priority... I'm sure I'm missing something, but anyway, here is what I see as being most important, with a few notes about things to not forget to include (not meant to be complete, just stuff you could forget but shouldn't):

1. Logos Greek, Hebrew, Aramaic, and Latin Morphology databases. Ideally other morph databases as well, at least providing easy access when not already available to the documentation provided by the authors.
2. Clause search database - Remember to document how you classify the subject of imperative verbs, and how to find the one actually doing the ordering using {Speaker <Person>}
3. Biblical Sense Lexicon
4. Factbook tagging - Include what is tagged and what isn't (e.g. abstract nouns, etc.). Include difference between something like <Event> and {Section <Event>}
5. Reverse Interlinear tagging - After reading the docs on this, users should know things like they need to use the SBLGNT and LHB for reliable searches, they should know what criteria are used to markup words as primary or secondary, etc.
6. Ancient Versions section of the Textual Variants tool (Rick Brannan already documented this in detail during the Logos 6 beta process on the Beta forum, it would be enough to rework that post).
7. Ancient Literature Tool (Rick Brannan already documented this in detail during the Logos 6 beta process on the Beta forum, it would be enough to rework those posts, and hopefully provide more detail about the automated logic that went in to the data mining of the Church Fathers references).
8. Journals tool - Document what is included and what not
9. LiteraryType dataset - Especially important as this dataset is really quite poor as a dataset, in part as it was not designed to be used as a dataset, and so has different criteria and quality than the other datasets, and so users need to know what they can expect
10. Cultural Concepts
11. Cross References Tool

As I said in a previous post, I don't think documenting these tools needs to be something out of this world, or even take that long. The basic question is what does an academic need to know to use these tools, who did them, including even interns, criteria used, and what their limitations are, what is missing in the tagging, etc.

BTW, in my opinion Rick Brannon is an example of someone who seems to understand and do these things already. He also has a keen sense of the importance of releasing complete and accurate data. Thanks, Rick! I'm sure there are many others as well, I have just particularly noticed that about him, so, thanks!

6. LiteraryType dataset - I've mentioned this before, but it's worth repeating. This dataset is particularly poor as a dataset. I have been happy with the other datasets in general, but not this one. I really think you should consider redoing it in house. At the very least, it should be a priority for documentation to warn users about its limitations, and explain why this dataset is not like the others....

7. Communication regarding incomplete datasets - In an earlier post, Sean earlier mentioned the Factbook Septuagint Deuterocanonicals dataset. For me, that dataset is actually an example of decent communication. It was always advertised as being only the Deuterocanonicals. Perfect. What is problematic is when datasets are published and it isn't clear what is missing. Think the Journals tool, Outlines Tool, Bible Book Factbook Entries, etc.

Don't get me wrong - I fully agree you need to advertise, be positive, etc. I don't think these limitations need to be bragged about. But at the same time the information needs to be available.

For tools like the Journal tool, the Outlines Tool, the Bible Book Factbook Entries, etc. that are currently being worked on, ideally I think there would be either a resource in the library that listed what works were included, or maybe a webpage with that info, or maybe both. When the datasets are updated to add more works, the list of works included is also updated.

Maybe as well on the Guides and Factbook sections there could be some sort of standard Info bubble, or help bubble that is available when documentation is available, that would bring you to the academic documentation / list of works included up to now, etc. Some tools would always have this info button (such as the Ancient Literature tool), others only when it applies (such as the Factbook entry for a Bible book).

Thanks everybody!

Just for the record, I fully agree with all of your qualifications Lew, and that has been one of my reasons to be somewhat leery when criticisms of the datasets come without specific examples, especially given how common it is on these forums for people to misunderstand how a feature works and misinterpret it as an error.

In the case of the LiteraryType dataset, here is a post with some examples of the type of problems this dataset has: https://community.logos.com/forums/p/94632/660593.aspx. Note these aren't really errors per se... they are more just a design that isn't optimized for the type of searches people would ideally like to do from a dataset like this. 

I would peronally still prefer having it than not having it, but I do think it is one where you have to be careful. And, the fact that Faithlife didn't design the data may have implications (?) as regards their ability to correct it or adapt it based on how it's being used??? I'm not sure.

Either way, as I've said before, if you keep in mind where something comes from and what its limitations are, it can almost always be useful (cf. https://community.logos.com/forums/t/92669.aspx for the source of the LiteraryType dataset, also present in the Info window of this resource, although it should also be in an Introduction to that resource as well).

EDIT: Looks like Sean had already said he was hoping to improve and correct this dataset: https://community.logos.com/forums/t/98894.aspx. Good news. If it is low on the priority list, hopefully documentation of its sources (and limitations) can be moved up high on the priority list, at least.

Thanks Fr. Devin for your concrete, actionable post. Because of different interests and goals in my use of Verbum/Logos, I have a few additional items and probably a different order of priorities but you are both spot on re: what is needed and how to inform Logos effectively of what is needed.

Lew Worthington, your point regarding never trusting someone else's categorization of the text is also spot on. The ability to put notes on nearly everything has finally given us non-scholars the ability to note our own questions and disagreements with the tagging offers. Now Logos needs to make those notes more visible