Feedback on Lemma in Passage

Mark Barnes said:

The grouping needs to be at two levels, not just one. That is, if I select the 'lemma' view, it should group by lemma, but then group by resource within each lemma. Equally, if I select resource view, it should group first by resource, but then by lemma within each resource.

I agree, and it's already in the works.

Mark Barnes said:

In the lemma view, it would be much better if the results were ordered by prioritisation. I know you might say this is a search, and therefore results can't be ranked by prioritisation, but because it's a searching within a milestone, I do think prioritisation should be used.

Good point. It may be deceptively difficult since the results are coming from a web service that may not have ready access to your personal priority rules. But we'll look into it.

Mark Barnes said:

It would be nice to have a link to a search. I presume the search criteria is something like this: (πνευματικός OR πνευματικῶν) WITHIN {Milestone <Bible ~ 1 Cor 12:1>} within type:bible-commentary, but it would be nice not to have to re-type it. Searches (particularly inline searches) are particularly useful when you want to scan through the results in context.

When/if we build a search provider, we'll include something like that.

We aren't doing a search for multiple surface forms. Rather, Rick wrote a much more sophisticated heuristic to compare a given surface form against candidate lemmas from the passage in question. This cuts down on false positives due to homographs and false negatives for spelling variations. His dataset then assigns an exact lemma to the selection in the resource. In some cases, it could assign an exact instance of a lemma, though we don't leverage that data just yet.

So your search would yield different results than the dataset.

Mark Barnes said:

Given (4), above, it seems odd that this search needs to be performed on the Logos servers. A locally-based dataset of acceptable declensions for each lemma should be all that's needed to make this local. And that dataset, if it doesn't already exist, could be useful for all sorts of things.

We pursued a solution that just maps inflections to lemmas. Such an approach is tolerable for NT Greek, less good for Greek beyond the NT, and embarrassingly awful for Hebrew and Aramaic. (We might still include this global mapping in the lemma list dataset for guessing lemmas for surface words in any resource. A sort of last-resort automatic parser, if you will.)

But commentaries are a special case. So instead we opted for a process that maps specific selections in a resource to specific lemmas -- that is, as if the tagging had been done in situ in the resource. This results in a much more accurate but much larger data set.

Given that, we can either a) create one very large (~ 1 gig and growing) supplemental data resource that covers every commentary we've ever produced, and then update it every time a new commentary is produced whether you buy it or not, or b) create one supplemental data resource for each commentary resource, or c) some number of supplemental data resources less than the number of commentary resources, say, by lumping commentary volumes together. A is a non-starter from the user perspective. B and C are a production and maintenance nightmare. It could be done, but then the feature would fail at cost/benefit and we wouldn't build it at all.

On the other hand, the library services architecture we've been building to support the web app already keeps one automatically-updated gigantic index for every resource that will ever exist and filters out results for a given query based on the user's licenses. 

Rick Brannan (Faithlife) said:

I'm unsure where that stands. 

It's in progress. Should come up in a beta real soon now.

Mark Barnes said:

However, given what you've said my preferred approach would be to add tagging to the existing lexicons and commentaries, so that your "supplemental data resource" is actually a part of the resources themselves. That way users could access it locally, but would only have that part of the database that was relevant to them.

That would be technologically simpler. However, to do that would require the book production team to do what we call a "recycle" on each of around 2,400 commentary volumes. Recycling a resource means taking it back to Xml source files, adding markup, and then re-compiling and re-publishing the book. Depending on 20 years worth of previous circumstances when the book was first built, recycling can be a very smooth process, or a lossy and fragile process — with no obvious way to tell which. (A good rule of thumb for computer files is, "if it ain't broke, DON'T TOUCH IT!")

This process, as I understand it, would take around two calendar years, or 100 volumes per month, and would impact the team's ability to do other more valuable things.

A second reason, more pertinent to the business factors than tech, is that this data is value added markup that we want to make available only to subscribers. If we embed it in the books, we currently have no way to list it as a separate line item for commerce. It's just part of the book, so 100% giveaway.

So in the end, the choice wasn't really between "in the books" or "on the server," it was "on the server" or "not at all."

Mark Barnes said:

It's much more that having all these separate databases severely limits my ability to construct my own searches that use data from a variety of databases, rather than simply through the guides and tools.

That's the situation at the moment, but we do plan to make these searchable eventually. The same technology that allows search extensions ({...}) allows us to build custom search providers to search data that is not stored in the local index.

But, as Bradley is fond of reminding me, the default state for any feature is to not exist. So that means that the immediate choice wasn't between "search" or "guides" but between "guides" or "nowhere."

Hope that makes sense!