CrowdSourcing: Is it time for a revival...

... of support/interest in crowdsourcing the typo/tagging problems?! Jonathan Pitts created this UserVoice suggestion over 4 years ago and it seems to have been forgotten. One thing that we can all do is vote for his suggestion, but I thought that perhaps we could also use this thread to "brain storm" how the system could/should work...

For example:

1) Could absolutely anyone make "corrections" or only a certain Group?

2) Would corrections/tags get reviewed by a committee (from the "crowd") or only by Faithlife?

3) Would corrections/tags get released to resource updates on a regular basis? (consider the no-restart-required updates being worked on in the beta release)

4) ???

Find more posts tagged with

Comments

Josh Hunt

This seems like a great idea to me

Reuben Helmuth

Some of my own thoughts are:

1) Could absolutely anyone make "corrections" or only a certain Group?

Users would need to join a Faithlife Group in order to gain access to the correct tools. This could be a closed group in the event that Faithlife felt like they needed ultimate control over who can edit. Edits would only be visible/available to those in the group until reviewed and approved for "official" release.

2) Would corrections/tags get reviewed by a committee (from the "crowd") or only by Faithlife?

Initially, a Faithlife employee would do ALL the reviewing/approving, but would grant this power to select users upon meeting certain requirements. The requirements could be ">100 edits submitted" & ">95% accuracy" (accuracy=correct identification AND tagging) My thought is that upon meeting BOTH requirements, a user would be able to review/approve the edits of other users. In this way the burden of reviewing thousands of edits could be spread out fairly quickly while still maintaining a very decent level of QC.

3) Would corrections/tags get released to resource updates on a regular basis?

"Administrator" approved edits should be released at least every six weeks (along with the normal release cycle). Once the updating of resources no longer requires a restart (and all the bugs are out!), I don't see why approved edits couldn't be released weekly.

EDIT: On second thought, it doesn't make sense to do a (even restart-less) resource update for only a single fixed typo, so perhaps resources should be updated every 10 edits OR every 6 weeks, whichever comes first.

Everett Headley

https://community.logos.com/discussion/comment/721139#Comment_721139

My initial thought is this: I have invested over $6,000 into my library with Logos. Many, if not all, of these are "value added." This means instead of the bargin at Kindle, I went with Logos. In the case of "From Heaven He Came and Sought Her" it was a $32.95 vs $3.99.

For that kind of money, I expect that all linkage be correct and complete. For me to pay a near 10X mark up I do not want to see the need for a crowdsourced editing; Logos should foot the bill for this and employ more/better QA/Editors.

This is why I have a thread that I regularly post in: Too often (and in the same book referenced above, just released this year) I find no hyperlinking. It is frustrating because this is why I have everything in Logos and why I pay the premium pricing. If I wanted to search for a refernce in a another book I would have used someone else.

Bottom Line: Logos needs to do its own policing and editing and increase (dare I say set any) the standard for hyperlinking.

Matthew C Jones

1) Could absolutely anyone make "corrections" or only a certain Group?

David Taylor, Jr.

https://community.logos.com/discussion/comment/721148#Comment_721148

While I like the intent of the idea, I think you would run into quality control issues.....

Rick

2) Would corrections/tags get reviewed by a committee (from the "crowd") or only by Faithlife?

I would think that only Faithlife would be able to review and approve changes. In some cases there are copyright issues. On more than one occasion I have seen someone report a typo/error, only to be told that it is actually not a typo but spelled that way in the actual resource. Users who don't have the original files to compare their changes with may actually be changing something in the resource that should not be changed.

Jacob Hantla

I've been begging for this for years. there is way too much latency (or sometimes it never gets done) on fixing typos or updating tags. Vyrso books would never get fixed apart from something like this. If they could incentivize large volume of quality link creation and typo fixing the customers would be happy and the quality of the resources greatly improved. Tagging is the main place this is needed

Jacob Hantla

https://community.logos.com/discussion/comment/721154#Comment_721154

by the way, in the meantime we can use community tags to sort of do this from Logos 6 (but the links aren't visible on mobile and only for people who enable community Tags)

Mark Barnes

Logos have moved in this direction with community tags. I'd love to see them adding community hyperlinks, so long as those hyperlinks were properly integrated into our library and could be searched, etc.

As far as quality control goes, I'd like to see Logos building a website where we could log-in with our Logos IDs, and vote for/against hyperlinks that other people had added. That would ensure quality, and whilst there'd be investment in building the site, there'd be little ongoing cost.

Reuben Helmuth

https://community.logos.com/discussion/comment/721142#Comment_721142

My initial thought is this: I have invested over $6,000 into my library with Logos. Many, if not all, of these are "value added." This means instead of the bargin at Kindle, I went with Logos. In the case of "From Heaven He Came and Sought Her" it was a $32.95 vs $3.99.

For that kind of money, I expect that all linkage be correct and complete. For me to pay a near 10X mark up I do not want to see the need for a crowdsourced editing; Logos should foot the bill for this and employ more/better QA/Editors.

I feel your pain! My own investment is dangerously close to 5 figures. [:S] I agree that crowdsourcing shouldn't be needed, but to me it looks like it is! Realizing that there would be some (many) who do not wish to "contribute," no one would be forced to be part of the group. On the other hand, the missing links that you so much dislike could be fixed in a very timely manner. I think that by the end of a year, the vast majority of missing links/errors could be fixed.

I do think that if (when) this gets implemented, we should have a commitment from Faithlife that they won't "get lazy" because of it!

Don't you wish that every typo that you report would be fixed within a month?! I think that (this is an added detail to the system)... the general typo reports of ALL users should be accessible to the Group, rather than relying on only Group Members to find issues.

Reuben Helmuth

https://community.logos.com/discussion/comment/721154#Comment_721154

I would think that only Faithlife would be able to review and approve changes. In some cases there are copyright issues.

In the case of potential copyright issues Faithlife could do a final review. There are a number of typos that I've seen that are (by all appearances) the result of OCR. Also, as far as hyperlinking goes, I'm not aware of any potential copyright issues, so there shouldn't be a problem with Faithlife forgoing an in-house review (given that adequate Quality Control was in place, of course).

Reuben Helmuth

https://community.logos.com/discussion/comment/721152#Comment_721152

I think you would run into quality control issues.....

There are multiple ways of overcoming this potential. Let's discuss possible ways.

Reuben Helmuth

https://community.logos.com/discussion/comment/721180#Comment_721180

As far as quality control goes, I'd like to see Logos building a website where we could log-in with our Logos IDs, and vote for/against hyperlinks that other people had added.

Thanks for your input, Mark. What do you see as the advantages of a vote based QC system versus an Admin-approval system? I'm open to any ideas, just wondering what your thought process was.

Thanks!

Mark Barnes

https://community.logos.com/discussion/comment/721219#Comment_721219

Thanks for your input, Mark. What do you see as the advantages of a vote based QC system versus an Admin-approval system? I'm open to any ideas, just wondering what your thought process was.

Simply that it's not always easy to find someone to take responsibility as an admin. It's potentially a lot of work.

JAL

https://community.logos.com/discussion/comment/721230#Comment_721230

Simply that it's not always easy to find someone to take responsibility as an admin. It's potentially a lot of work.

Possibly administration responsibility could be assumed/assigned at the resource level - still potentially considerable work though.

Mark Barnes

https://community.logos.com/discussion/comment/721250#Comment_721250

Possibly administration responsibility could be assumed/assigned at the resource level - still potentially considerable work though.

Then you need a super admin to assign administration responsibility for each of thousands of resources....

Reuben Helmuth

https://community.logos.com/discussion/comment/721230#Comment_721230

Simply that it's not always easy to find someone to take responsibility as an admin. It's potentially a lot of work.

I understand that if responsibility fell on one person, it would be overwhelming! My thought was that it shouldn't be long before there would be 6-10 people qualified as "administrators" who could approve/reject submissions. Am I wrong to think that the voting process would take quite a bit longer, since each submission would need to be reviewed by multiple people?

Would it get too complicated to have a hybrid system? Perhaps a submission could have two paths to approval...

1) power-user review

2) ~5/7 affirmative votes by group members.

Say, for instance, that Rosie submitted a hyperlink which you [Mark] then reviewed and approved. To me, it would seem ridiculous to require a bunch of other people (including "young'ns" like me) to also review/approve the submission. [;)]

If group members had access to the database of typo submissions, it would not take long at all for a sizable number of users to acquire a critical mass of edit submissions which could be assessed for accuracy by Faithlife before granting "open throttle" access to qualifying users (c.f. my proposal in my second post).

With a votes system, how many or what majority of votes do you suggest that an 'edit submission' should receive before approval?

Mark Barnes

https://community.logos.com/discussion/comment/721487#Comment_721487

Yes, Reuben, I think a hybrid system like that would be an ideal one if it wasn't too complicated to implement. Something like this:

"Ordinary" users can submit typos and vote on other people's typos.
If a typo has 90% acceptance after 10 votes, it gets automatically accepted.
When a typo is accepted, the user who submitted it gets a point added to their profile, if it's rejected they lose a point.
Users with more than 100 points become superusers, whose can fast-track other people's typos to acceptance, without needing 10 votes.

Having said all that, I think it more likely Logos will be amenable to letting us crowdsource adding links, than they would let us crowdsource typos (I can't see publishers necessarily agreeing to the latter).

Randy W. Sims

https://community.logos.com/discussion/comment/721496#Comment_721496

Would a visual filter like solution that displayed user typo corrections be subject to publisher restrictions?

Reuben Helmuth

https://community.logos.com/discussion/comment/721508#Comment_721508

Would a visual filter like solution that displayed user typo corrections be subject to publisher restrictions?

I believe the restriction that Mark was referring to, is where the actual text of the resource gets updated.

Concerning this restriction, I think it would be in Logos' best interest to still allow "crowd-sourcing" of the typo fixing, but they could then have these submissions filtered. Those occurring in resources that wouldn't have these restrictions (e.g. Lexham Press), could use the express lane. With some publishers Faithlife could negotiate allowing this with the condition that the publisher be provided an exhaustive list of corrections (benefit to the publisher!) For the remaining instances (where restrictions remained) Logos would at least not have to wade through multiple reports of the same typo and/or incorrect reports. Instead, they would only need review a single instance of legitimate reports, helping streamline things on their end.

Veli Voipio

https://community.logos.com/discussion/comment/721154#Comment_721154

In some cases there are copyright issues. On more than one occasion I have seen someone report a typo/error, only to be told that it is actually not a typo but spelled that way in the actual resource.

How about a visual filter functionality to correct typos in that case?

Reuben Helmuth

https://community.logos.com/discussion/comment/721540#Comment_721540

In some cases there are copyright issues. On more than one occasion I have seen someone report a typo/error, only to be told that it is actually not a typo but spelled that way in the actual resource.

How about a visual filter functionality to correct typos in that case?

I think this would be a WONDERFUL solution. My thought is that the filter could automatically cause the typo to be in a gray font with the correction following (perhaps in a capsule).

Sean

https://community.logos.com/discussion/comment/721142#Comment_721142

While I do think there may be a space for this sort of thing on a small scale, enthusiasm should be tempered in offering to do for free what we've already paid Logos Faithlife to do.

In other words, this:

My initial thought is this: I have invested over $6,000 into my library with Logos. Many, if not all, of these are "value added." This means instead of the bargin at Kindle, I went with Logos. In the case of "From Heaven He Came and Sought Her" it was a $32.95 vs $3.99.

For that kind of money, I expect that all linkage be correct and complete. For me to pay a near 10X mark up I do not want to see the need for a crowdsourced editing; Logos should foot the bill for this and employ more/better QA/Editors.

This is why I have a thread that I regularly post in: Too often (and in the same book referenced above, just released this year) I find no hyperlinking. It is frustrating because this is why I have everything in Logos and why I pay the premium pricing. If I wanted to search for a refernce in a another book I would have used someone else.

Bottom Line: Logos needs to do its own policing and editing and increase (dare I say set any) the standard for hyperlinking.

Matthew C Jones

https://community.logos.com/discussion/comment/721142#Comment_721142

For that kind of money, I expect that all linkage be correct and complete. For me to pay a near 10X mark up I do not want to see the need for a crowdsourced editing; Logos should foot the bill for this and employ more/better QA/Editors.

My sentiments exactly.

Matthew C Jones

https://community.logos.com/discussion/comment/721154#Comment_721154

I would think that only Faithlife would be able to review and approve changes. In some cases there are copyright issues. On more than one occasion I have seen someone report a typo/error, only to be told that it is actually not a typo but spelled that way in the actual resource. Users who don't have the original files to compare their changes with may actually be changing something in the resource that should not be changed.

I think this is the only reasonable excuse for having typos present. I was told it is rude to correct others' typos in the forums and I noticed the "(sic)" footnote in professional journalism. So maybe there is something to leaving source material typos intact.

Rick

https://community.logos.com/discussion/comment/721540#Comment_721540

How about a visual filter functionality to correct typos in that case?

Hi Veli, I honestly don't know what the best course of action to take would be. I really have not put a whole lot of thought into it.

Although I really would appreciate 100% accuracy, the few typos that I encounter don't really bother me. What really irritates me is when I find things that are completely omitted such as missing images, maps etc.

Disclaimer: I use Logos only for my personal study. I am not a student and hold no office or position in the church. I completely understand why those who use Logos as a student, elder or for professional reasons, need higher standards. I'm sure that I don't use and depend on it nearly as much as they do.

Rick

https://community.logos.com/discussion/comment/721580#Comment_721580

I think this is the only reasonable excuse for having typos present.

I agree. I believe that we all realize that mistakes happen, but, we should expect these mistakes to be fixed within a reasonable time. Sometimes, though, it appears as if Faithlife simply doesn't care. There are some resources that are simply horrible. An example, for me, would be "First Corinthians: An Exegetical-Pastoral Commentary". It was released over two years ago and from day one has had very bad formatting that makes grey areas look like some kind of a highlighting. See: https://community.logos.com/forums/t/67793.aspx

I know that this can be fixed because many other resources have the same kind of grey boxes that these highlights are supposed to be.

Eli Evans (Logos)

https://community.logos.com/discussion/comment/721508#Comment_721508

Hi, Eli Evans here. Head of interaction design.

Hyperlink tagging: I've added a card to our ideas board to extend Community Tagging to all data types, not just Factbook headings, and also to periodically "harvest" these tags and integrate them into the index (rather than keeping them segregated as Community Tags). The first part is pretty straightforward; the second part will pose some (surmountable but non-trivial) technical challenges.

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print.

But we could make improvements to the typo reporting and fixing workflow, as well as the hyperlink tagging process. We're coming up on a audit/review of both of those processes in a few weeks. (Don't know what will come of that yet. That's why we do the review. [:)])

The visual filter to replace text is an interesting idea. We've discussed it internally before; the main hesitation is that changing the surface text won't change the underlying index, so searching and other features will still work off of the unchanged data.

Such a filter would need a dataset of pending corrections to accompany it, and creating that dataset from the existing typo reports wouldn't be perfect. There would definitely have to be some kind of multiple-report threshold; you'd be surprised what some people will write into the correction slot (hint: not corrected text). Even then it's surprising how often people try to correct things that aren't wrong, like British spellings or references to Joel 4 that (correctly) point to the Hebrew versification. Plus many people leave the correction text blank. Sometimes that means the word should be deleted; sometimes it's an indication that the correction is self-evident. Computer programs can't tell the difference.

Anyway, I can't promise anything concrete, but this issue is on our radar.

David Taylor, Jr.

https://community.logos.com/discussion/comment/722054#Comment_722054

Hi, Eli Evans here. Head of interaction design.

Hyperlink tagging: I've added a card to our ideas board to extend Community Tagging to all data types, not just Factbook headings, and also to periodically "harvest" these tags and integrate them into the index (rather than keeping them segregated as Community Tags). The first part is pretty straightforward; the second part will pose some (surmountable but non-trivial) technical challenges.

This sounds more than reasonable!

MJ. Smith

https://community.logos.com/discussion/comment/722054#Comment_722054

I would love the equivalent of community tags to add links however there is an elephant in the room ... its called "missing datatypes" especially relating to church fathers from my experience but scattered widely elsewhere ... it also has a shadowy cousin called "duplicate inconsistent datatypes" where the same text is given different datatypes in different environments ... then there is the stepchild called "missing synonyms" which is agitating the current Factbook set.. I suppose the good news is that the elephant(s) in the room keep the population of wild and unruly elephants up ... the bad news is they give me headaches.

David Paul

https://community.logos.com/discussion/comment/722054#Comment_722054

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print.

Understood, but to reduce user frustration (at least to some degree), I suggest that Logos implement some sort of margin icon similar to the speaker icon that would indicate that a given typo has previously been reported and/or acknowledged but cannot currently be fixed. That will alert users to the situation and keep them from wasting time and help them know that the situation has been addressed even if not remedied.

Reuben Helmuth

https://community.logos.com/discussion/comment/722054#Comment_722054

to periodically "harvest" these tags and integrate them into the index

Great! I hope that this "periodically" would be no greater than the 6 week release cycle.

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print.

But we could make improvements to the typo reporting and fixing workflow,

Hopefully the suggestions in this thread can help chart the course! [;)]

you'd be surprised what some people will write into the correction slot (hint: not corrected text). Even then it's surprising how often people try to correct things that aren't wrong, like British spellings or references to Joel 4 that (correctly) point to the Hebrew versification. Plus many people leave the correction text blank. Sometimes that means the word should be deleted; sometimes it's an indication that the correction is self-evident. Computer programs can't tell the difference.

If I don't miss my guess, this is one of the major reasons for the (reportedly) huge backlog of typos. In my mind this would be a tremendous reason to allow the system we're proposing...the "corrections" that make it through to the legit status would be vastly reduced in number, meaning that Faithlife employees wouldn't be quite as "snowed under."

The visual filter to replace text is an interesting idea. We've discussed it internally before; the main hesitation is that changing the surface text won't change the underlying index, so searching and other features will still work off of the unchanged data.

If the visual filter was replacing adding text, wouldn't it be possible to have searching include the added text? For the visual filter I would propose a main filter called "User Corrections" that would have two subsets; 1) "pending approval" & 2) "verified"(but permanent). The "pending approval" would include all (group) approved corrections (mostly typos) that needed further review. After Faithlife reviewed these to determine their permissibility, they would be moved either into the index of "harvested" corrections, or into the second sub-filter "permanent."

Lee

https://community.logos.com/discussion/comment/722054#Comment_722054

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print...

If the typos are exactly as printed, I guess most users are willing to accept that.

Even then it's surprising how often people try to correct things that aren't wrong, like British spellings or references to Joel 4 that (correctly) point to the Hebrew versification. Plus many people leave the correction text blank. Sometimes that means the word should be deleted; sometimes it's an indication that the correction is self-evident. Computer programs can't tell the difference.

Consider the suggestion of having select few users who are more experienced or cleared for greater privileges.

MJ. Smith

https://community.logos.com/discussion/comment/722343#Comment_722343

Cold water alert: using the new concordance feature on two books, a statistically insignificant sample, I exported the word list into Excel and spellchecked it. I chose two texts I personally perceived as error-prone. To my surprise what I found was very few spelling errors. (Note this method doesn't catch valid spellings but an incorrect word). I suspect that the level of accuracy that Bob P. touts is, in fact, met. What I did find was a number of errors in language tagging.

Secondly, looking at community tagging thus far I am unimpressed with the quality - Israel the nation and Israel the person being frequently mistagged by the community tags in the particular book I am reading.

I like the idea of harvesting tagging and corrections and think it could ultimately work. But I suspect that as a solution it is far more complex that it appears on the surface.

I have an older sister who works as a proof-reader for an educational software firm. She's pushing into the "elderly" category and works from home. What I would like Faithlife to consider as a starting point is letting selected individuals prove their proof-reading skills (something I couldn't do) and allow them to work through assigned books in exchange for free books

Jacob Hantla

https://community.logos.com/discussion/comment/722054#Comment_722054

Hi, Eli Evans here. Head of interaction design.

Hyperlink tagging: I've added a card to our ideas board to extend Community Tagging to all data types, not just Factbook headings, and also to periodically "harvest" these tags and integrate them into the index (rather than keeping them segregated as Community Tags). The first part is pretty straightforward; the second part will pose some (surmountable but non-trivial) technical challenges.

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print.

But we could make improvements to the typo reporting and fixing workflow, as well as the hyperlink tagging process. We're coming up on a audit/review of both of those processes in a few weeks. (Don't know what will come of that yet. That's why we do the review. )

The visual filter to replace text is an interesting idea. We've discussed it internally before; the main hesitation is that changing the surface text won't change the underlying index, so searching and other features will still work off of the unchanged data.

Such a filter would need a dataset of pending corrections to accompany it, and creating that dataset from the existing typo reports wouldn't be perfect. There would definitely have to be some kind of multiple-report threshold; you'd be surprised what some people will write into the correction slot (hint: not corrected text). Even then it's surprising how often people try to correct things that aren't wrong, like British spellings or references to Joel 4 that (correctly) point to the Hebrew versification. Plus many people leave the correction text blank. Sometimes that means the word should be deleted; sometimes it's an indication that the correction is self-evident. Computer programs can't tell the difference.

Anyway, I can't promise anything concrete, but this issue is on our radar.

Thank you so much, Eli!

As youre considering this direction, consider mobile apps as a primary platform for reading.

Nathan Parker

It would be neat for typos. Also, for those that want a "Crowdsourced Study Bible", I setup the Community Study Bible Faithlife group. It's neat to check out. Needs some more interaction though.

JAL

https://community.logos.com/discussion/comment/722080#Comment_722080

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print.

Understood, but to reduce user frustration (at least to some degree), I suggest that Logos implement some sort of margin icon similar to the speaker icon that would indicate that a given typo has previously been reported and/or acknowledged but cannot currently be fixed. That will alert users to the situation and keep them from wasting time and help them know that the situation has been addressed even if not remedied.

Eli, I like David's suggestion. Some means of identifying known errors that doesn't violate copyright, or publisher contract, must be attainable.

Lee

https://community.logos.com/discussion/comment/722469#Comment_722469

I have an older sister who works as a proof-reader for an educational software firm. She's pushing into the "elderly" category and works from home. What I would like Faithlife to consider as a starting point is letting selected individuals prove their proof-reading skills (something I couldn't do) and allow them to work through assigned books in exchange for free books

I suggested something like this a couple of years ago. F.L. ought to consider this. For example, months ago someone who was learning/teaching Hebrew pointed out glaring errors in a Hebrew grammar. There's a lot to be said for incentivizing individuals like that.

David Paul

https://community.logos.com/discussion/comment/722569#Comment_722569

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print.

Understood, but to reduce user frustration (at least to some degree), I suggest that Logos implement some sort of margin icon similar to the speaker icon that would indicate that a given typo has previously been reported and/or acknowledged but cannot currently be fixed. That will alert users to the situation and keep them from wasting time and help them know that the situation has been addressed even if not remedied.

Eli, I like David's suggestion. Some means of identifying known errors that doesn't violate copyright, or publisher contract, must be attainable.

This would be the equivalent of an ebook sic. When quoting books with errors, sic is used to indicated the error is in the book and is being quoted "as is". It makes perfect sense that ebooks would indicate the same thing.

Reuben Helmuth

https://community.logos.com/discussion/comment/722603#Comment_722603

I've been working on a proposal/ideas document to submit to Logos and would welcome any feedback.

Crowdsourcing Proposal

This is an informal (and developing) proposal for a system that would exploit crowdsourcing for the correction of typos and the input/correction of hyperlinks throughout the Logos ecosystem of resources. Suggestions and feedback are welcome and can be directed to reubedu AT gmail

Pre-requisite: User requests permission to join the “typo/links” Faithlife group.

Typos

Group member submits a typo

This is immediately visible to other group members as a visual filter.

Group members vote on submissions.

The initial filter would be red and hovering would display both the original and proposed spelling with the option to submit a yes/no vote directly from the tag. Each user could only vote once for a given submission. Voting will be anonymous to group users to prevent biased voting).
After receiving a minimum of 10 votes with a 90+% acceptance rate, a tag is changed from red to green and the option to vote is removed. The submission is now “ripe for harvesting.”
In the event of receiving multiple negative votes, a submission could accept up to 20 votes in an effort to reach the required 90%, before being rejected.

A system of points will be put into place for several purposes. Firstly, it will prevent indiscriminate voting and promote caution and carefulness throughout (think QC). Secondly, it will provide a means for “compensation” (more on that later).

If a submission gets accepted, the original user who submitted the correction as well as those who voted in favor, will receive 1 point, while those who voted against will be docked a point.
If a submission gets rejected (doesn’t reach 90% approval by 20 votes), the original poster as well as those who voted in favor, will get docked 1 point and those who voted against will gain a point.

Approved/accepted submissions get harvested at the time of the six week release cycle.

In the case of typos, “harvesting” would be slightly more time consuming on the Faithlife employee end of things since each submission would need to be assessed against the original document because of permission issues. Bear in mind, however, that these submissions would not need to be weeded through for non-sense, British/American spellings, or other erroneous reports that are currently plaguing the typo-report database.
Typos which prove to be present in the original document (and aren't permitted to be corrected) would be moved to a separate visual filter document that would insert the correction immediately after the typo. The filter could also cause the misspelled word to be struck through or grayed out. THIS visual filter should be available to ALL users (like propositional outlines). It could obviously be turned off and thus should not violate publisher prohibitions for typo correction.
Searches should then support the text in the visual filter as though it were surface text (or whatever the filter was applied to).

Hyperlinks to Bible References (only minor differences from typos above)

Group member enters a link to a Bible reference.

This is immediately visible AND clickable for other group members.

Group members vote on accuracy of links.

Same steps as Typos section

System of Points

Same as typos section, but that each submission (and vote) would be worth 2 points instead of only 1.

Approved/accepted submissions get harvested at the time of the six week release cycle.

These could be harvested “wholesale” without manual intervention!!

Hyperlinks to Non-Bible References

Same as the Bible reference section except that each submission (and vote) would be worth 5 points.

Super-Users

Group members will accumulate “points” as they submit or vote on corrections.

Members who acquire (and retain) a total of 500 points will receive “super-user” status

Super-users could function much like regular users, but their votes would be worth half the total required votes for a submission. In other words, where it would take 10 regular users to approve a submission, 2 super-users could accomplish the same thing. Also, 1 super-user combined with 5 regular members would provide approval. In light of the ration required for approval, a super-users negative vote may not be able to carry as much weight as his/her positive vote.

Super-users could effectively use their track-record of reliability to “fast-track” submissions into the harvest-ready stage.

Compensation

Users will be compensated on a monthly basis.

Much like Faithlife is offering Logos credit as compensation to competent writers, group members will be compensated for their help on a per point basis each month. I propose the nominal fee of a penny per point per period. Each month a user’s new points will be converted to Logos credit at a rate of $0.01/point and applied to the users Logos account.

These credits will be non-expiring and can be allowed to accumulate in order to buy any Logos resources available on logos.com

David Taylor, Jr.

https://community.logos.com/discussion/comment/722603#Comment_722603

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print.

Understood, but to reduce user frustration (at least to some degree), I suggest that Logos implement some sort of margin icon similar to the speaker icon that would indicate that a given typo has previously been reported and/or acknowledged but cannot currently be fixed. That will alert users to the situation and keep them from wasting time and help them know that the situation has been addressed even if not remedied.

Eli, I like David's suggestion. Some means of identifying known errors that doesn't violate copyright, or publisher contract, must be attainable.

This would be the equivalent of an ebook sic. When quoting books with errors, sic is used to indicated the error is in the book and is being quoted "as is". It makes perfect sense that ebooks would indicate the same thing.

I see your point but it isn't exactly equivalent. You don't see people who re-publish a public domain work putting [sic] within the text.

At the end of the day publishers need to catch up to the digital age where it doesn't cost thousands of dollars to redo a typeset to fix an error or running a new edition. It is a couple of keystrokes combined with the submit button.

Brian Wilson

https://community.logos.com/discussion/comment/721496#Comment_721496

Yes, Reuben, I think a hybrid system like that would be an ideal one if it wasn't too complicated to implement. Something like this:

"Ordinary" users can submit typos and vote on other people's typos.

If a typo has 90% acceptance after 10 votes, it gets automatically accepted.

When a typo is accepted, the user who submitted it gets a point added to their profile, if it's rejected they lose a point.

Users with more than 100 points become superusers, whose can fast-track other people's typos to acceptance, without needing 10 votes.

and those with a certain amount of points get logos credit for future purchases.

Eli Evans (Logos)

https://community.logos.com/discussion/comment/726082#Comment_726082

Thank you everyone for the enthusiasm and willingness to pitch in.

Crowdsourcing hyperlink tagging might be a good idea, or it might not. Either way, it's definitely "before its time."

We've discussed similar ideas for allowing data collection for hyperlinks by users a number of times over the years. The hold up on our end isn't really the "How would we manage the collection?" (front-end) part of it so much as the "Do we have all the infrastructure and workflows necessary to do what we need to with the resulting data?" (back-end) part. We'd hate to build a hyperlink collection system that encouraged users to waste time tagging references that went into a black hole. We'd also hate to build a data integration system that would require rebuilding, re-downloading, or re-indexing resources too frequently, or en masse. We'd hate to build a system where you couldn't get remunerated for tags, or couldn't freely self-manage your level of contribution.

In the meantime, we've been building most of the infrastructure components that would be necessary for it to work for other reasons -- for example, Community Tags and a points system for remuneration, and lot of other plumbing stuff that you don't see. I still think we're a ways off, though. Some of the necessary pieces were just shipped in L6 and others are still on the drawing board.

We are definitely thinking creatively about the bigger problem (namely: how to tag more stuff more fast but not less good).