CrowdSourcing: Is it time for a revival...

Some of my own thoughts are:

Reuben Helmuth said:1) Could absolutely anyone make "corrections" or only a certain Group?

Users would need to join a Faithlife Group in order to gain access to the correct tools. This could be a closed group in the event that Faithlife felt like they needed ultimate control over who can edit. Edits would only be visible/available to those in the group until reviewed and approved for "official" release.

Reuben Helmuth said:

 2) Would corrections/tags get reviewed by a committee (from the "crowd") or only by Faithlife? 

Initially, a Faithlife employee would do ALL the reviewing/approving, but would grant this power to select users upon meeting certain requirements. The requirements could be ">100 edits submitted" & ">95% accuracy" (accuracy=correct identification AND tagging) My thought is that upon meeting BOTH requirements, a user would be able to review/approve the edits of other users. In this way the burden of reviewing thousands of edits could be spread out fairly quickly while still maintaining a very decent level of QC.

Reuben Helmuth said:

3) Would corrections/tags get released to resource updates on a regular basis?

"Administrator" approved edits should be released at least every six weeks (along with the normal release cycle). Once the updating of resources no longer requires a restart (and all the bugs are out!), I don't see why approved edits couldn't be released weekly.

EDIT: On second thought, it doesn't make sense to do a (even restart-less) resource update for only a single fixed typo, so perhaps resources should be updated every 10 edits OR every 6 weeks, whichever comes first.

Everett Headley said:

My initial thought is this:  I have invested over $6,000 into my library with Logos.  Many, if not all, of these are "value added."  This means instead of the bargin at Kindle, I went with Logos.  In the case of "From Heaven He Came and Sought Her" it was a $32.95 vs $3.99.  

For that kind of money, I expect that all linkage be correct and complete. For me to pay a near 10X mark up I do not want to see the need for a crowdsourced editing; Logos should foot the bill for this and employ more/better QA/Editors.

I feel your pain! My own investment is dangerously close to 5 figures. [:S] I agree that crowdsourcing shouldn't be needed, but to me it looks like it is! Realizing that there would be some (many) who do not wish to "contribute," no one would be forced to be part of the group. On the other hand, the missing links that you so much dislike could be fixed in a very timely manner. I think that by the end of a year, the vast majority of missing links/errors could be fixed.

I do think that if (when) this gets implemented, we should have a commitment from Faithlife that they won't "get lazy" because of it! 

Don't you wish that every typo that you report would be fixed within a month?! I think that (this is an added detail to the system)... the general typo reports of ALL users should be accessible to the Group, rather than relying on only Group Members to find issues.

David Taylor Jr said:

I think you would run into quality control issues.....

There are multiple ways of overcoming this potential. Let's discuss possible ways.

Randy W. Sims said:

Would a visual filter like solution that displayed user typo corrections be subject to publisher restrictions?

I believe the restriction that Mark was referring to, is where the actual text of the resource gets updated.

Concerning this restriction, I think it would be in Logos' best interest to still allow "crowd-sourcing" of the typo fixing, but they could then have these submissions filtered. Those occurring in resources that wouldn't have these restrictions (e.g. Lexham Press), could use the express lane. With some publishers Faithlife could negotiate allowing this with the condition that the publisher be provided an exhaustive list of corrections (benefit to the publisher!) For the remaining instances (where restrictions remained) Logos would at least not have to wade through multiple reports of the same typo and/or incorrect reports. Instead, they would only need review a single instance of legitimate reports, helping streamline things on their end.

Veli Voipio said:

Rick said:

In some cases there are copyright issues. On more than one occasion I have seen someone report a typo/error, only to be told that it is actually not a typo but spelled that way in the actual resource.

How about a visual filter functionality to correct typos in that case?

I think this would be a WONDERFUL solution. My thought is that the filter could automatically cause the typo to be in a gray font with the correction following (perhaps in a capsule).

Eli Evans said:

to periodically "harvest" these tags and integrate them into the index

Great! I hope that this "periodically" would be no greater than the 6 week release cycle.

Eli Evans said:

Typo fixing: Unfortunately, there's just no way around the legalities involved in letting users fix typos. Sad but true: Sometimes WE don't get permission to fix typos in the print. 

But we could make improvements to the typo reporting and fixing workflow, 

Hopefully the suggestions in this thread can help chart the course! [;)]

Eli Evans said:

you'd be surprised what some people will write into the correction slot (hint: not corrected text). Even then it's surprising how often people try to correct things that aren't wrong, like British spellings or references to Joel 4 that (correctly) point to the Hebrew versification. Plus many people leave the correction text blank. Sometimes that means the word should be deleted; sometimes it's an indication that the correction is self-evident. Computer programs can't tell the difference. 

If I don't miss my guess, this is one of the major reasons for the (reportedly) huge backlog of typos. In my mind this would be a tremendous reason to allow the system we're proposing...the "corrections" that make it through to the legit status would be vastly reduced in number, meaning that Faithlife employees wouldn't be quite as "snowed under."

Eli Evans said:

The visual filter to replace text is an interesting idea. We've discussed it internally before; the main hesitation is that changing the surface text won't change the underlying index, so searching and other features will still work off of the unchanged data. 

If the visual filter was replacing adding text, wouldn't it be possible to have searching include the added text? For the visual filter I would propose a main filter called "User Corrections" that would have two subsets; 1) "pending approval" & 2) "verified"(but permanent). The "pending approval" would include all (group) approved corrections (mostly typos) that needed further review. After Faithlife reviewed these to determine their permissibility, they would be moved either into the index of "harvested" corrections, or into the second sub-filter "permanent." 

I've been working on a proposal/ideas document to submit to Logos and would welcome any feedback.

Crowdsourcing Proposal

This is an informal (and developing) proposal for a system that would exploit crowdsourcing for the correction of typos and the input/correction of hyperlinks throughout the Logos ecosystem of resources. Suggestions and feedback are welcome and can be directed to reubedu AT gmail

Pre-requisite: User requests permission to join the “typo/links” Faithlife group.

1. Typos
1. Group member submits a typo 
1. This is immediately visible to other group members as a visual filter.
2. Group members vote on submissions.
1. The initial filter would be red and hovering would display both the original and proposed spelling with the option to submit a yes/no vote directly from the tag. Each user could only vote once for a given submission. Voting will be anonymous to group users to prevent biased voting).
2. After receiving a minimum of 10 votes with a 90+% acceptance rate, a tag is changed from red to green and the option to vote is removed. The submission is now “ripe for harvesting.”
3. In the event of receiving multiple negative votes, a submission could accept up to 20 votes in an effort to reach the required 90%, before being rejected.
3. A system of points will be put into place for several purposes. Firstly, it will prevent indiscriminate voting and promote caution and carefulness throughout (think QC). Secondly, it will provide a means for “compensation” (more on that later).
1. If a submission gets accepted, the original user who submitted the correction as well as those who voted in favor, will receive 1 point, while those who voted against will be docked a point.
2. If a submission gets rejected (doesn’t reach 90% approval by 20 votes), the original poster as well as those who voted in favor, will get docked 1 point and those who voted against will gain a point.
4. Approved/accepted submissions get harvested at the time of the six week release cycle.
1. In the case of typos, “harvesting” would be slightly more time consuming on the Faithlife employee end of things since each submission would need to be assessed against the original document because of permission issues. Bear in mind, however, that these submissions would not need to be weeded through for non-sense, British/American spellings, or other erroneous reports that are currently plaguing the typo-report database.
2. Typos which prove to be present in the original document (and aren't permitted to be corrected) would be moved to a separate visual filter document that would insert the correction immediately after the typo. The filter could also cause the misspelled word to be struck through or grayed out. THIS visual filter should be available to ALL users (like propositional outlines). It could obviously be turned off and thus should not violate publisher prohibitions for typo correction. 
3. Searches should then support the text in the visual filter as though it were surface text (or whatever the filter was applied to). 

1. Hyperlinks to Bible References (only minor differences from typos above)
1. Group member enters a link to a Bible reference.
1. This is immediately visible AND clickable for other group members.
2. Group members vote on accuracy of links.
1. Same steps as Typos section
3. System of Points
1. Same as typos section, but that each submission (and vote) would be worth 2 points instead of only 1.
4. Approved/accepted submissions get harvested at the time of the six week release cycle.
1. These could be harvested “wholesale” without manual intervention!!

1. Hyperlinks to Non-Bible References
1. Same as the Bible reference section except that each submission (and vote) would be worth 5 points.

1. Super-Users
1. Group members will accumulate “points” as they submit or vote on corrections.
1. Members who acquire (and retain) a total of 500 points will receive “super-user” status
2. Super-users could function much like regular users, but their votes would be worth half the total required votes for a submission. In other words, where it would take 10 regular users to approve a submission, 2 super-users could accomplish the same thing. Also, 1 super-user combined with 5 regular members would provide approval. In light of the ration required for approval, a super-users negative vote may not be able to carry as much weight as his/her positive vote.
1. Super-users could effectively use their track-record of reliability to “fast-track” submissions into the harvest-ready stage.

1. Compensation
1. Users will be compensated on a monthly basis.
1. Much like Faithlife is offering Logos credit as compensation to competent writers, group members will be compensated for their help on a per point basis each month. I propose the nominal fee of a penny per point per period. Each month a user’s new points will be converted to Logos credit at a rate of $0.01/point and applied to the users Logos account.
1. These credits will be non-expiring and can be allowed to accumulate in order to buy any Logos resources available on logos.com