CrowdSourcing: Is it time for a revival...

Page 1 of 3 (44 items) 1 2 3 Next >
This post has 43 Replies | 4 Followers

Posts 1998
Forum MVP
Reuben Helmuth | Forum Activity | Posted: Tue, Apr 28 2015 2:23 AM

... of support/interest in crowdsourcing the typo/tagging problems?! Jonathan Pitts created this UserVoice suggestion over 4 years ago and it seems to have been forgotten. One thing that we can all do is vote for his suggestion, but I thought that perhaps we could also use this thread to "brain storm" how the system could/should work...

For example:

1) Could absolutely anyone make "corrections" or only a certain Group?

2) Would corrections/tags get reviewed by a committee (from the "crowd") or only by Faithlife?

3) Would corrections/tags get released to resource updates on a regular basis? (consider the no-restart-required updates being worked on in the beta release)

4) ???

Posts 765
Josh Hunt | Forum Activity | Replied: Tue, Apr 28 2015 7:01 AM

This seems like a great idea to me

Posts 1998
Forum MVP
Reuben Helmuth | Forum Activity | Replied: Tue, Apr 28 2015 7:02 AM

Some of my own thoughts are:

Reuben Helmuth:
1) Could absolutely anyone make "corrections" or only a certain Group?

Users would need to join a Faithlife Group in order to gain access to the correct tools. This could be a closed group in the event that Faithlife felt like they needed ultimate control over who can edit. Edits would only be visible/available to those in the group until reviewed and approved for "official" release.

Reuben Helmuth:
 2) Would corrections/tags get reviewed by a committee (from the "crowd") or only by Faithlife? 

Initially, a Faithlife employee would do ALL the reviewing/approving, but would grant this power to select users upon meeting certain requirements. The requirements could be ">100 edits submitted" & ">95% accuracy" (accuracy=correct identification AND tagging) My thought is that upon meeting BOTH requirements, a user would be able to review/approve the edits of other users. In this way the burden of reviewing thousands of edits could be spread out fairly quickly while still maintaining a very decent level of QC.

Reuben Helmuth:
3) Would corrections/tags get released to resource updates on a regular basis?

"Administrator" approved edits should be released at least every six weeks (along with the normal release cycle). Once the updating of resources no longer requires a restart (and all the bugs are out!), I don't see why approved edits couldn't be released weekly.

EDIT: On second thought, it doesn't make sense to do a (even restart-less) resource update for only a single fixed typo, so perhaps resources should be updated every 10 edits OR every 6 weeks, whichever comes first.
Posts 943
Everett Headley | Forum Activity | Replied: Tue, Apr 28 2015 7:24 AM

My initial thought is this:  I have invested over $6,000 into my library with Logos.  Many, if not all, of these are "value added."  This means instead of the bargin at Kindle, I went with Logos.  In the case of "From Heaven He Came and Sought Her" it was a $32.95 vs $3.99.  

For that kind of money, I expect that all linkage be correct and complete. For me to pay a near 10X mark up I do not want to see the need for a crowdsourced editing; Logos should foot the bill for this and employ more/better QA/Editors.

This is why I have a thread that I regularly post in:  Too often (and in the same book referenced above, just released this year) I find no hyperlinking.  It is frustrating because this is why I have everything in Logos and why I pay the premium pricing.  If I wanted to search for a refernce in a another book I would have used someone else.

Bottom Line:  Logos needs to do its own policing and editing and increase (dare I say set any) the standard for hyperlinking.

Posts 8967
RIP
Matthew C Jones | Forum Activity | Replied: Tue, Apr 28 2015 7:34 AM

Reuben Helmuth:
1) Could absolutely anyone make "corrections" or only a certain Group?

No

Logos 7 Collectors Edition

Posts 3017
David Taylor Jr | Forum Activity | Replied: Tue, Apr 28 2015 7:45 AM

While I like the intent of the idea, I think you would run into quality control issues.....

Teacher, Ministry Leader, Student, Author, Husband

How to upload logs

Visit My Site: Reformed Truths

Posts 1647
Rick | Forum Activity | Replied: Tue, Apr 28 2015 7:48 AM

Reuben Helmuth:
2) Would corrections/tags get reviewed by a committee (from the "crowd") or only by Faithlife?

I would think that only Faithlife would be able to review and approve changes. In some cases there are copyright issues. On more than one occasion I have seen someone report a typo/error, only to be told that it is actually not a typo but spelled that way in the actual resource. Users who don't have the original files to compare their changes with may actually be changing something in the resource that should not be changed. 

Peace  Smile

Romans 14:19 (NRSV)
19 Let us then pursue what makes for peace and for mutual upbuilding.

Posts 2922
Forum MVP
Jacob Hantla | Forum Activity | Replied: Tue, Apr 28 2015 7:48 AM

I've been begging for this for years. there is way too much latency (or sometimes it never gets done) on fixing typos or updating tags. Vyrso books would never get fixed apart from something like this. If they could incentivize large volume of quality link creation and typo fixing the customers would be happy and the quality of the resources greatly improved. Tagging is the main place this is needed

Jacob Hantla
Pastor/Elder, Grace Bible Church
gbcaz.org

Posts 2922
Forum MVP
Jacob Hantla | Forum Activity | Replied: Tue, Apr 28 2015 7:50 AM

by the way, in the meantime we can use community tags to sort of do this from Logos 6 (but the links aren't visible on mobile and only for people who enable community Tags)

Jacob Hantla
Pastor/Elder, Grace Bible Church
gbcaz.org

Posts 13360
Forum MVP
Mark Barnes | Forum Activity | Replied: Tue, Apr 28 2015 8:40 AM

Logos have moved in this direction with community tags. I'd love to see them adding community hyperlinks, so long as those hyperlinks were properly integrated into our library and could be searched, etc.

As far as quality control goes, I'd like to see Logos building a website where we could log-in with our Logos IDs, and vote for/against hyperlinks that other people had added. That would ensure quality, and whilst there'd be investment in building the site, there'd be little ongoing cost.

Posts 1998
Forum MVP
Reuben Helmuth | Forum Activity | Replied: Tue, Apr 28 2015 9:08 AM

Everett Headley:

My initial thought is this:  I have invested over $6,000 into my library with Logos.  Many, if not all, of these are "value added."  This means instead of the bargin at Kindle, I went with Logos.  In the case of "From Heaven He Came and Sought Her" it was a $32.95 vs $3.99.  

For that kind of money, I expect that all linkage be correct and complete. For me to pay a near 10X mark up I do not want to see the need for a crowdsourced editing; Logos should foot the bill for this and employ more/better QA/Editors.

I feel your pain! My own investment is dangerously close to 5 figures. Tongue Tied I agree that crowdsourcing shouldn't be needed, but to me it looks like it is! Realizing that there would be some (many) who do not wish to "contribute," no one would be forced to be part of the group. On the other hand, the missing links that you so much dislike could be fixed in a very timely manner. I think that by the end of a year, the vast majority of missing links/errors could be fixed.

I do think that if (when) this gets implemented, we should have a commitment from Faithlife that they won't "get lazy" because of it! 

Don't you wish that every typo that you report would be fixed within a month?! I think that (this is an added detail to the system)... the general typo reports of ALL users should be accessible to the Group, rather than relying on only Group Members to find issues.

Posts 1998
Forum MVP
Reuben Helmuth | Forum Activity | Replied: Tue, Apr 28 2015 10:05 AM

Rick:
I would think that only Faithlife would be able to review and approve changes. In some cases there are copyright issues.

In the case of potential copyright issues Faithlife could do a final review. There are a number of typos that I've seen that are (by all appearances) the result of OCR. Also, as far as hyperlinking goes, I'm not aware of any potential copyright issues, so there shouldn't be a problem with Faithlife forgoing an in-house review (given that adequate Quality Control was in place, of course).

Posts 1998
Forum MVP
Reuben Helmuth | Forum Activity | Replied: Tue, Apr 28 2015 10:06 AM

David Taylor Jr:
I think you would run into quality control issues.....

There are multiple ways of overcoming this potential. Let's discuss possible ways.

Posts 1998
Forum MVP
Reuben Helmuth | Forum Activity | Replied: Tue, Apr 28 2015 10:10 AM

Mark Barnes:
As far as quality control goes, I'd like to see Logos building a website where we could log-in with our Logos IDs, and vote for/against hyperlinks that other people had added.

Thanks for your input, Mark. What do you see as the advantages of a vote based QC system versus an Admin-approval system? I'm open to any ideas, just wondering what your thought process was.

Thanks!

Posts 13360
Forum MVP
Mark Barnes | Forum Activity | Replied: Tue, Apr 28 2015 10:35 AM

Reuben Helmuth:
Thanks for your input, Mark. What do you see as the advantages of a vote based QC system versus an Admin-approval system? I'm open to any ideas, just wondering what your thought process was.

Simply that it's not always easy to find someone to take responsibility as an admin. It's potentially a lot of work.

Posts 623
JAL | Forum Activity | Replied: Tue, Apr 28 2015 11:06 AM

Mark Barnes:
Simply that it's not always easy to find someone to take responsibility as an admin. It's potentially a lot of work.

Possibly administration responsibility could be assumed/assigned at the resource level - still potentially considerable work though.

"The Christian mind is the prerequisite of Christian thinking. And Christian thinking is the prerequisite of Christian action." - Harry Blamires, 1963

Posts 13360
Forum MVP
Mark Barnes | Forum Activity | Replied: Tue, Apr 28 2015 11:10 AM

JAL:
Possibly administration responsibility could be assumed/assigned at the resource level - still potentially considerable work though.

Then you need a super admin to assign administration responsibility for each of thousands of resources....

Posts 1998
Forum MVP
Reuben Helmuth | Forum Activity | Replied: Wed, Apr 29 2015 5:54 AM

Mark Barnes:
Simply that it's not always easy to find someone to take responsibility as an admin. It's potentially a lot of work.

I understand that if responsibility fell on one person, it would be overwhelming! My thought was that it shouldn't be long before there would be 6-10 people qualified as "administrators" who could approve/reject submissions. Am I wrong to think that the voting process would take quite a bit longer, since each submission would need to be reviewed by multiple people? 

Would it get too complicated to have a hybrid system? Perhaps a submission could have two paths to approval...

1) power-user review

2) ~5/7 affirmative votes by group members.

Say, for instance, that Rosie submitted a hyperlink which you [Mark] then reviewed and approved. To me, it would seem ridiculous to require a bunch of other people (including "young'ns" like me) to also review/approve the submission. Wink

If group members had access to the database of typo submissions, it would not take long at all for a sizable number of users to acquire a critical mass of edit submissions which could be assessed for accuracy by Faithlife before granting "open throttle" access to qualifying users (c.f. my proposal in my second post).

With a votes system, how many or what majority of votes do you suggest that an 'edit submission' should receive before approval?

Posts 13360
Forum MVP
Mark Barnes | Forum Activity | Replied: Wed, Apr 29 2015 6:41 AM

Yes, Reuben, I think a hybrid system like that would be an ideal one if it wasn't too complicated to implement. Something like this:

  1. "Ordinary" users can submit typos and vote on other people's typos.
  2. If a typo has 90% acceptance after 10 votes, it gets automatically accepted.
  3. When a typo is accepted, the user who submitted it gets a point added to their profile, if it's rejected they lose a point.
  4. Users with more than 100 points become superusers, whose can fast-track other people's typos to acceptance, without needing 10 votes.

Having said all that, I think it more likely Logos will be amenable to letting us crowdsource adding links, than they would let us crowdsource typos (I can't see publishers necessarily agreeing to the latter).

Posts 2062
Forum MVP
Randy W. Sims | Forum Activity | Replied: Wed, Apr 29 2015 7:11 AM

Would a visual filter like solution that displayed user typo corrections be subject to publisher restrictions?

Page 1 of 3 (44 items) 1 2 3 Next > | RSS