Some info on community rating

An analogy with be like ratings from IMDb. Given IDMb score (community rating), does high IDMb score result in more likely one, or more personally I, like the movie/TV? From my experience It’s true for me most of the time, but I have rare cases that I absolutely hate a movie with a very high IMDb rating.

Lessons here:

• One should first check if the community rating closely follows one’s taste. In my case, a manual check shows IMDb and Logos community rating both resemble my taste to a large degree. This assumption can breaks, eg if a non English native, non US born person being shown a very good TV show in politics in US, they are very likely to absolutely hate it.
• This predictor (using community rating to predict personal taste) are virtually never to be 100% accurate, as in anything involved in Statistics.
• Wether this is useful or not depends, especially on the user. There’s another thread someone essentially is dismissing the value of community tag. The logic is the same—statistics never reflects 100% correctly on what I think about that (rate, tag, etc.) I think may be many people here are spoiled by the Logos style of doing things. I.e. Logos spends a lot of resources to manual tag something to create accurate dataset. The quality of this is superior to anything done automatically (such as using natural language processing.) So perhaps especially Logos users has a larger tendency to not accept that a certain search might give them fuzzy results.
• Yes, statistics never reflects 100% on your judgement on the issue. But then who is to say that one’s judgement can be trusted? I wonder how normally Logos users rate their resources, and how many of them. For me I’d like to rate all of them. The problem becomes one hasn’t read all of them but need to rate all of them. Which one should one do first? Rating all of them is doable while tedious, having each one read first and then rate it is just impossible (given typical Logos users buy base packages and has thousands of resources.) Now why would I trust my own rating if I haven’t happened to even finish reading it? One may base their rating by the word of mouth, but the same question can be asked (how can you be sure say D.A. Carson’s taste reflect yours?); or base on reading the excerpt (this is just arrogant for one to think that can results in an accurate rating.)
• What I personally did is to revisit my own rating vs. community rating and essentially they are mostly agreeing (in terms of trend, e.g. most of my 3 happens around communities' 4), and for those that they doesn’t agree, I’ve found they are mostly my mistakes. E.g. I rated something I wouldn’t use at all as 1, where one such resource is very reputable and has a community rating of 4. Another instance is I use relative rating to order which King James Versions I think are better. So I gave 2, 3, 4 rating to 3 of them, but in reality they aren’t differing that much to result in that much a difference in score.
• With enough data, much of the noise and bias can be removed. E.g. if the whole database of user rating are given, such as the rating distribution per resource over all users, you can do a ton of data cleaning to remove bias. E.g. with the prior knowledge some Logos users convention like to use 2 as default (default value bias is very common and is almost always one of the first things people try to remove), then one would first perform a data analysis to detect that and remove it (again it won’t be perfect but reduces biases.)

So take this thread as a glimpse into what data analysis into Logos/Faithlife’s data can provide, and how an API to this enable these, rather than a final verdict.

P.S. it is not impossible for default value of 2 can strongly skewed the averages. Imagine there’s a resource most Logos users don’t read and rate at all, and they assign a default value of 2. Then actual rating samples are of low quantities, hence the mean is essentially determined by the default. However, this doesn’t prevent one to conclude there’s 2 clusters here (one many argue from the graph, luckily the real ratings seems to tends to be high, i.e. centered and spreader higher than 2 to make this default bias not mingled within the actual distribution.) nor the usefulness of this information. One can infer since these resources is mostly not read and rated by the general Logos community, it might not worth my time exploring it. Not that you won’t find gems here, just less likely.

Denise said:

Well, Kolen, your work is admirable but, I tend to agree with NB. When they first intro'd, people were indeed using them in code-like fashion. The first rule in analytics is to know the basis for input, before interpreting output.

I think you didn't read my last reply and don't really understands what's going on... (Hint: read the P.S. part.)

First, the bug I mentioned isn’t really bug but a wrong syntax.

Case in point: rating:<2.2 ANDNOT rating:0 results in 100% agreement in my own tagging. i.e. if I use that as a rejection criteria (labeling “bad”), I’d have no false positives.

Note that 2.1 is the minimum between the 2 clusters and is used as a simple cut on clustering decision. Note that we aren’t even trying to do Logistic regression and provide my own supervised labels. The point is really to get something very useful in short amount of time (comparing to e.g. going through all resources and rating it by myself.)

It’s never completely accurate but a tradeoff. Wether that tradeoff is good enough varies on opinion and also on how closely it is to an individuals.

I kind of regret posting it here because there’s often people don’t understand and/or don’t get it. I’d have saved more time by just performing that and use it on my own.

David Ames said:

NB.Mick said:

I don't think so. Many users don't use rating at all. From forum discussions it seems that those who do, especially the oldtimers (and thus sometimes serving as a role model for the others) apply a system that uses 2 as a default rating 

Yes, some "oldtimers" use the "rating system" in way that has nothing to do with how good the resource is but if they want it to be included in a search.   [[0 New resource, 1 don't include, though 5 for PBB as in if I took the time to input that then I want it in all searches.] 

What if Logos looked at our flag on report community ratings. Then if we do not want to see how the community reports ratings then don't include our rating in the community ratings for others.     

Note that the discussion on the default of 2 is quite irrelevant here. His observation is based on a misunderstanding of the graph itself (what distribution are you really seeing?)

Equation-wise, one can model this bias:

Note that

• we are focusing one 1 single resource, and the mean is over the entire user pool Logos has access to in order to calculate the community rating.
• this is not an assumption but can be derived, assuming some users rated the resource at default 2
• depends on resource. i.e. if is the resource index,
• can be interpreted as the probability that a user rated that resource at 2. i.e. when , the estimator is unbiased. When , it is not difficult to see that the estimator biased towards 2.
• the distribution you’re seeing is , not , where s is the score, i is the resource index, j is the user index. You don’t got to see j, Logos doesn’t release this dependency.

Now, another question is how would distribute across resources. If it doesn’t vary too much, and is sort of random, its effect on is just biasing it towards 2, i.e. narrowing its variance towards 2, but not changing its relative order.

However, if there’s some strong correlation between to some , the situation can be different. The most extreme scenario one can imagine is this: there are 2 kinds of resources, one that Logos users generally care about and one doesn’t, such as free Faithlife eBooks. So for these kinds of resources, might distribute much closer to 1. i.e. the estimator is almost insensitive to the actual mean. In this case, it will results in a strong cluster around 2.

Essentially this is the detailed argument of the counterpoint I loosely describe above verbally. Even in this case, how will one label this kind of resources such that most users who rated it don’t even bother changing it from the default 2? “good” or “bad” may not be the right word, but you get the idea, these are the resource I shouldn’t spend too much time caring about either.

Also, try to think about this, why would the conventional wisdom of a default value to be 2? It reflects the relative ranking one would placed between something one cares or not. That’s why you don’t default it to 5, it must be pretty special to be regarded like that. That’s also why you don’t default it to 1, because it must be pretty special to be trashed like that.

And in reality no one really cares why it clusters like this, as long as it is a good predictor. As I’ve already mentioned, the minimum between the 2 clusters happened at 2.1. Comparing community rating < 2.2 has 100% agreement on my own manual rating (i.e. 1 or 2). As long as I move it to community rating < 2.3, my rating already doesn’t agree with that (2 of mine rated at 4 is at community rating of 2.2.)

So case in point it is entirely useful to me. Whether or not it is useful to you requires your own experiment, because the last verification step requires your own rating as the supervised label. (And the crossing point in this cluster depends on your library, i.e. the distribution of average ratings across all your available resources.)

And please if you can’t follow the arguments/equations above, please don’t even try to argue at all. Those are very introductory probability that if one can’t follow it, one almost certainly do not know how to interpret even what the graph means. Even for those who understand, with the available data this is as much as one can do. Unless we are given more data (the j index above), not much more can be said.

P.S. there’s a more subtle argument that the default value might not be at fault here. If you follow the model above, you’ll found that (with simple assumption on ) the left clusters should peaked above 2, not below it.