New Rating System Feedback

I'll reiterate that something that took into account the individual users average rating or typical rDev would be nice. I rate a lot harder than most people. I've tasted, and rated, enough beers that I think I have a pretty good sense of what is an "average" beer in most popular styles. And, if it's "average", it gets a 3 or close to a 3. This apparently makes me an "outlier" pretty frequently, and it's awfully frustrating being forced to justify an average rating.

It's also a bit frustrating if you rate on Untappd, or on paper, or on your own spreadsheet, and then bulk enter those ratings to BA that you can't predict in advance whether or not you'll be asked to justify yourself.

Finally, as a side note, I'd be really curious to know if the algorithm is tripping more frequently for low outliers than for high outliers. IMO, if it's not identifying approximately equal numbers on both ends, it's probably broken.

 

So this has been up now for more than a month. Is there any data on what percentage of people who choose to rate without doing a full review are just entering the same number across the board for all five categories? It looks to me like it is more than 50%. If so, I'd say that's very compelling evidence that people do not like the new system, we're not getting the anticipated benefit from doing this, and we should just go back to a single rating in quarter point increments like we had before this change.

 

I agree with this 100%. I've been asked a few times to justify a low outlier, but never a high one. It seems as if +/- 20% off average is the trigger for the low outlier (because I have some scores that are -16% and didn't have to justify), but I don't know what it is on the high end. I can't imagine it's 20%, is it?

 

Of course reviews are up, you're forcing people to review outliers. Naturally there going to go up.

 

Any word on when we will see this?

I found this very helpful:

 

Yes, but the threat of outliers likely causes a lot of untappd/offline raters to preemptively take notes that they wouldn't otherwise.

 

The horror...

 

Apparent subtle change in weighting this week? Looking at my reviews I posted on Friday night it looks like aromas/appearance got a bump and taste dropped some percentages. My reviews with stronger Aromas went up .02, stronger Flavor dropped .03.

 

RE: number of reviews are up...

I assume this includes 5-10 word reviews that almost no one would find all that useful anyway. Has there been any measurable uptick in "useful" reviews--reviews that are more than 10 words long? I'm pulling 10 words out of thin air (maybe others have a different threshold for usefulness), but when I'm looking for reviews, I skip over the 5-10 word ones to the longer, more thought out, more descriptive (and therefore useful) reviews to get the information I seek.