(Why) Do obvious outliers count the same as regular ratings?

Help Discussion in 'BeerAdvocate Talk' started by gcamparone, Mar 30, 2016.

Thread Status:
Not open for further replies.
  1. gcamparone

    gcamparone Pooh-Bah (2,131) Dec 6, 2011 Rhode Island
    Pooh-Bah Trader

    It seems that extreme outliers carry the same or close to the same weight as regular reviews.. I don't believe one bad experience from one user should affect raw rating, especially for beers with less reviews in which one bad rating will have a greater impact.

    For example, the beer Fundamental Observation just dropped in the top 250 from #13 to #24, largely due to one user giving it an extremely low rating relative to others. I do respect other's opinions but sometimes it's not the beer that's bad, but the environment (uncleaned taps, accidentally pouring the wrong beer, biased reviews, ulterior motives to skew a rating, etc)

    Just a thought. I know there used to be an outlier detection system in place and was wondering if this still exists, and if it affects raw score.
     
    #1 gcamparone, Mar 30, 2016
  2. elucas730

    elucas730 Initiate (0) Feb 5, 2010 New York

    Herding is already a documented problem with online rating systems. Now you want to make it worse by throwing out any review that deviates from the herd? Who's to say that review isn't the more "correct" review and the rest aren't the result of herding? What if a bunch of local homer fanboys give their favorite local beer/brewery artificially high reviews to hype it up, then you will just throw out reviews that don't agree with the hype? At the end of the day, this is an unscientific online rating system that should be taken with a large helping of salt.
     
    #2 elucas730, Mar 30, 2016
    PA-Michigander, TMoney2591, maximum12 and 2 others like this.
  3. drtth

    drtth Initiate (0) Nov 25, 2007 Pennsylvania
    In Memoriam

    @gcamparone, IIRC the outlier detection system went away when ratings without reviews stopped being counted as having equal weight with reviews/ratings.

    But from a statistical perspective, the change in ranking you are troubled by isn't really worth worrying about in the sense that rank orderings in that list are calculated on very very small differences in the weighted average score (as you can tell from the change that bothers you) which may show up numerically but may not mean much since mostly they are not significant. Effectively what you're looking at, especially towards the top of the list, is called "an equivalence class" where to all intents and purposes the items are pretty much indistinguisable on the bases of ratings.

    If you go look at the description at the bottom of that page about how the numbers are crunched the use of a very small n is going to create volatility in the list. IIRC one aspect of the thinking behind this is that it gives new beers/breweries a chance to competitively stand out from the crowd without having to have hundreds of reviews and to help give them a shot at being recognized for quality. If that rated quality holds across multiple reviews/reviewers/conditions the beer will survive and thrive. (There are lots of examples of such beers on the Beers of Fame list.) If that rated quality of the beer proves to be artificial for any reason the beer will soon drop off the list and back into obscurity.
     
    #3 drtth, Mar 30, 2016
    gcamparone likes this.
  4. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    The law of averages, our weighted ranking system and other factors come to play, but I'm not convinced that the review you mentioned was solely responsible for it's drop in the Top 250. There were quite a few ratings after it, and other beers may have joined the list or received more ratings and higher scores.

    As for the outlier detection system, it was only in place to encourage reviews. But it's no longer needed thanks to more recent updates.

    And what @elucas730 said. Crippling or throwing away outliers only caters to the herd mentality.
     
    #4 Todd, Mar 30, 2016
    gcamparone likes this.
Thread Status:
Not open for further replies.