Are beer ratings biased?

Discussion in 'Beer Talk' started by BeerPugz, Apr 21, 2017.

Thread Status:
Not open for further replies.
  1. papposilenus

    papposilenus Grand Pooh-Bah (3,208) Jun 21, 2014 New Hampshire
    Pooh-Bah Society Trader

    There have been variations on this topic before with some BA's saying you should rate to style and as many others that you should rate according to your tastes - which leads to the inevitable conclusion that the ratings, in aggregate, are a muddled mess of methodologies.

    Which is fine, so long as you understand how the rating that you're looking at was derived. I've been using the BA ratings long enough to understand, intuitively, that a DIPA with a rating of something-like 4.40 is qualitatively equivalent to a bier de garde with a rating of something-like 3.80. When in doubt, I look at the comparative ratings within the style category - which is not a difficult thing to do unless you're in a busy store and don't want to look all geeky.
     
  2. bbtkd

    bbtkd Grand High Pooh-Bah (7,326) Sep 20, 2015 South Dakota
    Pooh-Bah Society Trader

    If I had to guess, I'd say the demographic of 60%+ of BA site members is white guys from the US, age 30 to 60. Certainly most of that 60% started by drinking BMC (to excess) and eventually moved on to craft, many exclusively. The two favorite styles are probably stout/porter variants and pale ale variants. A significant percent drink their favorite styles exclusively. All of this creates biases.

    My biases; I tend not to score anything below 3 unless it seriously offends me, and I am wary of scoring anything over 4 that I would not seek out again. I prefer stouts, but like a number of other styles which I drink only occasionally. I try every stout I can find. I want to like pumpkin beers of any style, try every one I find, but most suck. I feel guilty scoring pale ales because I am really not a fan of the style, and am likely punishing them by scoring them lower than they deserve.
     
    #22 bbtkd, Apr 21, 2017
    Last edited: Apr 21, 2017
  3. TongoRad

    TongoRad Grand Pooh-Bah (3,848) Jun 3, 2004 New Jersey
    Pooh-Bah Society Trader

    Well, I guess people could rate with style in mind, the way the website suggests. Somehow that never happens, though :slight_smile:.
     
    beergoot, BeerPugz, Squire and 3 others like this.
  4. TongoRad

    TongoRad Grand Pooh-Bah (3,848) Jun 3, 2004 New Jersey
    Pooh-Bah Society Trader

    Still higher than its rating, but not by as much.
     
  5. nc41

    nc41 Initiate (0) Sep 25, 2008 North Carolina
    Trader

    The numbers can be generally useful. If I see an IPA with a score of say 90 I'd think it was a pretty average IPA at best. Most of the better ones do score much higher and are in the mid to high 90s and into 100. It's hard to find a Pils that top 90 and if I did I'd think it was most likely a stellar beer and worth looking for. Two styles same score mean different things.
     
    BeerPugz and papposilenus like this.
  6. SFACRKnight

    SFACRKnight Grand Pooh-Bah (3,274) Jan 20, 2012 Colorado
    Pooh-Bah Society Trader

    My personal bias can be seen through my ratings over time. My first ratings were biased high because beer just tasted so damned good. Now that I have settled down, and have become familiar with these flavors, that bias is less obvious.
    The same could be said for yourself and saison. Because you're unfamiliar with the style you may over rate because of new and exciting flavors you encounter, or you may self regulate to the point of under rating that beer. I know I did.
     
  7. drtth

    drtth Initiate (0) Nov 25, 2007 Pennsylvania
    In Memoriam

    I agree it mostly does not happen, but that seems to originate from a few factors, among them, "rate with style in mind" is ignored by some and misinterpreded by others to mean the same as "rate to style, just as if you were a trained judge following the BJCP." Another factor that contributes to it is lack of experience with lots of examples within a particular style. Still another is what I think of as the "Pilsener" effect, i.e., lots of folks know that the big breweries call their AALs "Pilsener," and they let their dislike of the big breweries and their beers shape their judgement.
     
  8. TongoRad

    TongoRad Grand Pooh-Bah (3,848) Jun 3, 2004 New Jersey
    Pooh-Bah Society Trader

    Even Pilsners are filtered through the American Craft Palate, so that 90 isn't as indicative of a stellar beer as you'd think. I certainly wouldn't go out of my way again for Hill Farmstead Mary, and found it to be a solidly second tier beer.
     
  9. TongoRad

    TongoRad Grand Pooh-Bah (3,848) Jun 3, 2004 New Jersey
    Pooh-Bah Society Trader

    Right, all of that is happening, so we're back to @cavedave 's advice to just use the reviews and ignore the overall numbers.
     
    ikapaMK and papposilenus like this.
  10. drtth

    drtth Initiate (0) Nov 25, 2007 Pennsylvania
    In Memoriam

    As you know you and I generally agree upon the importance of the reviews compared to the ratings but I suggest that the ridiculousness of using the numbers for any other purpose than complain about them actually depends quite a bit on how much one knows about the strengths and weaknesses of the numbers and how to adjust one's thinking about the meaning of the numbers. For example, just becuse there is a rating scale from 1-5, lots of folks think that means there should be a mean of zero. Unfortunately that is based on and only applies to a unique set of conditions that don't exist in the way ratings are used on this site. Similarly I'd bet there's not 1 in 10 on this site who have learned enough about numbers, etc., to understand the meaning of the "pDev" that accompanies the Mean associated with each beer (let alone how to calculate one :slight_smile:). Similarly I'd bet there's not 1 in 100 on this site who know what a Baysian estimate is or who have read the explanation of how the rank orderings on each "Top Rated Beers" within a particular style are calculated.
     
    #30 drtth, Apr 21, 2017
    Last edited: Apr 21, 2017
    LeRose, ikapaMK and Beer_Economicus like this.
  11. Harrison8

    Harrison8 Grand Pooh-Bah (5,771) Dec 6, 2015 Missouri
    Pooh-Bah Society Trader

    One of my biggest goals when I review a beer is to try and remove potential bias, or if there is a hearty bias present in my review, note it in my after thoughts (where I put what type of glass I used, if it was draft or not, etc.)

    The things I think about to try and reduce those biases:
    - Do I have expectations of this beer? If so, why? Dump them.
    - What's the style? Rate wholly within the style. A Gose doesn't have much head formation? Okay, that's fine. An IPA doesn't have much head formation or retention? Poor execution.
    - Has anything gone on in my day to make this beer taste better or worse?
    - Am I rating based on how I feel about this beer? Or how it tastes? Or am I rating based on the stylistic guidelines?

    It's not a fail safe operation, and there are still things I'm not totally aware of effecting my taste buds or thought process that day, but it at least keeps me aware of how I'm approaching a beer.
     
  12. drtth

    drtth Initiate (0) Nov 25, 2007 Pennsylvania
    In Memoriam

    Are we?

    I'd say for many folks that's a good general strategy and useful to boot.

    But I'd still disagree that the numbers are meaningless or using them is ridiculous. I'd say that depends on the knowledge one brings to understanding what the ratings and numbers actually mean and how they can be fruitfully used even on a site such as this where there are some biases that are not controlled for, especially when some of those biases are pretty obvious to many.
     
    #32 drtth, Apr 21, 2017
    Last edited: Apr 21, 2017
    Beer_Economicus likes this.
  13. jesskidden

    jesskidden Grand Pooh-Bah (3,071) Aug 10, 2005 New Jersey
    Pooh-Bah Society Trader

    :astonished: And non-deities who don't "know" can simply subtract 9 months from the "Best by" date on the packaging. :grinning: (Last fall's jpeg):
    [​IMG]
    As for the time in transit on freighters, the brewery claims it takes only 21 days via train to Hamburg and then in "cooled" shipping containers at O°C (32°F) to the docks in NJ.

    After that, the beer is supposed to remain refrigerated at the distributors' warehouses --- but the system routinely breaks down when it hits most retailers and the MC reps don't seem to care, either.
     
  14. cavedave

    cavedave Grand Pooh-Bah (4,145) Mar 12, 2009 New York
    Pooh-Bah Society Trader

    Indeed. Number ratings are an easy scorecard for tickers to use, and number ratings often are all that a ticker requires, or desires, to remember a specific beer they've tried.
     
    Harrison8 likes this.
  15. drtth

    drtth Initiate (0) Nov 25, 2007 Pennsylvania
    In Memoriam

    Yep, that one way some folks use them. There are other uses, some of which are even more broadly useful than that.
     
    Harrison8 likes this.
  16. papposilenus

    papposilenus Grand Pooh-Bah (3,208) Jun 21, 2014 New Hampshire
    Pooh-Bah Society Trader

    I want to be able to do that but I have a short attention span and can only look at a couple of the most recent before my mind wanders. I confess that I use the numbers a lot.
     
    beergoot, TongoRad and drtth like this.
  17. NickTheGreat

    NickTheGreat Maven (1,336) Oct 28, 2010 Iowa
    Trader

    Yes since it is a subjective system, it will be biased.

    Sidenote, I also rate per style. I can't stand people who don't like a Hefeweizen because it's not hoppy enough. Or a IPA that is too bitter, from a dude who doesn't like hops.

    I mean, I wouldn't order guacamole at a restaurant and complain that it tastes nothing like apple pie.
     
  18. cavedave

    cavedave Grand Pooh-Bah (4,145) Mar 12, 2009 New York
    Pooh-Bah Society Trader

    I suppose you're right, and that is my own bias :slight_smile: for the pragmatic that causes me to limit my consideration of ratings and reviews to that.

    I mean, when you say broadly useful, I assume you mean in an academic sense?
     
    Harrison8 likes this.
  19. Lurchus

    Lurchus Zealot (733) Jan 19, 2014 Germany

    I basically stopped really rating when I stopped ticking, around 7,8 years ago.......
    That being said, it's all about the frame of reference. How many authentic, as in czech, svetly lezaks did you have before, in good codition?
    I'd say, seen in the context of the dozens excellent examples out there, at least for me personally, bottled pilsner urquell is below average for its "style". Fresh,espacially unfiltered,unpasteurized, urquell is another thing though, but I would still argue there are way better examples around...
     
  20. drtth

    drtth Initiate (0) Nov 25, 2007 Pennsylvania
    In Memoriam

    No, actually in a very personal sense. Just as I can use the reccomendations of someone whose judgment I trust (e.g., you, @TongoRad and others) the numbers are a useful filter in choosing among alternatives before I spend the time and effort to read reviews.

    I'll never live long enough or make enough disposable income to have a meaningful serving of all the beers in the database, so why not devote most of my time and money to the list of top beers within a style?

    The top 20 from a list all have different ranks, but is that a meaningful difference? If not, why spend my time and money looking at and for the highest ranked beer when I can look at and easily find any of 5 others in my local shops?

    Etc.
     
    ikapaMK, BeerPugz, TongoRad and 3 others like this.
Thread Status:
Not open for further replies.