Dismiss Notice
Subscribe to BeerAdvocate magazine for only $9.99! (Limited time offer, US delivery only)

Are beer ratings biased?

Discussion in 'Beer Talk' started by BeerPugz, Apr 21, 2017.

  1. BeerPugz

    BeerPugz Initiate (161) Dec 4, 2016 Wisconsin

    As the title suggests, I wonder, how many ratings are biased? I don't care the website/platform. Is the rater aware but doesn't care?

    Right now I am drinking a Pilsner Urquell, and I gotta say, I'm enjoying it much more than I did Surly Xtra-Citra (the 4-pack was a total drain pour, and drunk within 7 days after canning). I feel Pilsner Urquell is a much better representation of its respective style compared to Surly Xtra-Citra. I could say the same about Pilsner Urquell representing its respective style much better than many stouts and stouts are my favorite style. For me, it's hard to rate a pale ale/stout better than a pilsner just because it's a pale ale/stout, which got me thinking, how do you rate beers? I know, ASTMO. But ASTMO still seems like a joke to me if someone is picking a stout over a pilsner just because "it tastes better" or "has more flavor" instead of how it represents the respective style.

    Are your ratings based on the overall taste, respective style, a combination?

    Cheers! Go Brewers!
     
    PatrickCT, CrimeDog and bbtkd like this.
  2. bbtkd

    bbtkd Poo-Bah (1,525) Sep 20, 2015 South Dakota
    Supporter Subscriber

    It is impossible to keep bias out. I'm not a fan of pale ales in general, due the typical bitterness. Still, I make it a point to try the higher rated ones, particularly if they don't have off-the-chart IBU ratings. I like a few of them, but most I don't. Hard as I try, since I can't relate to the style, I expect I'm rating them lower due to my bias against pale ale bitterness vs rating them within the spectrum of that style.

    To help overcome this bias, a few months ago I set out to try all 104 styles. I'm stuck at 97 until I can hit some out-of-state bottle shops where brand distribution is better/different. I may never get through all 104, but during this journey I have enjoyed many of the styles, and encountered only a few where I would not seek to try them again.
     
  3. SFACRKnight

    SFACRKnight Meyvn (1,069) Jan 20, 2012 Colorado
    Beer Trader

    I prefer to rate to style for this reason. There are AALs out there that could be a 5/5 for the style, but when rated against things like ipa or imperial stout those numbers skew.
    @bbtkd that's quite the quest you're on. It's inspiring, I may have to force myself to buy a new style each week just to break out of my comfort zone.
     
    SaltofOH, ikapaMK, bubseymour and 9 others like this.
  4. Scott17Taylor

    Scott17Taylor Meyvn (1,210) Oct 28, 2013 Iowa
    Beer Trader

    I rate based on how much I enjoy the beer. I'll usually give a bump if it blows me away for the style, but you'll still see my top beers being mostly stouts and IPAs or pale ales.
     
    mikeinportc and BeerPugz like this.
  5. JrGtr

    JrGtr Devotee (400) Apr 13, 2006 Massachusetts

    I don't think ratings are biased, necessarily (and depending on the system that may change) but I think that RATERS are biased, in general.
    Now, Don't get me wrong, many raters will do the best they can to be fair, but there are a lot of factors that skew the ratings of many beers.
    First, is the taster themselves. Do they like this style, or not? Can they be fair to a style they don't care for? Then there's the style itself. Like it or not, BBA stouts, sours and IPAs (especially NEIPAs) these days get a boost in their score for being a BBA stout, sour or IPA.
    Something like a Pils may get marked artificially down, possibly unfairly, by virtue of not being a hazy hop bomb.
    Lastly, I think the venue its being rated in plays a big difference. I see beers being rated at a fest all the time - and when you're drinking 2oz of something in a crowd of a few thousand closest friends, when sampling 20 or 30 beers that night, you are not going to get an accurate rating, no matter how good you think you are. Slightly better is at a bar - you may be drinking a pint, but with food and probably other beer, and with friends, again will skew your numbers.
    Versus sipping a beer in your living room couch, no other distractions, you can pay better attention to what's going on there.
    I also forgot to mention correcting for the condition of the beer - what was it stored, shipped, transported, how old is it, etc. Often, especially with beers from the other side of the ocean or even country, what we hold in our hands doesn't bear much resemblance to what the brewer intended.
     
  6. bbtkd

    bbtkd Poo-Bah (1,525) Sep 20, 2015 South Dakota
    Supporter Subscriber

    Go for it, you're at 46 so almost half way there! Back in early February I realized I was at about 55 without even having tried. I created a spreadsheet listing the remaining ones, along with the better and easiest to get representatives of each. I carry that list and even ran across one Japanese style at a Chinese restaurant! I found Kvass at a Russian grocery and the owner said Kvass is not even considered a beer in Russia due to the low alcohol content. For a few weeks, all I drank were new styles. I'm amazed to get to 97 in South Dakota where the number of breweries distributed is quite limited, particularly foreign styles.
     
  7. BeerPugz

    BeerPugz Initiate (161) Dec 4, 2016 Wisconsin

    Reading the replies so far, I guess a better question would be how biased are beer ratings?

    For me, rating a saison more than 4* is difficult because I still don't quite understand how the style should taste. The 4* rating probably appears blasphemous to those whose favorite style is saison. When I'm drinking a new style I typically bump it .25 or .50 considering I might have an unknown bias. I typically try to drink it more than once.

    Haha. Cheers!
     
    utopiajane likes this.
  8. Oktoberfiesta

    Oktoberfiesta Aspirant (291) Nov 16, 2013 New Mexico
    Beer Trader

    Rating to style is so unfair to like 95 of the styles out there. If one enjoys something, rate the hell out of it. Yet here I am with a world class lager that I am hesitant to rate above 4.25. Yet 100 other ipas get checked in at or near 4.25 without hesitation.

    So yes. Biases are real. They suck and I myself find it hard to change it
     
    ikapaMK and cjgiant like this.
  9. nc41

    nc41 Meyvn (1,482) Sep 25, 2008 North Carolina
    Beer Trader

    Of course they are, the more hops the bigger the score. Delicate styles don't fare so well.
     
  10. cjgiant

    cjgiant Poo-Bah (3,632) Jul 13, 2013 District of Columbia
    Supporter Subscriber Beer Trader

    There's definitely a bias in individual ratings. There are those who try to limit/eliminate it, and those who don't. Hopefully, there's enough of a community here, that we balance ourselves out and get a fairly true representation (via rating/score) of a beer.

    Until we reach whatever constitutes a "statistically significant" sample, beer ratings will be skewed. Best for now is just to understand it. A local IPA at 4.5 with 5 ratings probably isn't Pliny or Heady quality. An established German Pilsner at 87 is probably as good or better (within style) than an BBA stout rated 100.

    Is this "right" or good? Idealistically not, but there;s probably no good "fix " or it, either.
     
  11. Ranbot

    Ranbot Devotee (463) Nov 27, 2006 Pennsylvania

    The ratings on this are generally ok within the same style, but the bias for certain styles becomes really obvious when comparing one style to another style.
     
  12. GetMeAnIPA

    GetMeAnIPA Zealot (562) Mar 28, 2009 California
    Beer Trader

    Just look at the top 100 beer list. Uh, see a pattern? I don't see any pilsners, bitters, porters, saisons, brown ales etc.

    More hops, more alcohol, more haze, more adjuncts, more difficult to acquire = higher ratings.
     
  13. alucard6679

    alucard6679 Aspirant (259) Jul 29, 2012 Arizona

    This website is host to a, for the most part, specific demographic. Take a world class DIPA and a world class brown ale, which do you think would have the better overall score?

    It's not really a bad thing, it's completely natural. It just means that if you're going to let reviews and scores guide your purchases on this website you should take some things with a grain of salt.
     
  14. HorseheadsHophead

    HorseheadsHophead Meyvn (1,100) Sep 15, 2014 New York

  15. drtth

    drtth Poo-Bah (3,223) Nov 25, 2007 Pennsylvania

    Well...., it depends.

    Biased at the level of one person or at the level of the overall scores?

    Biased within a particular style or between styles?

    Biased compared to what alternative?

    Is a bias, if there, really worth fixing?

    Etc.

    The answer is: Yes or No or Maybe.
     
    #15 drtth, Apr 21, 2017
    Last edited: Apr 21, 2017
    utopiajane likes this.
  16. drtth

    drtth Poo-Bah (3,223) Nov 25, 2007 Pennsylvania

    There actually are fixes, but most of them would "cost" more time/money/work than most people are willing take on. They would also radically change the site and it's not clear that those changes would make a more useful site.
     
    Prince_Casual and cjgiant like this.
  17. cavedave

    cavedave Poo-Bah (2,255) Mar 12, 2009 New York
    Beer Trader

    Folks who are smart, like me, ignore the numbers, and look for reviews that show
    -reviewer has a palate similar to mine
    -reviewer gives honest reviews
    and look for reviews done by people like this from whom to take my advice about beer selections. And of course smart people give reviews of this kind for others to use.

    Numbers are ridiculous to use as anything except a number to complain about in a thread on BA. Reviews and number ratings have only one purpose, and that is to give information to us consumers to help us buy beer we like. Numbers, even if there were a way to remove "bias", don't do this, so it is kind of ridiculous to hear folks complain they don't do what they cannot do. Even more ridiculous to complain when there are plenty of reviews and honest reviewers whose written beer descriptions can be trusted
     
    #17 cavedave, Apr 21, 2017
    Last edited: Apr 21, 2017
    CaptainHate, LeRose, gibgink and 14 others like this.
  18. moshea

    moshea Devotee (427) Jul 16, 2007 Michigan
    Beer Trader

    Yes... yes they are
     
  19. sosbombs

    sosbombs Initiate (171) Jan 12, 2016 Vermont

    Pilsner Urquell may be better then it's rating- at it's source. Not after sitting in warehouses and freighters for lord knows how long.
     
  20. utopiajane

    utopiajane Poo-Bah (2,389) Jun 11, 2013 New York


    You have just defined one of the "biases". I have noticed that my first reviews of any style ( styles I have not tried I mean) seems a bit high. But I think the review is the place to really get the sense of the beer, not the number score. I think the reason you do not see a pilsner in the top 250 YET is because it's a common style. It's everywhere now and a few years ago it wasn't. The better you know the style the better you will be able to review it. I am harder on my locals and on the styles I know well. I also have noticed that most commercial beer is not that bad or below a 3 in any category. It's rare in other words to get a truly awful beer out there.
     
    BeerPugz, Squire123, Reef and 3 others like this.
  21. papposilenus

    papposilenus Defender (648) Jun 21, 2014 New Hampshire
    Beer Trader

    There have been variations on this topic before with some BA's saying you should rate to style and as many others that you should rate according to your tastes - which leads to the inevitable conclusion that the ratings, in aggregate, are a muddled mess of methodologies.

    Which is fine, so long as you understand how the rating that you're looking at was derived. I've been using the BA ratings long enough to understand, intuitively, that a DIPA with a rating of something-like 4.40 is qualitatively equivalent to a bier de garde with a rating of something-like 3.80. When in doubt, I look at the comparative ratings within the style category - which is not a difficult thing to do unless you're in a busy store and don't want to look all geeky.
     
  22. bbtkd

    bbtkd Poo-Bah (1,525) Sep 20, 2015 South Dakota
    Supporter Subscriber

    If I had to guess, I'd say the demographic of 60%+ of BA site members is white guys from the US, age 30 to 60. Certainly most of that 60% started by drinking BMC (to excess) and eventually moved on to craft, many exclusively. The two favorite styles are probably stout/porter variants and pale ale variants. A significant percent drink their favorite styles exclusively. All of this creates biases.

    My biases; I tend not to score anything below 3 unless it seriously offends me, and I am wary of scoring anything over 4 that I would not seek out again. I prefer stouts, but like a number of other styles which I drink only occasionally. I try every stout I can find. I want to like pumpkin beers of any style, try every one I find, but most suck. I feel guilty scoring pale ales because I am really not a fan of the style, and am likely punishing them by scoring them lower than they deserve.
     
    #22 bbtkd, Apr 21, 2017
    Last edited: Apr 21, 2017
  23. TongoRad

    TongoRad Poo-Bah (2,033) Jun 3, 2004 New Jersey
    Supporter Beer Trader

    Well, I guess people could rate with style in mind, the way the website suggests. Somehow that never happens, though :slight_smile:.
     
    beergoot, BeerPugz, Squire123 and 3 others like this.
  24. TongoRad

    TongoRad Poo-Bah (2,033) Jun 3, 2004 New Jersey
    Supporter Beer Trader

    Still higher than its rating, but not by as much.
     
  25. nc41

    nc41 Meyvn (1,482) Sep 25, 2008 North Carolina
    Beer Trader

    The numbers can be generally useful. If I see an IPA with a score of say 90 I'd think it was a pretty average IPA at best. Most of the better ones do score much higher and are in the mid to high 90s and into 100. It's hard to find a Pils that top 90 and if I did I'd think it was most likely a stellar beer and worth looking for. Two styles same score mean different things.
     
    BeerPugz and papposilenus like this.
  26. SFACRKnight

    SFACRKnight Meyvn (1,069) Jan 20, 2012 Colorado
    Beer Trader

    My personal bias can be seen through my ratings over time. My first ratings were biased high because beer just tasted so damned good. Now that I have settled down, and have become familiar with these flavors, that bias is less obvious.
    The same could be said for yourself and saison. Because you're unfamiliar with the style you may over rate because of new and exciting flavors you encounter, or you may self regulate to the point of under rating that beer. I know I did.
     
  27. drtth

    drtth Poo-Bah (3,223) Nov 25, 2007 Pennsylvania

    I agree it mostly does not happen, but that seems to originate from a few factors, among them, "rate with style in mind" is ignored by some and misinterpreded by others to mean the same as "rate to style, just as if you were a trained judge following the BJCP." Another factor that contributes to it is lack of experience with lots of examples within a particular style. Still another is what I think of as the "Pilsener" effect, i.e., lots of folks know that the big breweries call their AALs "Pilsener," and they let their dislike of the big breweries and their beers shape their judgement.
     
  28. TongoRad

    TongoRad Poo-Bah (2,033) Jun 3, 2004 New Jersey
    Supporter Beer Trader

    Even Pilsners are filtered through the American Craft Palate, so that 90 isn't as indicative of a stellar beer as you'd think. I certainly wouldn't go out of my way again for Hill Farmstead Mary, and found it to be a solidly second tier beer.
     
  29. TongoRad

    TongoRad Poo-Bah (2,033) Jun 3, 2004 New Jersey
    Supporter Beer Trader

    Right, all of that is happening, so we're back to @cavedave 's advice to just use the reviews and ignore the overall numbers.
     
    ikapaMK and papposilenus like this.
  30. drtth

    drtth Poo-Bah (3,223) Nov 25, 2007 Pennsylvania

    As you know you and I generally agree upon the importance of the reviews compared to the ratings but I suggest that the ridiculousness of using the numbers for any other purpose than complain about them actually depends quite a bit on how much one knows about the strengths and weaknesses of the numbers and how to adjust one's thinking about the meaning of the numbers. For example, just becuse there is a rating scale from 1-5, lots of folks think that means there should be a mean of zero. Unfortunately that is based on and only applies to a unique set of conditions that don't exist in the way ratings are used on this site. Similarly I'd bet there's not 1 in 10 on this site who have learned enough about numbers, etc., to understand the meaning of the "pDev" that accompanies the Mean associated with each beer (let alone how to calculate one :slight_smile:). Similarly I'd bet there's not 1 in 100 on this site who know what a Baysian estimate is or who have read the explanation of how the rank orderings on each "Top Rated Beers" within a particular style are calculated.
     
    #30 drtth, Apr 21, 2017
    Last edited: Apr 21, 2017
    LeRose, ikapaMK and Beer_Economicus like this.
  31. Harrison8

    Harrison8 Poo-Bah (1,700) Dec 6, 2015 Missouri
    Beer Trader

    One of my biggest goals when I review a beer is to try and remove potential bias, or if there is a hearty bias present in my review, note it in my after thoughts (where I put what type of glass I used, if it was draft or not, etc.)

    The things I think about to try and reduce those biases:
    - Do I have expectations of this beer? If so, why? Dump them.
    - What's the style? Rate wholly within the style. A Gose doesn't have much head formation? Okay, that's fine. An IPA doesn't have much head formation or retention? Poor execution.
    - Has anything gone on in my day to make this beer taste better or worse?
    - Am I rating based on how I feel about this beer? Or how it tastes? Or am I rating based on the stylistic guidelines?

    It's not a fail safe operation, and there are still things I'm not totally aware of effecting my taste buds or thought process that day, but it at least keeps me aware of how I'm approaching a beer.
     
  32. drtth

    drtth Poo-Bah (3,223) Nov 25, 2007 Pennsylvania

    Are we?

    I'd say for many folks that's a good general strategy and useful to boot.

    But I'd still disagree that the numbers are meaningless or using them is ridiculous. I'd say that depends on the knowledge one brings to understanding what the ratings and numbers actually mean and how they can be fruitfully used even on a site such as this where there are some biases that are not controlled for, especially when some of those biases are pretty obvious to many.
     
    #32 drtth, Apr 21, 2017
    Last edited: Apr 21, 2017
    Beer_Economicus likes this.
  33. jesskidden

    jesskidden Meyvn (1,289) Aug 10, 2005 New Jersey
    Subscriber

    :astonished: And non-deities who don't "know" can simply subtract 9 months from the "Best by" date on the packaging. :grinning: (Last fall's jpeg):
    [​IMG]
    As for the time in transit on freighters, the brewery claims it takes only 21 days via train to Hamburg and then in "cooled" shipping containers at O°C (32°F) to the docks in NJ.

    After that, the beer is supposed to remain refrigerated at the distributors' warehouses --- but the system routinely breaks down when it hits most retailers and the MC reps don't seem to care, either.
     
  34. cavedave

    cavedave Poo-Bah (2,255) Mar 12, 2009 New York
    Beer Trader

    Indeed. Number ratings are an easy scorecard for tickers to use, and number ratings often are all that a ticker requires, or desires, to remember a specific beer they've tried.
     
    Harrison8 likes this.
  35. drtth

    drtth Poo-Bah (3,223) Nov 25, 2007 Pennsylvania

    Yep, that one way some folks use them. There are other uses, some of which are even more broadly useful than that.
     
    Harrison8 likes this.
  36. papposilenus

    papposilenus Defender (648) Jun 21, 2014 New Hampshire
    Beer Trader

    I want to be able to do that but I have a short attention span and can only look at a couple of the most recent before my mind wanders. I confess that I use the numbers a lot.
     
    beergoot, TongoRad and drtth like this.
  37. NickTheGreat

    NickTheGreat Zealot (508) Oct 28, 2010 Iowa

    Yes since it is a subjective system, it will be biased.

    Sidenote, I also rate per style. I can't stand people who don't like a Hefeweizen because it's not hoppy enough. Or a IPA that is too bitter, from a dude who doesn't like hops.

    I mean, I wouldn't order guacamole at a restaurant and complain that it tastes nothing like apple pie.
     
  38. cavedave

    cavedave Poo-Bah (2,255) Mar 12, 2009 New York
    Beer Trader

    I suppose you're right, and that is my own bias :slight_smile: for the pragmatic that causes me to limit my consideration of ratings and reviews to that.

    I mean, when you say broadly useful, I assume you mean in an academic sense?
     
    Harrison8 likes this.
  39. Lurchus

    Lurchus Aspirant (240) Jan 19, 2014 Germany

    I basically stopped really rating when I stopped ticking, around 7,8 years ago.......
    That being said, it's all about the frame of reference. How many authentic, as in czech, svetly lezaks did you have before, in good codition?
    I'd say, seen in the context of the dozens excellent examples out there, at least for me personally, bottled pilsner urquell is below average for its "style". Fresh,espacially unfiltered,unpasteurized, urquell is another thing though, but I would still argue there are way better examples around...
     
  40. drtth

    drtth Poo-Bah (3,223) Nov 25, 2007 Pennsylvania

    No, actually in a very personal sense. Just as I can use the reccomendations of someone whose judgment I trust (e.g., you, @TongoRad and others) the numbers are a useful filter in choosing among alternatives before I spend the time and effort to read reviews.

    I'll never live long enough or make enough disposable income to have a meaningful serving of all the beers in the database, so why not devote most of my time and money to the list of top beers within a style?

    The top 20 from a list all have different ranks, but is that a meaningful difference? If not, why spend my time and money looking at and for the highest ranked beer when I can look at and easily find any of 5 others in my local shops?

    Etc.
     
    ikapaMK, BeerPugz, TongoRad and 3 others like this.
  • About Us

    Founded in Boston in 1996, BeerAdvocate (BA) is your go-to resource for beer powered by an independent community of enthusiasts and professionals dedicated to supporting and promoting better beer.

    Learn More
  • Our Community

    Comprised of consumers and industry professionals, many of whom started as members of this site, our community is one of the oldest, largest, and most respected beer communities online.
  • Our Events

    Since 2003 we've hosted over 60 world-class beer festivals to bring awareness to independent brewers and educate attendees.
  • Our Magazine

    Support uncompromising beer advocacy and award-winning, independent journalism with a print subscription to BeerAdvocate magazine.