Are beer ratings biased?

Discussion in 'Beer Talk' started by BeerPugz, Apr 21, 2017.

  1. Leebo

    Leebo Initiate (119) Feb 7, 2013 Massachusetts

    Taste is subjective. I don't rate or read reviews ( gasp) The only taste that matters is my own. Drink fresh, drink local.
     
  2. JAStheAce

    JAStheAce Initiate (153) Apr 25, 2009 Florida
    Beer Trader

    In general, no. However, when it is a beer that is not easily obtainable aka rare, then the rating becomes more susceptible to bias. It is just human nature to subjectively rate such a beer a bit higher than your normally would because the fact that you are drinking it and others are not makes it taste better. Remember, the most delicious flavor in beer - is rare and I believe this skews ratings.
     
    BeerPugz likes this.
  3. beersite

    beersite Initiate (124) Nov 8, 2016 Illinois

    kinda hard to not let bias creep in. that being said, I can appreciate a well structured lager as much as a stout. does one have more flavor? most would say yes. I would say that is also subjective. my favorite style of beer is properly conditioned cask pale ale/bitter which doesn't beat you over the head on the first sip. but a well made beer will keep you ordering pints and you will pull nuances out of ever sip that you didn't find previously. As long as you are enjoying it I say be as biased as you want!
     
    BeerPugz and Harrison8 like this.
  4. Harrison8

    Harrison8 Meyvn (1,340) Dec 6, 2015 Missouri
    Beer Trader

    I read a review the other day that just said something along the lines of "wow, this has a lot of alcohol. I'm still recovering." and they rated it low. How does that help? No notes on the look, aroma, flavor profile, or feel. Did it just taste like alcohol? Like boozy bourbon alcohol or a fresh, harsh wine alcohol? Fortunately, since it was a review, it was made available to report and clean out of the system. It makes me wonder how many others are dumping a low rating on a beer because they just simply don't like it. That thought makes me a little leary to lean wholly on just the numbers. I just really like how the review gives the ratings a tangible quality, especially since there are several interpretations on how to rate a beer on here despite there being site guidelines.

    I do agree the numbers have a place though. It's a quick check for how a beer performs, at least within this community. If I'm on the fence between buying two different beers, I'll check their ratings first. If they're within 0.15 or so from each other, I'll read a couple reviews to see if one sounds better then the other and then make my purchase.
     
  5. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    Your examples are exactly why some reviews should be ignored and, as @cavedave recommends, we should be paying most attention to reviewers who share tastes similar to our own.
     
  6. beersite

    beersite Initiate (124) Nov 8, 2016 Illinois

    im with you on the reviews. and to be honest...I like to look at the lowest ratings and read peoples gripes about something for the reason u mentioned. someone drinks a big 12% beer and whines about it tasting of alcohol, yeah disregard that review imo. numbers are good when there is a large sample size. im not sure id use it to pick one over another, but I will take a look (after I am already interested!) and see if its worth my money and time.
     
    Kaapstadmk, BeerPugz and Harrison8 like this.
  7. Urk1127

    Urk1127 Savant (976) Jul 2, 2014 New Jersey
    Beer Trader

    Everything in life is biased. Somebody's experience with the beer, like if they had it in an awesome time and place could subconsciously effect a rating
     
    BeerPugz and Harrison8 like this.
  8. FFreak

    FFreak Defender (634) Nov 10, 2013 Vermont
    Beer Trader

    I rate appearance, smell, and mouthfeel mostly based on style, but I'm totally biased when it comes to taste. I like what I like. Plus I rate mostly for my own reference.
     
    JimKal and BeerPugz like this.
  9. BrewNoob1

    BrewNoob1 Devotee (401) Jan 8, 2015 Minnesota
    Beer Trader

    It definitely has a lot of bias especially if you are trading for a beer. Some like to score a beer off the top 250 and may rate it higher because it was difficult to obtain, rare, and/or the hype centered around the beer. Also, I've seen some reviews that always compare a beer to another within the same style. If using say, Heady Topper, for your "control" beer, all the other ratings/reviews will be biased.

    I also agree with a lot of the responses on styles. If it's not loaded with hops, ingredients from local businesses/farmers, or aged in the best spirits, the simpler styles get a lower rating. I've seen a ton of low reviews for lagers because they do have an overpowering flavor/aroma because people don't keep style into mind.

    Also, location bias. Some rate local beers higher to support their local watering hole. Experience can be another bias. If you're newer to the craft beer scene and only had a hand full of local things, you may rate/review higher because you don't know what's out there.
     
    Ninjakillzu, BeerPugz and Harrison8 like this.
  10. StoutElk_92

    StoutElk_92 Meyvn (1,225) Oct 30, 2015 Massachusetts

    It depends what you call biased. I think most people rate on how much they like the beer, and most raters on here seem to like the taste of intense IPAs and stouts more than a pilsner or hefeweizen for examples. I do see a lot of straight 5s when someone likes a beer, and in my glass it doesn't look a 5 or feel a 5, etc to me. Everyone is different, some people will like a beer and others not so much. I like reviews that describe the actual beer and not the experience around it, and when the individual ratings seem fair. Generally I think a lot of people underrate, and I probably overrate. If a pilsner looks, smells, tastes, and feels really good to me then I will rate it as high as I feel accurate to my senses, not a 4.25 because it's only a pilsner. It's all subjective but I like to think that some ratings on here are honest and fair. It doesn't mean everyone will like or dislike a beer though, because everyone's palate and preferences are different. I usually rate overall based on drinkability and at least somewhat respect to the style.
     
    #50 StoutElk_92, Apr 21, 2017
    Last edited: Apr 21, 2017
    BeerPugz, Barracudas and utopiajane like this.
  11. kculrich

    kculrich Zealot (574) Jun 21, 2014 New Hampshire
    Beer Trader

    And this is another way in which the ratings are, if not biased, then at least skewed. Personally, I seek out highly-rated beers of styles I enjoy - DIPA's and RIS's. Assuming the ratings are pretty much justified, I will likely rate it highly myself. Extend this phenomenon to thousands of other BA's doing the same and behold the result.
     
  12. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    Yes, that's a known bias that comes with not requiring that the reviews/ratings be done blind.

    But there are also a few other things at play. A reasonable number of folks will express their disappointment that the beer didn't live up to their expectations or justify the hard work or mony the spend in finding it by scoring more harshly. So this is where large numbers of reviews/ratings come into play and where the numbers can be useful. The overall ranking or mean score will be based on the reactions of lots of folks with different biases so at least some of that first bias winds up getting lost in the noise.

    It's also where sorting reviews can be helpful, I'd say that the top reviewers and the lowest rated reviews are likely to provide a useful counterbalance. The top reviewers have been around the block more than once and are less likely to just heap praise on a beer when others have but where it isn't really due. The lowest scores given by reviewers/raters of the beer (and the size of the pDev on the beer's page) give a sense of what are the disappointments/disagreements, what is the level of disagreement and some of the reasons both groups offer for reviewing/rating as they did.
     
    BeerPugz likes this.
  13. tlema1

    tlema1 Meyvn (1,220) Nov 19, 2008 South Carolina
    Beer Trader

    I try to rate to style, and I try everything, even beers that I don't like, so I can give my customers honest feedback, but it's tough for me with sours and wild ales, since I despise them so much
     
  14. zid

    zid Crusader (743) Feb 15, 2010 New York
    Beer Trader

    Please don't stop your mission with the style list on this site.
     
    bbtkd likes this.
  15. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    I'm very curious then, so a question if I may? Why do you even bother to use this, or any, site that is centered around reivews/ratings?
     
  16. zid

    zid Crusader (743) Feb 15, 2010 New York
    Beer Trader

    Of course ratings are biased. They always will be, even if rating "to style." It's inherent and it's fine.

    This topic has been explored many times and there are multiple schools of thought on rating to style:
    1) Anyone not rating to style is breaking the system and doing it wrong.
    2) I'll never rate the greatest AAL higher than a great barrel aged stout because in reality there is an inherent hierarchy.
    3) I rate according to taste regardless of whether or not it is to style.

    None of these approaches are really wrong even if they aren't compatible. Group ratings that reflect all of the views above will be inherently more interesting than a singular approach.

    Personally, I view beer through the lens of style, but I think "rating to style" is a terribly flawed idea. When @TongoRad specifically says "with style in mind" he is not being a pendant nor is he being casual with his word choice. It's a deliberate choice and it makes all the difference (for the better). You don't judge a Helles lager according to the same standard as a stout, but there is no genuine standard for a Helles lager or a stout. People tend to treat "rating to style" as a gold standard of a wide viewpoint, but it's really a narrow (or misguided) viewpoint. Rating according to personal taste is also a narrow viewpoint by definition.
     
    herrburgess, gibgink, cjgiant and 6 others like this.
  17. dennis3951

    dennis3951 Champion (819) Mar 6, 2008 New Jersey
    Subscriber Beer Trader

    That's good. Since the things you despise about sours and wild ale are likely the things fans of sours and wild ale love about them you are giving useful feed back to your customers.
     
    tlema1 likes this.
  18. Leebo

    Leebo Initiate (119) Feb 7, 2013 Massachusetts

    Beer releases, current news, New England happenings, beer meet ups, events and beer discussions, style and otherwise. Went to Newport RI last weekend. Where to find good beer and places to eat. Here of course. Do you only look at ratings?
     
    Dravin and drtth like this.
  19. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    Thanks.

    No, quite the contrary, I don't look only at ratings. I make a much heavier use of reviews in addition to such things as beer discussions (there have been times that the review-like comments from certain people in such discussions are all I need to give something a try). That is partly conditioned by having had, until the last couple of years, to purchase beer only on draft, only at exorbitant prices, or only by the case. So that is one reason I find reviews useful.

    As an example of the usefulness I've found in reviews, few years ago I was confronted with a decision. Boulevard started distributing to PA. In my local retailer, case only at the time, I saw a very fresh case of the 12 oz bottles of Tank 7. So I had three choices. Leave it sit until after I finally found someone offering it on draft. Buy an entire case having no idea what it tasted like. Read the reviews from some reviewers whom I knew to be careful, thoughtful and have tastes similar to mine.

    I came home, read some carefully chosen reviews and went back the next day to buy that case before someone else bought it (after that case it took me another six months to find a really fresh case to buy as a repeat so I was glad I didn't leave it sit very long before buying the first case). That beer is still among my favorites list and if it hadn't been for the reviews I might never have had it at all.

    Similarly the place reviews are very helpful when traveling to areas where I don't know anyone, have only a day or two free, and want to find somewhere to go beer shopping or spend time having a beer. Again that has proven very useful in saving me time, money and helping me to find good beer that is fresh. (The beers are often local even when I'm not. :))
     
    #59 drtth, Apr 21, 2017
    Last edited: Apr 21, 2017
    gibgink and utopiajane like this.
  20. andy712

    andy712 Initiate (146) Jul 23, 2016 Oregon

    I hope this isn't too off topic, but I'm curious given that you're from Vermont and rate based on, among other things, on appearance and mouthfeel. How do you rate the appearance of the super cloudy NE style IPAs that are so popular right now? Do you rate relative to other NE IPAs or relative to the general IPA style? Same re mouthfeel. Personally, I have had a number of NE style IPAs that I have liked (although I prefer IPAs with a little malt backbone), but I find the look off putting and would rate lower in appearance than, say, a Pliny the Elder. Taste is, of course, most important, but I am befuddled by reviews giving 5s for appearance for beers that look nothing like the traditional look of that beer style (I am presuming IPAs were initially mostly clear, but I could be wrong about that).
     
  21. CrimeDog

    CrimeDog Aspirant (212) Dec 31, 2015 New York
    Beer Trader

    Great post/question....

    Earlier today, I bought a 4 pack of Putting Out Fires from Sand City which got me thinking about the same thing (I think)...

    "Fires" is an awesome session, so in my opinion it should be judged against other sessions - not big DIPA's etc...

    But at the same time I understand someone elses post about how much they enjoy the beer in front of them at that moment....damn...Im rambling now...

    Have a great weekend everyone #LGM
     
  22. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    Agree, "rating to style" on a site such as this one is basically flawed, if only because almost none of us have the training or experience to do it well and virtually nobody reviews blind. I'd say that's why both @TongoRad's comment and the instructions on this site about "keeping style in mind" when rating are so important.

    But I would suggest that there is indeed a standard for an Helles lager or a different standard for a Stout. But for each currently used/definted style there is not a single point on each dimension used in categorization that is the only target. The categorization dimensions are fuzzy dimensions with each dimention having a range of values associated with or "typical" foreach style. For example, there is a range of ABVs that are said to typical for an APA but there are examples of APAs that fit most or all of the other dimensions of categorization but that have an ABV that falls outside the normal range. Etc.
     
    TongoRad and zid like this.
  23. zid

    zid Crusader (743) Feb 15, 2010 New York
    Beer Trader

    I mainly agree, but to a point ;). To take it a step further - if standards of beer styles are ranges rather than points, and one is rating to style, then any beer that falls within those ranges w/o flaws (flaws can even be subjective as well) would be a perfect "5." If the ranges are large, then there's little point in using that approach. Rebel IPA, Redhook Long Hammer, Harpoon IPA, Racer 5, and Two Hearted might all fit within the acceptable ranges for American IPAs (and if they don't, then perhaps there's an issue with the ranges rather than the beer), but should everyone rate them evenly?
     
    TongoRad likes this.
  24. rgordon

    rgordon Defender (664) Apr 26, 2012 North Carolina

    If nothing else, various rating (systems) are an interesting yardstick for current trends. In the wine world, if, say, "Barnstormer" Zinfandel scored a 95 at the Cloverdale Fair in 1995, that 95 score redounds into eternity on bottle neckers, box graphics, and company propaganda. I take almost all ratings with a grain of salt. There are some voices that I do trust. Ratings are indeed biased.
     
    dennis3951 likes this.
  25. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    Well, first lets abandon the idea of "rating to style" which is different than rating "with style in mind" and I think we can agree that the former will never be the norm on sites like this, if only because there is no way to force people to do blind ratings.

    Next let's factor in the fact that people themselves have different patterns of behavior when it comes to using rating scales, regardless of what is being rated. There are some folks who almost never use the extreme scores and there are others who seldom use the values in between. So a "perfect" 5 across all raters is simply not going to happen, and is an unrealistic expectation about human behavior.

    Sure, if the ranges are wide rather than narrow there's likely to be a lot more "slop" in the system but that doesn't really mean they are not useful or that there can not be multiple beers falling into the "equivalence category" created by the range and still display differences within that range. Further, that doesn't mean their scores/ratings, etc. are not useful.

    Rather it means people have to live with variability. Which is actually something we do regularly and quite well in many circumstances. To use a loose analogy, we often speak about people who are "redheads" and are well understood when we day that, but there is acutally wide range of shades that fall into that category. To do allow anything else would put us in the position of having to have a different category name for every individual different shade and the location of a line would still be difficult to decide.

    So our choices boil down to either a few potentially imperfect categories with fuzzy boundaries and ranges that may be of different widths or it leaves us with an almost totally unworkable multiplication of descriptors, each referring to a different category. In addition we'd still have a problem determining which criterion of difference-sameness to use. For the example of hair color, would it be human ratings limited by the sensitive of the human eye and differences in terminology, or would it be the unambiguous wave length of light which can be assessed very precisely but contains many subdivisions not detectable by the human eye.

    All of which is a long, overly detailed way to geting to "No, we should not expect every individual to rate exactly as does every other individual."
     
    #65 drtth, Apr 21, 2017
    Last edited: Apr 21, 2017
    SFACRKnight likes this.
  26. Junior

    Junior Aspirant (259) May 23, 2015 Michigan
    Beer Trader

    Not really bias, but I find it interesting that the average rating on this site is somewhere near 3.75. Shouldn't it be closer to 2.5 or 3.
     
    SFACRKnight likes this.
  27. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    Only if there is a normal distribution of ratings centered around one of those numbers. (3 since there is no 0 score to be assigned.) But that requires assumptions and/or procedures not in place.
     
  28. Ninjakillzu

    Ninjakillzu Disciple (331) Oct 5, 2015 Washington

    Bias will always exist. My ratings are almost always skewed high, because if I like a beer's aroma/flavor, the rating for those will never go below 4.5. It's not often that I dislike a beer. My ratings are usually a combination between flavor and style correctnesd.
     
  29. SFACRKnight

    SFACRKnight Savant (976) Jan 20, 2012 Colorado
    Beer Trader

    That's funny, I find rare beers never live up to the hype and I usually rate them lower. KBS was one of those beers. I had just built it up in my head and when I finally tried it I was let down. Same with red poppy. Same with a bunch of others too.
     
    cavedave likes this.
  30. 1ale_man

    1ale_man Aspirant (273) Apr 25, 2015 Texas

    For what it's worth, I really "listen" to what I read in the ratings on this site! All of you have way more experience than I. These ratings may be somewhat skewed, but I still "listen". We all have our favorites, and no matter how hard we try, we will lean toward them! So please!!! Don't lead me too far astray! By the way. I don't rate porters and stouts! Not my favorite! Y'all keep on rating!!!!

    Cheers:):D
     
    cjgiant and zid like this.
  31. StoutElk_92

    StoutElk_92 Meyvn (1,225) Oct 30, 2015 Massachusetts

    The American IPA is nothing like the original IPAs. Why does a beer need to look traditional to be a 5? If it looks good it looks good that's about it. What if the modern interpretation of a style looks better than the original traditional interpretation? Do we rate it lower because it's not how it is suppose to look?
     
    Squire123 and cavedave like this.
  32. andy712

    andy712 Initiate (146) Jul 23, 2016 Oregon

    Well, I guess that's what I am trying to get at. Is there any objective measure of appearance or is it entirely subjective? E.g., I like NE IPAs; this looks like a NE IPA; I therefore rate it a 5 in appearance. If we don't "rate it lower because it's not how it is suppose to look" what's the point of rating appearance? I suppose it's a matter of deciding what the criteria are. Maybe NE IPAs are an anomaly, or at least a substandard with its own criteria for appearance. In the end, it's a pretty minor component of what makes a good beer, I guess.
     
  33. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    It is, as you say, a matter of deciding what the criteria are. In the case of the NE IPA does one use the long standing tradition of clarity or allow murky appearance as a good thing. That really won't be resolved until the larger debate over whether NE IPA is a new style or not has been resolved. The statement of using different criteria should, however, be made by the reviewer if they choose to ignore the long standing criteria of clarity and a fluffy white head that leaves lots of lacing. These did not come about arbitrarily, if only because a fail on either or both might be an indication of a brewing flaw.

    However, this appearance business is not such a minor matter that we should dismiss it out of hand, the way some folks do as being irrelevant to their enjoyment of the beer. (And which you are clearly not doing in this thread.)

    Appearance can have an effect. Not only can it indicate a brewing flaw it can be shown to have an effect on people's expectations of flavors and their enjoyment, etc. This is one reason food production companies actually worry about such things as clarity in some beverages.

    http://www.foodbusinessnews.net/articles/news_home/Beverage_News/2014/01/The_value_of_beverage_clarity.aspx?ID={1DB752F4-D4F8-466E-B250-4FD4AEE47479}

    Until the debate is resolved about whether the NE IPA is "a new style with it's own criteria or still part of an older style that violates one or more widely accepted criteria," my own approach is to be quite clear in my review about what my reasons are for assigning the numerical value that turns my subjective experience into an objective number. That way any one who happens to read the review can make their own judgment about what I've done and why.
     
    #73 drtth, Apr 22, 2017
    Last edited: Apr 22, 2017
    cjgiant, kculrich and utopiajane like this.
  34. TongoRad

    TongoRad Poo-Bah (1,753) Jun 3, 2004 New Jersey
    Beer Trader

    I look at it more like multiple dimensions rather than larger ranges. Some beers are delicate, some are bold, some are inventive, some are classic, and on and on. All can be seen within the construct of keeping style in mind.

    So, although I don't think all of the beers you listed are on the same level, I do allow for multiple beers being on the super elite (not "perfect") 5 level.
     
    utopiajane likes this.
  35. drtth

    drtth Poo-Bah (2,867) Nov 25, 2007 Pennsylvania

    Hmm, but if you allow for multiple dimensions rather than ranges, where is your either/or dividing line between delicate and bold? :)

    I think you've clarified part of what I was hoping to say as well. There are both the problem of deciding which dimensions to use and how broad the dividing line (i.e., the range) is to be.
     
    TongoRad likes this.
  36. TongoRad

    TongoRad Poo-Bah (1,753) Jun 3, 2004 New Jersey
    Beer Trader

    To really simplify it, the question is "what are they going for, and how well did they do? ". In reality, though, the answer isn't all that simple to answer, and can take a few times with a new beer to wrap ones head around it.
     
    cavedave and drtth like this.
  37. dlcarst

    dlcarst Aspirant (237) Aug 21, 2015 Illinois
    Subscriber Beer Trader

    Absolutely, although I've found myself becoming more balanced lately. I used to give 4.5+ ratings to double IPAs and coffee or barrel aged stouts, now I've found myself giving (only!) a 4 for highly rated/hyped beers and 4.5 for a superb Irish stout or Vienna lager. However, I still can't seem to break out of rating almost every single beer I drink 3.5-4.5. Part of it is that I know how to avoid mediocre beer, but when I look at my ratings I often can't believe that I rated two beers so close to each other when I clearly enjoyed one much more.
     
    Squire123 and utopiajane like this.
  38. Giantspace

    Giantspace Devotee (494) Dec 22, 2011 Pennsylvania

    I rate by how I like what I am drinking. If drinking a Hefeweizen I think to the Hefeweizen I have had before and compare. If it's a style I don't know about I will still rate to my taste , correct to style or not. I try to read reviews before using the scores to decide if I would buy.

    I use a 3.0 score for PBR as my base for all beers.

    Enjoy
     
    Squire123 likes this.
  39. StoutElk_92

    StoutElk_92 Meyvn (1,225) Oct 30, 2015 Massachusetts

    What I'm trying to get at is I think we should just rate based on how good it is to us, in this case looks. I've had hazy unfiltered IPAs that don't look nearly as good as say most Trillium or Tree House beers. If you like it you like it, never mind what it's suppose to be like. If it's good it's good. That's how I feel about rating.
     
  40. Beer_Economicus

    Beer_Economicus Initiate (149) Apr 8, 2017 Indiana
    Beer Trader

    I have read through all these posts, and I feel like everyone is providing solid feedback. Here'S my take (with some overlap to previous posts).

    There will inherently be bias for myriad reasons, most notably:
    1) Everyone is not starting at the reference point. Some people come into a review with 50 beers, some with 250. This will dramatically alter your perspective.
    2) Everyone has different tastes. I do not mean that some people like IPAs and some don't, I mean literally that taste is different. Someone that says "I have rarely had a beer that I cannot drink" will likely have higher avg scores than someone who is significantly more selective.
    3) regarding #2, if you only drink "high regarded" offerings, your avg score will also be different, as you have more high end beers to compare to (even if still comparing to high end beers)
    4) people are likely not only comparing within "style", but also within class. I can see a lot of people comparing KBS largely to other "well regarded BA stouts" rather than just other "BA stouts." This will make a big difference.
    5) There is no guide for people to use for rating, and as a result most of the scale does not get used. One person may use 1 as "the worst beer ever", while someone else might say that their worst beer ever was still drinkable, and therefore a 3. Same silly logic you see in graduate school grades where you see clumping at 90%. Tells you very little about where everyone falls compared to the others in that group - instead it just says "competent."

    Few others, but these seem to carry most the weight (for me).
     
    Phoodcritic, cjgiant and StoutElk_92 like this.
  • About Us

    Your go-to website for beer (since 1996), publishers of BeerAdvocate magazine (since 2006) and hosts of world-class beer events (since 2003). Respect Beer.
  • BeerAdvocate Microbrew Invitational

    Join us June 2-3, 2017 in Boston, Mass. for beer, cider, mead, kombucha and sake from over 70 small producers.

    Learn More
  • Subscribe to BeerAdvocate Magazine

    Support uncompromising beer advocacy and award-winning, independent journalism with a print subscription to BeerAdvocate magazine.

    Subscribe