What rDev and Histogram says about an individual's beer rating tendencies

Discussion in 'BeerAdvocate Talk' started by Limazulu, Feb 22, 2020.

  1. Limazulu

    Limazulu Initiate (69) Jan 4, 2020 Idaho

    I am interested in hearing wisdom from those on the high end of the Beer Karma spectrum about how best to use the metrics on my beer ratings to make sure my ratings are legit. I've been geeking-out a bit on whether my rDev is wacky... high ratings too high, and/or low ratings too low. I use the community's pDev as the range within which I generally expect my ratings to land. When my rating is well-outside of this range (either more favorable or more critical), I go back and start questioning whether i missed something good or bad about a brew that caused me to deviate. While this may help me to quality control my ratings, I fear it will lead to what data analysts refer to as "central tendency error"... where I start to rate everything right around average. My histogram will be a perfect bell-curve, but I won't allow myself permission to go way outside of the boundaries on occasion when I have a uniquely positive or negative reaction to a beer. One the other hand, I'll often go and drink a second bottle just to double check my opinion. And hey, if drinking a second beer is wrong, then I don't wanna be right!
    mikeinportc, hottenot and rclangk like this.
  2. Insomniac

    Insomniac Initiate (28) Nov 5, 2019

  3. beertunes

    beertunes Poo-Bah (7,009) Sep 24, 2007 Liechtenstein
    Society Trader

    Don't worry about trying to make you numbers :fit". Just do the reviews fairly and honestly, and let the numbers fall where they may.
    VanTyne, maximum12, RyanK252 and 23 others like this.
  4. readyski

    readyski Aspirant (259) Jun 4, 2005 California

    Believe in the law of averages (and the theory of chaos). They will get you to where you need to go.
    VanTyne, GuyFawkes, jrc1093 and 3 others like this.
  5. zid

    zid Meyvn (1,416) Feb 15, 2010 New York
    Society Trader

    There's probably a fine line between learning from the impressions of others and being a sheep in the flock. :slight_smile:
    VanTyne, beergoot, denver10 and 8 others like this.
  6. WunderLlama

    WunderLlama Poo-Bah (2,304) Dec 27, 2010 Massachusetts
    Society Trader

    Dont drink to rate beers for the mean or median, drink for the taste, aroma, appearance. Savor and rate it to your senasations for the beer. Rate for the style of beer but dont trash a beer a for offering the style it is supposed to represent.

    “Ewww, that Berliner Weisse was so tart and sour, I gave it a 1.0 for taste”

    Let your perceptions be your guide to that beer and outliers to the mean are always welcome. I’d ask that you complete your reviews so that meet the minimum for ratings.


  7. unlikelyspiderperson

    unlikelyspiderperson Meyvn (1,228) Mar 12, 2013 California
    Society Trader

    I agree with others that it is only muzzling yourself to worry about rating with the herd.
    I think that the best use of.your own rDev is to tell you what beers (and styles) you tend to like much more/less than the population at large. This can basically help you to translate the aggregate ratings by learning your own "handicap" for a given style. For instance if you find you always have a high positive rDev with Vienna lagers when you look at a new to you Vienna lager's rating you can mentally add some points and not be scared off by a 3.6 rating.
    The pDev is mostly useful to identify love/hate beers. If a beer has a high pDev that tells me I should ignore the score and try it if it sounds interesting. A beer with a narrow pDev tells me that the rating is probably reasonably accurate and I can apply my personal style handicap.
    But seriously, just give each beer you want to review your earnest intention and record your honest thoughts and enjoy what.you learn
  8. Scott17Taylor

    Scott17Taylor Poo-Bah (1,773) Oct 28, 2013 Iowa
    Society Trader

    Just rate and review to your own specifications. You’ll keep getting better and better the more you do it. Don’t worry about outliers, rating have a way of evening out in the long term.
  9. Bitterbill

    Bitterbill Poo-Bah (7,069) Sep 14, 2002 Wyoming
    Society Trader

    I love it when I favour a beer that gets lousy ratings and vice versa. :slight_smile:
    GuyFawkes, beergoot, dcotom and 6 others like this.
  10. PapaGoose03

    PapaGoose03 Poo-Bah (2,633) May 30, 2005 Michigan

    The science of Statistics is a very precise medium because you're dealing with math and numbers. However, the source of those numbers can make the 'science' imprecise, thus the calculations can lean toward being meaningless.

    The source of the rating numbers that are used to calculate the standard deviations here on BA come from the members. There are two basic methods of rating beers -- rating a beer to style, or rating a beer to personal likability. Both methods are very subjective and have their faults. (Bias toward certain styles, trying to rate the first beer in a new-to-you style, etc.)

    The problem comes from that fact that all of these numbers are thrown into a common pot here on BA so that you have stats that cannot be considered good for accurate calculations. Maybe as the law of large numbers kicks in the credibility of the numbers goes up, but some beers that you rate may have only a smattering of ratings, thus no chance of the larger-number law kicking in. Deviation calculations try to adjust for this, but look at the source of the numbers used in those formulas.

    That's the long way to say (like everyone else above), just go ahead and rate the beers the way you think is correct, and do your best based on your experience in the various styles that you choose. We all appreciate that you're rating the beers to add to this site.
    beergoot, dcotom, mikeinportc and 3 others like this.
  11. SLeffler27

    SLeffler27 Poo-Bah (2,201) Feb 24, 2008 New York
    Society Trader

    This is a great approach, even if I hadn’t thought of it first. I may have to rethink my awesomeness... nah, still my hat is off to unlikelyspidyperson.

    Aside from this, what everyone else has said. Trust your impression, consider style guides, be fair minded, and let the judges do there thing. After all the one benefiting most from your reviews is you yourself.
  12. SLeffler27

    SLeffler27 Poo-Bah (2,201) Feb 24, 2008 New York
    Society Trader

    hottenot likes this.
  13. rgordon

    rgordon Meyvn (1,036) Apr 26, 2012 North Carolina

    That covers it nicely. I simply record my impressions when I do reviews and never worry an ounce.
  14. mikeinportc

    mikeinportc Champion (887) Nov 4, 2015 New York

    This^ , along with what the rest have said. You are who you are , just let it flow.

    For myself, in looking at a new-to-me beer, I don't (mostly) go by the numbers . I look at the experience the reviewers describe. In deciding how to trust/weight a particular review, I also look at the BAs other beers, especially the ones we have in common. What does that person tend to like or dislike? If we have similar experiences with beers in common, then I'd guess it's likely I'll have a similar experience with the unknown beer.

    Sooooo....... when I'm doing reviews, I put more effort and concern into describing it, than with the #s. Hoping that gives someone reading it some help in deciding .
    4.0 or 4.25? :thinking_face: If I have to think about it that much, then I'll tend to give the brewer the benefit of doubt.

    I haven't run into any styles that I don't like, just individual beers.:wink: I do love sour styles, Belgians, and funky wild ales. That does produce some outliers. As @WunderLlama suggests, there are a lot of reviews by people that just don't like(or get) a given style. Also, I get the impression that a lot of people are drinking the beer too damned cold , and some that are way past prime. Point is, if your review is vastly different than most, just explain it , and that will be helpful . Even bad reviews of old beers can helpful ,if noted, because that might be what is found in the wild. It can give a picture of whether or not a given beer on the shelf might be worthwhile.

    The only way to be able to do it, is just do it.
    Just give it your best shot, & caveat emptor for the reader. :sunglasses:
  15. dbl_delta

    dbl_delta Poo-Bah (2,488) Sep 22, 2012 Pennsylvania

    Don't sweat the rDev and pDev. Don't look at the other reviews. Just give it your best shot and be honest.
    I try to treat the scoring system as my own personal "journal" - what I thought of a beer, admittedly as influenced by my prejudices and preferences. That means my ratings are consistent as they apply to me. , and gives me a good idea of what I like and don't like. That sounds kind of selfish, but it eliminates all the variability of BA reviewers.
    I do usually check my score against the BA average after the fact, but I never go back and change my scores.
  16. Bluecrow

    Bluecrow Poo-Bah (1,514) Jul 16, 2012 New York

    I think about the/my “apparently high avg rating score” but also recognize that I don’t often drink anything that I expect to dislike. I do not drink a brew that I do not want, just to add a rating. My ratings are skewed thusly.
    I also appreciate and enjoy a wide range of styles and try to rate those within the realm of a similar style. Perhaps I am justifying “grade inflation” but I think not.
    My physics students would never/rarely characterize their experience with me that way.
  17. StoutElk_92

    StoutElk_92 Poo-Bah (2,570) Oct 30, 2015 Massachusetts

    The histogram and your avg rating all depends on how good the beers you drink are as much as how high or low you rate. The more higher rated beers you drink and rate, the higher your avg rating will be. If you drink a lot of crappy beers that you don't like, then your histogram will show with a lower avg score. Like a lot of folks have said, don't worry too much about the end result, focus on the minute details, rating each beer to your fairest and highest degree. Everyone rates differently and prefers different styles, so just rate how you think is fair and good. I have plenty of beers I have rated well above the rDev, and when I look back I never change them because to me the ratings are accurate, and when I retry a beer and look back at my ratings, I am open to but hardly ever change my rating because I have such a set standard now that it is what it is when I rate a beer. Over time you will likely find your standard for what each interval stands for and what beers hit those marks. Stay open minded. Cheers :beers:
    Bluecrow likes this.
  18. beertunes

    beertunes Poo-Bah (7,009) Sep 24, 2007 Liechtenstein
    Society Trader


    What the existing average is should have no impact on his review.
    PapaGoose03 likes this.
  19. StoutElk_92

    StoutElk_92 Poo-Bah (2,570) Oct 30, 2015 Massachusetts

    Generally speaking, highly rated beers get higher ratings, thus if he is rating highly rated beers more than likely his ratings for those will be higher than a lower rated beer, though his rDev could be off, sure, just generally speaking.
  20. bbtkd

    bbtkd Poo-Bah (5,231) Sep 20, 2015 South Dakota
    Society Trader

    OP, your reviews are meant to be subjective, don't overthink them. That said, reviewing styles that you don't like, then rating them low for the characteristics they were created for would not be "fair". Like rating IPAs low for being bitter, or sours for being sour. Before rating a new-to-you style, take the time to read the description of it under Beers/Beer Styles.
    beergoot, Bitterbill and StoutElk_92 like this.
  21. mudbug

    mudbug Defender (618) Mar 27, 2009 Oregon

    Just remember, if all beer drinkers (or even a large percentage of them) in America joined BA and rated their favorite beer the top rated beer would be Bud Light. A vast majority of BAs that rate/review beer on this site only rate beers they like.
    dcotom likes this.
  22. Limazulu

    Limazulu Initiate (69) Jan 4, 2020 Idaho

    I think Beertunes' point is, if you only drink beer that you really like, then by definition you will not have a bunch of low ratings in your histogram.
    unlikelyspiderperson likes this.
  23. beertunes

    beertunes Poo-Bah (7,009) Sep 24, 2007 Liechtenstein
    Society Trader

    My point was more that if folks only drink beer that other people like (highly rated), they're going to highly rate that beer. They shouldn't. A given person may not have the same experience, and shouldn't worry if their review ends ends up as an outlier.

    Don't look at scores before you review. Don't follow the flock. Don't succumb to peer (beer?) pressure.
    Bitterbill likes this.
  24. hottenot

    hottenot Aspirant (298) Aug 13, 2018 North Carolina

    My reviews run the curve of way too high or way too low.
    Occasionally spot on with the others.

    It matters not to me if anyone cares what I think about a beer.
    Since it's me who's opinion that matters. To me.
  25. dcotom

    dcotom Poo-Bah (2,362) Aug 4, 2014 Iowa
    Society Trader

    I'm usually so far off the reservation that it'd take all the resources of The Ohio State University Alumni Association to track me down. I give out 5's like Halloween candy. A lot of my low-rated beers are AAL's, Euro pale lagers, and light lagers. Although these are styles I don't particularly like and ones I wouldn't normally drink even if they were free, my rDevs for them are all over the map. I probably represent an outlier in more than a few samples here, on high and low sides alike. But I always try, maybe not always successfully, to rate to the style guidelines and to take into account the brewer's intent. For example, I gave DKML and Hopslam straight 5's because I thought Founders and Bell's absolutely nailed them across the board. I never second-guess myself based on others' reviews (although I do feel vindicated now and again, especially when folks like @superspak and @woodychandler agree with me :grinning:).

    Trust yourself. You are the world's foremost expert on what you think.
  26. StoutElk_92

    StoutElk_92 Poo-Bah (2,570) Oct 30, 2015 Massachusetts

    My point was that higher rated beers are usually “better” and that’s why they get high ratings, and why most people like them more and rate them high. I wasn’t talking about rating close to the beers avg rating because it is supposed to be high, I just meant that it is likely that beer is good and you will rate it higher because it is good, despite other people’s reviews or ratings, but with the freedom we can rate good beers low and bad beers as high as we please.
    Limazulu likes this.
  27. beertunes

    beertunes Poo-Bah (7,009) Sep 24, 2007 Liechtenstein
    Society Trader

    But going into it knowing that a beer is highly rated will influence how it's perceived, by a lot of people. That internal pressure will then (either consciously or not) effect how they rate the beer. How many folks are able to say "well, I don't think this beer is great, but, there's 2000 people on the internet that thinks it's awesome, so I must be wrong", and not have it have an influence?
    Bitterbill and PapaGoose03 like this.
  28. StoutElk_92

    StoutElk_92 Poo-Bah (2,570) Oct 30, 2015 Massachusetts

    Everyone is different. What do you do when you are the first one to rate a said good beer? I was thinking of it as a personal issue, for instance I have rated over 200 Trillium beers because they are local and pretty easy to get, and now I go to their brewery more than I buy beer in stores. I rate a lot of these beers highly not because I think they "should be" good, but because I think they are. Conversely, if I drank no Trillium beers and only drank Sam Adams beers that are generally lower rated, my avg score would be lower, unless of course I was rating Sam Adams beers above 4.5 which is not what I usually rate them. Of course there are people like you though that go in drinking a beer like King Sue and come out giving it a 3.3 rating. It doesn't always work that way, but usually highly rated good beers get higher ratings from people that like the style, unless maybe you have a bias against high ratings and certain styles, and tend to underrate beers.
  29. beertunes

    beertunes Poo-Bah (7,009) Sep 24, 2007 Liechtenstein
    Society Trader

    Well the more bees ya drink, the more similar each example of a given style become. Therefore, more are closer to average, than being on either end.

    With that, when you've had a couple hundred of a given style, if you're the first to review a new beer, you've got a solid baseline to compare against. So, the first person should be able to slot it in appropriately.

    Given that BA scores on a 1-5 scale, the average beer (which is a damn tasty quaff) should score around 3ish, with 4+ scores being rare.
    PapaGoose03 likes this.
  30. StoutElk_92

    StoutElk_92 Poo-Bah (2,570) Oct 30, 2015 Massachusetts

    That's another thing, how people rate. Everyone is different again there. I like to go by a grading approach, so 5 might be 100/100, a 1 rating is a 20/100 (should anything really be less than 20?), and 3 is in the middle, you could say a 60/100, which would be a D- grade, and anything under a 3 would be an F grade. Not everyone rates like this, but some people do, and from my perspective most craft beers shouldn't be rated less than a 3 unless it is a seriously crappy brew, so most of my avg decent beers lay closer to around a 4 than a 3 or 3.5, and whatever is under a 4 I honestly didn't really like that much, and under a 3.5 I really didn't care for, so it's hard to compare ratings when people can rate so differently. I guess we have to just believe that it all balances out.
  31. beertunes

    beertunes Poo-Bah (7,009) Sep 24, 2007 Liechtenstein
    Society Trader

    See, to me, a 4 is (to use a term I hate) World Class. A 4.1 is rarified air, and above that we're in unicorn territory. There's just no way I believe that folks are drinking the very best beers in the world ad their daily drinkers. I do believe they're pumping their egos, by pumping scores.
    PapaGoose03 likes this.
  32. unlikelyspiderperson

    unlikelyspiderperson Meyvn (1,228) Mar 12, 2013 California
    Society Trader

    But if a 5 is the pinnacle of a given style then shouldn't there be many beers that are 80%+ of the way to expressing that style perfectly (a 4 rating)? Especially for older styles like barleywine or brown ale. Lots of people make very solid versions of those styles and if 5 is perfect then a 4 is still a good beer without having to approach perfection
  33. StoutElk_92

    StoutElk_92 Poo-Bah (2,570) Oct 30, 2015 Massachusetts

    Well what are the best beers in the world if they're not from the top rated best breweries in the world, or world class rated beers? Is there something better out there that we're all missing? Everyone doesn't drink the same beers but at some point there has to be a standard for what is the best. If a 4.5 is a 90/100, that should be the line for some of the top rated/best beers, or a 4.75, a 95/100. In the end I guess just rate what you feel and the numbers will speak for themselves.
  34. beertunes

    beertunes Poo-Bah (7,009) Sep 24, 2007 Liechtenstein
    Society Trader

    I don't think so. There should only be a very small number of 5 scores. Sure, there will be more 4s, but still a relatively low number.

    By definition, most things can't be either above or below average, most things are near average.
  35. unlikelyspiderperson

    unlikelyspiderperson Meyvn (1,228) Mar 12, 2013 California
    Society Trader

    But that's not how the number scale on this site is set up. 3 isn't average. This is what BA says the beer rating numbers mean;
    Its not so much in comparison to all other beers but instead looked at against a sort of platonic ideal of the style. There is absolutely no reason that a style like English mild couldn't be populated by mostly 4s and above since few people make them, its a very simple style where the ideal is very achievable, and many of the folks making them have probably been doing so for a long time.

    My point is more so that the way that you rate isn't the only way to rate (in fact it's not even the "official" way to rate). Any site that aggregates ratings of consumer product/experience has to accept that people bring their own program to bear on rating things and the data is only valuable if you have a high enough number of ratings to sort of balance things out.

    For me, since BA is where I learned to rate/review beer, and since I seek out good to great beer, I rarely have stuff that is below a 3.5. I'm making a concerted effort to drink beer that is better than "OK" and I'm half decent at achieving that goal.
    Limazulu, PapaGoose03 and StoutElk_92 like this.
  36. Bitterbill

    Bitterbill Poo-Bah (7,069) Sep 14, 2002 Wyoming
    Society Trader