Are beer ratings biased?

Two kind of super-obvious points about this...

1) BeerAdvocate probably doesn't attract a lot of people who are routinely drinking and rating below-average beers.

2) The ratings are not weighted by volume of beer brewed. There are probably, what?, less than fifty high-volume macro labels in broad distribution compared to at least tens of thousands of craft and regional microbrewer labels. So, in aggregate, a Bud Lite has the same overall weight as a Trillium DDH Melcher Street even though there are ten-hundred-million-bazillion more gallons of Bud Lite being brewed and consumed.

It probably didn't need to be said. Sorry.

 

No, and the simple reason that is easily proved is that people on this site tend to only rate the beers they enjoy. How can I prove it? Easy, look at the amount of reviews for Bud Light (The number one beer sold in the US) about 1400, verses Pliny the Elder (Not available almost everywhere)over 3000. Now that leads to only a few conclusions. One is that there are 1500 BAs that have never had a Bud light but had a PtE (preposterous) Or that there are a lot (IMHO the majority) of BAs that choose to only rate beers they like. Hence the average tends positive.

 

There is bias in everything that humans do that is subjective, and especially something that is as diverse and evolving (RAPIDLY) as beer, especially in the "micro" segment. You thinking ASTMO is "a joke to me if someone is picking a stout over a pilsner because 'it tastes better'" is akin to someone else saying, "It's a joke to me that someone thinks Pilsner Urquell is a better representation of its respective style than Surly Xtra-Citra - just tell me which one TASTES BETTER!" Both points of view are viable.

Though I will say that reading that Xtra-Citra was a drain pour certainly raised my eyebrows a bit - I understand that a beer isn't to someone's taste, but a drain pour IMHO is a pretty strong term for most beers, especially one that rates as "Very Good" on a statistically significant number of reviews/ratings.

 

You've discovered a correlation, but that does not imply causality.

It is an equally viable explanation that people who use this site only seek out and buy beers they think they will enjoy, so we're working with a biased sample of beers bought in the first place. For example, if I don't even want to try Bud light why should I buy it in the first place let alone review it? But if all the people I know on this site rave about PtE I'd certainly buy it and review it if I could.

Similarly the sample of people doing the reviews is possibly even more biased, if only because those who drink only AALs would not be using this site. Similarly many who use this site either don't review at all or review only occasiionally. For example, if I buy Bud Lite but I'm not a site user I'll not be rating any beers at all beers. And if I don't review anything....

So if I don't buy it to review it in the first place and/or if I'm not even using this site, there will be no review from me on the Bud Light.

 

Well, I'd accept the causality/correlation argument except for the overwhelming statistical evidence. Bud Light is by magnitudes more consumed than PtE. Are you seriously saying that it's plausible that there are 1500 members of BA that have never had a Bud Light? I have often agreed with some of your argument, I am sure that if every beer drinker in the US joined BA and rated their favorite beer Bud Light would be #1 with no hope of anything else coming close

 

No, among other things I'm seriously saying that many don't choose to review it because they either don't even want to try it in the first place or because they'd have to buy it again so don't have one in hand to review. After all, we are specifically asked to not review only from memory. If you were arguing that people only buy beers they expect to enjoy that would change things.

Yes, if every But Light drinker joined BA the odds are good that Bud Light would score higher. But if that's the only beer you buy....

 

Maybe we are talking past each other here and agreeing on almost everything. My take is I was tying to demonstrate with numbers why the average ratings trend more positive than the mean nothing more. And we both agree on the basic facts. (plus I challenge you to find 1500 BA's, or for that matter 1500 beers drinkers in the US that have not drank a Bud Light) Review by memory a Bud Light? LOLOLOL In my sleep if I chose to.

 

No doubt biases will be part of the reviews. They are still fun, and good to use as a rough guide, no matter the style.

 

Only partly talking past each other. But that is what sometimes makes discussions worth having. Sometimes clarification helps.

There's no need to find 1500 beer drinkers that have not had Bud Light or 1500 BAs who have not had a Bud Light. But it would be quite easy to find 1500 BAs who have had Bud Light but have no desire to make a repeat purchase with the intention of reviewing it because they have moved beyond.

There is no question the average reviews tend to be more positive, but there are other factors that come into play.

One is that most brewers will not release a beer that s/he does not think is worth drinking (even if they are wrong).

Another is that most BAs are simply not going to buy and review a beer they don't expect to like and that doesn't mean the buy a beer they don't enjoy and chose not review it and express their disappointment.

Another is that even highly rated beers have negative reviews.

Finally a statistical point. The average review score is based on calculation from the ratings but there is no ideal average independent of that score, e.g., there is no reason the mean should be, say 3.0, since the necessary constraints and corrections are not in place that are required to produce a normal distribution centered around 3.0.

So my concern is with your wording that says BAs only review beers they like and the implication that they buy beers and don't review the one's they don't enjoy. I'm suggesting they tend to buy the beers they expect to enjoy and, if they review at all, they review both beers they like and beers they don't like.

 

The obvious answer to the title, as previously stated is yes.

1+1=2

This is an objective fact.

An apple taste better than a pear.

This is subjective and based on personal bias.

What makes a good beer is subjective, but that doesn't make ratings meaningless. The ratings on BA, and I dare say more than other sites, are from people with a better than average understanding of what is a poor, good or great beer.

Your doctor is biased. But that doesn't mean you ask your vets opinion. Best to check with another doctor. Ask 1,000 doctors and the data becomes meaningful. 10,000 even better.

When it comes to beer, the best ratings are blind. Why? Because while it it still bias based on your personal preferences, you have removed the bias of, "I like/don't like this brewery", "I don't like Belgium styles, not brewed in Belgium", "The brewer is a nice guy", etc.

Most of the people that rate and review here are fully aware that bias exists in their ratings. Many do their best to reduced bias their ratings. But knowing they exist, view all ratings with healthy skepticism.

In the end, the only ratings that really matter, are your own.

 

OK, that makes sense, how about changing the wording to MOST BAs only Rate/ review beers they like? It doesn't conflict your post at all .
As far as you statement about brewers only releasing enjoyable beers I give you this:
https://www.beeradvocate.com/beer/profile/27642/88082/?ba=mudbug#review

This has been a most enjoyable debate. Cheers!

 

Better.

But notice, I didn't say brewers only release enjoyable beers. I said they only release beers they think are worth drinking. That doesn't mean you or I will find them enjoyable, only that the brewer did not think it was a bad beer. Brewers have been known to be wrong about that.

Yes, helpful discussion. Enjoy!

 

I read the first page only; this has all been discussed many times before. Reviews are biased by the name of the brewer, by rarity, by how much work is required to obtain the beer, by trade value, by the expectations of the reviewers, by the kind of beer the reviewer likes and doesn't like, by "annointed" styles v "reviled" styles, by "exciting" styles v "boring"styles, etc. , etc.

It is what it is.

So, how about a slightly different take on the subject, specifically, what is a style? @bbtkd says...

A nice goal no doubt.

But by who's measure are there 104 styles? Of course, I know he is referring to the style list on this site. But, that other beer site says there are only 78 styles. The BJCP says there are 119 styles. And, the Brewers Association says there are 151 styles. Next year there will probably be more in at least one of those lists.

As laudible as your goal is, @bbtkd, it may be a bit like trying to grab the pot of gold at the end of the rainbow.

 

Yes BA, since as I review beers it counts the styles. I understand that opinions differ and BA's list is not definitive, but this is the site I use to do my reviews, etc, so I'm going with the styles they have, and not worrying about whether others believe in more or less.

 

Yes!

 

I will say, it's nice being a newbie to beers.
For one, BA gives a general score of how well-received a beer is (both by the general forum and by the brothers). As someone stated above, looking at the number of reviews, std dev, and pdev of each beer/score/personal rating gives a good sense of how refined the opinions are/mean is, how far from the norm an extremely good/bad review is, and how well-known a beer is or how popular a style is. These metrics have all helped me in discerning where to start for a style or, in conjunction with the written review, what to expect from a specific beer.
That being said, I've also learned to toss everything out the window when it comes to personal preference. I've met diehard IPA-ers. I've met rabid Porter/Stout-ers. Me? I tend to prefer something in the middle - a good amber ale, pilsner, lager, or brown ale, preferably on the maltier side - and enjoy watching the campists froth at the mouth in incredulity.

The main point being - Like what you like and review how you review. If you don't like the beer, say so - the mean will be found eventually. If you have a bias, announce it up front. Last year I reviewed a fresh hop IPA. I openly admitted that I'm hot a hop bomb fan, but I gave it the review that I felt best fit. After all, you can taste good quality, even if it's not your preferred style.

 

I think that's largely because of people trying to be nice. In general, people don't tend to give negative reviews to the same absolute value as they do positive reviews (e.g. very good = median +1.75, but very bad = median -1)

 

But then we're not having the same conversation, because that's what I'm addressing.

Perhaps I am drastically misinterpreting the idea of "rating to style." I've always taken that to mean that a pilsner is judged according to the idea of what a pilsner should be. I don't know why such an approach would need to be blind.

My friend, maybe you are misinterpreting me here. I am saying that if a beer is judged solely against what a style should be... and if those style parameters are wide enough to accommodate the beers within the style... then logically (and hypothetically) speaking, most beers should be 5s since they adhere to those parameters and that's the metric of judgement. This isn't about realism. How can an IPA be considered a bad IPA if it is an IPA? Personal taste is one way to arrive at this, but if we are making a distinction between personal taste and rating to style... and we are honing in on rating to style, then there would need to be a standard that all IPAs were being judged against.

Because they are partially or entirely rating according to personal taste or belief.

Generally speaking, I believe that rating to personal preference plays a bigger role in the entire exercise than is given credit.

Personally, I don't like the diacetyl in Pilsner Urquell, and I don't rate it highly in my imaginary scale because I simply don't like that quality. If I was rating strictly according to a style standard, then it would have to be a 5 since style guidlines are written with this beer in mind as a definitive example (no matter how silly it is to have a definitive example). Would I have to give Pilsners without diacetyl a lower rating if this is the gold standard? If I was rating according to brewer intention then I'd have to consider the diacetyl in PU as favorable, and the lack of diacetyl in other Pilsners as favorable in those instances. More on intention further below.

The quote below was in response to another member, but I feel like it relates to our discussion.

There will never be a "resolution" of whether or not a style is a "real" style. There are brewers labeling their beers NE IPA and there are drinkers who are looking for NE IPAs. There are attributes that people and brewers use to justify the tag. The loose contract is there. It's a style. Maybe not one for the ages, but at least for the moment. Debates on the matter are impotent. You mention that an IPA has a "long standing criteria" for "lots of lacing." Is this really the case? Who decided this? If Harpoon IPA didn't leave lacing, should its score reflect this because it wasn't "to style."

Skipping to my other partner in crime, TongoRad:

I agree that it isn't that simple, but because the metrics aren't that simple. If the brewer was going for a "decent" beer, and they achieved this splendidly, then what? One could argue that any batch released to the consumer (rather than dumped) reflects intention since the brewery deemed it so. Who am I to say that a brewer only partially achieved what they were going for? Going more concrete and less philosophical, if a brewer released a Citra dry-hopped Pilsner and called it a Pilsner, how would one rate that if they were trying to ignore personal preference? Would they rate it poorly because it isn't a good example of what a Pilsner "should be"... or would they rate it highly if it perfectly embodied what the brewer aimed for?

 

Well, I think there is more to it than that. Many years ago people were drinking and loving beers like Hopslam, Zombie Dust and Jai Alai. These are all great beers. But today there are appreciably better local options that one might rate as high or better than these beers. Is that to say that Zombie Dust didn't deserve a 5 rating when first introduced? Does it still deserve one now? I am not going to try an answer that question. But we are not going to go back in time and re-rate those old ratings. In my opinion it is like trying to rate top athletes in any sport over different eras. Is Kareem Abdul Jabar better than Michael Jordan or Lebron.James?

If ratings were without bias an average beer would be a 3 not a 3.75.

 

Lots of interesting thoughts.

I'm not saying a perfect 5 is not theoretically possible, I'm saying that empirical research has found that some raters regardless of what is being rated almost never or eve never use a 5, no matter how much they like what ever it is that is being rated. Therefore there won't be perfect 5s in lots of individual cases. Further, when averaging across several individuals there won't hardly ever be an average rating of 5. It has nothing to do with biases about beer, but rather biases about rating and using the the extreme numbers.

More tomorrow when I've a physical keyboard.