New Rating System +/-

No, they were arbitrarily selecting 4.25 or 4.00 instead. The uncertainty between scores was less in the old system, but it was still there. As such, you picked the "wrong" score less often, but it had greater impact when you did.

Suppose you have a bathroom scale that rounds to the nearest pound and is quite certain about how many pounds you weigh. You get a new one that rounds to the nearest tenth. The old scale says you weigh 185. The new one says you weigh 185.3 until you shift your weight a little, then it says 185.2. You exhale and it says 185.4.
Is the new scale wrong? No.
Is it more precise? Yes.
Is either scale less accurate? No.
Does the new scale invalidate the record you've kept of your weight on the old scale? No.

Suppose you're throwing darts at a board with 10 concentric rings of equal area. The outer ring is 5 points, and each ring closer to the center is 5 more for a 50 point bullseye. Replace it with a target with 50 concentric rings, starting at 1 and going up to 50. In the long run, you'll score the same on both dart boards. The first board will have more high scores and more low scores, but you'll score the same in the long run.

Adding precision can be pointless, it can't be harmful.

No, I'm arguing for which system I think makes more sense. Whether there's more value in improvement (and I realize you dispute that it's an improvement) or in consistency for the sake of consistency is a different argument and one that I feel much less strongly about.

 

I think an even bigger issue is the temptation for breweries to use shills. The problem with shills at sites like Yelp is well-documented. In this upvote/downvote system, what's to prevent a brewery from getting shills to upvote all the favorable reviews of its beers and downvote all the unfavorable ones?

 

You can already do this. Click the "Display Reviews Only" box.

 

it's a real problem. the vast majority of the brewers i've met would never think of doing this, but i think every longtime reviewer has at least one story of "local brewer sent me highly unprofessional email [usually an attack] after i posted a bad review."

unfortunately i think moderation is the only thing. it's a pain in the ass, but it's the door you open if you introduce up/down votes on a site that has market influence.

---

edit: i mean, i still don't mind the feature and don't think it will be a huge deal (especially for the majority of beers). but it's absolutely a concern that site maintainers want to have in mind, IMO.

 

I'm worried that this might be getting a bit off-topic, but I want to try one more thing. I'm glad you brought up the scale analogy for two reasons:

1) It differs from scoring beers in that a scale is built to measure, rather than evaluate. A scale isn't providing a judgement of the weight of it's occupant, it's simply ascribing a numerical value to that weight. Comparing a scale's measurement precision to a person's perceptual precision is akin to comparing IBUs to bitterness. IBUs measure a specific chemical feature of beer, but don't necessarily measure actual bitterness, which varies from person to person based on their individual perceptual capabilities.

2) But perhaps more importantly, the fact that a scale is measuring rather than evaluating gives me another way to illustrate the harm of pointless precision. Say you throw out both the old and new scale and buy an even newer scale that displays six decimal points. The first scale says you weigh 185. The second scale says you weigh 185.3. And the third scale says you weigh 185.325181. The problem is that the likelihood that a plain old store-bought scale is going to actually be able to measure weight to the millionth of a pound is effectively zero. So what are those extra five decimals measuring? Functionally, they are measuring nothing, because the tool that is providing the numbers has no actual basis for the numbers. So ultimately, the scale is less precise, because it is providing random numbers in lieu of actual data.

If a user is choosing to rate a beer 4.21 instead of 4.20 without any real basis for the difference, then that hundredth of a point is useless, making the rating less precise. You might be right that in the long run, it will average out, but I'm skeptical, mainly because as noted, the new system removes the normalization of how scores are determined.

 

I think that its hardly a concern. If someone was going to go through the trouble of attempting to manipulate rankings, they'd do it with the ratings, not the "usefulness", since it actually affects the rAvg.

Once "Hads>Ratings" were implemented, that was the point that it would have become easier to have a brewery or store/bar/brewpu manipulate their rating. Prior, even if you created a bunch of "dummy" accounts, you'd still need to take the time to write multiple unique fake reviews. Much less effort to rate a "5" multiple times. That barn door has long been open.

 

Damn my aging eyes! I see it here on the PC, but usually look on my phone or iPad and it is freaking tiny.

 

But the extra digits are of reduced significance in accordance with their reduced certainty. If the second and third scale are equally accurate, and you weigh yourself 10 times on each, you'll get the exact same average on each. Too many digits can be an eyesore, and can mislead you into thinking you're more precise than you really are, but they cannot actually make your evaluation worse. If the digits on the third scale beyond the tenths place are indeed totally random, it is not less precise, it is equally precise (both precise to the tenth pound). The third scale would still be more precise than the first assuming its tenth place actually works; its flaw is just that it may lead you to mistakenly believe it is more precise than it really is.

That is the worst that can be said about the precision in the new system. That someone will naively believe a 4.73 beer is certainly and meaningfully better than a 4.70 beer, rather than just more likely to be better than it is to be worse. In that sense still, I don't think anything has changed.

 

These long-time BAs described those categories because they were there to follow.

I read those reviews, matched them with scores, and learned.
And I didn't just learn about how to write about beer, but how to taste it, too.

Without categories, some reviewers are going to stop writing about them (indeed, some already have.)
Then maybe BAs who are still learning don't see, say, aroma mentioned in recent reviews and miss out how important aroma can be.

No categories could make it harder for noobs to get educated on what makes good beer.

 

To be clear, my primary issue is with the false precision that is imparted by allowing users to choose their scores across a range of 401 ratings. And the fear that this false precision will ultimately lead to lower accuracy (I misspoke at the bottom of my last post and said less precise). And maybe in the long run, it will even out, as some people inaccurately rate some beers above and others rate them below the "correct" rating.

But the key for me is that this change, while it's not a huge detriment (outside of the removal of preexisting data), is that it's change that doesn't really add much benefit. A change not being catastrophic isn't a good enough reason that it should be undertaken. There should be a clear purpose for the change, and this one just doesn't seem to have much of one.

 

My concern for the "Reddit style" up and down voting system is that people will vote on reviews that favor their own opinion, and are not necessarily more useful or insightful.

Additionally there are already camps of people who prefer specific breweries and beers. This encourages promotion of favorable beer reviews within these camps. For example, if you feel that a highly rated beer is average by your own standards you would normally just rate it accordingly and move on. If a majority of people like the beer and feel that your review doesn't align with their own belief they will down vote you. I have a feeling that people will start creating "hive mind" reviews just so they can be validated by a few up votes.

 

honestly, if people "just like more control" (whether they're misguided or not), if there are few real drawbacks to the change, i think that's a good enough reason to make it.

when it comes to so-called accuracy, the big picture questions are of agency and boundaries. specifically,

1) how much agency is involved on the part of the user to craft a good review (as opposed to built in structure and quality provided by the system)?

2) where is the boundary past which the user is responsible for the quality of the review, and we must simply throw up our hands and say, "it's on them, for better or worse"?

to talk about (2) first, this always came up with the so-called objective or "to-style" guidelines for appearance, smell, taste, mouthfeel. the site went so far as to say "COMPARE IT AGAINST OTHER BEERS OF THE SAME STYLE." but what defines a style? and how much research do you need to do to be qualified to make this comparison in the first place?

the answer to both questions, per (2) is pretty clearly: it's a free, consumer review site. you want to rigorously define styles? you want to research a lot before you review? great, hopefully that will make your reviews better. or shit, maybe it will make them worse. either way, it's on you, and whatever we get, that's what we asked for. (edit: to be a bit clear, there's the boundary: "please grade to style, but after that imperative, any further questions and definitions have to be up to you.")

(1) is highlighted by minimum review lengths, number precision, etc. the agency i'm talking about would be limited, for example, if mods periodically deleted reviews where someone gave a 5 for aroma and in the review wrote "this smells bad". all the more if the structure did this for the mod in advance (forcing people to grade by a certain increment, for example).

the new system gives the reviewer a little bit more agency than before, particularly regarding how to define number scores. i think it will be used (and ought to be used this way, hence me being okay with this change too!) to be like, "well, a 4 feels high to me, but how high? ehhhhhh maybe 3.9. no, even more, 3.8." you know? there is no standard, and the increments are really vague, but you can put in that extra "this isn't .5 better than that last beer's mouthfeel, it's only like .1 better!" i'm with that.

i could write a lot more about this, but i think (1) and (2) are interesting to think about if you want to consider how a review site can be structured. i'll withdraw for now. (edit: bringing it back to "accuracy," if you can define just what we're trying to accurately reflect, you then hit (1) and (2) to try to figure out how to best get that reflection)

 

I honestly have no idea what now constitutes an inappropriate review. With the 5 character minimum length, does the review still have to mention any characteristic of the beer?

I quickly found examples of extreme negative and positive reviews, yet that give no information about the actual beer to justify the rating:

Michelob Ultra:
1/5 rDev -45.4%
this embarrassing to say I even drank one at one time

Stone Arrogant Bastard:
5/5 rDev +19.9%
THE BOMB!!!!!!!!

Is this quality really what Beer Advocate wants?

 

The quality continues...


Crushed Green Wit
5/5 from
rDev: +6.4% | rAvg: 4.7
I was skeptical. That made it all the better.


Cold Press Paw Paw Belgian
5/5 from
rDev: +16.8% | rAvg: 4.28
Can you come up with a more creative summer beer?

Tecate
5/5 from
rDev: +99.2% | rAvg: 2.51
My first beer when I was 15 in Mexico I will give it a 5 because in Mexico its a whole different beer than what they sell in the states

 

Why not have the best of both worlds, two websites, one for the noobs and one clicks, and one for the rest of us. BeerAdvocate and BeerAdvocate Classic, you know, just like Coke and Coke Classic, right after Coke found out that getting rid of what brought them to the dance, was a grave mistake.

 

I think this point is well made, and does bring up how the people running the site, and separately and equally, how the community wish to view ratings. As many have pointed out, no one can really be forced to follow the guidelines if a user's goal is to get an overall rating where they want. The fact that people are doing it and not following the guidelines I think is part of why the Bros made the change.

What I think would be great is reinstate and separate the 4 "objective" scores to attempt to get an "objective score" for a beer (one equally weighted for everyone, as you state, regardless of what that weighting is), and a separate Overall/Enjoyment score that users can do whatever they want with. Obviously, each beer could show both averages, but alternatively (as someone else mentioned) that Overall rating could be for one's own use, maybe even only visible to the rater.

While I understand your point of what level can someone differentiate at, some of us do want to differentiate. I review[ed] just over half my beers, which could get me ratings of, let's say 3.67. For those beers I did a straight rating for that I felt was equivalent to that 3.67, I could not indicate so (in this I am agreeing with @RichardMNixon). What I want to say is that I may have a desire to differentiate between the 30 3.75s that I chose because I couldn't do 3.65 or 3.85, and my reason is to say that, relatively, I thought beer X was some degree better than beer Y that is less than the forced values in the old system. Granted, I'd probably be fine with ratings to one decimal point, as well.

 

Like another has posted with random awful 'reviews' I found that alot of people who are now suddenly reviewing beers are giving out 5's left and right and saying the beer is "AWESOME!". As already mentioned...that is not a review. Not even close. By knocking down the character limit you (the Bros) HAD to have seen stuff like this coming, so it just perplexes me that you would go from seemingly wanting solid user reviews to that dribble.

When there was categories someone had to put the mental thought in to 5 different things to give a AWESOME 5 to, now, new reviewers have zero context on what to rate the beer off of. All the one liner reviews I've seen trend more towards personal preference, and less on the beer's style. So instead of having a website filled with reviews of style vs personal preference those reviews seem to have left style on the wayside.

 

Not to add fuel to the fire of the beer reviewing veterans (I'm in no place of having exceptional reviews myself, I tick a lot since there is a lot I don't know about certain styles and maybe I had the beer too long ago and could only place a tick to remind myself of hopefully some day coming back to it to further investigate it's characteristics).

Yes, there are times I may have sounded inadequate in some of my reviews but after talking with seasoned reviewers I think I'm getting better. With that in mind, I think I need to go back to some older reviews to perhaps clarify my wording better.

Last thoughts: I have only been here for about 3 months but if someone like me can write something on each beer characteristic, I can't see why it would be hard for anyone to come up just something that they observed in the beer.

Just my two cents, I know I'm still a noob here so I'll go back to my hole now.

 

As a noob, I totally agree with this. No offense to the Bros, but people like me who are new and want to know more and be better reviewers will be part of a system that has been dumbed down too much.

Just a rhetorical question to the Bros and the mods:

Is this what you want?