1. The wait is over! Download the BeerAdvocate app on iTunes or Google Play now.
  2. Get 12 issues / year of BeerAdvocate magazine for only $9.99!

Updates: Rating Scale & BOS

Discussion in 'Announcements' started by Todd, May 26, 2013.

  1. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    The following are visual updates to our 1-5 rating scale pulldowns for reviewing Beers and Places and BeerAdvocate Overall Score (BOS). No data has been altered.

    Rating Scale with Words
    We've also applied words to the core 1-5 whole numbers, which are neither negatively or positively weighted and retains 3.00 as the average; .25 increments are intact. We believe this will allow members to more effectively express their opinion when rating.
    • 1.00 - awful
    • 2.00 - poor
    • 3.00 - average
    • 4.00 - good
    • 5.00 - awesome
    Good/Very Good
    We've broken up the 80-90 BOS range.
    • Good was 80-89, now it's 80-84.
    • Very Good is now 85-89.
    Zero Padding
    Numbers are now zero padded for better readability; when viewing a pulldown. Example: 2 becomes 2.00, 2.5 becomes 2.50.

    We hope you find these subtle changes useful.

    Cheers!
     
  2. Any small tweaks that you believe will improve the site experience are welcome, but I would have used a more superlative word to describe a 4.00.

    Too many people can't comprehend that "average" can also be "good," and as a result, those people rate solid but unremarkable beers as above average out of some sense that rating a beer as "average" is the same as calling the beer mediocre and personally insulting the brewer's family. In my mind, a score of 4.00 should be "very good" or at least "above average."

    People are already reluctant to use the lower half of the rating scale for anything less than a drainpour or an adjunct-laden lager, and characterizing the upper scores as merely "good" could only further this score inflation.
     
    beertunes likes this.
  3. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    Overall, members have been using the system according to the above 1-5 scale since it was introduced well-over a decade ago. Changing 4.00, which has always been known as "good," to "very good" or "above average" would positively or negatively unbalance the rating scale.
     
    ehammond1 likes this.
  4. fredmugs

    fredmugs Champion (830) Indiana Aug 11, 2012

    Pretty sure people can figure out if 3 = Average then a number higher than 3 is above average.
     
    KS1297, duceswild, Purp1eOne and 3 others like this.
  5. zachary80

    zachary80 Savant (325) Indiana Nov 29, 2009

    I see where devilben02 is coming from. I'd say a huge proportion of the reviews leave me scratching my head, most people seem to think everything is above average. For example, read the following paragraph and figure out how this is considered a 3 (for the taste category) by the reviewer:
    "The flavor of this brew is an absolute trainwreck. The first couple of sips arnt that bad and by a couple of sips I mean an ounce or 2. The vanilla inclusion is present but it blends into solvent alcohol that immediately tastes like burnett's bottom shelf vodka. (You know, that vodka that costs like $5 a gallon.) The interplay between artificial tasting vanilla, neutral alcohol and faint hop inclusion makes for a very clashing brew that just isnt that good."
     
  6. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    This reinforces that all of this (consumer opinions/ratings) is highly subjective anyway; and expected.

    We're not all going to agree, nor should we expect to.
     
  7. szmnnl99

    szmnnl99 Advocate (675) Michigan Apr 3, 2006

    I think this is because the term "average" can be relative to the beers you DO taste or "average" of all beers you COULD taste. I would say the former is 3.75 and the latter is around 3.00. Unless it is real cheap or unusual, I won't buy a 3.0 beer.
     
    martyl1000, woosterbill and Craigory like this.
  8. nsheehan

    nsheehan Savant (485) Texas Jul 3, 2011

    I agree but also see Todd's point about keeping things consistent for a decade. I'm happy about the good/very good distinction to BOS. However, I don't like the word for 5.00.

    Awesome sounds as if the audience is high school girls. I admit, I use the word "Awesome" more than I should. Also, I'd like to note I am pro women in craft beer/brewing (if they're 21 in the US of course), so I'm not trying to alienate women by belittling a word used by a sub-population.

    I think a word like superb, perfect, or amazing would be better-suited to represent 5. Especially because BA's hardly ever rate a beer a 5, and I've seen BA's kind-of-bragging about only rating 1 or 2 beers ever a 5.00 in both taste in smell.

    P.S. I have no full reviews on here because I am organizing my beer review papers and typing them this year.
     
  9. kevanb

    kevanb Advocate (680) Illinois Apr 4, 2011

    Not sure what site you are visiting, but there are an extreme abundance of 5.00 reviews on this site, ESPECIALLY from the hads.
     
    Beerandraiderfan likes this.
  10. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    It'll be next to impossible for everyone to agree on a set of words for a rating scale.

    The current set is simply a play off of this very common and balanced set:
    • 1.00 - very poor
    • 2.00 - poor
    • 3.00 - average
    • 4.00 - good
    • 5.00 - very good
    I'd have no problems using it as is, however if we were to address concerns about using the word "average," something like this might work:
    • 1.00 - very poor
    • 2.00 - poor
    • 3.00 - okay
    • 4.00 - good
    • 5.00 - very good
    Personally, I don't mind using "average," but I realize that many view it as a negative these days. "Okay" is very common and straight forward; acceptable; neither good or bad.
     
    jgluck, Retsinis and ehammond1 like this.
  11. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    We could also give 1.00 and 5.00 a bit more distinction, which was our original goal instead of using "very."
    • 1.00 - terrible
    • 2.00 - poor
    • 3.00 - okay
    • 4.00 - good
    • 5.00 - excellent
     
    mattafett, Rohkey, frothyhead and 5 others like this.
  12. Purp1eOne

    Purp1eOne Advocate (580) Illinois Oct 23, 2010

    Why don't you use what you have already then 3.00 will make people feel good about drinking it I guess.
     
  13. nickfl

    nickfl Advocate (745) Florida Mar 7, 2006

    Any chance of allowing users to add a beer with a had instead of a full review? I know this was talked about a while ago as an eventual update, is it any closer to being implemented?
     
  14. dbrauneis

    dbrauneis Site Editor (950) North Carolina Dec 8, 2007 Staff Member

    This was mentioned in another thread as something that was being considered in the site redesign - they would be changing the way in which beers were submitted/added and it seems like they were thinking of disconnecting that from the actual review/quick review.
     
  15. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    I removed your random quoting of text from various non-forum pages. Please explain this a bit more. Thanks.
     
  16. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

  17. nsheehan

    nsheehan Savant (485) Texas Jul 3, 2011

    I stand corrected. I just looked through a few of the pages of 60K recent ratings. I should start off by saying that I pretty much filter out Hads, so that does something to make my statement MUCH less wrong. Pretty much all the 5's I've seen recently (sporadically over the last few months) on the Beer home page have been Hads, which I don't really count.

    Looking at the recent ratings, most of the beers with 5/5 (review and hads) have a % deviation of positive double digits, which would tend to signify a 5/5 rating is quite a bit above the average rating for that beer. Potentially even just the had-ers way of saying "This beer is excellent, I'm just going to check 5 and enjoy this amazing brew, instead of thinking for 1 minute to see where between 4 and 5 (inclusive) this beer falls." Nothing inherently wrong with that, just something I wouldn't do.

    I should've stated "I never see a beer getting lots of 5's." But I do see the 'I-don't-give-5's' line about once a week on the forums. That said, I don't look at the beer homepage or look up tons of world-class beers for fun, so I'm not exposing myself to the part of BA that would enable me to see 5.0 ratings regularly. Also interesting how many of the 5/5 beers I saw in the recent ratings pages I looked at were beers that most BA's will probably never get to try more than once, if ever.
     
    Cubatobaco likes this.
  18. socon67

    socon67 Advocate (660) New York Jun 18, 2010

    With some debate over what "average" means in regards to a BA review I am mildly curious what is the average rating score for a review. I've looked at the ratings by style in the past and based the rating a beer has against its contemporaries to determine what people think of it. For example; Hennepin is the saison with most reveiws and has a rating of 4.18. Noticing others like Tank 7 & Sorachi Ace are ranked similarly I'd expect that any saison rated 4.10 and higher would be very good. But a similar score for a DIPA would not necesarily indicate the beer to be as noteworthy since there are numerous imperial IPA's rated that high.

    No matter what verbiage is assigned to the rating values there will still be subjective influences that affect the rating the reviewer gives.
     
    Cubatobaco and nsheehan like this.
  19. Purp1eOne

    Purp1eOne Advocate (580) Illinois Oct 23, 2010

    I just took your bullet points with the review of one of the beers on BA (Founders Breakfast Stout) which indicates its a "world class" beer and this moved good to average which could be more acceptable to some.
     
    nsheehan likes this.
  20. Purp1eOne

    Purp1eOne Advocate (580) Illinois Oct 23, 2010

    - Couldn't edit previous reply, walked away from my desk and couldn't update with the 15 minute timeout.

    So your scale would be:
    • 1.00 - terrible
    • 2.00 - poor
    • 3.00 - good
    • 4.00 - very good
    • 5.00 - world class
     
  21. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    That's an unbalanced scale that favors/encourages positive ratings. It needs to be balanced; 2 negative, 1 neutral, 2 positive. Your's has 2 negatives and 3 positives.
     
    CelticAleMan likes this.
  22. Purp1eOne

    Purp1eOne Advocate (580) Illinois Oct 23, 2010

    Some synonyms for neutral are: "middle of the road", "on the fence" or "poker-faced".

    Just throwing out ideas. I am good with either "average" or "okay".
     
  23. nsheehan

    nsheehan Savant (485) Texas Jul 3, 2011

    I know it's just words and the numbers matter more, but I think part of what Purp1eOne gets at is good. Making the words with the numbers match other aspects of the site. Calling a 5.0 World Class matches better with how full reviews work.

    Looking at some Boston Beer Co and Stone reviews, it looks like (roughly) 3.5 is the cutoff between average and good, and 4.0 is the cutoff between very good and exceptional. Scores with 4 and above correspond to the 90's (exceptional), scores in between 3 and 3.5 (Average), scores in between 3.5 and 4 (good and very good).
    So a scale closer to full reviews would be:
    • 1.00 - terrible
    • 2.00 - poor
    • 3.00 - average
    • 4.00 - Exceptional
    • 5.00 - world class
    The above scale I think is pretty fair and corresponds well to other site aspects.
    Also, the BA scale is only 4 full point values (1 to 5 inclusive) so 3 is dead middle and is rightly called average. But 5 is an endpoint, so for the 4 full point value range I don't think it'd be crazy to have two negative and two positive words (terrible, poor, good, exceptional), then have the end point 5 be world class.
     
  24. zachary80

    zachary80 Savant (325) Indiana Nov 29, 2009

    My comment was more about people misusing average than changing the scale again. I like your first scale of this thread as well as your fourth one (terrible -> excellent) and would probably lean towards the latter. I would be concerned that moving to the third scale, okay instead of average, would make future reviews skew higher.
     
  25. Todd

    Todd Founder (1,440) Colorado Aug 23, 1996 Staff Member

    So a hybrid:
    • 1.00 - terrible
    • 2.00 - poor
    • 3.00 - average
    • 4.00 - good
    • 5.00 - excellent
    However, I still think changing "average" to "okay" could be helpful. They both mean the same thing, but the latter seems to be more accepted.

    More thought is required. Keep the feedback coming ...
     
    szmnnl99 likes this.
  26. dbrauneis

    dbrauneis Site Editor (950) North Carolina Dec 8, 2007 Staff Member

    Works for me!
     
  27. beertunes

    beertunes Poobah (1,035) Washington Sep 24, 2007

    How about "typical" for "average"?
     
    flexabull and Kerrie like this.

  28. This one seems the most transparent and straightforward. "Okay" might be a little better because as others have pointed out, "average" is a little ambiguous.
     
  29. drtth

    drtth Champion (860) Pennsylvania Nov 25, 2007


    This list is most like what one sees in the professional literature written by people who develop rating scales. Since they have researched which word choices work best and cause the least confusion among the most people, seems like their work should be adopted.

    Edit: Also I think some of your statistical procedures assume that the rating scale has some of those common and balanced characteristics you refer to. Leading me to wonder how robust they are to violations of those assumptions.
     
  30. Cubatobaco

    Cubatobaco Advocate (620) Virginia Jan 27, 2013


    I have no problem with the above rating system. Thanks for keeping us in the loop and asking for the feedback of the community.
     
  31. nsheehan

    nsheehan Savant (485) Texas Jul 3, 2011

    Regardless of the word for 3.00, I think good and excellent for 4.00 and 5.00 is not adequate. A beer with an average score of 4.00 is either Very Good or Excellent when looking at reviews. This beer is a near ideal example as it has enough reviews to be significant (pDev is a little high IMO and Dubbel isn't a very large style) http://beeradvocate.com/beer/profile/222/2508

    Calling a 5.00 "excellent" is a big understatement to me. A score of 5.00 in any category is damn hard to get. It's not to a lot of BAs, but maybe I just think a little more when I review/tick. I still advocate calling 5.00 World Class.
     
  32. This sounds pretty good to me, but I agree that "excellent" might not be quite superlative enough to convey the full weight of a 5. Since there are twice as many possible scores now with the quarter-point increments (which I love, btw), why not include twice as many descriptions?

    For example:

    1.00 - undrinkable
    1.50 - terrible
    2.00 - poor
    2.50 - disappointing
    3.00 - average
    3.50 - okay
    4.00 - good
    4.50 - excellent
    5.00 - superb

    I can see how this could make the dropdown overly cluttered, but it could help people get a better feel for how the numbers should apply. Just my .02.

    Cheers!
     
    Burkbom23, buffs9, nsheehan and 4 others like this.
  33. nanobrew

    nanobrew Initiate (0) California Dec 31, 2008

    Just to throw out some more terms:

    • Terrible
    • Sub-par
    • Standard
    • Outstanding
    • World Class
     
    2beerdogs and sliverX like this.
  34. duceswild

    duceswild Advocate (740) Maryland Feb 8, 2010

    If something is terrible would you still drink it? I wouldn't. Though I like the 1/2 point scale words.

    On a separate note a true 5.00 should be labeled super-de-duper.
     
    Beerontwowheels likes this.
  35. Would I drink a terrible beer? Maybe not, but that's beside the point. The difference between undrinkable and terrible, in my mind at least, is the difference between "could you?" and "would you?". Do I think, say, Natty Light is a terrible beer? Yes. Is it undrinkable? No. Perfect candidate for a 1.5.

    Now, Rogue Voodoo Doughnut? That I literally couldn't finish a 2 ounce pour of: 1.
     
    Michigan and 2beerdogs like this.
  36. 2beerdogs

    2beerdogs Champion (765) California Jan 31, 2005

    Ahh, Wooster Wisdom. I agree completely.
     
    woosterbill likes this.
  37. 2beerdogs

    2beerdogs Champion (765) California Jan 31, 2005

    I kinda like the 3 as average, it has a more objective feel vs. okay, which sounds like a junior high conversation.
     
    woosterbill likes this.
  38. Michigan

    Michigan Initiate (0) Michigan Oct 24, 2012

    What is the average across the whole site?
     
    woosterbill and 2beerdogs like this.
  39. Kerrie

    Kerrie Savant (485) Michigan Aug 24, 2012

    For me, 1s are drainpours. I've rated some macros slightly higher because I can at least get through a can when they're given to me for free at a family gathering and such. I'd never buy them, however. As for the higher end of the spectrum, 5s are those that I'd drink every day, if given the chance (usually not possible), no matter the circumstance. My latest is the BA Old Rasputin XV. A friend brought it to one of my parties and I'm forever grateful. I may not ever have it again, but wow... that's a 5 for me.
     
  40. 2beerdogs

    2beerdogs Champion (765) California Jan 31, 2005

    Michigan had a good question. What is the average across the reviews?
     
    Michigan likes this.

Share This Page