New Rating System +/-

Talk Discussion in 'BeerAdvocate Talk' started by travolta, Aug 14, 2014.

Thread Status:
Not open for further replies.
  1. markdrinksbeer

    markdrinksbeer Initiate (0) Nov 14, 2013 Massachusetts

    If what they wanted was to make their ratings more valuable, accurate, truthful and meaningful, so that perhaps some local beer shops around my area for example, would use their ratings on the shelves instead of some other site that rates beer, then I would have to say no, that is not what they wanted.
     
  2. doktorhops

    doktorhops Pooh-Bah (2,065) Jan 12, 2011 Australia
    Pooh-Bah

    Yes, good points! You know how many beers I've rated an overall 5? None. Why? Because when I first started reviewing on this site the 101 posts the Bros made stated "A score of 5 is for a perfect beer" - just wait and see what will happen to the Top 250 with all these 5/5 reviews coming through... food for thought.
     
  3. doktorhops

    doktorhops Pooh-Bah (2,065) Jan 12, 2011 Australia
    Pooh-Bah

    Wow, never seen an rDev this big - that person must love Bud Light:

    Bud Light
    5/5
    rDev +170.3%

    In fact, and further to my last post there's been a few 5/5 beers rated over the past couple days - some perfect beers there...

    -08-21-2014Bud Light
    Anheuser-Busch
    5+170.3%

    R08-21-2014Miller Lite
    Miller Brewing Co.
    4.75+120.9%

    -08-20-2014Corona Extra
    Grupo Modelo S.A. de C.V.
    5+116.5%

    -08-20-2014Sol
    Cervecería Cuauhtémoc Moctezuma, S.A. de C.V.
    5+104.9%

    -08-21-2014Budweiser
    Anheuser-Busch
    4.76+104.3%

    R08-20-2014Tecate
    Cervecería Cuauhtémoc Moctezuma, S.A. de C.V.
    5+99.2%

    R08-21-2014Dixie Jazz
    Dixie Brewing Co.
    5+90.1%

    -08-21-2014Miller High Life
    Miller Brewing Co.
    5+87.3%

    -08-20-2014Heineken Lager Beer
    Heineken Nederland B.V.
    5+84.5%
     
  4. TheDoctor

    TheDoctor Grand Pooh-Bah (3,484) Mar 7, 2013 Canada (QC)
    Pooh-Bah Trader

    I agree and don't see why the new system would stop this. My assumption is that people did that because they saw their standard deviation. Something that is still shown under the new system.
     
    kemoarps, BeerBob and markdrinksbeer like this.
  5. rudzud

    rudzud Initiate (0) Apr 28, 2010 Massachusetts
    In Memoriam

    Someone reviewed an oktoberfest from Mexico, Indio, and wrote as his review: "Mexico"

    Bro's, come on...are you joking?

    This is what constitutes a review? Would that be considered an "inappropriate" review?

    Pandoras box has been opened gentlemen.

    I feel like this whole thing is some absurdly late April Fool's joke.
     
  6. markdrinksbeer

    markdrinksbeer Initiate (0) Nov 14, 2013 Massachusetts

    It feels great to be part of this site that actually PROMOTES this type of "notes" for a beer instead of categorically rating and reviewing the beer:

    Crushed Green Wit

    Moody Tongue Brewing Company
    Witbier / 5.00% ABV

    4.75/5
    rDev: +0.4% | rAvg: 4.73

    Ahhhhhhhh refreshing.



    08-21-2014 14:56:29
    [​IMG]

    Cold Press Paw Paw Belgian

    Moody Tongue Brewing Company
    Belgian Pale Ale / 5.50% ABV

    4.5/5
    rDev: +3.7% | rAvg: 4.34

    Great name. Great taste.
     
    kemoarps, BucannonXC5 and LaneMeyer like this.
  7. jmdrpi

    jmdrpi Grand High Pooh-Bah (8,989) Dec 11, 2008 Pennsylvania
    BA4LYFE Society Pooh-Bah

    Like you, out of my 1500+ reviews/ratings, I have zero perfect "5"s.

    The issue of the recent "grade inflation" I think is unrelated to the new review system, it's just a product of a ton of newer users not understanding the point scale and that 1's and 5's were supposed to be rare, not common.

    I know that when the old "Hads" turned into "Ratings" that counted towards the overall scores, the Bros stated that the average of the Ratings and Reviews was similar. But in the few years since, I don't think this is the case anymore.

    Was it just in this update that the only wording shown with the rating numbers is "3.0- okay"? I can't remember.
     
  8. Brenden

    Brenden Grand Pooh-Bah (3,436) Feb 25, 2008 Ohio
    Pooh-Bah Trader

    Maybe there's something I'm missing, but that seems to be the case.
     
    TheDoctor likes this.
  9. markdrinksbeer

    markdrinksbeer Initiate (0) Nov 14, 2013 Massachusetts

    Where are you seeing this rating of 3.0 - okay?
     
  10. jmdrpi

    jmdrpi Grand High Pooh-Bah (8,989) Dec 11, 2008 Pennsylvania
    BA4LYFE Society Pooh-Bah

    The drop down window for rating any beer.

    I know at one point it gave reference words for the 1, 2, 4, and 5 scores also.
     
    kemoarps likes this.
  11. markdrinksbeer

    markdrinksbeer Initiate (0) Nov 14, 2013 Massachusetts

    As far as I remember (and that only goes back about 10 months), I recall only seeing the Okay in line with the 3.0 and not any other scores.
     
  12. jmdrpi

    jmdrpi Grand High Pooh-Bah (8,989) Dec 11, 2008 Pennsylvania
    BA4LYFE Society Pooh-Bah

    There were a couple of iterations of changing the wording, to try to give people a frame of reference for the number scores.
     
    kemoarps likes this.
  13. jmdrpi

    jmdrpi Grand High Pooh-Bah (8,989) Dec 11, 2008 Pennsylvania
    BA4LYFE Society Pooh-Bah

    I think it's a overall problem of "Ratings" scores being higher on average than "Review" scores.

    Just for shits and giggles, I pulled up the recent scores for a popular, highly regarded beer, Stone Enjoy By:

    The last 20 Reviews: 4.7, 3.7, 4.75, 4.45, 4.45, 4.49, 4.48, 4.01, 4.18, 4.01, 5, 4.08, 4.18, 4.59, 4.2, 4.58, 4.68, 4.88, 4.89, 3.91 Average - 4.41
    The last 20 Ratings: 4.28, 4.25, 4.25, 4.3, 5, 3.96, 4.44, 4.25, 4.68, 4.72, 5, 4, 4.56, 4.75, 5, 4.75, 4.5, 5, 4, 4.64 Average - 4.52

    A bit of a difference, and probably mostly due to the 4 perfect "5" Ratings, vs. only 1 perfect "5" Review.

    Looking at Heady Topper, you can see the possible effect of the new "short" Reviews

    The last 20 Reviews (not including new "short" Reviews): 4.65, 5, 4.5, 4.24, 5, 4.68 4.98, 4.35, 4.85, 5, 3.19, 4.48, 4.64, 4.63, 5, 5, 5, 4.8, 4.33, 4.01 Average: 4.62
    The last 20 Ratings: 5, 4.72, 5, 5, 4.5, 4.5, 4.75, 4.25, 4.75, 4.95, 4.8, 5, 4.75, 5, 5, 4.7, 4.2, 4.5, 5, 5 Average: 4.77
    There have only been 10 "short" Reviews so far: 4.5, 4.5, 5, 5, 5, 4.75, 5, 5, 4.5, 4.75 Average: 4.80

    Caveat: I understand these are small samples sizes, but I'm sure the Bros can generate averages across the whole database.
     
  14. Roguer

    Roguer Grand High Pooh-Bah (7,811) Mar 25, 2013 Connecticut
    Mod Team Society Pooh-Bah Trader

    Remember, this changed fairly recently, as well. Good vs Outstanding, for example. Now it only shows "Okay" as 3.0. This changes from time to time to better reflect the way people are rating beers, if I understand correctly.

    And this is why the new review system (well, a reason, anyway) is a problem. I think adding a comment box to the "Had" tick - an Untappd kind of check-in - would be fine. That would allow little notes like those, and I don't think anyone would care. "Ahhhhhhh refreshing" may not be particularly informative, but if it's just a quick tick, it wasn't particularly informative, anyway, so no love lost.

    But now, "Show reviews" shows these reviews, which are absolutely worthless. The obvious reply is that you can downvote those reviews (or even flag them, if you think they are inappropriate, which is of course subjective and debatable).

    I think the perfect answer (as I and others have already suggested) is to incorporate both systems. Allow people to post short, 5+ character comments with "Had" ticks. Allow a more robust review system that gives us our categories back. Only the latter would be counted as a full "Review," but people are free to completely avoid it, while still utilizing precise scoring and comments, if they so choose.

    Best of both worlds; no drawbacks of which I can currently conceive.
     
  15. jmdrpi

    jmdrpi Grand High Pooh-Bah (8,989) Dec 11, 2008 Pennsylvania
    BA4LYFE Society Pooh-Bah

    Found it.

    Words were added to the Rating scale on 5/26/13. After debate, 6/12/13 it was updated to:
    Rating Scale Words
    • 1.00 - awful
    • 2.00 - poor
    • 3.00 - okay
    • 4.00 - outstanding
    • 5.00 - perfect
     
  16. John_M

    John_M Grand High Pooh-Bah (6,849) Oct 25, 2003 Washington
    Mod Team Society Pooh-Bah Trader


    Since the site moved to a rating system that doesn't require a narrative as part of the review, I've found that you increasingly need to "drill down" in order to make sense of some of the more unusaul review scores. Increasingly, a lot of folks will "tick" a beer or place by giving it a preconceived score that has little relevance to the quality of the beer or the place (I've seen a number of folks who give every place they visit, and every beer they sample, a score of 5). In my opinion, this practice is unfortunate, as it tends to skew the review scores on this site.

    In the case of the review you're referencing, the first thing I noticed is that the reviewer just joined BA today, and that this is his first review. Based on such limited data, I think it's impossible to know what to make of that review; maybe he's just ticking the beer, maybe he's unfamiliar with the reviewing process (and figured who gives a rat's ass if he screws up a review of bud light), maybe bud light is the only beer he's every had, or maybe he just really, really, really likes bud lite.

    My point being.... I'm not sure how much I would take from a perfect 5 score, without knowing more about the person who gave the beer that score.
     
    doktorhops likes this.
  17. SitkaSteve

    SitkaSteve Initiate (0) May 19, 2014 Idaho

    Totally agree and was trying to say what I said in a tactful way.
     
  18. chinabeergeek

    chinabeergeek Pooh-Bah (1,837) Aug 10, 2007 Massachusetts
    Pooh-Bah

    all i know is that the longer a final decision is delayed, the more likely things will end up staying the way they are.
     
    kemoarps, Jugs_McGhee and SitkaSteve like this.
  19. SitkaSteve

    SitkaSteve Initiate (0) May 19, 2014 Idaho

    All due respect to you. Any advice on how we can go about getting this corrected? I really love this website and all that it has to offer and what it set out in the ' 90s to accomplish. I mean how are we really "respecting beer" writing one liners?
     
    kemoarps and BeerBob like this.
  20. Dope

    Dope Pooh-Bah (2,925) Oct 5, 2010 Massachusetts
    Pooh-Bah

    Now I understand why every beer on the planet has a 90+ rating on the other site.

    Dope
     
Thread Status:
Not open for further replies.