Attn New BA Score, Beer Ranking & More Updates

Discussion in 'BeerAdvocate Talk' started by Todd, Sep 19, 2017.

Thread Status:
Not open for further replies.
  1. Todd

    Todd Founder (5,518) Aug 23, 1996 California
    Staff

    New Updates: https://www.beeradvocate.com/community/threads/new-ba-score-histograms-more-updates-part-two.538899/

    ----------

    Greetings!

    The following updates will be rolling out over the next few hours, so we wanted to give everyone a heads-up just in case you come across an update in the works.

    ----------

    The TL;DR
    The BA Score is now the percentage of raters who gave the beer a 3.75 or higher. We're now telling a more complete story behind a beer's ratings, and beers are being scored on their own merits vs. being scored against all other beers.

    The basics have been applied to breweries and other places, too. Keep on reading to learn more.

    ----------

    Goodbye, BeerAdvocate Overall Score (BOS).
    The BOS was a proprietary weighted point system that represented the final overall score for a beer. Based on a weighted rating, its purpose was to help make some sense out of the beer's average across all ratings and provide consumers with a quick reference when comparing one beer to another.

    Why we're ditching it:
    • Its 30-100 point scale was a seemingly arbitrary, dated standard that was borrowed from the wine and spirits world. In my opinion, this point scale is designed for shelf talkers and tends to favor higher scores.
    • Many confused it with being a percentage.
    • It failed to clearly or quickly tell consumers what the rating community really thinks.
    • Beers were being scored against all qualifying beers, instead of being scored on their own merits.
    We also assigned text (world-class, outstanding, very good, good, okay, poor, and awful) to the BOS ranges to further help explain the overall consensus. We're saying goodbye to this, too. Instead, we're going to tell a more complete story about the beer's ratings and empower consumers to make a better, more informed, decision for themselves.

    Hello, BA Score. It's Just Math.
    The BA Score for beers is now loosely inspired by Rotten Tomatoes' Audience Score and is represented as the percentage (rounded to the tenth for depth) of raters who gave the beer a 3.75 or higher.

    (number of ratings >= 3.75 / the number of ratings) * 100

    After analyzing the data and reviews we determined that a 3.75 or higher was a clear indication that the beer was, without a doubt, liked. It also matches the bar that's been raised by consumer and industry expectations in recent years.

    We also anticipate that each user will eventually create their own percentage threshold of acceptance. For example: I might consider a beer with a BA Score of 48.7% as worth a try as nearly half the people who rated the beer really liked it, whereas Jason might not consider trying any beers without more of a consensus (say 60% or higher), and you might be totally fine exploring beers that are under 25%. But we're going to allow you to decide what's "world-class, outstanding, very good, good, okay, poor, and awful."

    A beer must still receive 10 or more ratings before a BA Score is applied.

    Liked
    The number of users for the given item who rated it a 3.75 or higher.
    • Liked This Beer: across all ratings/reviews for the beer
    • Liked Their Beers: across all ratings/reviews for all active beers
    • Liked This Place: across all ratings/reviews for the place
    The number of liked ratings appears under Beer or Place Stats accordingly for reference.

    Ranking
    The BOS is gone, but we didn't ditch the underlying weighted rating (WR) system. All qualifying beers (active with 10 or more ratings) are now given a ranking based on their WR. This directly correlates to our main top 250 beers, and allows readers to quickly determine if a beer lands within the top 250 (a highly requested feature) or where it lands among all qualifying beers.

    Additionally, we plan on taking snapshots of a beer's ranking to add even more depth to a beer's rating history here on BA.

    Rankings are updated on a daily basis.

    Beer Average
    Previously, the beer average was weighted based on each user's rating history. It's back to a straight average now, which considers the ratings of all users equally. This will no doubt shake-up the beer rankings.

    Beer Shelf Talkers
    Have been updated to support the above changes, but are rounded to a whole number for readability and to offset minor shifts to the tenth, which will give them more longevity.

    Top Beers Threshold for Consideration
    Previously this was the average number of ratings across the data set, which was used throughout the WR calculation. We still use this number in the calculation, but the threshold to be listed is now 10 or more ratings to support the new ranking and to help flush out lists that had small data sets.

    Bros Score
    Based on reviews by Jason and I, our Bros Score has been retired from focus on lists and beer pages. It's still displayed under a beer's stats as a historical nod to the site's origins, but we felt that it's important to put all the main focus on the rating community's opinion (BA Score). And when displayed, it's just a rating or average now.

    That's it for now. Browse around and feel free to let us know what you think.

    Cheers!
     
  2. jhavs

    jhavs Poo-Bah (1,649) Apr 16, 2015 New Hampshire
    Premium Member Beer Trader

    This is a lot of info to take in, I look forward to checking out how everything will look!
     
  3. AZBeerDude72

    AZBeerDude72 Meyvn (1,379) Jun 10, 2016 Arizona
    Premium Member Beer Trader

    I just literally started looking but I like what I see. Was checking out the rating and like the way it looks, lots to digest and take in but so far so good. Cheers :slight_smile:
     
    Tshane, Guttaskulk, jhavs and 2 others like this.
  4. fthegiants

    fthegiants Poo-Bah (1,535) Nov 20, 2013 California
    Premium Member Beer Trader

    Every time i get close it gets changed.......
     
    Lochnessmonster, M_chav and Narthax like this.
  5. dbrauneis

    dbrauneis Site Editor (7,101) Dec 8, 2007 North Carolina
    Premium Member Beer Trader

    Not sure what you are getting close to that has changed??? The lists are still the same, this is just more about how the information is displayed on the individual beer pages and brewery pages:
     
  6. fthegiants

    fthegiants Poo-Bah (1,535) Nov 20, 2013 California
    Premium Member Beer Trader

    Fortunately it only knocked me down 1 beer on the top 250. It knocked my buddy down 8 beers. I also go after the top 100 in american double IPA's, Stouts, and wild ales... Knocked me down about 5 beers in each.
     
  7. Harrison8

    Harrison8 Poo-Bah (2,485) Dec 6, 2015 Missouri
    Premium Member Beer Trader

    Hopped into a beer profile earlier and was wondering what some new data categories were, but now it all makes sense.

    Lots of info, but I dig it! My biggest gripe was with the old BOS. I never pulled much value from that number. I always scrolled past that to get the average rating out of 5. Although most folks in my daily life would pull up a profile and quote the BOS. Pretty confusing. Thanks for switching it up and clarifying everything.
     
    Beer_Economicus likes this.
  8. Mike

    Mike Senior Web Developer (755) Nov 17, 2010 New Hampshire
    Staff Beer Trader

    There was a slight adjustment due to the weighted average becoming a straight average.
     
  9. jcflorian

    jcflorian Devotee (476) Oct 27, 2014 Connecticut
    Beer Trader

    I gained two more so I like it.
     
    fthegiants likes this.
  10. cryptichead

    cryptichead Savant (928) Jul 3, 2014 Illinois
    Beer Trader

    One of the reasons I enjoy this site is that there is always a push for improvement and adaptation. Looking forward to BA 4.0 in its full glory!

    Some initial thoughts of the new system:

    1) Score averages: I like the straight average score that was just implemented. I always thought the old system giving more weight to different users and reviews often skewed the data, especially for low sample-size beers.

    2) New BA Score Percentage: my initial thought is that it is not as helpful as the previous score. I think this new system makes an assumption that users rate beers to style, which is mostly not the case. For example, some beers (i.e., AALs) are now showing BA scores of zero, even though they may have a solid average rating for AAL (i.e., mid-to-low 3s).Thus, there will be lots of beers in certain styles with pretty low BA scores, but solid average ratings for those styles. This may inadvertently influence purchasing decisions.

    Ultimately, rating an AAL at 3.5 sends a different signal for that beer because it is closer to that style's rating ceiling than rating an IPA at 3.5.

    Lastly, knowing that 98% of people rated the beer 3.75 or above provides me with less information than the previous score of 98. Now, looking at the average score will provide more complete information than the BA score.
     
    #10 cryptichead, Sep 19, 2017
    Last edited: Sep 19, 2017
  11. RobH

    RobH Disciple (370) Sep 23, 2006 Maryland

    For statistics geeks and purists, and thoroughbred beer geeks on this site, the new percentage rating system makes sense.

    That said, I believe that these ratings "scores," in a retail store shelf-talker context where they are viewed and referenced by average retail consumers, this new system will be confusing and perhaps not as useful. For one thing, in the retail shelf-talker context the viewer will not have the benefit of the in-depth explanation of how the numbers were calculated. And in the broader context of the wine and spirits 100-point ratings system that is commonly used and understood by consumers, there will be confusion.

    Being a beer enthusiast and BA subscriber since '06, I can appreciate the thought and approach behind this new rating system. But being in the retail industry I care about its viability with the average consumer.
     
  12. Mebuzzard

    Mebuzzard Poo-Bah (3,500) May 19, 2005 Colorado

    I dunno. I still go by (mostly) rating a beer according to how closely it matches its style. I despise coconut, but I won't trash a rating b/c of it. Coffee is another. I appreciate the flavors, just don't like them.
    On the other side, some NE IPAs may taste good, but their appearances go against IPA guidelines.
     
    FlandersNed likes this.
  13. dbrauneis

    dbrauneis Site Editor (7,101) Dec 8, 2007 North Carolina
    Premium Member Beer Trader

    I did not notice any differences in the Top Beers list or any of the others that I quickly checked though I'm sure there were a few that had differences (none significant).
     
  14. darktronica

    darktronica Meyvn (1,443) Aug 29, 2014 Indiana
    Beer Trader

    If I may provide some unsolicited feedback about the new stats, as well as the new summary display on a beer's page:

    [​IMG]

    - There's a lot of duplicated information now, and more numbers become confusing. For example, "4.05 w/ 258 Ratings": both numbers are repeated just below. Also, why not write out "with" if you're going to keep that line? There's more than enough room, even for beers with five figures' worth of ratings.

    - I'm not personally a fan of adding the Ranking here (or at least not having it as the first stat listed), because I don't have any reference point for what that means. Someone who has no idea how many beers are in the database might see a beer that's ranked over #1,000 and think that must mean it's not very good ("it's not even in the top 1,000!"), then get confused by dissonance between that ranking and the "Liked" score. Someone who doesn't know how the rankings are calculated is going to wonder why their favorite amazing-but-retired beer doesn't even merit a ranking. (I dig the idea of being able to find out how a particular beer is ranked, but I might suggest doing that via a Search function on one of the Top Beers pages. Alternatively, maybe hide some of these stats but provide a clickable option to display Detailed Stats that would include stuff like the overall Ranking, Ranking within the beer's style, Alstroms ratings, maybe even pDev.)

    - Adding Liked: to the Stats list also seems superfluous and confusing, because it can be calculated from the Ratings in combination with the new BA Score above. Someone who's rating a beer might also wonder why the process never asks them if they liked the beer.

    - That brings me to why I am cautious about the Liked score in the first place. Any threshold is going to be somewhat arbitrary, and without a readily accessible rubric, people will always have their own individual interpretations of what a 5-point scale means. The site now provides normative guidance that 3.75 should be the dividing line between beers you like and beers you don't like. I frequently see reviews with qualitative descriptions indicating that someone really enjoyed a beer, but their overall score is in the mid-3 range. I don't think it's anywhere close to a majority of reviews with scores 3-3.75 that do that, but it's also not uncommon, and it's just validation that people utilize the 1-5 range differently. I would suggest watching closely to see if the statistics show a movement in the next few months towards more clustering around 3.75; that would suggest some people tweaking their new scores along the lines of, "well, I liked it, would ordinarily score it a 3.3, but BA says if I like a beer it should be at least a 3.75."

    - I'm in favor of de-emphasizing the Bro scores in the page layout, but I don't really like putting it in the list of stats, right below a technical metric like pDev, and calling it "Alstroms." That makes it sound like an even more arcane metric (i.e. an "angstrom" is already a unit of length) and might confuse people who don't know who you guys are.

    That was longer than I originally planned, but in short, I think 1) the layout could be cleaned up more, and 2) some of the changes appear clearly oriented towards power users, potentially to the detriment and confusion of new or more casual contributors to the site. The latter point is not necessarily a "bad thing," but it at least should be a conscious choice, so I'm just pointing it out.

    Cheers, and thank you for all the hard work making useful content and summaries in the first place! I hope this doesn't sound overly critical, as I am grateful for the continuous efforts to improve the usefulness of the site--just trying to provide feedback at this step along the way.
     
    #14 darktronica, Sep 20, 2017
    Last edited: Sep 20, 2017
  15. darktronica

    darktronica Meyvn (1,443) Aug 29, 2014 Indiana
    Beer Trader

    Yes, it appears that the BA Score is just as the tooltip describes it: the number of ratings 3.75 or higher divided by the total number of ratings. For example, in my sample image in the previous post (3 Floyds' Lazersnake), the 89.5% score is just the number of Likes (231) divided by the total Ratings (258).
     
    Todd likes this.
  16. Todd

    Todd Founder (5,518) Aug 23, 1996 California
    Staff

    Correct. There was a typo in my post (long day with lots of moving parts), but the tooltip and linked help post were accurate. The typo has been fixed, and thread cleaned to avoid further confusion. (Sorry about that!)

    I'll be getting back to my beer now...:grimacing:
     
  17. eldoctorador

    eldoctorador Zealot (579) Dec 12, 2014 California

    I like this better than before, especially that it shows the beer ranking.

    Looking forward to the beer ranking snapshots
     
  18. beertunes

    beertunes Poo-Bah (5,175) Sep 24, 2007 Washington
    Beer Trader

    Gonna have to re-read when sober, but, not sure this affects me. Carry on.
     
    Eddiehop likes this.
  19. bbtkd

    bbtkd Poo-Bah (2,055) Sep 20, 2015 South Dakota
    Premium Member

    My count of most popular beers took a big jump
     
  20. Narthax

    Narthax Crusader (785) Mar 23, 2014 Texas
    Beer Trader

    249 to 243.
     
  21. rgordon

    rgordon Champion (809) Apr 26, 2012 North Carolina

    I'm clear-eyed and fresh, and this is all like Chinese and Mayan script to me. But yet interesting. I'm beginning to understand the contest piece and competitiveness of beer experience that is quantifiable. It's not for me, but I know it really matters to people. I'm still amazed by the printing press.
     
  22. thebeers

    thebeers Poo-Bah (2,109) Sep 10, 2014 Pennsylvania
    Premium Member Beer Trader

    I usually use the ratings and reviews for purchasing decisions, so I don't see the new score impacting that. Will be interesting to see if I start paying attention to it.

    I do think I'll probably start paying attention to whether my rating of a beer is a 3.73 or 3.77.

    Long term, it would also be interesting to figure out scoring within style in addition to just across the whole wide world of beer. (I guess the ranking is one tool, but unless it's near the very top, I don't find it particularly helpful information. Maybe list rankings as "X of Total"?)
     
    VABA, TheDoctor, jakecattleco and 3 others like this.
  23. emannths

    emannths Aspirant (215) Sep 21, 2007 Massachusetts

    What analysis did you do? Do you have a histogram or anything that shows how many beers fall into the new BA score deciles?
     
  24. fthegiants

    fthegiants Poo-Bah (1,535) Nov 20, 2013 California
    Premium Member Beer Trader

    Now that each beer has a ranking would it be possible to see the complete list? Not just the top 250.
     
    paresis likes this.
  25. Chuckdiesel24

    Chuckdiesel24 Champion (847) Jul 6, 2016 Illinois
    Beer Trader

  26. KWD

    KWD Initiate (25) Oct 6, 2008 Tennessee

    @Todd

    I am appreciative of all the hard work you guys put in but...

    Am I the only one that really liked that at-a-glance word ratings of "world-class, outstanding, very good, good, okay, poor, and awful?

    It really helps me when I'm standing in a store googling on my phone (yea im that guy).

    Now I don't see BA's % any better than any other place's in terms of having to infer meaning from percentage. You mention that we get to decide if we want to try a 50% beer, but we got to decide if we wanted to try an okay or poor beer before so that hasn't changed. It's free will.

    You guys were my gold standard. But alas, you still are. I can't give up on you. I'm hooked...

    I will accept change. I appreciate you guys trying to tighten the formula for a true-er rating but I also think losing the word rating system is a step in the wrong direction. I know it's a dumb complaint but I like good beer and I don't want to have to study too hard before I grab something out of my local store's fridge and check out.
     
  27. bwarner2015

    bwarner2015 Initiate (109) Mar 25, 2016 Connecticut

    I've been following BA and it's beer ratings for years now, but just only posted my first review this week.. My take on the new rating system will be that the reviewers will begin to use 3.75 as a major judging threshold. It will be my barrier between a perfectly drinkable but completely average (say Sierra Nevada Pale Ale) beer, and one that I really want to seek out and try. That being said, I would recommend removing the new "Percentage Liked" from the top to the bottom, and put the AVG/5 on the top. The average will be my new go-to criteria for deciding if a beer is worthy enough to seek out, as the BOS was my former criteria. Based on beers I have LOVED and their current averages, I would say a 4.20 will be my Outstanding beer, and a 4.33 will be World Class. But to each is own! Thanks BA for all you do!
     
    Thomas_Wikman and Lucular like this.
  28. zid

    zid Champion (845) Feb 15, 2010 New York
    Beer Trader

    It's great that your team is always trying to improve things. Since you are asking for opinions, I feel like there's too much of a mismatch between the way that beers have been rated here and arriving at a percentage of people who "liked" a beer. It's not really an accurate inference. If users were simply clicking "liked" or "disliked" when logging beers here, then displaying that percentage as a stat would be perfect.

    I feel like this is at odds with your goal of "telling a more complete story behind a beer's ratings," because the new story is a binary one (liked vs disliked) but the numerical data from reviews is far more nuanced without that layer of interpretation.

    By necessity, users will start thinking of anything below a 3.75 as a "dislike" and that really constricts the numerical range for "liked" beers. Maybe I'm alone, but as a beer lover, I like beers far more often than I dislike them.

    Speaking beyond the numbers, I think the "like" terminology can loosely encourage simple ratings over full reviews - which runs counter to some previous goals.
     
  29. zid

    zid Champion (845) Feb 15, 2010 New York
    Beer Trader

    I should add that the terminology is limited to "liked" and does not actually include "disliked," but I feel like the implication is there... and even if it isn't, reviews will start to use the system as such.
     
    WesMantooth likes this.
  30. zid

    zid Champion (845) Feb 15, 2010 New York
    Beer Trader

    No offense, but if there is now a new beer in the "top 250" that you haven't had, will it not be worth your consideration to drink... did the one no longer in the 250 become a wasted effort? I just don't get the concern. :slight_smile:
     
  31. fthegiants

    fthegiants Poo-Bah (1,535) Nov 20, 2013 California
    Premium Member Beer Trader

    Ha.... Thats not the point. I have been trying to complete the top 250 list for many years.... I am now at 246 with 3 in my cellar...... (need Oude Fermier). Its not a wasted effort as i enjoy the beers and the hunt, but id love to complete the list at least once. Changes make that harder.
     
    zid likes this.
  32. zid

    zid Champion (845) Feb 15, 2010 New York
    Beer Trader

    @Todd Would a hypothetic beer with five 3.75 ratings and five 1.00 ratings receive the same BA Score as a beer with fifty 5.00 ratings and fifty 3.50 ratings?
     
  33. mig100

    mig100 Savant (913) Aug 3, 2014 Texas
    Beer Trader

    I like some of the changes, but still feel that only reviews (not ratings) should count towards the average.
     
    Sabtos likes this.
  34. donspublic

    donspublic Poo-Bah (1,619) Aug 4, 2014 Texas
    Premium Member Beer Trader

    I guess it is nickel here nickel there type of thing. I guess I place more value on the numerical rating vs how many considered it acceptable. Most style will not do well on this, probably both numbers will reflect it, but some styles you need to dig into the reviews of people you trust. I am still holding out for the ability to download my list of reviews :slight_smile:
     
  35. ovaltine

    ovaltine Poo-Bah (2,118) Apr 6, 2010 Indiana
    Premium Member Beer Trader

    "Hi, I'm ovaltine and I approve this post."

    Thanks for the cogent post.
     
    donspublic and Todd like this.
  36. bwarner2015

    bwarner2015 Initiate (109) Mar 25, 2016 Connecticut

    Based on the current averages of many beers I have had, and either Loved, Liked, or Disliked, I will be basing my reviews along the lines of the following letter grade system:
    4.33-5.0 = A+ World Class
    4.20-4.32 = A Outstanding
    4.00-4.19 = A- Very Good
    3.75-3.99 = B+ Good, Enjoyable
    3.5-3.74 = B Average, Drinkable
    3.25-3.49 = B- Meh
    3.0-3.24 = C+ Not awful
    <3.0 = Avoid
     
  37. MNAle

    MNAle Savant (968) Sep 6, 2011 Minnesota

    I've never liked the "tomatometer" style of either "certified fresh" or "rotten." Just as with movies, all beer is not either "fresh"/"liked" or "rotten"/"disliked". To date, my average rating on BA is 3.74, pretty close to your "liked" number, but...

    I rarely buy beers I'm pretty sure I won't like, hence the average is skewed due to a lack of low numbers. And, I can name you dozens of beers I have rated below 3.75 that I like and re-purchase from time to time. There is even one one-time brewed beer that I rated just below 3.5 that I liked enough to go back and buy up all that was left on the shelf to cellar.

    There aren't any below 3.4 that I would likely purchase again.

    Now, I don't expect you to re-align your system to mine, but this will give me pause if I am "correctly" rating beers. After all, 3.0 is the numerical midpoint of the 1-5 scale, but you're suggesting the scale should be used more like 3.75 is the midpoint.

    I would suggest that you figure out how to avoid the implication that by saying 85% "liked" a beer means the other 15% "disliked" it.
     
    #37 MNAle, Sep 20, 2017
    Last edited: Sep 20, 2017
    meefmoff, beertrip, FBarber and 4 others like this.
  38. Sabtos

    Sabtos Poo-Bah (4,623) Dec 15, 2015 Ohio
    Beer Trader

    I like the changes, but at the same time, I agree with this sentiment.

    Carrying my thought process over from a past life using the 5-star systems being a critic, I've always considered 3.5 as a basic "like" score: not great, but good. I was never one of those to approach reviews with a school-grade sort of mindset, so 3.5 did not have a negative connotation to me.

    There are plenty of beers I've given a 3.5 to that I did enjoy, but not enough to really feel inspired or fully satisfied by. In this new light, it might appear to many that I downright didn't like a good number of beers.

    That said, I'm not going to alter the way in which I rate beers for the sake of breaking the "like" threshold. I'm sure the site will continue to evolve. Either way, it sure has been fun seeing these updates, especially coming with such full disclosure as this. I appreciate the efforts.
     
    Ozzylizard and MNAle like this.
  39. rypkr

    rypkr Champion (818) Nov 19, 2011 Pennsylvania
    Beer Trader

  40. dbrauneis

    dbrauneis Site Editor (7,101) Dec 8, 2007 North Carolina
    Premium Member Beer Trader

    Not exactly the place to do this, if you think a beer is a duplicate please feel free to submit an update and provide the beer you think to be the duplicate and it will be looked at by the Site Editors.

    In this case, there is not a generic BA Ten FIDY - there is a Bourbon Barrel Aged Ten FIDY (as well as a bunch of other types of barrels) which is different than the Whiskey Barrel version.
     
    Ozzylizard likes this.
Thread Status:
Not open for further replies.