Proposed Updates to Top Rated Beer Lists

Discussion in 'BeerAdvocate Talk' started by Todd, Dec 28, 2021.

Thread Status:
Not open for further replies.
  1. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    Check out: End of Year Site Updates

    ###

    In the past, I'd make a massive update that would impact our top rated beer lists, collect feedback, and make adjustments. For this update, I'd like to discuss one update that's already occurred and propose two major updates for discussion before anything is applied.

    Occurred: The Raw Average Is Back

    In the before times, we switched from using the beer's average to a truncated (trimmed) mean, wherein we were essentially discarding any ratings that were outliers equally on both ends of the 1-5 rating scale. While it seemed like good practice at the time, I don't like the idea of discarding opinions on a beer simply because they're numerical outliers. I'd rather let the raw average come to play, and then let the weighted rank sort it out.

    This update has already been applied.

    Proposed: Consider Active Beers Only

    Only consider "Active" beers (not retired and rated within the last two years), when ranking beers. This will no doubt help to remove beers that are no longer available/produced from camping the lists, but it'll significantly limit our pool of ranked beers. Why? See below.

    Proposed: Lower the Minimum Threshold
    Since basically forever, the minimum number of ratings that a beer needed to be assigned a BA Score and to be considered for top rated beer lists (ranked) has been 10. Today, there are over 9,000 breweries in the US alone releasing hundreds of thousands of beers a year, we've seen a decline in ratings over the years, less new beers being added in recent years, and most beers being entered today are one-offs (brewed once and retired after a year of being entered on the site).

    Given all of this, it would make sense to lower the minimum to 5 ratings. This would immediately double the pool of ranked beers, give more beers a BA Score, and help users discover new beers. If we don't, it's going to increasingly become more difficult (or impossible) for new beers to ever make the lists unless they're regular offerings that gain wider distribution. Even then, it could take a very long time.

    Note: I understand that some of you who chase/tick the Top 250 and whatnot will oppose any significant changes that would thwart your efforts, but I'm looking for feedback that goes beyond that and considers the bigger picture.

    Thanks in advance for your thoughts.
     
  2. Sheppard

    Sheppard Poo-Bah (2,057) Mar 16, 2013 Massachusetts
    Society Trader

    I am not a Top 250 ticker and I am someone looking out for new local options that stand out, so these changes would certainly be welcome for me. I've mentioned this in the Top State Beers thread, but it's hard to discover new breweries (unless you just go to whatever is out there) since the lists are already congested or even dominated by established breweries. Hopefully, this evens the playing field a little and allows for someone who does something really well to breakthrough. Based off my criteria, I welcome these changes.
     
    Edgewise, DrOfGolf, scott451 and 18 others like this.
  3. beergoot

    beergoot Poo-Bah (7,852) Oct 11, 2010 Colorado
    Society Trader

    I like your ideas for the raw average plus listing only using active beers for the top rankings. The minimum rating, I can take that or leave it. I suppose it's worth trying out and seeing how it impacts ratings.

    I imagine the beers of fame still will include retired beers, correct?
     
  4. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    It's never (intentionally) included retired beers.
     
  5. Harrison8

    Harrison8 Poo-Bah (4,877) Dec 6, 2015 Missouri
    Society Trader

    Per the 'active' - I'd say allow active & inactive to be apart of the lists. If anything, beers landing in 'inactive' can be more easily flagged by BA users to 'retire' them. Keeping this process manual means there could be some more evidence for when a beer is truly retired vs. simply not active on BA.

    Personally, I'm less than thrilled with bumping the BA score threshold down to five, but ultimately, I think it's the right move. Even the more frequently rated/reviewed beers from breweries local to me are only hitting ≤ 8 ratings at best. That's after several re-releases.

    It could create more of a mess than anything, but anyway to supplement beer listings under 10 ratings/reviews with an averaged Untappd score? Probably a PITA to link them all, with the different naming conventions and all that, but wanted to throw it out for consideration.
     
    Edgewise, ChicagoJ, Whyteboar and 8 others like this.
  6. JohnnyHopps

    JohnnyHopps Poo-Bah (2,658) Jun 15, 2010 Indiana
    Society Trader

    I’m a little concerned about a one-off that gets (only) five reviews getting an exaggerated placement in the top 250. 5 reviews at a popular a brewery is not a lot. In the long run, I don’t think this helps more obscure breweries get recognized.
     
    ADTaber, defunksta, cjgiant and 13 others like this.
  7. zid

    zid Poo-Bah (1,671) Feb 15, 2010 New York
    Society Trader

    Personally, from a conceptual standpoint, I think a minimum of 10 ratings for a beer to be on a communally created global top beers list is too low rather than too high. If 10 is too high from a practical standpoint, then I’d question the purpose of even having such a conceptually compromised consumer driven list.
     
    ADTaber, hopley, wspscott and 19 others like this.
  8. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    I've been a big proponent for a unified BeerAdvocate/Untappd brewery/beer database, and I think we'll eventually get there, but it's a massive project and there are bigger priorities at the moment.
     
    Edgewise, Junior, Mikexw and 15 others like this.
  9. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    Just like you don't see any beers with 10 ratings in the Top 250, you're not going to see beers with 5 ratings in the Top 250. You'll see them in some regional and styles lists though.
     
    Junior, Mikexw, ChicagoJ and 12 others like this.
  10. UrbanCaveman

    UrbanCaveman Meyvn (1,056) Sep 30, 2014 Ohio
    Society Trader

    Regarding the "active beers only" idea, I've run across several beers marked as retired that have apparently been pulled out of mothball status for re-release, the most recent being Listermann's Maple Chickow. How is retirement status currently determined, for beers that aren't one-offs but also aren't necessarily on a regular schedule?
     
  11. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    The minimum can and should be flexible based on activity. You see this with IMDB (we use a similar weighted rank system). Though they no longer disclose their criteria, it's changed over time.
     
  12. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    Beers are retired manually (by a report from a user or brewer) or automatically retired after one year if the beer was entered as a one-off.
     
  13. paulish

    paulish Poo-Bah (2,339) Feb 2, 2014 New York
    Society Trader

    Sadly... Welcome 250 kinds of Pastry Stouts.
    My suggest - The beer with 5 ratings will be to able to include in BA rankings but not more one year. If the beer cannot collect more than 10 for year the beer will remove from any BA rankings.
     
    Sammy, rodbeermunch and JohnnyHopps like this.
  14. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

  15. zid

    zid Poo-Bah (1,671) Feb 15, 2010 New York
    Society Trader

    Perhaps you should consider modifying this with the addition of “inactive.”
     
    Beer_Economicus likes this.
  16. PapaGoose03

    PapaGoose03 Poo-Bah (3,293) May 30, 2005 Michigan
    Society

    A little bit of a tangent here, but a long time ago there was discussion about some sort of 'handicap' factor to be applied to less popular styles that would help great beers in those styles reach the 'top' lists. Did that ever happen, or should it be part of the discussion in this thread?
     
  17. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    As in auto-retire them? If so, that assumes that the beer is no longer being brewed. But running with this for a moment, we could merge inactive and retired into simply inactive. One less tab, less confusing, and an inactive beer could easily become active again with a simple report if needed.
     
    laketang, DrOfGolf, scott451 and 13 others like this.
  18. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    No. This never happened. Big can of worms, IMO. But feel free to start a new thread.
     
  19. Coronaeus

    Coronaeus Poo-Bah (1,657) Apr 21, 2014 Canada (ON)
    Society Trader

    While I understand the hesitation from some regarding lowering the necessary ratings to 5, this proposal might actually help get the Canadian list (as well as others I’d presume) move out of 2014.

    Here, we suffer from too few users rating or reviewing beers. I’m not sure how that will ever change. We also see mediocre breweries that distribute to the US take far too many spots on the list. The breweries that most local beer drinkers would mention as being ‘among the best’ are almost completely absent from the list. I don’t know if the proposed changes will help make the list more reflective of the current beer scene here, but they might.

    If you take one of the most popular styles, NEIPA, you’ll find next to none of the generally acknowledged best Canadian makers of the style on the list currently. Just from Ontario and Quebec alone, Messorem Bracitorium, Bas Canada, Wood Brothers, Badlands, Barncat, Third Moon should dominate the list. They don’t. This would be like Other Half, Tree House, Monkish etc., not making a top list in the US, because Celis or Pete’s Wicked Ale are still holding down spots.


    One other thought. For the tickers, perhaps an annual archived Top 250 list would be a nice thing to have easy access to.
     
  20. PapaGoose03

    PapaGoose03 Poo-Bah (3,293) May 30, 2005 Michigan
    Society

    I can see it being a big can of worms to create factors to use, and you don't need that right now. Anyone who wants to see top beers in a particular style can go to the style page and find the top beers there.
     
  21. LesDewitt4beer

    LesDewitt4beer Savant (978) Jan 25, 2021 Minnesota
    Society

    It all makes sense to me in the ever changing landscape and revolving doors of beers. Especially in now times, as you mention, just the sheer number of new US breweries and offerings alone not to mention distribution issues/disruptions we sometimes face now. I embrace all 3 main points: Raw Average raking in all numerical ratings, the proposed "active only beers"(2 yrs is IMO a fair amt of time although some beers are very slowly rotational vs 'one off') which makes way for your 3rd point proposing a lower minimal number of ratings to 5 thus creating more area/visibility for new beers. Sensible changes Todd for Top Rated Beers. Cheers!
     
  22. Rug

    Rug Meyvn (1,467) Aug 20, 2018 Massachusetts
    Society Trader

    Yeah I feel like this is important. I see many local and smaller breweries in general who are just shy of that 10 review cutoff on some beers
    I also agree with this. Wouldn't even be too hard to go back and archive previous years. Maybe when you navigate to the top 250 it defaults to the current year but there's a drop down box where you could check the year you want
     
  23. IPAExpert69

    IPAExpert69 Aspirant (225) Aug 2, 2017 Pennsylvania

    And this little snippit is exactly why this change is warranted, give small brewers an opportunity to be discovered. Heck there were so many breweries right in my local area I was completely blind to until I became active on BA. And isn't that kinda the entire point?
     
  24. cryptichead

    cryptichead Poo-Bah (1,709) Jul 3, 2014 Illinois
    Society Trader

    All the proposed changes make sense to me. I say go for it!
     
  25. FBarber

    FBarber Poo-Bah (5,619) Mar 5, 2016 Illinois
    Moderator Society Trader

    This is where I think this change is going to be the biggest boon for BA and for us users. It will allow for my dynamism in the regional, state and style listings which are lists that I personally use on occasion to find new stuff.

    Edit: @IPAExpert69 beat me to this point.
     
  26. papposilenus

    papposilenus Poo-Bah (1,901) Jun 21, 2014 New Hampshire
    Society

    My issue here - and I don't know if I've actually raised it before or only said it in my head - is that you have way more raters throwing up straight 5's than 1's. Using raw averages would seem to result in a higher average than might be otherwise be warranted.

    Many times beers that were initially added to the database as one-offs are subsequently brewed again, as an anniversary beer, or go into rotation. Whenever I come across one of these - not infrequently - I submit an edit request but I notice that other reviewers have been adding reviews in the meantime. I assume that there are a fairly huge number of these that I haven't stumbled across. For this reason, whenever I add a beer to the database, I add it as a rotator pretty much regardless - I apologize if this is dickish.

    100% yes. I often feel lonely and irrelevant drinking all these low-traffic northern New England and Québec beverages and crave validation.
     
  27. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    Depends. I found the reverse when looking at the Top 250. Many of those averages dropped.
    It's certainly not helpful, accurate, and will ultimately result in more reports/works for mods. But it's not always easy to tell if a beer is a one-off either.
     
  28. JohnnyHopps

    JohnnyHopps Poo-Bah (2,658) Jun 15, 2010 Indiana
    Society Trader

    Messing with the top 250 list really throws some people’s world in the chaos. How about expanding it to the top 500!?! … I bet that would get the trade forums moving!
     
    hopley, Sammy, REVZEB and 4 others like this.
  29. Bhubbard34

    Bhubbard34 Aspirant (262) Mar 4, 2016 Pennsylvania
    Trader

    I Like the proposed changes Todd.

    Just a little confused because I thought only active beers appear in the top lists now as it is. Currently retired beers are not eligible anyway, so not sure how this is a change? What am I missing?

    lowering the minimum number of ratings to 5 is a good idea, and it is not going to affect the Top 250. I don’t think people understand there is an “unknown weighted formula” Todd uses to get the rankings. Just because a new beer gets “5” perfect 100 scores, does not mean it’s instantly jumping into the top 10. It still is put through a weighted formula to get ranked.
     
  30. Smakawhat

    Smakawhat Poo-Bah (8,046) Mar 18, 2008 Maryland
    Society

    I think the change from 10 to 5 is a good idea since it will open up people to more beers and breweries, just as Todd mentioned.

    Often at times many of the inactive beers I find aren't inactive either, they just haven't been rated in a long time, so I just find them when I happen to review them and sent a note to the staff to un-retire the beer.
     
  31. Tucquan

    Tucquan Poo-Bah (3,315) Oct 11, 2007 Pennsylvania
    Society Trader

    I don't know if it is already the case but my suggestion is to only consider beers that meet at least the minimum number of characters to be considered a review.
     
    Whyteboar likes this.
  32. Bhubbard34

    Bhubbard34 Aspirant (262) Mar 4, 2016 Pennsylvania
    Trader

    Ratings vs reviews are two different things. I believe all of these changes are directly related to ratings, not reviews
     
    Beer_Economicus and mactrail like this.
  33. Gkruszewski

    Gkruszewski Poo-Bah (2,676) Nov 1, 2013 New York
    Society Trader

    Can we please have a top 100 list for retired beers… or beers that are only brewed annually…
     
    hopley, ZebulonXZogg, REVZEB and 8 others like this.
  34. jvgoor3786

    jvgoor3786 Poo-Bah (2,224) May 28, 2015 Arkansas
    Society Trader

    I had this issue last night. I went to a brewery out of town and entered a new beer. I had no idea if it was a one-off, rotator, or even a regular menu item. I just make my best guess.
     
  35. plaid75

    plaid75 Poo-Bah (4,566) Jan 13, 2005 New York
    Society Trader

    Since likely most of the beers to which you refer are subregional, I don't see the value of featuring them on the top beer list if the intended purpose for doing so is to expose the beer to more people.

    Perhaps dealing with this on a regional basis would better serve the purpose of exposing top beers to people who are actually in the position to drink them.
     
    PapaGoose03 and BBThunderbolt like this.
  36. dbrauneis

    dbrauneis Poo-Bah (9,951) Dec 8, 2007 North Carolina
    Moderator Society Trader

    When adding a beer, the default is now “One-off” and I think that is reasonable unless you know it will be rotating or year-round. There are tons of beers that have been added as rotating but were one-offs in the database that have yet to be reported (hence the inactive lists for beers that have not been rated/reviewed in over two years).
     
  37. dbrauneis

    dbrauneis Poo-Bah (9,951) Dec 8, 2007 North Carolina
    Moderator Society Trader

    Only active beers currently appear in the top lists - previous to the addition of inactive there were only two choices (active or retired). With the addition of inactive (not retired on BA but not reviewed/rated in over two years), there are now three choices. The proposal is only to consider active beers and exclude retired and inactive (not reviewed/rated in over 2 years).
     
    PapaGoose03, Bhubbard34, Todd and 2 others like this.
  38. misteil

    misteil Savant (930) Feb 21, 2019 Ireland
    Society

    Please do this! I’ve wanted this for yonks but never actually mentioned it. The Irish top rated beers list is just ridiculously inaccurate, because the best stuff being made these days is never getting more than 10 reviews. This change would really help I suspect. Thank you.
     
  39. Todd

    Todd Founder (6,564) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    Correct. This update is looking at the numbers (ratings), not text (reviews, which all have a rating attached to them).
     
  40. woodchipper

    woodchipper Meyvn (1,451) Oct 25, 2005 Connecticut
    Society

    I am generally in favor of your proposals, but like everyone here, wondering what the real results will be. Have you considered creating a sandbox version of this that you and maybe some moderators can review and look for unintended consequences before going live with it?
     
    Beer_Economicus, o29, Todd and 4 others like this.
Thread Status:
Not open for further replies.