Tweaks to the Top Rated Beers

Blog Discussion in 'BeerAdvocate Talk' started by Todd, Sep 11, 2018.

Thread Status:
Not open for further replies.
  1. Todd

    Todd Founder (13,254) Aug 23, 1996 Finland
    Staff Super Mod Pooh-Bah Society Trader

    Last week @Chuckdiesel24 pointed out a potential bug with underdeveloped data sets found within certain top rated beer lists. After some testing over the weekend, I've determined that the criteria being applied to our algorithm were too strict. This resulted in inconsistencies, which impacted certain regional and style lists the most, and the formation of stagnant (BORING!) lists.

    This morning I applied tweaks that not only fixed the issue, but I think we've struck a solid balance between recognizing popular beers and allowing newcomers a chance to climb the ranks and be discovered. Basically, more diverse lists that better reflect the current industry and criteria that we can easily revisit as our data evolves over time.

    You can view the Top Rated Beers here: https://www.beeradvocate.com/lists/top/ and see the sidebar for more lists. And the sidebar FAQ has been moved into its own help post to better explain how the lists are developed.

    Enjoy!

    Update:
    https://www.beeradvocate.com/community/threads/tweaks-to-the-top-rated-beers.587706/#post-6183347
     
  2. Amendm

    Amendm Pooh-Bah (2,305) Jun 7, 2018 Rhode Island
    Pooh-Bah Society

    Nice, I just looked at The Beers of Fame and Most Popular Beers, good stuff.
    Lots of Trillium and Tree House on the Top Rated Beers New.
     
    Harrison8 and IPAExpert69 like this.
  3. Chuckdiesel24

    Chuckdiesel24 Pooh-Bah (2,924) Jul 6, 2016 Illinois
    Pooh-Bah Society Trader

    Sorry if my suggestion sent you down a rabbit hole!

    This actually swings the pendulum toward beers with fewer ratings more than I'd personally want, but as you said, it's probably more in-line with the industry. Some of the best beers in <insert state> might be from a brewpub that doesn't distribute and you've never heard of - and is maybe worth checking out if you're traveling there...
     
    #3 Chuckdiesel24, Sep 11, 2018
    Last edited: Sep 11, 2018
  4. unclekeith

    unclekeith Pooh-Bah (1,602) Sep 4, 2012 Alaska
    Pooh-Bah Deactivated

    I applaud the effort for constant improvement and to make better lists. Personally, I'd like to see the minimum ratings needed to qualify for "Beers of Fame" be higher. There are some beers on the new list I have never heard of and have less than 2 ratings/year (average) - to me this does not scream "Fame".

    Thanks for the tweaks and the explanation.
    Cheers!
     
    Thomas_Wikman, fthegiants and Amendm like this.
  5. Todd

    Todd Founder (13,254) Aug 23, 1996 Finland
    Staff Super Mod Pooh-Bah Society Trader

    No worries. It was good bug catch.
    This has been tweaked.
     
  6. fthegiants

    fthegiants Pooh-Bah (2,300) Nov 20, 2013 California
    Pooh-Bah Society Trader

    Can you please explain why a beers are higher on one list and lower on others? For example Henna b3 which is #260 on the top beer rankings and #38 on the top american imperial stout list, where BA Popinski (BAPBM) is #345 on the top beer rankings but higher #34 on the top imperial american stout list. Thank you
     
    paulish likes this.
  7. Todd

    Todd Founder (13,254) Aug 23, 1996 Finland
    Staff Super Mod Pooh-Bah Society Trader

    Different data sets with different criteria. For example: the average score across the entire data set, which is used in the algorithm that generates the beer's weighted rank for the list, can be drastically different from list to list.

    Also, the ranking # displayed on the beer's page is its ranking against all beers that were considered for the top 250.
     
    FBarber likes this.
  8. Sabtos

    Sabtos Grand High Pooh-Bah (7,486) Dec 15, 2015 Ohio
    Pooh-Bah Society Trader

    Wow I lost 20 some beers from the top list. It does make me cringe a little to see some more stouts by JWB and Tree House on there. Some of the new beers also seem to have very few ratings. It does seem to reflect current trends without being completely overwhelmed by NE IPA or pastry stouts.
     
    Chuckdiesel24 and Narthax like this.
  9. Skellbop

    Skellbop Maven (1,283) Feb 26, 2015 New York
    Trader

    Same, all my hard work down the drain!
     
    Narthax likes this.
  10. generallee

    generallee Initiate (0) Apr 5, 2008 Virginia

    Is there a Tree House beer that is not in the top 250?
     
    Sabtos likes this.
  11. Snowcrash000

    Snowcrash000 Grand High Pooh-Bah (6,017) Oct 4, 2017 Germany
    Super Mod Pooh-Bah Society Trader

    I really mean no offense, but if drinking beer is "hard work" for you then maybe it's time to find another hobby. My apologies if you were being sarcastic, it's kinda hard to tell sometimes...
     
  12. Lucular

    Lucular Grand Pooh-Bah (3,943) Jun 20, 2014 Maryland
    Pooh-Bah Society Trader

    I'm kind of sad to see zero New Glarus beers on the list. Then again, if I were to guess about the list's composition without having beers like the one in my avatar in mind I wouldn't necessarily expect to see any 4% (non-wild, non-BA, not very sour) fruit beers making an appearance. However, I did notice that there are two NG beers in the top 25 on the beers of fame list. Interesting that being a top 25 beer all-time is not enough to land you in the current top 250 overall.
     
    TheDoctor, NickSMpls, Junior and 3 others like this.
  13. Skellbop

    Skellbop Maven (1,283) Feb 26, 2015 New York
    Trader

    It WASN'T sarcasm....:slight_smile:
     
    TheDoctor likes this.
  14. jcflorian

    jcflorian Initiate (0) Oct 27, 2014 Connecticut

    I went from 125 to 109. Was all excited getting to 125. Took me a long time with a lot of trading. All these beers I never heard of suddenly on the list. Liked it way better before, sorry guys.
     
  15. hopley

    hopley Pooh-Bah (2,608) Feb 24, 2010 Massachusetts
    Pooh-Bah Society Trader

    I applaud improvements and corrections to the algorithms but do feel bit bummed for similar reasons as others have stated - my top beers went down by about 20% and I now have no idea what they were before the change. This isn’t the first time a change in Top 250 formula has resulted in downward adjustments and while I hate to admit it, it just kinda takes the fun out of the chase.

    Giving newcomer beers a chance to punch into the ranks, while a noble and admirable gesture, seems off-kilter to me – isn’t that the point of the Top 250 New Beers? Shouldn’t Top 250 Beers reside somewhere between Top New Beers and Beers of Fame?

    I would actually rather see some kind of a waiting period before new beers can get entered into the Top 250, and would rather a higher minimum number of beer ratings to qualify (I counted 3 beers in the top 50 with 75 or less ratings). And including Limited / One-Off beers like Trillium Permutations in the Top 250 has always seemed off to me, even if for only a year – this makes no sense to me at all. Maybe give them their own list, maybe include them in Top 250 New Beers, but certainly not Top 250.

    I guess either I don’t fully agree with these changes, or I don’t understand them or get the rationale. I’m personally less concerned with whether these changes reflect industry standards, but I’ve also come to respect that you guys (BA) know your shit and are more about leading the industry rather than following, so I’ll go with these changes and assume they’re based on more insight than I am aware of (pretty easy assumption!). However, please take my unvarnished opinions and suggestions in good stride and keep up the good work. Cheers!
     
  16. Todd

    Todd Founder (13,254) Aug 23, 1996 Finland
    Staff Super Mod Pooh-Bah Society Trader

    UPDATE ALERT
    I woke up at 3am this morning for some reason. Checked the site. Saw your post. Agreed. Immediately had to do something about it.
    Great feedback and idea. We've also needed better separation between the Top 250 and Top New. And now we have it.

    Update Notes:
    • Top 250 are now beers added a year ago or more. This reduces the size of the data set, but provides much more validity to the list.
    • Top New are now beers added within the last year. This increases the size of the data set, and makes the list more exciting and distinct from the Top 250. This is where limited (brewed once) beers will thrive.
    • All beers (new or not, limited or not) will still show up on other lists; style, state, etc.
    • This also helps to address concerns from those chasing the Top 250. In theory, you have a much better chance of doing so now vs. before this update.
    Let me know what you think. And thanks for waking me up at 3am. :grimacing:
     
  17. Chuckdiesel24

    Chuckdiesel24 Pooh-Bah (2,924) Jul 6, 2016 Illinois
    Pooh-Bah Society Trader

    I think this is a great idea. I'd add that it works really well because it matches the timeframe where you guys retire one-off beers. Also, in today's brewing world, brewers often have significant batch to batch variation when they first come out with a beer. Keeping one-offs or even very new beers off the list for one year helps lessen the impact of that variable. I'd advocate making this change for all the top 100 lists - definitely styles and probably states too.

    @hopley 's points summed my thoughts up fairly closely. There was about 30% of me that liked the change and about 70% that didn't. I'd raise the minimum (both to qualify for the lists and in your bayesian formula) to 20 or 30 because I don't think 10 is statistically significant. You could have one reviewer checking the wrong beer and it still having a decent impact at that level.

    Here's an example - I've been trying to complete the IL top 100 list, and More's BA Karma is now on that list after the change with 12 check-ins. I live one mile from More, but to my knowledge they've never even had it at their taproom; they've only tapped that beer at two different festivals. I have no doubt it's a great beer, but it's hard to imagine the 12 people who attended either FOBAB or GI's Stout Fest two can effectively judge that beer compared to a beer like Deth's Tar that has been out for a number of years. Based on the timing, I frankly think one of the 12 ratings is an error from More's non-BA version of the beer.
     
    #17 Chuckdiesel24, Sep 20, 2018
    Last edited: Sep 20, 2018
    hopley and FBarber like this.
  18. Todd

    Todd Founder (13,254) Aug 23, 1996 Finland
    Staff Super Mod Pooh-Bah Society Trader

    Raising it to an arbitrary number will only create issues; see my original post. That said, and as previously mentioned, we can easily revisit the minimum criteria as our data evolves over time.
     
    Chuckdiesel24 and FBarber like this.
  19. JLK7299

    JLK7299 Pooh-Bah (2,729) Apr 21, 2016 North Carolina
    Pooh-Bah Society Trader

    Great ideas!
     
  20. smartassboiler

    smartassboiler Pooh-Bah (2,482) Apr 9, 2012 Illinois
    Pooh-Bah Society Trader

    While I understand what you guys have done with the Top Rated Beers (Top 250) list after reading through this thread, today officially ends my pursuit of completing this list after realizing the criteria has changed yet again (granted, this time, due to a perceived defect). I started this quest in earnest about 3-4 years ago, and then about a year ago I realized I had made great progress and started aggressively trading for/buying what I hadn't had yet. I also reviewed the list about once a month or so to to keep up with the new beers that appeared after having enough ratings to make the cut. I got it down to 9 that I hadn't tried or didn't currently have in the cellar, and now that number has ballooned back to about 30.

    I appreciate new beers coming out and becoming deserving of a spot on the list, but the seemingly constant evolution of what defines the list has become frustrating. Some people may find this evolution exciting, but unfortunately I do not as someone who was trying to complete it.

    Obviously this is a #fwp post that doesn't mean much in the grand scheme of things; just some feedback from someone who's used your site to track everything since I started this hobby back in 2012.
     
    JLK7299 likes this.
Thread Status:
Not open for further replies.