Updates: Top Rated Beers

We've been getting a growing amount of constructive feedback regarding our Top Rated Beers lists. All of it basically boils down to the minimum number of reviews (what we've referred to as m) for a beer to be considered and how ours is too low (10). I've been struggling with the list myself and share much of the same concern, so I spent the last week soaking it in, analyzing our data and focusing on a better way to present a more realistic set of lists.

Now the problem with m is if it's too low it allows beers with high ratings and few ratings to dominate. And if m is too high it'll exclude many worthy beers of ever having a shot. By having a very low fixed m we basically hit this problem. On top of this, our previous system of calculations and equations was far too complicated and restrictive.

TL;DR: I had to remind myself to keep it simple, allow the data and math to do the work, and try to cater as many users as possible.

So, m is now dynamic vs. fixed. That is it'll change based on the rating activity of our users over time. I've also applied a new Bayesian-based formula that better takes into consideration (weighs) the number of ratings for each beer and works more efficiently with a dynamic m.

Other things worth mentioning:

• Individual ratings and beer stats were not altered.
• The formula, averages and minimum number of ratings are no longer displayed.
  
• The weighted rank/rating (WR) is no longer displayed on the lists. It was pointless, not displayed anywhere else, caused confusion, and by focusing on the beer's average and number or ratings it provides a better and more consistence reference.
• The "Top Rated Beers: New" criteria has been extended to include more new beers.
• The "Beers of Fame" criteria has been adjusted to better represent the list.
• A "Most Popular Beers" list has been added; simply sorted by number of ratings.

Personally, I'm very happy with the lists now. And unless you're chasing beers with high ratings and few ratings/reviews, there's really not much to complain about.

Check it out: https://www.beeradvocate.com/lists/top/ and see the sidebar for more lists.

Cheers!

 

Update: Please note that I'm currently working on striking a balance with our formula and m due to the impact that this update has had on some of the less active (in brewers/beers and user ratings) states/countries.

The lists will change multiple times over the next 24 hours, so there's no need to discuss the changes until the process is complete.

Thanks for your patience.

 

Update: I've reviewed the updates with @Jason and @Mike. We're good to go and don't anticipate any further changes for the foreseeable future.

m is significantly higher than 10 now, but still dynamic and we're still keeping everything simple by allowing the math and data to work for us. This will also allow the lists to organically grow and evolve over time.

(And yes, previous comments were scrubbed. They were useful, but no longer apply.)

Thanks again for your patience, and thanks to everyone who chimed in here or privately with feedback.

Cheers!

 

Now, m is the average number of ratings across the dataset (not disclosed, or anywhere close to 10). Before this update m was fixed at 10 reviews to be considered for the list and basically ignored ratings (not a good thing).

Yes. But individual beer stats are weighted toward reviews based on multiple, undisclosed factors.

 

It has 9 ratings and 2 reviews, which are simply ratings with a min of 150 characters attached to them.

 

Correct, but now it's based on ratings. It still runs through our weighted rank system before generating the BeerAdvocate Overall Score (BOS), though. And the thinking behind setting it to ratings (and other changes going forward) is: use the damn data (!!!) and present the user with more helpful information.

 

A "new" toggle for each list is something we're considering simply for the sake of discovery.