Proposed Updates to Top Rated Beer Lists

This.

 

Is anyone here, or does anyone know an actual statistician who might be able to give an informed opinion from a mathematical perspective?

If not, maybe contacting someone in the field about the question would be worth our time.

 

I understand that I am not as integrated into the BA review system as many, so perhaps this more “outside” view is irrelevant, but…from my perspective it seems there are several issues. Aside from a lack of reviews (especially for new beers), it seems like the problem is one of time: How do BA lists stay relevant given the # of breweries, # of new beers, and lack of ratings?

It seems to me like the answer is to focus on timing. Let me explain by highlighting what @Todd mentioned a few posts above:

Why not have a “Top 250”, a “beers of fame” and a “Best New beers (250 or 500)”, the idea being that it would use the date added and first review to start how long it *could* enter AND stay on the list. (I think you need both because some beers, like bourbon county, get announced and added early, but may not be released for 1-6 months). This means that at most a beer would be on the list for no more than a year.

By lowering the threshold of reviews to 5 and creating this new list, you have hyper focus on new, “hip” and “hyped” beers, regardless of where they are or whether they can knock the guardians off their post. You could even throw a factor into the algorithm that takes into account the size of the brewery and whether that brewery is a local, regional, or national distributor. That way, even if Other Half (since someone mentioned them above) releases 40 new IPAs in a year, and even if they all get 25 ratings, they won’t just populate 40 of the spots because # of people to review (ie size of brewery) is taken into account. On that note, you could even put a cap on how many of the brews from a single brewery could be on the list (5? 10?).

Some of the other discussion is still relevant for potentially “fixing” the top 250, but I think the best thing that can be done to solve the issue is to actually create a new list that has beers constantly rotating in/out.

 

Hello! Former property-casualty actuarial analyst here! Ideally, you'd want to apply a credibility factor when aggregating your ratings. More recent experience is presumed to be more credible than older experience and is weighted accordingly. Any experience older than X years isn't credible, other than as a curiosity, and is either marginalized or excluded altogether. Larger numbers are more predictive than smaller numbers. If you happen to share ownership with another company with compatible data, pull that in, too. Todd seems to already be doing some of this but is shy about explicitly saying so. Considering how long it took me to think of the word 'compatible,' and then spell it, I'd probably be wanting a relatively high abv beverage to consider the matter further.

 

I am on team more lists the merrier, so why not have top active, top retired, etc?

 

So essentially a combination of the current Top New and Trending lists?

 

Kind of, but I think the devil is in the details. You look at those lists, and if you pay even a little attention on BA, you’ll recognize a huge number of those beers. Most are from VERY known breweries If you don’t include a measure to offset how available a beer is, size of brewery etc., you’ll continue to see nothing but certain breweries. I mean, that’s cool, I appreciate those lists at times. But I don’t feel like having those solves the issues raised here.

Or, I could always just be missing the point. Missing the trees through the forest, so to peak.

 

I wonder if people like the top rated lists on Untappd. If so, would it make sense to mirror that?

 

No because untappd lists every year a beer is released as a separate entry/vintage

 

Bayesian updating would be an interesting way to help refine older beers staying on lists by taking into account more recent ratings, particularly if they are retired beers. Does BA do anything like this?

 

Is there any potential for this? Seems like a good idea given that first ratings are often exaggerated, and as time goes on ratings usually drop (some) due to age. Seems like it would make some sense, even if you don’t put it into effect until a beer has reached a certain age, for example.

 

ie Bought the beer off the shelf of my local beer store

 

A store I was just at would qualify as well. I think 90% of the beer there was really functioning as insulation for the walls rather than purchasable product. It’s crazy.

 

I read all of the above, even though I now realize I failed to give a "like" to some good comments. I'm good with the proposed changes as long as there is an indication of the number of reviews that went into the rankings. Clearly, a beer that has only 5-6 ratings/reviews is subject to having an overall score skewed by inclusion of (for want of a better term) an uneducated or unrealistic reviewer. When the minimum number or ratings is 10 that is somewhat less likely to happen. I have no answer to the dilemma of how to exclude individual ratings made by morons, fools, or those who don't give a hoot what the intended rating system is supposed to be based upon, like the rating from a person who assigns a "2" to a sour when they already know they hate all sours. How about for starters, we require 5 reviews, rather than 5 ratings. A review can at least provide insight to the validity of a score, although I recognize that reading all the reviews is unrealistic.
I'm a little conflicted about a change to now include scores that are more than +/- "x" standard deviations from the norm. I somewhat support excluding outliers, although the threshold for what constitutes a real outlier could be reviewed.

 

That alone makes it sound like a really fun exercise if there was someone like a BA volunteer who could grind out the numbers for free. I think it would be as interesting as heck to know how average scores from reviewers compare to those of tickers, at least for a subset of beers, especially those which have high standard deviations in scores. However, I fully understand the need to prioritize things when resources are limited. I like the opportunity to provide input on these ideas, though

 

5 is too low, and can manipulated by a local tasting of 5 drinkers. As a real scientist, I can tell tell you 15 is absolutely the minimum for any significance to compare across different regions, styles, etc. and even then with a minimum of different geographic locations. But why not have a minimum of 5 local Prince Poo-Bahs ratings, where the top poo-bah of a geographic region is the Prince.

 

Another vote for 5 ratings being to low. I would prefer the minimum to be 20.

 

Just another thought. Is there any reason why all beers have to have the same # of ratings? Any reason this can’t be staggered based on size of brewery or distribution footprint? I feel like this could be relatively straight forward with just 3 tiers of more was too complicated (eg national, regional, local).

 

Personally, I think 5 ratings for a beer are essentially meaningless. Granted, I think 10 ratings for a beer are also essentially meaningless. For as many beers as people
drink, those are numbers that are N= negligible.

I’m all for getting more beers BA scores, and all for promoting reviews and subsequent recognition of beers for being highly rated by reviewers, but I think including beers with such limited ratings in any “best of” or “Top #____” list is futile, promotes a “whalez bro” culture, and bastardized the definition of a “best of” list.

I’d say go for the opposite, a higher requirement to be considered on any “Top X” list, and a move towards promoting more of a Beers of Fame list than a flavor of the month list.

Just me 2 cents.

 

I knew the coding would be a problem, but as someone who had done coding I figured there would be an easy way to do an if-then statement to get 90% based on first beer reported, # beers reported on Ba and so on. Then people could submit edits as needed.

If this is a completely unrealistic expectation, my apologies. Just the line of thinking I was following.

 

Bear in mind the distinction between 'rating' and 'ranking.' It may only take five ratings to enter the sweeps, but the number of ratings still determines the ranking. A perfect 5 with ten ratings will rank higher than a perfect 5 with only five ratings. And if I understand Todd's previous correctly, a perfect 5 with a review will weigh more heavily than a perfect 5 without a review. I am content.

 

As a long time member of BA, I can say the Top 100 list popping up on a Google search is what led me here (it used to be top 100, not top 250).

However, I honestly haven't looked at the lists at all in a long time. And that has less to do with BA, and more to do with there being >9000 breweries in the country now. And who knows how many beers are being brewed by each...just thinking about breweries like Tree House and Other Half makes my head spin...

Maybe this discussion isn't for me then, but I would like there to be more value in the lists here on BA. I always enjoyed them in the past. Personally I can't think of a way to make them more meaningful, but I applaud the effort by @Todd and the team for trying, and continuously asking for feedback.