Updates to the BA Score, Rankings, and Top Rated Beers

Discussion in 'BeerAdvocate Talk' started by Todd, Jul 22, 2019.

  1. MNAle

    MNAle Poo-Bah (1,570) Sep 6, 2011 Minnesota
    Society

    I'm not so sure about the notion of reduced weighting and/or dropping old reviews.

    I understand the reason for it (beer landscape has changed, competition is tougher, etc.)

    But the disadvantages, it seems to me, are not insignificant.

    First, it would make everything even more hype-driven than it already is.

    Second, it would (unfairly, IMO) penalize beers that have established a long record of excellence but aren't attracting the attention of the kool kids. (Maybe 1st and 2nd are opposite sides of the same coin.)

    Third, it silently drops an individual's review. IOW, if I review a beer years ago, I don't usually update that review unless something changes with the beer. Yet, I COULD re-review if I knew the old one had been dropped. Notifying everyone who's review has been dropped does not seem feasible.

    Just seems to me like the gain (KBS is hard to catch, so let's fix that by dropping some of those high reviews, under the guise of them being old ...) is not worth the loss.
     
    stevepat, hopley, pmarlowe and 4 others like this.
  2. Harrison8

    Harrison8 Poo-Bah (3,608) Dec 6, 2015 Missouri
    Society Trader

    Perhaps, but just because something has been, doesn't mean it always has to be. There are so many sources for information on beer now, meaning potential users who don't want to deal with the sky's falling hysterics can go an alternate route. That seems like a dangerous precedence to set for a resource I rarely hear mentioned in local craft beer (outside of mentions of the Top 250).

    This thread is restoring my hope a little bit (some good discussion without bashing and has remained on topic).
     
    AlcahueteJ, hopley and cavedave like this.
  3. officerbill

    officerbill Disciple (350) Feb 9, 2019 New York
    Society

    If a beer has to “struggle” to get 10 reviews over the course of several years...

    How about a cut off at a certain number of reviews? Say, 10-25 reviews=all reviews count, over 25 only reviews less than 5 years old are considered.
     
  4. cavedave

    cavedave Poo-Bah (2,603) Mar 12, 2009 New York
    Trader

    I am in favor of any change(s) that keeps this site relevant and popular. I've read folks' comments about it, and I can't see how losing the historical records of individual beers improves anything in those two ways. OTOH where else can one go to find historical records of this kind provided by BA members over the years?
     
    MJSFS, stevepat, REVZEB and 5 others like this.
  5. officerbill

    officerbill Disciple (350) Feb 9, 2019 New York
    Society

    That's why I suggested 5 years, it would level out any hype factor and the cool beers would have time to lose their new beer frenzy.

    Formulas, tastes, experiences change. Is that Outstanding beer you had 7 years ago still outstanding given what you know and have available today; is it now just above average, if you haven't had it in several years do you know if it still tastes like you remember? Your 4.5 (or 2.5) rating might fade out, but if the beer has been consistent someone else's 4.5 review will take it's place.
     
  6. beertunes

    beertunes Poo-Bah (6,343) Sep 24, 2007 Kyrgyzstan
    Trader

    Can't rate a zero here, the scale is 1-5. I'm not in favor of it because, as someone who tends not to give out as many blue ribbons as most, it will likely mean a lot of my review scores will get tossed out. Why bother then?
     
  7. Harrison8

    Harrison8 Poo-Bah (3,608) Dec 6, 2015 Missouri
    Society Trader

    Struggling to get 10 reviews over several years is not uncommon and part of the reason some are ditching Beer Advocate for rating sites with a larger populous of input. A specific local example is Martin City, which has been opened since 2014, and yet only has 9 of their entire line-up with enough ratings/reviews for a BA score. That means only a portion of their line-up feature a quick score for users to reference when making a purchase, pushing users to where they can see a more complete picture. That means only a portion of their line-up is available for shelf talkers for store owners and social media bragging rights for the brewery - which are free advertisements for Beer Advocate. Take one of the local examples and multiple it across thousands of small breweries across the nation and it becomes clear why some are seeking out ratings elsewhere. Cutting away the old ratings/reviews heightens that issue (in Martin City's example, at quick glance, they'll lose 2 beers's BA scores by this time next year pending any additional reviews/ratings, and 1 of those is a distributed flagship.)

    At first thought, a cut off at a certain rating threshold could work. It could help local breweries 'stay on the map'.

    edit: this post does sound like the 'sky is falling' hysterics I referenced earlier, but I just do not think it's wise to cut out old reviews and ratings as it weakens the already softening rating database. Although my personal opinion is that the reviews here are better than elsewhere, the '10 minimum' ratings/reviews stresses the importance in achieving and maintaining 10 user scores - regardless of rating/review.
     
    ian_graf, REVZEB, hopley and 2 others like this.
  8. officerbill

    officerbill Disciple (350) Feb 9, 2019 New York
    Society

    Okay, I see where you're coming from. I wasn't thinking in terms of ratings site competition and brewery exposure. You make a good point.
     
    Harrison8 likes this.
  9. Harrison8

    Harrison8 Poo-Bah (3,608) Dec 6, 2015 Missouri
    Society Trader

    Lots to consider with potential changes. I'm happy we have a platform and user input to discuss it.
     
    kuhndog, meefmoff, AlcahueteJ and 2 others like this.
  10. heysuz

    heysuz Crusader (743) Nov 18, 2010 Indiana
    Trader

    Got it! Thank you very much! Totally missed that.
     
    AlcahueteJ and FBarber like this.
  11. MNAle

    MNAle Poo-Bah (1,570) Sep 6, 2011 Minnesota
    Society

    How do you know I wouldn't rate it the same? Declaring something invalid because it is old is a fallacy. Is the unstated objective is to allow newer beers to get their due hype more easily?
     
    Chuckdiesel24 and beertunes like this.
  12. AZBeerDude72

    AZBeerDude72 Poo-Bah (1,794) Jun 10, 2016 Arizona
    Society Trader

    Agree with the struggle to get 10 reviews. I look at Arizona and places like Tombstone and Arizona WIlderness, Wren House, etc. They are making world class beer but no reviews outside myself and a handful of others since the population on BA is heavy on East Coast etc. I really think a two tier system should exist that allows beers with low ratings/reviews to get maybe a link so folks can see them and the reviews posted in order to see what they are doing and if its good. Seems like a lot of places will be overlooked or not even mentioned and gives way to why a lot of folks to utilize other sites to get info, its the only way to see things about a beer/brewery due to more users etc.
     
  13. Chuckdiesel24

    Chuckdiesel24 Meyvn (1,136) Jul 6, 2016 Illinois
    Trader

    Agree with this, even more strongly than OP. I think there really aren’t any advantages. There’s too easy movement into these top lists based on hype and rarity. At one point Todd said they’ve made changes to let new beers have a chance, but it’s way past that to where older beers that aren’t the new fad of the day don’t really have a chance.

    You’re thinking about this incorrectly. Dropping ratings after X years would only impact beers that are older. It would do nothing to stop newer beers from getting ratings and making a list.
     
    beertunes likes this.
  14. AlcahueteJ

    AlcahueteJ Meyvn (1,089) Dec 4, 2004 Massachusetts

    I know, that's why I wrote in parenthesis "just a hypothetical scale".

    Let's use 1 - 5. If a beer has all ratings between 2 and 4, 1 and 5 would potentially be the outliers.

    Are your reviews THAT different than all of the reviews?

    For example, Tree House Julius is a 4.68. If you gave that beer a 3, I don't think it would get flagged as an outlier. But a 1 might.
     
  15. hopley

    hopley Meyvn (1,358) Feb 24, 2010 Massachusetts
    Society Trader

    I think pretty much everyone thinks yesterday's changes are permanent, there's really nothing that suggests otherwise - if there was a strong disclaimer suggesting otherwise, the discussion would have been completely different. Ironically, the notion of permanency in this case is a good thing because it sends a much stronger message about how people value these lists. To the notion that nothing is permanent, slight tweaks and changes to these lists over the years have been all in the spirit of constant improvement, eliminating them completely and replacing them with something completely different is an entirely different thing.
     
    Chuckdiesel24 and Todd like this.
  16. Gkruszewski

    Gkruszewski Poo-Bah (1,841) Nov 1, 2013 New York
    Society Trader

    The top 250 lists are going to be missed , to say the least....it really was something that separated BA from untapped.... I never considered or took untapped serious.... I personally took the lists serious and would strive to capture the top brews( according to BA)....... with all the new beers out there, (just too many)... the top ones were the must have.... please reconsider bringing back both formats ( new and old).....its like eliminating all the hall of fames and record books...
     
    JLK7299 and Holderness like this.
  17. beertunes

    beertunes Poo-Bah (6,343) Sep 24, 2007 Kyrgyzstan
    Trader

    Yeah, a lot of my reviews are in the 3.-3.25 range.

    And, to tie in with another angle, a lot of my reviews are from small, local/regional breweries. I've added hundreds of beers to the site, and a fair number of them have less than 10 reviews. I live in a city of 90-ish thousand people. Only 6-8 folks are active on BA, and I'm really the only one that's very active on the forums and doing reviews. The 10 review minimum just delegitimizes those breweries efforts, and then to throw out the high and low? Why bother?
     
    REVZEB, hopley and AlcahueteJ like this.
  18. hopley

    hopley Meyvn (1,358) Feb 24, 2010 Massachusetts
    Society Trader

    I agree with the use of median and means from a statistical perspective, but in this case, I don't agree with eliminating outliers as such. What's wrong with giving a Quad a 2 because you don't like hoppy beers, or a Gose a 2.5 because it's salty - who are you doing these ratings / reviews for - you, me, or the BA community? I get your point, and sometimes I struggle to rate a Gose, in particular, according to my personal tastes, or the tastes of what one would expect of a Gose (I'm not a big fan of Goses, but can appreciate what a good one tastes like). So I think the bottom line is you have to rate beers according to your own personal tastes, not what you think others think or expect => keep the outliers.

    Re. keeping / ditching historical ratings. I'm for keeping them. While I think underweighting them, deleting them, etc. unfairly penalizes "those that have come before"...I also think that the last thing you want to do is get rid of data. Keep it, use it, refine it, figure it out, etc. More is better.
     
  19. AlcahueteJ

    AlcahueteJ Meyvn (1,089) Dec 4, 2004 Massachusetts

    Reading the OP again, I can see how some might have thought that. But also based on what I've bolded from @Todd 's posts below, I saw it as something that wasn't permanent just yet.

     
    officerbill, hopley and Todd like this.
  20. officerbill

    officerbill Disciple (350) Feb 9, 2019 New York
    Society

    How do you know you would rate it the same, how often do you go back and update a beer you rated 6, 8, 10 years ago?
    It's not for newer beers to ”get their due hype” because that would even out over time. It's for beers that might have been great (or awful) back in the day to show they've maintained or improved their quality against the new competition.

    To use sports as a metaphor; take Mohammed Ali, Michael Jordan, or Tiger Woods. When they first came on the scene they were phenomenal, with style and ability that hadn't been seen before. They had no real competition at the beginning, but soon a new crop of athletes appeared and these guys were forced to prove they were still the best on a regular basis. Sure, you won the championship three years ago, but do you still have what it takes?
     
    AlcahueteJ likes this.
  21. MNAle

    MNAle Poo-Bah (1,570) Sep 6, 2011 Minnesota
    Society

    Old reviews are to be removed because they haven't been re-done? That's completely circular.
     
    cavedave likes this.
  22. MNAle

    MNAle Poo-Bah (1,570) Sep 6, 2011 Minnesota
    Society

    There has been a fair amount of discussion about point #1.

    Note the word "outliers." This has a statistical meaning, and is not just "throw out the top and the bottom."

    If the calculation is actually identifying outliers and tossing them, then it is a valid statistical method. The objective is to limit the effect the outliers have on the calculated score.
     
    AlcahueteJ, officerbill and Harrison8 like this.
  23. Snowcrash000

    Snowcrash000 Poo-Bah (2,202) Oct 4, 2017 Germany
    Moderator Society Trader

    I wouldn't dream of using Untappd for purchasing decisions and I wouldn't recommend it to anyone else as well, or else you're going to miss out on a lot of great beer. Quality over quantity, I'd rather trust 5 BA reviews than 500 Untappd ratings.

    My experience with Untappd has been that all the cool, hyped styles and breweries get around 4+ ratings no matter how good or bad the beer is and all the less cool, more subtle styles get around 3.5 or less ratings no matter how good the beer is.

    I've had terrible NEIPAs that were rated very highly on Untapped and I had great beers than were rated around 3.5 or less. For example, I recently purchased a set of six beers from Belgian brewery Siphon that were brewed in collaboration with various other non-hyped breweries for their 3rd anniversary and each of those beers was excellent, yet none of them scored higher than 3.6 on Untappd.

    I really get the impression that the vast majority of Untappd users are hipsters and tickers that know fuck all about beer appreciation.
     
  24. officerbill

    officerbill Disciple (350) Feb 9, 2019 New York
    Society

    The community. I'm one of those who firmly believes that beers stood be reviewed, rated, and ranked within style. To base your review of a porter on the profile of a weißen is pointless and a disservice to readers.

    I rarely drink IPA's, I don't care for the flavor; however, I do know and recognize what a good IPA is supposed to be. Should I go on BA and start giving 4.0 IPA's 2's because they aren't to my personal taste?

    Is the purpose of publicly posted ratings & reviews to let others know whether or not you like pilsners or whether or not this is a good pilsner?

    I agree with that, don't get rid of the data.
     
    AlcahueteJ and beertunes like this.
  25. TheIPAHunter

    TheIPAHunter Poo-Bah (2,107) Aug 12, 2007 Pennsylvania
    Society Trader

    I love the change. Thank you, and cheers!
     
    cavedave, officerbill and Todd like this.
  26. officerbill

    officerbill Disciple (350) Feb 9, 2019 New York
    Society

    No, update a review and the calendar starts over. Every few years you go back and edit ”first had this beer 2 years ago and nothing has changed, just as good as ever”, ”rated this beer 3 years ago, at that time it was well above average, now it's in the middle of the pack”, “haven't had this beer for 4 years, they've changed the formula and now it tastes like dishwater”.
    The countdown starts over because your review is, once again, timely.
     
    AlcahueteJ likes this.
  27. MNAle

    MNAle Poo-Bah (1,570) Sep 6, 2011 Minnesota
    Society

    I think you just repeated what I said... declaring data invalid merely because it is old, with no evidence to back that up other than it is old.
     
    cavedave likes this.
  28. fthegiants

    fthegiants Poo-Bah (1,571) Nov 20, 2013 California
    Society Trader

    The Top 250 list is Beer Advocate.... Please don't jump the shark
     
    hopley, Gambyt20 and JLK7299 like this.
  29. officerbill

    officerbill Disciple (350) Feb 9, 2019 New York
    Society

    A 10 year old review has historical relevance, a snapshot of how a beer ranked against the competition in 2009. That review has gone stale, like a restaurant review from a decade ago.
    I don't know if your old opinion is still valid or not and neither do you, so where's the harm in requiring that a beer be revisited every few years to verify if, given the current state of the industry, it still deserves that score?
     
    rightcoast7 and AlcahueteJ like this.
  30. beertunes

    beertunes Poo-Bah (6,343) Sep 24, 2007 Kyrgyzstan
    Trader

    What about a beer you had once, 10 years ago?

    The purpose of a review is to reflect one persons experience with one beer, at one point in time, in one point in space. A 10 year old review is no less valid than a 10 minute old one.
     
  31. MNAle

    MNAle Poo-Bah (1,570) Sep 6, 2011 Minnesota
    Society

    Where's the harm? Really? You with a grand total of 53 reviews are saying that? Really?
     
    FBarber likes this.
  32. REVZEB

    REVZEB Poo-Bah (5,615) Mar 28, 2013 Illinois
    Society

    As someone who has friends constantly trying to get him to switch to untappd, I couldn't agree more. Quality>hype. Next time they try, I am going to send them this post!
     
    Snowcrash000 likes this.
  33. hopley

    hopley Meyvn (1,358) Feb 24, 2010 Massachusetts
    Society Trader

    I think we're going to have to disagree on this one - but with a caveat. Yes, of course, why would you ever rank a beer against another style?! Agree. But I question how you could honestly and accurately rank the taste of a good IPA if you don't like IPAs - it's contrary to the point, and theoretically you wouldn't have had enough IPAs to know the difference. Same goes with me and Goses. So I just settle out on ranking the taste a 3 or something not egregiously bad, just not really all that great either - it doesn't do a disservice to the brewery / beer, but it also enables me to go back and look at my review to know if I liked it or not. Additionally, the ratings on Look, Feel, Smell should have less bearing on Taste and will usually be enough to bring it up to a reasonable level if it's a decent beer. Even better, if I don't like a Gose, or any other beer, I often write a review describing why.

    But the larger question is a good one - are BA reviews intended to inform the public of whether it's a good Pilsner, or of whether you liked that Pilsner or not - I still vote on the side of the latter. Firstly, the information I enter into this site is my opinion, and based on the legalese I probably signed when I opened my account, I realize others can read it, and BA can use it. Secondly, the public is smart and discerning enough to base its opinion on multiple users, not just one. This goes for this forum as much as any particular beer review. I would much rather see people rank beers based on their personal tastes, and even better write about them, than gloss over them by trying to rank them based on how they think others within that style will appreciate them. Cheers!
     
  34. AlcahueteJ

    AlcahueteJ Meyvn (1,089) Dec 4, 2004 Massachusetts

    Someone can correct me if I'm wrong, but I believe Untappd is more widely used for reviews than Beeradvocate, it was likely part of the impetus for this change.

    Google reviews can have the best reviews in the world, but if there's only 5 reviews on there for an item I want to purchase, and there's 20,000 on Amazon, I'm going to trust the website with more data.

    Plus, it's not like people on here are all BJCP certified judges and Cicerones.

    How is that different than what Beer Advocate's ratings were like before they made these changes?

    Dominance from Tree House/Trillium IPAs, Toppling Goliath pastry stouts...etc.

    And Pilsners with low ratings in comparison.

    At least in my experience, this is wrong.

    All kinds of people use Untappd. It's widely considered a great tool for keeping track of what beer you drank and where, what your friends have had, and has far more reviews for beers than many here.

    And for the record, I don't use Untappd either. I'm applauding Beer Advocate for recognizing a need for change.

    For a very long time it was only the Top 100.

    To add to your point, how often do head brewer's leave to start their own brewery? Or breweries expand and have to scale up?

    These all can, and have caused changes to beers over time.

    I don't think @officerbill was advocating to delete all old reviews. He wanted to keep them, but simply have them not count towards the overall score after a certain period of time.

    I believe the Bros. have always desired to have the reviews on this site be to style. @Todd
     
  35. stevepat

    stevepat Crusader (784) Mar 12, 2013 California
    Trader

    It seems to me that a compromise might be to allow additional reviews of the same beer from the same user to both count. This way, users committed to keeping their impression of beers fresh and (in their mind) relevant could add a new review that would be counted in the beers score along with their original, and still valid, impression. But you don't have to remove the older reviews
     
    officerbill likes this.
  36. cavedave

    cavedave Poo-Bah (2,603) Mar 12, 2009 New York
    Trader

    No. What a person tastes and reports is what a person tastes and reports. I won't mind if you quote and reply to this so long as you address what I just said.
     
    FBarber, beertunes and stevepat like this.
  37. beertunes

    beertunes Poo-Bah (6,343) Sep 24, 2007 Kyrgyzstan
    Trader

    Again, Why? That review from 10 years ago isn't invalidated; it still reflects that persons thoughts at that time. This attitude, bluntly, just seems to be a method to help the newest kool kid breweries get the highest scores, so that the score tickers can brag about having had all "best" beers.
     
    FBarber, cavedave and Chuckdiesel24 like this.
  38. drtth

    drtth Poo-Bah (4,006) Nov 25, 2007 Pennsylvania

    Require?? How would you propose to enforce that?
     
    colby600, FBarber and beertunes like this.
  39. beertunes

    beertunes Poo-Bah (6,343) Sep 24, 2007 Kyrgyzstan
    Trader

    Even on UT, only your first check-in counts towards the beers score. You can check it in as many times as you want, changing the score each time, but only the first one counts. try it sometime with a brand new brew from a local brewery that only has a handful of check-ins.
     
  40. Bitterbill

    Bitterbill Poo-Bah (6,837) Sep 14, 2002 Wyoming
    Society Trader

    Quantity has its advantages at times. Like when you have a beer that isn't listed on BA or RB or, more importantly, on the brewer's website. I was seeing all these beers locally that I couldn't find any information aboot except for on UT. That's why I added it to the beer centric sites that I joined. Shrugs.
     
    AlcahueteJ likes this.