Possible bug with underdeveloped data sets in the top rated beer lists?

Help Discussion in 'BeerAdvocate Talk' started by Chuckdiesel24, Sep 7, 2018.

Thread Status:
Not open for further replies.
  1. Chuckdiesel24

    Chuckdiesel24 Grand Pooh-Bah (3,208) Jul 6, 2016 Illinois
    Society Pooh-Bah Trader

    Has there been any thought to raising the number of minimum entries to qualify for a list? I think it takes a little bit more than 10 ticks to have what most people would think is a valid rating (It used to be ten full reviews). Plus, you get weird situations on the less developed lists - like this in Nebraska.

    62 Stupefy (3.76, 11 ratings)
    Lucky Bucket Brewing Co.

    63 Pernicious (3.77, 15 ratings)
    Blue Blood Brewing Company

    The beer with the higher rating and more ratings is lower on the list. I think this can happen at the bottom end of a set when the number of observations in the formula differs from a minimum to be allowed into the set.
     
    #1 Chuckdiesel24, Sep 7, 2018
    drtth and FBarber like this.
  2. rudiecantfail

    rudiecantfail Pooh-Bah (1,927) Aug 9, 2011 Pennsylvania
    Pooh-Bah

    Well, there's nobody with 11 ratings showing up anywhere near the top of any lists. You're safe.
     
    #2 rudiecantfail, Sep 7, 2018
  3. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    Yes. We've tested raising the min along with many other tweaks over the years. Where it's at now is based on this and subsequent user feedback. We have no desire to change things right now, but we do review it periodically.

    You're also looking at the BA Score vs. the weighted score; not displayed, but used to rank the beers.
     
    #3 Todd, Sep 7, 2018
    FBarber, Harrison8, Chuckdiesel24 and 1 other person like this.
  4. Chuckdiesel24

    Chuckdiesel24 Grand Pooh-Bah (3,208) Jul 6, 2016 Illinois
    Society Pooh-Bah Trader

    Thanks for answering! I understand that but a beer with a higher rating and more ratings than beer #2 shouldn’t be behind any other beer. This only happens on he bottom half in what I’d call “less developed” lists, where I’d guess there aren’t that many beers with over 10 and the beers that make the lower part of the list must be below the average of that specific population.

    That probably doesn’t matter as much, but I do think there’s a chance of local homerism in the lower end. Anyways just thoughts. Thanks!

    Yeh this is mostly an OCD thing of mine - I fully admit!
     
    #4 Chuckdiesel24, Sep 8, 2018
    drtth and FBarber like this.
  5. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    After some coffee and digging around, this does appear to be the case for less developed data sets. But it's more of a byproduct of our algorithm, which I think does need some tweaking.

    As for the min: lowering it could fix this once the algorithm is tweaked, but that would result in a list full of beers with barely any ratings. And raising it too much will only further restrict the data set. It's a balancing act.

    I've updated the thread title to better reflect the issue, and I'll post an update once I sort it out.

    Thanks for pointing this out.
     
    #5 Todd, Sep 8, 2018
    FBarber, Chuckdiesel24 and seakayak like this.
  6. Chuckdiesel24

    Chuckdiesel24 Grand Pooh-Bah (3,208) Jul 6, 2016 Illinois
    Society Pooh-Bah Trader

    Can't really speak for anyone other than myself, but lowering the min definitely isn't what I'd advocate. I've generally viewed these lists as "beers meeting this criteria (style or state/country) you can find ranked". Or could have reasonably found if you were there at the right time for something like a seasonal. That's simplifying it a bit, but basically, if I travel to somewhere Alabama - I check out the list of top Alabama beers and use that to help figure out if I want to buy something local.

    Lowering the min would limit the usefulness of these lists - I think you'd get a lot of taproom one-and the like jumping into the lists of less developed states and styles.
     
    #6 Chuckdiesel24, Sep 10, 2018
    FBarber and Lucular like this.
  7. ICTguy21

    ICTguy21 Initiate (0) Jul 18, 2017 Kansas

    Did the minimum get done away with entirely? I was checking out the KS top hundred beer list and it’s completely different now. Beers are on the list with only one rating. It’s definitely not indicative of what the top state beers actually are anymore imo. I understand you guys have bigger fish to fry running this site and you can’t make everyone happy but I’d hate to see one review be the new baseline for making a best of list.
     
    #7 ICTguy21, Sep 11, 2018
    Chuckdiesel24 likes this.
  8. shelby415

    shelby415 Pooh-Bah (2,098) Oct 10, 2011 Oregon
    Pooh-Bah Trader

    A number of the top lists by style have beers with only one rating as well.
     
    #8 shelby415, Sep 11, 2018
  9. Chuckdiesel24

    Chuckdiesel24 Grand Pooh-Bah (3,208) Jul 6, 2016 Illinois
    Society Pooh-Bah Trader

    I think from what Todd said above they are running some fixes and my guess is that this is temporary. At least that's what I hope...
     
    #9 Chuckdiesel24, Sep 11, 2018
  10. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    #10 Todd, Sep 11, 2018
    ICTguy21 likes this.
Thread Status:
Not open for further replies.