Blocking AI Bots, Scrapers, and Crawlers

Blog Discussion in 'BeerAdvocate Talk' started by Todd, Jul 4, 2024.

Thread Status:
Not open for further replies.
  1. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    Just a heads up. Using a relatively new Cloudflare feature, I've been blocking known artificial intelligence (AI) bots, scrapers, and crawlers from using (stealing and monetizing) our collective data to train their generative models. We've had this feature under our business plan for a while now, but Cloudflare just released it for all free plans.

    If you're feeling nerdy, you can learn more here: https://blog.cloudflare.com/declari...ots-scrapers-and-crawlers-with-a-single-click

    Will this change down the road? Unknown. All of this AI shit is still in its hype phase, moving very fast, changing daily, and raising new concerns and challenges with every change.

    In the meantime, this feature is currently blocking upwards of one million requests per day. I anticipate the inadvertent blocking of users to be very low, but false positives happen. Hit me up if you run into any issues.

    Thanks.
     
    #1 Todd, Jul 4, 2024
    train_wreck, yokels, JLaw55 and 78 others like this.
  2. micada

    micada Grand Pooh-Bah (3,960) Jul 13, 2015 New York
    BA4LYFE Society Pooh-Bah Trader

    People are jumping on the AI bandwagon much too freely, IMO. Block away!
     
    #2 micada, Jul 5, 2024
    JLaw55, Victory_Sabre1973, FLBeerGuy and 17 others like this.
  3. REVZEB

    REVZEB Grand High Pooh-Bah (7,686) Mar 28, 2013 Illinois
    BA4LYFE Society Pooh-Bah Trader

    I feel like there is a beer name inspiration in here somewhere…
     
    #3 REVZEB, Jul 5, 2024
    ChicagoJ, NorsemanOne, mactrail and 8 others like this.
  4. jvgoor3786

    jvgoor3786 Grand Pooh-Bah (4,222) May 28, 2015 Arkansas
    Society Pooh-Bah Trader

    Oh. Trying to ID bots is one of my favorite things to do here. :slight_frown: Guess I'll have to go back to posting the beers I'm drinking! :sunglasses:


    As always, thanks for everything you do to make this the best site on the internets, Todd.
     
    #4 jvgoor3786, Jul 5, 2024
    ChicagoJ, NorsemanOne, mactrail and 6 others like this.
  5. tasterschoice62

    tasterschoice62 Grand Pooh-Bah (3,949) May 14, 2014 Rhode Island
    BA4LYFE Society Pooh-Bah Trader

    Probably why I will always be here. You keep this site safe.
     
    #5 tasterschoice62, Jul 5, 2024
    ChicagoJ, jvgoor3786, BillAfromSoCal and 8 others like this.
  6. vurt

    vurt Grand Pooh-Bah (4,504) Apr 11, 2004 Oregon
    Society Pooh-Bah Trader

    #6 vurt, Jul 5, 2024
    ChicagoJ, jvgoor3786, MutuelsMark and 2 others like this.
  7. 2beerdogs

    2beerdogs Grand Pooh-Bah (5,682) Jan 31, 2005 California
    BA4LYFE Society Pooh-Bah Trader

    Much appreciated.
     
    #7 2beerdogs, Jul 5, 2024
    ChicagoJ, jvgoor3786, MutuelsMark and 2 others like this.
  8. beershrine

    beershrine Pundit (819) May 29, 2004 Idaho

    It would be nice to see governments crack down on stealing private internet businesses data. BA is a private concern and rules to prevent sucking up private data should already have been established. These tech companies won't even disclose what there doing, The future is looking bleak.
    Thanks Todd for putting up a fight.
     
    #8 beershrine, Jul 5, 2024
    ChicagoJ, mactrail, MutuelsMark and 2 others like this.
  9. chinchill

    chinchill Grand Pooh-Bah (5,541) Feb 29, 2008 South Carolina
    Society Pooh-Bah

    Thanks for jumping on this before the AI surge took over.
     
    #9 chinchill, Jul 6, 2024
    ChicagoJ, Victory_Sabre1973 and MutuelsMark like this.
  10. MutuelsMark

    MutuelsMark Grand Pooh-Bah (5,787) Jan 23, 2015 Kentucky
    BA4LYFE Society Pooh-Bah Trader

    @Todd are you saying there are nearly 1 million requests being blocked on BA?
     
    #10 MutuelsMark, Jul 6, 2024
    ChicagoJ likes this.
  11. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    Yes. Upwards of one million requests are being blocked each day.
     
    #11 Todd, Jul 7, 2024
    ChicagoJ, Jaycase, mactrail and 3 others like this.
  12. beerrat

    beerrat Pooh-Bah (1,739) Nov 28, 2013 Virginia
    Pooh-Bah Trader

    No AI just humans here please.
     
    #12 beerrat, Jul 7, 2024
    ChicagoJ likes this.
  13. MutuelsMark

    MutuelsMark Grand Pooh-Bah (5,787) Jan 23, 2015 Kentucky
    BA4LYFE Society Pooh-Bah Trader

    That is mind boggling!
     
    #13 MutuelsMark, Jul 7, 2024
    ChicagoJ and Victory_Sabre1973 like this.
  14. Orca

    Orca Grand Pooh-Bah (4,710) Sep 18, 2010 Washington
    Pooh-Bah Trader

    I won’t even pretend to understand what AI bots, scrapers, and crawlers are or what they do, but based on the fact that they give me kind of an itchy, creepy-crawly feeling when I think about them infesting this website, I’m going to assume that blocking them is a good thing. Thanks for doing that.
     
    #14 Orca, Jul 8, 2024
    ChicagoJ, BBThunderbolt, PapaGoose03 and 1 other person like this.
  15. BillAfromSoCal

    BillAfromSoCal Pooh-Bah (2,415) Aug 24, 2020 California
    Society Pooh-Bah

    I can see some potentially interesting uses for the BA data by third parties, but only if they pay BA for the right to download the massive piles of data.
     
    #15 BillAfromSoCal, Jul 9, 2024
    beergoot, ChicagoJ and PapaGoose03 like this.
Thread Status:
Not open for further replies.