Experiencing intermittent site errors? It's BA, not you.

Blog Discussion in 'BeerAdvocate Talk' started by Todd, Sep 10, 2023.

Thread Status:
Not open for further replies.
Page 1 of 2
  1. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    For the last 10 days or so I've been monitoring an increase in server errors related to our database. The issue is intermittent and doesn't last long, but you might run into a blank, error, or Cloudflare page when it occurs.

    Just wanted to let everyone know that it's being monitored, investigated, and that we hope to resolve it soon.

    In the meantime, I recommend writing your reviews locally so you have a copy. We backup everything, but I'd hate for anyone to lose their review during the posting process. (Forum posts get saved as drafts every 60 seconds and are stored for up to 24 hours from the last save.)

    Sorry for the inconvenience.

    Updates:
    1. Login required for search and other pages again to reserve resources for core users
    2. Adjusted daily cron jobs (update scripts) to run at better times
    3. More cron job tweaks
    4. Looks like the tweaking worked
     
    #1 Todd, Sep 10, 2023
    Rug, TCgenny, mpruden and 32 others like this.
  2. ATL6245

    ATL6245 Grand Pooh-Bah (3,984) Aug 16, 2018 Georgia
    Society Pooh-Bah

    Thanks for the heads up! I've been wondering if it was my provider or not.
     
    #2 ATL6245, Sep 10, 2023
    TCgenny, mpruden, beergoot and 5 others like this.
  3. Shanex

    Shanex Grand Pooh-Bah (4,960) Dec 10, 2015 France
    Mod Team Society Pooh-Bah Trader

    It has worked perfectly fine for the last ten days for me. None of the above issues. Is it affecting everyone, or just a majority of users very punctually or something?
     
    #3 Shanex, Sep 10, 2023
    TCgenny, mpruden, ChicagoJ and 5 others like this.
  4. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    There's a chance of impacting users who are active (clicking/posting) during the periods that are disrupted, but otherwise the vast majority of users won't be impacted.
     
    #4 Todd, Sep 10, 2023
    mpruden, ChicagoJ, beergoot and 6 others like this.
  5. AZgman

    AZgman Pooh-Bah (1,858) Dec 22, 2011 Arizona
    Society Pooh-Bah

    :+1:
     
    #5 AZgman, Sep 11, 2023
    mpruden and ChicagoJ like this.
  6. LesDewitt4beer

    LesDewitt4beer Grand High Pooh-Bah (6,315) Jan 25, 2021 Minnesota
    Society Pooh-Bah Trader

    Thanks a ton @Todd. You're a hard workin' dedicated man. I rcvd a few short-lived error messages last week but I'm simultaneously logged into multiple ad-filled sites and figured I just needed to clean sweep, shut down/power up. All smooth now. Thanks for your efforts!
     
    #6 LesDewitt4beer, Sep 11, 2023
    Shanex, mpruden, ChicagoJ and 2 others like this.
  7. micada

    micada Grand Pooh-Bah (3,960) Jul 13, 2015 New York
    BA4LYFE Society Pooh-Bah Trader

    Have to refresh pages three or four times.
     
    #7 micada, Sep 18, 2023
    jmdrpi, Shanex, mpruden and 2 others like this.
  8. GuyFawkes

    GuyFawkes Grand Pooh-Bah (5,630) Apr 7, 2011 Illinois
    BA4LYFE Society Pooh-Bah Trader

    Been pretty brutal, but I've been powering through, delivering the "likes" that my fellow Beer Advocates DESERVE!!

    :-)
     
    #8 GuyFawkes, Sep 18, 2023
    Rug, beergoot, eppCOS and 8 others like this.
  9. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    Woke up to the mods discussing this in Discord. Sounds like it was indeed brutal last night and probably the worst that it's been. It's very frustrating, but the DevOps team is investigating and I'll alert them about this last round of site interruptions.

    In the meantime, I've once again locked down search (ours is very resource intensive) and other pages (especially ones that bots/scrappers love) for logged in users only in order to help reserve resources for core users.

    Thanks for your patience, everyone.
     
    #9 Todd, Sep 18, 2023
    rclangk, Rug, russpowell and 41 others like this.
  10. PapaGoose03

    PapaGoose03 Grand High Pooh-Bah (6,057) May 30, 2005 Michigan
    BA4LYFE Society Pooh-Bah

    Things seem normal now. Thanks for your efforts, Todd and mods.
     
    #10 PapaGoose03, Sep 18, 2023
    Bitterbill, TCgenny, muchloveforhops3 and 5 others like this.
  11. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    They will for the most part, and regardless of my recent updates, but I'll be monitoring.
     
    #11 Todd, Sep 18, 2023
    Bitterbill, FBarber, TCgenny and 3 others like this.
  12. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    Adjusted our daily cron jobs to run a better times. Should help to smooth things out a bit.

    Speaking of which, it looks like at least one of the daily updates failed last night. Some data or lists will seem off until the next run.
     
    #12 Todd, Sep 18, 2023
    Bitterbill, FBarber, russpowell and 7 others like this.
  13. Whyteboar

    Whyteboar Grand Pooh-Bah (4,286) Jun 7, 2008 Michigan
    BA4LYFE Society Pooh-Bah Trader

    Oh man, been decades since I had to think bout CRON jobs. UNIX rocked though, good times.
     
    #13 Whyteboar, Sep 18, 2023
    Bitterbill, TCgenny, PapaGoose03 and 1 other person like this.
  14. Orca

    Orca Grand Pooh-Bah (4,710) Sep 18, 2010 Washington
    Pooh-Bah Trader

    This is kind of reminding me of when the IT guy tries to explain what’s wrong with my computer. I don’t understand or care, dude, please just fix it! :grin:

    Thanks Todd and mods for maintaining a (mostly) issue-free site.
     
    #14 Orca, Sep 18, 2023
    Bitterbill, ChicagoJ, JZH1000 and 5 others like this.
  15. hopley

    hopley Pooh-Bah (2,912) Feb 24, 2010 Massachusetts
    Society Pooh-Bah Trader

    FYI Todd - I'm not seeing any data for the different beer styles load fyi - i.e.

    Beers by Style - Bocks load fine so I know that I've review 7 out of 100 Dopplebocks on the list but Browns don't load...for American Brown Ales I've reviewed 0 beers, top 100 not achieved due to small data set...

    Same for all the rest with the exception of Wild / Sour Beers. Hope this helps...!
     
    #15 hopley, Sep 18, 2023
    Bitterbill, ChicagoJ, Rug and 5 others like this.
  16. jjamadorphd

    jjamadorphd Pooh-Bah (2,496) Jul 21, 2012 Florida
    BA4LYFE Society Pooh-Bah Trader

    i've been frequenting this site since 2012, first time EVER I have seen an outage such as last night. I would say that's purty darn good record - at least from my perspective. Keep up the good work!
     
    #16 jjamadorphd, Sep 18, 2023
    ChicagoJ, JZH1000, riptorn and 7 others like this.
  17. TCgenny

    TCgenny Grand Pooh-Bah (3,555) May 26, 2021 New York
    BA4LYFE Society Pooh-Bah Trader

    All is good, lol I was just… about… to have… a beer… last night
    I like to scope out WBAYD before having one, and alas I couldn’t get on.
    Plan B, paid attention to my wife and watched tv
    Thanks Obama!
     
    #17 TCgenny, Sep 18, 2023
    ChicagoJ, JZH1000, Todd and 2 others like this.
  18. BigIronH

    BigIronH Grand Pooh-Bah (3,762) Oct 31, 2019 Michigan
    Pooh-Bah Trader

    Never happened before except this morning.
     
    #18 BigIronH, Sep 19, 2023
    ChicagoJ, TCgenny and Todd like this.
  19. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    Top Rated Beers by Style are missing a bunch of lists, but it should be repopulated soon after 6 AM UTC.
     
    #19 Todd, Sep 19, 2023
    Rug, TCgenny, NorsemanOne and 2 others like this.
  20. Todd

    Todd Founder (13,518) Aug 23, 1996 Finland
    STAFF Mod Team Society Pooh-Bah

    From mine too, and especially when you consider the age of everything under the hood and how much duct tape has been applied over the decades.
     
    #20 Todd, Sep 19, 2023
    Bitterbill, ChicagoJ, Rug and 9 others like this.
Page 1 of 2
Thread Status:
Not open for further replies.