Update: BeerAdvocate Severs & Intermittent 50x Errors

Some of you may have experienced a "50x error" here or there. It's us, not you.

And I believe that I've finally tracked down the issues.

1. Sometimes one of our many Content Delivery Network (CDN) nodes will get overloaded for a region, which creates some disruption in services until the CDN handles the issue. This is out of our hands, but it's usually resolved quickly.
2. Although we upgraded our web servers, we're still having some intermittent issues during high traffic/loads. After much research and trail and error with settings, I think I've found the solution and applied the changes moments ago. Mutex? Semaphores? Anyone? After much digging, I discovered that it was a known Apache issue and there was a fix for our specific platform.
3. We've outgrown our database server, and if you apply the above it certainly doesn't help. After we host American Craft Beer Fest we'll begin the process of upgrading our database server to a monster server.

I'll continue to monitor the situation, and keep everyone posted with our progress, but I'm very confident that recent updates and a future upgrade will create a much more stable environment.

Cheers!

 

So after we sorted out the above web server issue, I discovered another set of issues.

1. Our database settings we're not optimal, and arguably too aggressive.
2. Our old code wasn't scaling well with our ever increasing traffic.
3. One of our top used database tables had yet to be converted to a more efficient engine type.

And if all 3 met, our resources could get eaten up within minutes.

So this past week I've been researching and applying fixes, which explains the disruption in services; service restarts were an unfortunate necessity. And yesterday our host's support team confirmed the underlying issues and assisted as much as they could.

Good news. Today I ...

• Applied new settings; so far, so good.
• Started the process of updating old code, and with each update things become more efficient.
• Converted that highly used table to a better engine type for increased stability.

I still have more work and monitoring to do, but it's time for a beer now.

 

I'm still optimizing, however, I'm stoked to report that our database server is vastly more stable than it was a week ago.

More updates to follow ...

 

Early this morning (3am MDT) I made some significant updates to several highly used database tables, as well as some supporting code. I'm extremely happy with the increased performance and stability.

Back to more optimizing ... It's addictive.