status.metafilter.com

Network maintenance

The colocation facility where the MeFi servers are sitting is undergoing maintenance right now, so the site is unreachable until they are done (should be 10-15min).

update: ooof, we're at 45 minutes of downtime and the colo just sent me the following message:

We apologize for the inconvenience, but the server is currently being monitored as down due to network maintenance being performed on equipment connected upstream from your server. We will inform you once the maintenance has been completed. We appreciate your patience, and hope to resolve the issue as quickly as possible.

And, a rocky day to follow

We spent most of the day working on uptime issues, trying out new settings and trying out different tweaks to the server, but in the end we're about where we were yesterday, with server restarts every hour or so (kicked off automatically when there are memory leaks/crashes). There were a few hours of strangeness on the front page of MeFi not showing all posts, but that has been fixed.

We'll be working on server issues likely next week as well, but hopefully things improve as traffic goes down over the weekend.

Oops, rocky night

For the past 24hrs or so, we've been testing out some monitoring software to get a better handle on what is causing some intermittent crashes, but it seems like the added load of the monitoring got the better of the server, causing several crashes per hour for the last 12 hours. We've rolled things back to how they were before the night began, so things should get a bit better today.

Getting better...

We're pretty much done with the integration of the basics on the new servers, and everything is running a lot better. We have a couple runaway processes we're keeping an eye on, but for the first time in a couple weeks, the server is stable for 12+ hours in a row.

The last things left to do are the following, and we hope to bring them all back next week:

• The internal search engine for all sites
• Recent flickr photos, recent twitter posts, recent last.fm music

In the home stretch

After some minor tweaks, we've had the first consistent super fast uptime in almost two weeks, and the new server is online and kicking well.

Tonight we'll bring down the site for about an hour to do one last software upgrade, then we'll be set for a while.

2nd try on moving to the new MeFi server

Ok, I've just thrown the DNS switch on the server server, so in the next 20min or so, most browsers should be loading things off the new server.

We think the problem last time was with a bad private network connection to the database, which caused extremely slow loading of data that bogged the server down. We think we fixed it, so we're throwing the switch and moving to the new hardware.

MeFi down for a few

We're updating some network drivers on the new database server, so for the next few minutes MeFi will be down while we fix that.

update: aaaand, a few minutes later we're back.

One step back

I've reverted DNS to the old server, at 74.53.68.130 and restored posting capabilities to it. It should update for most people within the hour. The old web server is running off the new database hardware, so we'll essentially be going to back to how things were a few days ago (with some slowdowns every few hours),while we work on the new web server.

As soon as the new server comes back online and is looking good, I'll move DNS to the new server and we'll give this another try.

Some apache refactoring...

We eventually figured out the problem seems to be with an unsupported version of apache we've been running on Fuelly for the past six months, but is causing problems on the MeFi box. We're first going to revert to a 32-bit official 2.2 version, and if that doesn't solve the issues, we'll move back to the 2.0 versions that have worked fine on the old MetaFilter server for years.

Sloooooooow

So things were going swimmingly for the first few hours on the new server, but as the east coast hit the office, some slowdowns began and we're currently digging in to figure out what is causing the bottleneck. The servers appear to be idling along with lots of extra processor power and memory so it's just a matter of hunting down what is hanging up the operation and causing time-outs and pageloads to take up to a minute instead of milliseconds.