Just as I was about to post the new podcast, both servers went down, and we can't seem to bring them back. I'm talking to the hosting company right now trying to get to the bottom of it, but it seems like a network/infrastructure issue at the moment that is hopefully resolved soon. In the meantime, MeFi is totally offline.
update: and we're back, after about 15min of downtime. Checking the logs to see what's up but it appears to be a temporary network issue.
Posted at 11:28 AM | Permalink
The colocation facility where the MeFi servers are sitting is undergoing maintenance right now, so the site is unreachable until they are done (should be 10-15min).
update: ooof, we're at 45 minutes of downtime and the colo just sent me the following message:
We apologize for the inconvenience, but the server is currently being monitored as down due to network maintenance being performed on equipment connected upstream from your server. We will inform you once the maintenance has been completed. We appreciate your patience, and hope to resolve the issue as quickly as possible.
Posted at 10:49 PM | Permalink
We spent most of the day working on uptime issues, trying out new settings and trying out different tweaks to the server, but in the end we're about where we were yesterday, with server restarts every hour or so (kicked off automatically when there are memory leaks/crashes). There were a few hours of strangeness on the front page of MeFi not showing all posts, but that has been fixed.
We'll be working on server issues likely next week as well, but hopefully things improve as traffic goes down over the weekend.
Posted at 03:26 PM | Permalink
For the past 24hrs or so, we've been testing out some monitoring software to get a better handle on what is causing some intermittent crashes, but it seems like the added load of the monitoring got the better of the server, causing several crashes per hour for the last 12 hours. We've rolled things back to how they were before the night began, so things should get a bit better today.
Posted at 08:25 AM | Permalink
We're pretty much done with the integration of the basics on the new servers, and everything is running a lot better. We have a couple runaway processes we're keeping an eye on, but for the first time in a couple weeks, the server is stable for 12+ hours in a row.
The last things left to do are the following, and we hope to bring them all back next week:
Posted at 07:49 PM | Permalink
After some minor tweaks, we've had the first consistent super fast uptime in almost two weeks, and the new server is online and kicking well.
Tonight we'll bring down the site for about an hour to do one last software upgrade, then we'll be set for a while.
Posted at 07:56 PM | Permalink
Ok, I've just thrown the DNS switch on the server server, so in the next 20min or so, most browsers should be loading things off the new server.
We think the problem last time was with a bad private network connection to the database, which caused extremely slow loading of data that bogged the server down. We think we fixed it, so we're throwing the switch and moving to the new hardware.
Posted at 02:23 PM | Permalink
We're updating some network drivers on the new database server, so for the next few minutes MeFi will be down while we fix that.
update: aaaand, a few minutes later we're back.
Posted at 01:22 PM | Permalink
I've reverted DNS to the old server, at 74.53.68.130 and restored posting capabilities to it. It should update for most people within the hour. The old web server is running off the new database hardware, so we'll essentially be going to back to how things were a few days ago (with some slowdowns every few hours),while we work on the new web server.
As soon as the new server comes back online and is looking good, I'll move DNS to the new server and we'll give this another try.
Posted at 09:17 AM | Permalink
We eventually figured out the problem seems to be with an unsupported version of apache we've been running on Fuelly for the past six months, but is causing problems on the MeFi box. We're first going to revert to a 32-bit official 2.2 version, and if that doesn't solve the issues, we'll move back to the 2.0 versions that have worked fine on the old MetaFilter server for years.
Posted at 08:35 AM | Permalink
