Sites should be down for a bit, starting... now.
update: Everything should be back! If you spot any problems, drop it here.
2nd update: Crap, we did something wrong that needs to be completely reconfigured, so you might see another 30-40min of downtime at 1pm
3rd update: Ok, we're done for the day and everything is (seemingly) back to normal.
Friday, April 18, we'll be doing some server maintenance between 10am and 2pm Pacific Time. If all goes well, downtime will be on the order of 20-30minutes, at worst, several hours. We'll post updates here during that time and keep everyone up to date on the progress.
While trying to fix some minor networking bug, the database server went unreachable after a reboot, so my ISP is walking over to kick the box in the next few minutes. Hopefully the downtime won't be too long, but MeFi has been down about 15 minutes now and will likely be unavailable for at least another 15 or so.
Update (25 min later): we're back online!
MetaFilter appears to be down, but unfortunately, the problem is out of my hands as it is a network issue on my ISP's side. Here's their word on it:
Dear Customer,
We are currently experiencing a network interruption that is affecting a a wide section of The Planet's network.
Our network engineers are diligently working to resolve the issue and we will update you with additional information shortly
Thank you for your patience.
Thank you,
LaurenceS
Technical Support
www.theplanet.com
UPDATE: We're back online, looks like it was about 2-3 hours of downtime total.
Things are back up, and kind of slow. I had to take two steps back due to an internal java freakout that couldn't be recovered from. I'll be tweaking small things for the rest of the week and leaving the big stuff until late nights when fewer people are online.
Sorry about this and thanks for bearing with me.
Well, sorry to say but while tweaking some settings, the server crashed pretty hard. I'm putting things back together but I don't know how long it'll take. Could be 5 minutes, could be hours at this point.
Doing some more server upgrades, which should make MeFi unavailable or throwing errors for the next hour or so.
I'm updating the server now, so there should be about 30 min of downtime starting....now.
While playing around with 427kb UI scripts, I figured it was time to institute some form of compression on the server, so the latest ajax features didn't require a simple webpage to blossom into a 1/2 megabyte download.
It was pretty simple to do on windows and I based most of the rules on this post. My 600kb test page went immediately below 200kb by simply compressing the javascript and css files. RSS files are also being compressed and I've noticed the site has seemed a bit more perky since I enabled it. Bandwidth isn't a problem (my current host allows for 4Tb of data monthly), but it will be nice if the site can be a little bit faster thanks to compressed downloads.
Christ on a crutch, was that ever a bad server update. The long story short is that uninstalling MS SQL Server 2005 (32-bit) and installing 64-bit versions over it is nearly impossible, requiring a dozens hand edits of the registry and all sorts of phantom processes.
What was supposed to take 20 minutes took over two hours, but we're back online and running. Uptime should be steady and constant after today, going forward thanks to all the tweaks, patches, backups, and monitoring we've setup.
