Things slightly better

It's been a few days, and while response times on metafilter aren't perfect (still way too slow in the daytime under loads, and jrun timeouts here and there), things are much better after a server check revealed that one jvm setting was the culprit killing the server every few hours for the past couple weeks. I had followed the advice of other coldfusion techs in setting up some extra flags on the java service, one of which was tuned to a dual processor machine, spreading the load out of lost threads to a second processor, which wasn't there. As a result, the server would eventually overload the memory with running threads no longer used until the whole thing would die.

So that's fixed, and Ask Metafilter's front page as well as all three sections' comment pages have been drastically streamlined, with shared queries stored in memory cache. Basically this results in faster pages when connections are established. I had to tune down the maximum number of connections while troubleshooting, and I've seen periods where I'm sitting for 20 seconds waiting to connect to mefi, then BAM! a page loads instantly. So fast when you can get to it, but too slow waiting to get there.

I've got rewritten homepages in the works for metafilter and metatalk that should reduce the load considerably and they'll hopefully be up this weekend. I'll also tweak the maximum concurrent connection limits to see if queue time can be reduced.

If by the end of June things aren't accpetably better, I'll look into beefing up the web server box to something dual processor, though I fear more web connections will overload the separate db server, and I'm not quite ready to move to db server clustering just yet.

Also, I am going to use this site more. I haven't posted anything since Feb and tons of things have changed.

Advertisements