As those who follow our twitter feed are already aware, we had a planned outage recently that took quite a bit longer to complete than hoped.

The short version, for those interested:

• Our hosting facility needed a power upgrade to meet updated building codes
• When the power came back online, we discovered that a PMU on our rack was marginal, preventing hosts from coming back online
• Once this was fixed, there were some networking issues (firewalls and routers not keeping pre-power-down state)
• And our NAS volume had bad RAM that kept the main server’s filesystem from mounting

Our host was very professional and worked hard to restore service lost as a result of these unfortunate events, and is working with us on compensation for the downtime.

My apologies for the extended outage; watch here and the Engine Room for updates in the near future that should limit the impact of future outages.