A test of our emergency broadcast system
We’re currently testing the system to see where this goes. Float!
Server5 has suddenly become unavailable and is being powercycled by the datacenter. We’ll have a complete investigation being emailed to any clients affected. The system appears to have locked up as of 7:43 AM EST.
I suspect the worst as far as hardware is concerned but it could be mere software glitches or an abusive user causing the issue.
Edit & Update: As of 8:02 AM EST the system is now fully available, we’ll be sending out a detailed email when the system has been fully audited.
De ja vu: March 4th, same time. The particular user responsible is being suspended on a permanent basis & this won’t reoccur, hopefully. We’ll bring in an additional system administrator to audit the server just in case there’s something we’re overlooking.
System should be back online momentarily.
As of 11:25 AM EST on Saturday, server4 & server8 became unavailable along with a few other systems in the datacenter. We’re currently investigating and having the on-site folks take a peek. We’ll have more updates available momentarily.
Update: It appears that everything is online however some users may be experiencing some routing issues. The network engineers are sorting the issue out and it does appear to be occurring with a significant number of ISPs that I’ve personally tested it from.
Update @ 12:20 PM EST: It appears the issue is the result of one of the facilities core distribution switches going offline leaving the remaining switch to cope with a significant amount of load. A large number of our systems aren’t effected, roughly about 43 clients are at this time (out of 1,200). If you’re affected an email has been sent out to you regarding the issue.
We’re awaiting an update on an ETA of the switch being replaced, we’ll post it here the instant it becomes available.
Update @ 1:02 PM EST: Everything is now back online, the distribution switch disappearing was supposed to automatically failover to the secondary however that didn’t occur. The total downtime was one hour and thirty four minutes.
Subscribe to the blog