
Good afternoon folks,
As of about fifteen minutes about (1:25 PM EST) server6 became unavailable, we’ve got our crew investigating and additional updates will be available shortly. You can check for them here.
Update: System is now back online as of 1:41 PM EST, we’ll keep a close eye on it and post a full review of what occured here shortly. We’ve narrowed it down to a specific user & we’re seeing what we can do to mitigate the cpu usage in the meantime.
All sorted!
The hardware gods hate us. We’re currently investigating once again, we brought the system down as it spiked suddenly to 250+ odd load making it unmanageable. We’ve got some piece of bad hardware in it, it’s just a question of what.
We’ve got a replacement system coming in shortly, however in the meantime we’re working on replacing whichever piece of this one needs sorted.
p.s., just glanced at the calendar and realized it was a Monday – no surprise there.
Update: 4:54 PM EST – The system is online in some capacity, email, ftp and http should function fine but I don’t expect it to remain swell until we have the hardware investigated more thoroughly. We’re scheduling a couple of quick replacements — initially, just the RAID card as it does not appear to want to properly initialize. Hopefully that will calm things down.
Final update, for now: Things have been solid since the the system came online at 4:54 PM EST so we’re going to consider this resolved in the meantime. The raid array is reinitializing which means that there will be some slight latency but nothing visible to end-users.
Today’s earlier issue still seems 100% entirely hardware related not being aided much by obscene amounts of cpu & resources being allocated to certain sites on the system and spammers / comment spammers in full swing today. As noted earlier in the post we’ll be fully switching this server out for a replacement come about Wednesday, more information will be sent out to you via email. The new system is quite a bit more potent and has faster drives which will offset some of the IO overall. The switch will be seamless & we’ll keep in full touch during the migration.
For now, enjoy your evening.
I still cannot access my site at all. Hopfully this will be resolved quickly as I have some things to get done before I leave for a week tomorrow.
Hurry Please!
![]()
We’re always in a hurry Billy.
![]()
Thanks for keeping us posted David. Annoying (the server issues, not your blog posts
) but just one of those things.
Fingers crossed for a swift resolution.
Kind regards
John
I’m crossing fingers, as well.
The way email kept connecting, then not connecting, felt more like a software (system) problem then a hardware problem….
Well — hardware problems do frequently cause software related problems, especially when it’s something drive related. In this case the RAID card is now reinitializing, so as long as it rebuilds there should be no further problems.
We never should have put any core2quads online — both of the ones we have in our shared fleet are showing signs of age already.
Any additional info? As of 3:40 Pacific all my stuff is still down.
Thanks
Brian,
I’ve just sent you an email, I’m afraid yours is entirely unrelated to this issue — sorry!
Any updates?
All of our stuff is down, 4 PM Pacific.
(I think ours is related to this server6 issue)
John,
I can see everything loading on this end, if you can’t — please fire me a ticket.
I suspect it’s firewall related.
Is server6 still having issues? my website is running up and down or sometimes slow
![]()
Hit or miss here. Still getting timeouts or ‘No Database’ error messages at 7:00pm Pacific (10:00 Eastern).
Would the RAID rebuild still be working away?
Tony & Robert,
Not sure if you two got our email blast about 25 minutes ago — mentioned we’d be replacing the RAM in the system for a brief 5 minutes. If you’re not the primary contacts on the account, check with them as they’ll have the full details.
It’s all back online now though.
Thanks David.
I am still at work chasing down a dying server of my own and haven’t been keeping an eye on e-mail for the last couple of hours. Sure enough, you guys are on top of it and had I looked I’d have known what was going on.
Keep on fighting the good fight! Good luck.
As of 7:45 Pacific this morning, email at all our offices are down again.
Website is up.
Hi Guys,
As of 9.03 (Central African Time) this morning our mail server is still not working. Will this problem persist?I have sent test mail to our numerous mail accounts and nothing is being received. I have also attempted to send mail and nothing…
We’re at a crucial point for our business here – I’m awaiting emails that could mean a lot of money to us. Please help.
Rolf
Rolf,
As I noted in the ticket response, your local pc’s are accessing the wrong server so you’re “looking” in the wrong spot, per se. You’ll need to flush your dns cache by performing:
1. start, run, type: cmd
2. ipconfig /flushdns
3. try reloading the mailIf that doesn’t work, please respond to the ticket you have open and we’ll dig deeper — but the issue in this case is on your end.
Subscribe to the blog