Late in November and into the first week of December, in between the deadlines for the magazine, I had a client site go belly up unexpectedly in the wee hours of the night. We’ll call it a “Website Doom”.
My conversation with the server and web browser went something like this with a strong coffee in hand:
Browser: “There is no website, are you crazy?”
Me: “No, look again. There is a website. I have not imagined this client for the last three years. Here, try this recovered backup from 11 pm last night, since you’re not switching to the failover.”
Server: “yeah, nah – I’m not sure I have it in me this morning to start up. It’s been a very rough coupla years y’know” (and don’t we all just have a second of completely identifying with this poor server yeah?)
Me: “c’mon you bugger, just friggin startup.”
Browser: “oh , you mean that website – why didn’t you say so – but your ssl is clearly fraudulent”
Me: “It’s not. Try again.” (hissing through clenched teeth)
Browser: “oh all right – but what is this, amateur hour or what? Because there’s way too many redirects for me to serve this content, which way am I going?”
Me: “Duck sake just serve the damn content please” (now yelling at the laptop)
Browser:
Me: “that’s it I’m done with you, moving to another browser, one that doesn’t hoard cookies like gollum in a secret stash, that will be my main browser from now on”.
Browser: “ok, ok, no need to be rash and make rushed decisions – here’s the site you want we’re all good. What a beautiful SSL it is as well, well done! Thanks for playing “Website Doom – The end of year edition. Just don’t switch to another browser yeah?”
Turns out – on deeper investigation of the logs that I was dealing with an introduced plugin conflict that caused a memory leak, and a rather aggressive brute force hacking attempt. All at once. Things got ugly quickly after that.
And still I managed to keep the site up, and running from that point on for the next 7 days while I tracked and mitigated hits, then checked and rechecked, and tweaked fail2ban, the CDN and firewall settings, and finally got it to the point that I didn’t feel the need to check it 5 times a day. It has not gone down since. So much for the Website Doom! Ha!
Start a website business they said.
Pfft.
It’s so easy they said.
Pffft.
