A Freaky Friday (Or The Day Our Blog Went Offline)

There are some days when everything happens which is opposite to what was expected and in a not so good way.

Friday 17 September was one such day.This was the day when this blog went offline for a while.

Here is how it all began :

We could no longer login to admin panel for ihaveapc.com or see the home page.

The first thing that we thought was that our ISP was having problems as it happens at times, so we kicked in the redundant internet connection hoping that it was really an ISP specific issue.

Close but no cigar.

Turns out that this redundant ISP too couldn’t really reach ihaveapc.com most of the times or extremely slow loading it.

This got us all pumped up (Fridays are usually boring especially the evening time :))

Next we ran a network trace (Start > Run >cmd and tracert in Windows or mtr / traceroute in Linux Mint) to find out what was messed up.The first few hops showed that packets were being sent ok from our ISP end but the last hop (the one which hits the domain provider/web host) showed something funny. The message was of a network interface error (1332) to be precise on the web host’s end.

What this meant was that the request to access the blog was being sent correctly but the destination end was all messed up resulting in site time outs.

This made things a bit clear, so immediately opened up a support ticket with our web host Go Daddy mentioning the error messages being displayed.

Meanwhile, connectivity to the site was intermittent or non-existent at times.The reply received from web host support was a fun one – basically it said that they are aware of the issue which is on their end but couldn’t really provide a time line on when it will be fixed.

Fast forward a few hours later, the connectivity seemed better but not exactly reliable.Submitted another support ticket, the reply this time was much more specific -  basically a malicious javascript had injected into php code for sites running WordPress and had escalated the attack to about 150 sites or so but the threat was now removed.The support staff recommended to empty all cache and update plug-ins to latest version (we already keep them updated).

By the way the web host’s security team did a good and quick job cleaning it up considering how messy such attacks can get in a very short period of time.

We still were having problems reaching the site as well as the admin panel.Logged on to a different machine and fortunately got hold of the admin panel, cleared all the cache and tried again.

Hallelujah – it worked and we were finally back to normal.

Moral of the story :

1. Make sure if the web site problem is on your end or not before escalating to your web host.(A relatively simple way to check if a website is down or not is mentioned in this article).

2. Always make regular backups.

3. Never panic – it is not the end of the world. [Unless it is 2012 and the world is really going to end 😉 ]

4. Try to gather information if anyone else is facing similar problems, usually the best source for this is the web hosting provider’s site or a simple web search.

5. Be better prepared next time 😉 [This means keeping latest backups at hand and ready to use plus estimating the time needed to restore the site from it].