You are at netAirspace : Forum : NOTAMs - Site Related Forums : About netAirspace

You may well have noticed that netAirspace was offline for several hours today. I believe that the outage lasted roughly 10 hours.

The trigger for all this was a power failure in our hoster's datacentre. At the moment I don't know how long it was out for.

When power was restored, our server came up cleanly but certain configurations had been lost. It appears that this is the reason that the outage was so long.

Chief among those configurations were the firewall rules allowing incoming web traffic.

The nightly scheduled tasks (counter resets, etc.) did not run, but with no traffic there were no views to count anyway... I triggered those tasks manually, and think I know why they were not run automatically. It seems that the cause of that is also responsible for the default font appearing in photo copyright bars; that will also fix itself once the underlying cause is removed.

You'll see other weirdness as we work to recover from this, so I apologise in advance.

At some point we will want to reboot the server intentionally to ensure that we have learned all the lessons from this incident. We'll let you know about that in advance.

Press on. I logged on at 0400 (Chicagoland time) and got the message "This Page Cannot Be Displayed".

Figured you'd handle it. I don't think we pay you enough.

ShanwickOceanic wrote:The trigger for all this was a power failure in our hoster's datacentre.

Surprised they don't have a UPS set-up?

I didn't actually notice the outage myself but got a couple of messages about it on facebook. Bit of a shame as it will put something of a dampener on what has been a fantastic month stats-wise, but these thing can't be helped I suppose.

JLAmber wrote:Surprised they don't have a UPS set-up?

This is the core of a lengthy conversation between me and the hoster, that has slowly reached 'nastygram' status by now.

Their initial narrative was: "Hey, we had a power outage, but things were restored within 2 hours and now everything smells of roses again!"

Since our server did not nearly behave as expected, I grew suspicious and asked for shutdown protocols. Which a good UPS should produce.

Surprise, surprise: they did not exist.

They then told me that the problem was caused by a faulty distributor that was located between the UPS and our server.

But since it was not just our server, but an entire colocation center that went down, I really begin to wonder about how they have their infrastructure set up.

I now suspect that there was a surge, and that our server received some excess voltage. I will certainly escalate this further.

JLAmber wrote:Bit of a shame as it will put something of a dampener on what has been a fantastic month stats-wise, but these thing can't be helped I suppose.

Stats must have been very good indeed, lot of shots in the screening queue and when looking at my views I'm amazed to see how it progressed (even if I uploaded more shots recently)...

All right, it looks like I've chased down everything that affects you guys, although there's still some work to do on the server-admin side. Once I've nailed that down, I'll find a quiet time to babysit this thing through a reboot and make sure everything comes back up as it should.

You'll possibly still see the odd broken image, or one with a boring font in the copyright bar, but those should go away in the early hours of Friday morning European time. Otherwise, everything should be OK.

Well, I thought that, then woke up to find baby-blue pages all over everywhere. May be coincidence, may be the hoster's idea of an April Fool's prank, but best guess is that it's some residual fallout that took some time to shake itself out.

I think I've fixed it, and it seems to have had no ill effect, but still not the sort of thing I enjoy waking up to!

If you happen to see a page with a light blue background, grab its URL and PM me.

ShanwickOceanic wrote:If you happen to see a page with a light blue background, grab its URL and PM me.

Saw one on my phone last night but it's back to normal now