If you tried connecting to this blog in the last few days, you might have noticed some downtime…

This was caused by multiple screwups by my ISP and was prolonged by them not being able to provide proper phone support (“we can send someone over in the next few days”)…

The first thing that happened is that “a node very high up [in the network hierarchy]” crashed (I was unable to get more information than this), which apparently caused an outage for the whole town. The support person said that it might take up to a full day to fix the problem, at which point I decided to call it a night, amused that they don’t seem to have failovers for “stuff very high up”.

Luckily, I woke up to a working internet connection. But for some reason my monitoring was still mostly red and my servers unreachable. After checking everything from the servers up to the router provided by Vodafone, I saw the problem. The IP had changed.

That’s kind of weird, considering that we pay extra for a static one, so I called support once again. They were as perplexed as me, because apparently the static IP option “somehow vanished” (their words) from the contract.

So, I asked them to “re-add” the option (they said without further charges, but I doubt that), after which we got a totally different static IP (and curiously enough also a different way of getting the IP) and I had to configure everything from scratch.
After I followed the guide they mailed me and nothing worked, I called support a fourth time.

This time I asked if they could go through the documentation together with me to make sure that I didn’t miss anything. Apparently they don’t even have higher level phone support because the only solution they saw possible was to send someone over to help me.
Since I didn’t want to wait for “a few days” for some dude to come to my house only to double check some router settings I refused that and went on a multi-hour Google adventure trying to get the information I needed myself.

The solution (as always) was stupidly simple when you finally found the right ancient forum post: Configure everything statically, ignore the ISPs docs.

(Except the connection of the provided router of course)

In hindsight I should have ignored the docs in the first place. That’s what I usually do with things like that and I trusted the documentation just because I wanted it to work again as fast as possible, which probably cost me more time than necessary.

Let’s see how long things keep working until the next time something magically reconfigures itself on their end…

Update: Just as I suspected we were charged double for the new static IP addresses. But at this point I don’t really care as long as it works. We were also charged double for the first Month. Their accounting is about as good as their technology…