Internet, we had a problem.
This post summarizes everything that was written to the temporary Netlify version of this site the two weeks it was offline from August 21 to September 3. Updates are posted in chronological order, with the most recent updates on the bottom.
www.vegard.net is offline due hardware failure. I’m not sure if and when it’ll eventually be back online.
Update @ 20:38 CET
VBOX4 is dead. It started to log CPU errors this morning, and locked up a couple of times before it went belly up for the last time. The CPU is integrated, I’ve tried to track down an identical replacement motherboard, but that turned out to be a dead end.
Instead of spending time and money on building a new VBOX, I’ve purchased a second hand Raspberry Pi 4B 4GB that is more than powerful enough to host the site. A VPS might have been an option, too, but I enjoy hosting the hardware myself.
Expect a couple of weeks downtime. The Pi has to arrive, and I have to restore everything from backups. In the meantime, you can browse what once was with the help of the Internet Archive mirror I’ve been maintaining.
Some internal services are back up and running on a Pi that’s usually dedicated for Pi-hole. It could probably have handled the load of the entire site, and since I got the internal stuff online again, getting the rest isn’t that much work. I’m not sure if I want to spend time on setting up everything twice, however, both on this Pi and the one that’ll arrive some time next week.
The second hand Pi I purchased could, for some reason, not be delivered to my door by the courier service. Now it’s bouncing around in their system. I’ve got a feeling this can take a while.
It’s taking a while.
The Pi has arrived. It’s a miracle.
I’ve started the process of rebuilding the site on the 5th incarnation of VBOX. Me from 10 years ago - or perhaps even 6 months ago - would have devoted all his spare time to get everything online again as quickly as possible. Me as of right now? Not so much.
Update @ 21:42 CET
I watched two episodes of Better Call Saul instead.
I’ve salvaged a ton of files from one of the SSD disks that was in the RAID1 array on VBOX4. I didn’t actually need the files form the disk, because everything was backed up every night, and, yes, I check by backups - at least semi-regularly. But it still saved me some time.
The SSD drive has been repartitioned and reformatted, and now is connected to the Pi with a USB to SATA cable, and will be doing all the heavy storage lifting. It will serve the Git repositories, databases, everything that Apache serves, home folders, etc.
The next step now is to move the internal services I use regularly from the their temporary home at the Pi-hole Pi to their new VBOX5-home. But that’s an adventure for another day, now I’m going for a walk instead.
I got my second covid shot yesterday, so today I’ve been drifting in and out of sleep. In the evening, I felt better, and managed to move the internal services from the temporary Pi, get Jenkins installed, and a completed a few other minor steps towards profit. This evening I’ve learned that there is a special place in hell reserved for whoever decided that Snap was the package manager of choice for Ubuntu. Also, AppArmor should be shot straight into the sun.
Turns out using Hugo installed from the Snap repo with Jenkins is fucking impossible when your home directories isn’t mounted on the usual /home path. Had to build Hugo from source for ARM instead because the Pi comes with an ARM CPU and there is no Hugo extended binary available, which I need for SCSS. I’ve shared three bottles of wine with the kindergarten dads, not sure if I should do this now. The potential for catastrophe is quite high with sudo powers. But I’m too worked up to go to sleep.
Update @ 23:23 CET
Hey, look at that. The Hugo site got built and published successfully by Jenkins. Turns out three bottles of wine isn’t such a bad idea after all. None of the links work, and the stylesheets are borked, but that’s because Hugo is configured with www.vegard.net as its root domain, and that’s currently hosted by Netlify to make sure you can read what you’re reading now.
Anyway, this is a positive development. I might be able to get everything back to where it was prior to VBOX4’s hardware failure during the weekend.
Exactly two weeks after VBOX4 crashed and burned spectacularly, VBOX5 is now online with all feature and all services that was previously running on VBOX4. Two weeks downtime is pretty miserable for a self-proclaimed nerd, but that’s where we ended up. The main thing to take away from this is that no-one died.
Go give someone you love a hug.