At some point, not today, I should blog about the multiple sites we are operating here. Today, though, I am staving off a sysadmin disaster.
As usual, at an inopportune moment I decided to upgrade the server operating system from one long-term service version to another long-term service version. I had a few years I could have stayed, but there were some related upgrades – specifically getting php onto version 7.2.x, of which I wanted to take advantage. And, as usual, Ubuntu broke stuff. In this particular case, libraries for multi-byte strings, a certain encryption library, and the cURL module. Without these my highest-priority service, cloud.saewyc.ca, is mostly dead in the water.
Which was bad enough I dropped everything and attempted to fix it. And it looked like I had; I found the appropriate entries in the repos, installed via dpkg… and Apache was inexplicably masked.
Having one site effectively offline is bad. Having 14 down because the server is unstartable is disastrous.
That, by the way, was Friday. On Sunday, having purchased a mega disc and backing up the essentials one
damneddarned site at a time, I gave in and re-installed the new system. This did not work.
Yes, yes, it has to work, but it did not. Basically, I was unable to get OpenSSH running properly using TaskSel on Ubuntu 18.0.4.x. So I degraded to 16.0.4 again. Nice clean install, things are going along well, I had opted for self-selected software because I really liked how MariaDB worked and wanted to use it again. I went to harden the db server and…
MariaDB is supposed to ship with a root user installed without a password. One of the first things you do is harden it up by removing all the various accesses, and set a good password on root. This install apparently set a password without telling me what it was, and it was none of the usual suspects.
After futzing with that for a couple hours I gave up, uninstall/purge, replace with the far more ubiquitous MySQL… which refuses to install because a more-recent version had previously been installed. This is a ‘feature’ of the war between MySQL and MariaDB in the open source world.
I reinstalled the server software. Nice clean install…
And, actually, from this point forward things have been progressing as they should. I pull the database from the backup, import to the spiffy-clean db server, go knit for an hour or two… come back to set up the vhosts, security certificates, et voilà! the site re-appears!
I am leaving the more-troublesome sites which needed some TLC before this event for last. Hopefully I can get the big ones done today, and the littler/newer projects with less content and history (all I have to say is: 2.6 TB of data in the cloud, with versioning going back 5 years…) should go faster.