Down time
Today we experienced a significant outage of approximately 8 hours. Which kinda sucks.
What happened…
Following an upgrade of the proxy server we discovered a significant number of webserver configuration files were failing to load, and the type of failure suggested filesystem corruption. An attempt to repair them in situ resulted in most of the file system structure being blanked.
A sector-by-sector recovery attempt also failed; there wasn't much hope for it in the first place as this was a microsd.
The microsd was reflashed and the server reconstructed from scratch, with new certificates and manually created virtualhosts, which was more time consuming than difficult. And that is why we lost hours of uptime.