Cloud maintenance

So it turns out that having unlimited storage creates unforeseen issues for gc routines in Nextcloud.

One of the wonderful features of our cloud software is the ability to track recent revisions and deletions. That is, you decide to edit a file, say a picture. It keeps track of one version in the last second, one version every two seconds for the last 10 seconds, for the last minute one version every 10 seconds, for the last hour one version every minute, for the last 24 hours one version every hour, for the last 30 days one version every day, beyond the last 30 days one version every week. So, if you had a file being edited every second, a log file for example, you might have 170 versions after one year.

Disc icon from Nuvola icon theme for KDE 3.x., via Commons.Wikimedia.Org

And, if you delete a file, the last version is kept in an archive for at least 30 days, then deleted on an as-needed basis.

But clean-up of these copies and archives is apparently triggered by using more than 50% of your allotted storage. So if you have, for example, 100 GB, the software ignore cleaning up until your total of files, versions, and archives reaches 50 GB. Then the garbage collection maintenance tasks would permanently delete the oldest deletions and versions until your total is less than 50%.

So how much is 50% of unlimited?

Well, it should be 50% of the total available storage for the system. The flexibility of Nextcloud, however, allows any user to add remote drives to their account, which themselves might have further remote storage. So the system does not track the total available storage (which can change with every attempt to measure.)

That meant versions and deletions continued to accumulate since this system was moved to this hardware, eventually filling all storage.

Which is a very long-winded way of getting around to saying: I have manually run a purge of deletion archives and versions, freeing up more than 5 TB of storage. I am also limiting all accounts to one degree or another, to help the cloud maintenance routine avoid storing too many versions and deletions. Sorry if you lost some recently deleted files/versions of things.