Since I discovered BorgBackup
(short: Borg) I have felt way more confident
about my backups. Borg gives you a space efficient storage of backups, data
is saved encrypted, and you get good versioning of backups. I solved all my
previous inconveniences when just using rsync. I lacked the way to keep
track and order different versions. Compression and encryption as a great
plus even when I just keep my backups locally.
I have felt a huge safety net concerning my personal devices backups and my
my cloud services backups. With such confidence, today I was crazy and
decided to dump my VPS and reinstall it from scratch for an upgrade. After
a full day of dealing with installation it just didn’t work, server was not
online. I could not get redis server to run on my openvz instance, because
my service provider just had an old kernel on my host machine and it
wouldn’t work with the latest version of Debian stretch(I upgraded from
Debian Jessie, which provided old enough packages for everything to work).
Having suffered a lot from my openvz VPS, because it had an old kernel and
I cannot put swap memory into it, I had had enough. Cheap does not
compensate for the frustration. I thus moved to a KVM, and everything went
smoothly regarding installation, I could use my Ansible playbook from the
sovereign
project with some of my tweaks. Almost everything worked with
little extra effort. Or maybe I was just in the mindset of fixing stuff and
dealing with the problems of the openvz server, that setting the new server
felt easy.
Not everything is wonderful when doing a migration, I did hit some
setbacks. I changed my infrastructure, service provider, server location. I
had to update my DNS configuration in my nameserver, which took a long
while. It is also a manual process, you must change the DNS config via a
web interface from the service provider. I would be awesome if I could have
Ansible change that configuration as well. Ansible is amazing, it
configured my VPS faster than DNS would update and that was an issue with
getting my Let’s encrypt certificates.
This is where having a backup saved my first, I could not get new SSL
certificates from Let’s encrypt because my new server did not responded to
the queries. The DNS still pointed to old server. But, why get new
certificates? I can use the old signed ones that are in the backup. Just a
simple manual copy and done.
I did experience some other few obstacles regarding the
configuration. Specially since I also upgraded to Debian 9 which is not yet
fully supported by the sovereign project. Some pull requests might end
there too.
When I was restoring nextcloud. I had done it before for a private server
from a customer. But when going through my own backups I could not find the
database backups. All files where there, except the database. The backup
script was indeed doing the database dump to a file, but Borg was never
instructed to back it up.
What is lost? not much really. I’m still the single user of my nextcloud
instance and I have multiple copies of the files, locally and in the
backup. Thus I only uploaded the files and had nextcloud index all of them
again
. I did loose the history and changes of this files. Yes, the backup
does have the multiple versions as nextcloud saves them, but I just feel is
to complicated to try to recover this functionality without the information
from the database and for things I care about versions I use a proper
version control system. Honestly, I have never tried to recover a earlier
versions of my files using nextcloud or its previous iteration owncloud.
Lessons learned:
- Test your backups. Is not enough to have them, you have to test they
work. A new task for me would be to test: how fast can I restore from
total disaster to a new working server? Now it looks pretty bad, about 20
hours. That is keeping in mind I had hope on the openvz working.
- Don’t dump a service before the replacement is working. This is kind of
obvious but since I’m the single user and I could afford downtime on this
and dedicate time until everything is fixed I took my risks on it.