Aside from all the terrible things that are associated with the current year, it is also an even year, and it’s past April, so that means we got a new Ubuntu LTS release: Focal Fossa. A few weeks ago, after much anticipation, the upgrade path from Bionic was released which meant it was time to upgrade my VPS. In this post I’ll go over how it went, what went wrong, and what went right.

My VPS is hosted at TransIP, a smallish Dutch cloud- and hosting provider. It hosts a few applications that I’d like to keep functioning, but most importantly it is my email server. This means that I’d prefer to keep my downtime low. This reliability issue is exactly why Canonical waits until the first “point release” before pushing the new LTS to everyone, and this year they even waited two more months.

As an aside, I will acknowledge that I’m a slow writer and that Groovy Gorilla came out as I was writing this article. It’s not an LTS release so I don’t feel too bad about it.

The plan

As this is not exactly the first time I’ve upgraded a Ubuntu server to the next LTS version, I started out by laying out the steps I was roughly going to follow. I won’t go into too much detail for each step as there are plenty of wonderful guides out there that will tell you how to do these things.

  1. sudo do-release-upgrade
  2. Repair anything that’s unexpectedly broken
  3. Thoroughly check my mail server
  4. Ensure nginx uses the new php-fpm version
  5. Check all running web-apps
  6. Upgrade PostgreSQL
  7. Check all running apps again
  8. Celebrate

Now, I’ve worked with computers for long enough to know that this plan was not going to work out without issue. However, thanks to some precautions, everything mostly worked out in the end.

Precautions

Doing an upgrade like this is inherently risky as various bits can — and will — break. That means you need to have proper backups. My hosting provider has a rudimentary snapshot system that is a bit inconvenient to use if you’re used to the luxuries from AWS, but it does the job.

It’s also not recommended to perform the update over SSH, as there are many things that can potentially break up your connection and make it impossible to reconnect. apt will generally finish up whatever it’s doing without you present, but in the case of a release upgrade, user interaction is sometimes required and then it will lock up.

My hosting provider provides a web interface to a serial console which almost works well. ncurses doesn’t quite understand it though, so it gets the number of columns wrong and messes up certain dialogs in the installation process. This might also be the screen-session (started by do-release-upgrade) not dealing well with being nested in tmux. I didn’t diagnose further as I was more worried about my system making it through the update after my screen became a mess. Which it did, thank you for asking.

How it went

After preparing as much as I could, it was time to pull the trigger. And most of it worked fairly well, some things were subtly broken, and other things went up in flames. In the interest of documenting the issues and helping others with the same issues, let’s go over all of them.

What went terribly wrong

There is really only one thing that really messed up, is that my server’s internet connection broke halfway through the update. Upon further investigation, this was because the network reconfigured itself at some point, and DHCP no longer worked. I’m still not sure what broke this and whether it’s my ISP or the new version of netplan but something broke.

In the end, I duct taped over the problem by configuring the interface statically. This worked well enough, as it’s a server with a static IP address anyway.

Things that were subtly broken

When your software advances two years into the future, some things are bound to change and thus quite a few things didn’t work as well as they used to. This is a non-comprehensive list of the things I noticed.

PostSRSd didn’t start

The syntax for specifying different domains to skip rewriting changed from space-separated to comma-separated. The daemon refuses to start if the wrong format is used, with the following extremely helpful error message:

1

/var/lib/dpkg/info/postsrsd.config: 14: /etc/default/postsrsd: example.org: not found

Changing spaces to commas worked just fine.

OpenDKIM no longer signed emails

In a similar vein to PostSRSd, OpenDKIM no longer properly signed emails because I had multiple domains configured for signing with the same key. Its man-page is not as good as the one provided for PostSRSd, but luckily you can simply configure the main domain to be signed and enable the Subdomains option to sign the subdomains as well.

Dovecot requires DH parameters for SSL now

This one is fairly obvious, but it is important nonetheless. The newer version of Dovecot no longer worked with built-in Diffie-Hellman parameters and instead requires you to generate unique ones of your own. This is a good idea in general and a wake-up call for me that I had neglected to do so before. Helpfully, the error message tells you exactly what to do.

1
2
3
4

config: Warning: please set ssl_dh=</etc/dovecot/dh.pem
config: Warning: You can generate it with: dd if=/var/lib/dovecot/ssl-parameters.dat bs=1 skip=88 | openssl dhparam -inform der > /etc/dovecot/dh.pem
# Followed by a lot of the following:
imap-login: Error: Failed to initialize SSL server context: Can't load DH parameters: error:1408518A:SSL routines:ssl3_ctx_ctrl:dh key too small: user=<>, rip=127.0.0.1, lip=127.0.0.1, session=<dQw4w9WgXcQ>

Unattended upgrades has lots of new options…

…which is useful but also means that you need to go over the options. I do recommend doing so as it includes some nice new tweaks.

Lots of software waiting to be removed

Due to the messy interactions between old dependencies and new dependencies, some package may get installed or not properly removed when upgrading. This upgrade was no exception, although the list was actually fairly short for me:

  • apache2 got installed somehow during the upgrade process. This causes fun times because I actually run nginx and they compete for ports.
  • php7.2 left some traces around.
  • python3.6 remained partially installed despite Ubuntu moving to 3.8 for the rest of the distro.
  • Finally, PostgreSQL always leaves its old version behind, but that’s a feature, not a bug, so you can more easily migrate between them.

In conclusion

The Ubuntu upgrade process is still a bit hairy. There’s just a lot of big updates happening all at once, which makes it difficult to determine what exactly caused what problem. The number of packages is simply too large to look over.

That said, it has been reasonably comfortable, and setting aside a weekend ever two years to do so is not too much trouble. Now it’s time to let this rest and wait for the next animal out there, which should start with a “J” if I can count. Joyful Jellyfish anyone?

Oh, and if you’re like me and you’re wondering what a fossa is and what they look like, it’s a small carnivorous kind of ferret cat which coincidentally is its literal name in Dutch, and they look like this:

What a Fossa looks like

I think they’re cute.