Upgrading to Ubuntu 22.04 "Jammy Jellyfish"
While it may not always feel like it, two years have passed and and a new Ubuntu release somehow has rolled around. That means it’s time for me to upgrade my VPS again. Contrary to last round, the first point release actually lined up with my planned time off quite well, so I got to upgrade straight away. It went quite smoothly this time, but I’d like to go over the peculiarities that happened.
And as an aside before we actually get into my points, I’d like to boast that I got the animal codename right two years ago while reviewing Focal. Joyful would have been so much better than Jammy though.
With that out of the way, let’s go over the major breakages that happened. The methodology hasn’t
really changed since last-time so I won’t go over it again. We simply
do-release-upgrade and follow where it leads us.
Roundcube decides to uninstall itself
Roundcube is an easy-to-install, PHP-based webmail client that I use to host some mail boxes for some non-technical people that still don’t want to have a random gmail address for their event.
The problem that occurred was that Roundcube decided to uninstall itself during the initial package
installation section of the update. I did spot this in advance, but
easily let you change its plans. I simply made sure that the database configuration remained intact
and reinstalled it afterwards.
The upgrade of the configuration was slightly painful, since they finally decided to move to the new1 PHP array syntax. There very few actual differences to merge, but every single array in the file had to be updated, resulting dozens of minor diffs to be inspected. At least they probably won’t move back.
PostgreSQL now defaults to
The new default configuration for PostgreSQL no longer allows
md5 password authentication, instead
opting for the
scram-sha256 method. This is in many ways a great move, except that it doesn’t work
with users that were created with MD5-hashed passwords. In other words, none of your existing users
can still log in to the database. There are two solutions to this problem:
- You update all the users to have a
scram-hashed password, or
- You restore MD5 functionality to your database login
The first one is obviously the proper solution, but it also is more work. When your system is not in
a working state, you might not want to go the long way around. Restoring MD5 login support is as
simple as updating the
pg_hba.conf file for your cluster and restarting it:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Database administrative login by Unix domain socket local all postgres peer # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all peer # IPv4 local connections: host all all 127.0.0.1/32 md5 # IPv6 local connections: host all all ::1/128 md5 # IPv4 local connections: host all all 127.0.0.1/32 scram-sha-256 # IPv6 local connections: host all all ::1/128 scram-sha-256 # Allow replication connections from localhost, by a user with the # replication privilege. local replication all peer host replication all 127.0.0.1/32 scram-sha-256 host replication all ::1/128 scram-sha-256
N.B. it is important that you put the
md5 entries above the
Otherwise, logins will still be disabled and your logs will show the following message:
1 2 2022-08-17 11:52:06.725 CEST  your_user@your_db DETAIL: User "your_user" does not have a valid SCRAM secret. Connection matched pg_hba.conf line 99: "host all all ::1/128 scram-sha-256"
Missing OpenDMARC schema upgrade
In a previous post, I’ve talked about how to set up OpenDMARC to validate incoming mail for authenticity.. The authentication part survived the upgrade just fine, with some minor refactoring of the configuration files. The reporting bit of our Domain-based Authentication, Reporting, and Conformance did not.
DMARC has over the years gotten a new mechanism, ARC, which among other things allows for mailing lists to actually pass DMARC verification.2 Support in OpenDMARC has now landed in Ubuntu, which is great, but it also requires some schema changes in the reporting. Unfortunately, these schema changes are not currently implemented. This Debian issue tracks the missing migration but nothing has been implemented yet. This is reasonable as it’s relatively easy to work around (see below) but it does result in in error messages like this:
1 2 3 4 5 /etc/cron.daily/dmarc-reports: opendmarc-import: failed to create table ID: Field 'repuri' doesn't have a default value opendmarc-import: failed to insert message: Unknown column 'arc' in 'field list' opendmarc-reports: can't extract report for domain example.org: Unknown column 'messages.arc' in 'field list' opendmarc-expire: DELETE failed: Table 'opendmarc.arcauthresults' doesn't exist
It should be possible to manually alter the schema to reflect the new requirements, but since there is not too much value in historical data , I’ve instead:
- created a second database,
opendmarc_jammy, from the stock schema script3,
- granted the existing
opendmarcuser access to this database, and
- updated the reporting script to use this database instead.
On the bright side, due to updates on the side of MariaDB, the schema no longer requires any modifications and you can simply execute the existing script as-is after changing the database name.
Minor repeat offenders
Not everything can go smoothly, but all in all most issues I had last time simply were fixed. Still, for posterity, here are the things that are still a bit annoying:
Apache still decides to install itself, even though you’re using Nginx. This one I saw coming, so I preemptively masked the systemd unit so it couldn’t be started and wouldn’t fight Nginx for control over the HTTP(s) ports.
The PHP FPM socket path still includes the PHP version, so you’ll need to update any (Nginx) configuration that refers to it. The PHP configuration in general is messy because Ubuntu installs a different set per PHP version. To me this means that I just have to bite the bullet and put my configuration in Ansible so I can quickly reinstall it.
Diffing changed configuration files is still more painful than it needs to be. It’s a bit jarring that half the packages handle it through a simple shell prompt while others go through a fancy curses GUI. Neither of which can hold a candle to
pacdiffwhich actually helps you resolve the conflicts.
As mentioned in the introduction, I think the upgrade as smooth as it has ever been. There were some minor speed bumps, but all in all things appear to work as intended.
Really the only casualty is Selfoss. I’ve been using it as my RSS reader of choice for the past few years, and now it has finally broken. This is apparently already fixed, but it’s still waiting to be included in a release. I can probably hold my breath for that.
So it seems we will be the next two years jellyfishing. The new PHP 8.1 seems like a big performance
improvement, the new systemd features look nice, OpenSSL 3.0 is a lot more secure, PostgreSQL 14
brings some minor convenience, and finally
zstd can be used here and there as an upgrade over
xz. This is also the first LTS release that gets the phased updates which I’m not sure about. It
will probably turn our fine, and if not, there’s always room for 24.04: Nifty Nightingale.
It’s not exactly new any more; it was introduced in PHP 5.4, but many projects haven’t changed for backwards compatibility. The main things is that
$array = array(1, 2, 3);can be shortened to
$array = [1, 2,3];. ↩︎
ARC provides a mechanism where any relaying email server can securely claim “Yes, I verified this before changing it.” I really should do an article in the future that explains how SPF, DKIM, DMARC, and now ARC work together to make email slightly less insecure ↩︎
The schema can be found in
/usr/share/dbconfig-common/data/opendmarc/install/mysql, managed by Debian’s