While it may not always feel like it, two years have passed and and a new Ubuntu release somehow has rolled around. That means it’s time for me to upgrade my VPS again. Contrary to last round, the first point release actually lined up with my planned time off quite well, so I got to upgrade straight away. It went quite smoothly this time, but I’d like to go over the peculiarities that happened.

And as an aside before we actually get into my points, I’d like to boast that I got the animal codename right two years ago while reviewing Focal. Joyful would have been so much better than Jammy though.

With that out of the way, let’s go over the major breakages that happened. The methodology hasn’t really changed since last-time so I won’t go over it again. We simply sudo do-release-upgrade and follow where it leads us.

Roundcube decides to uninstall itself

Roundcube is an easy-to-install, PHP-based webmail client that I use to host some mail boxes for some non-technical people that still don’t want to have a random gmail address for their event.

The problem that occurred was that Roundcube decided to uninstall itself during the initial package installation section of the update. I did spot this in advance, but do-release-upgrade doesn’t easily let you change its plans. I simply made sure that the database configuration remained intact and reinstalled it afterwards.

The upgrade of the configuration was slightly painful, since they finally decided to move to the new1 PHP array syntax. There very few actual differences to merge, but every single array in the file had to be updated, resulting dozens of minor diffs to be inspected. At least they probably won’t move back.

PostgreSQL now defaults to scram-sha256 authentication

The new default configuration for PostgreSQL no longer allows md5 password authentication, instead opting for the scram-sha256 method. This is in many ways a great move, except that it doesn’t work with users that were created with MD5-hashed passwords. In other words, none of your existing users can still log in to the database. There are two solutions to this problem:

  1. You update all the users to have a scram-hashed password, or
  2. You restore MD5 functionality to your database login

The first one is obviously the proper solution, but it also is more work. When your system is not in a working state, you might not want to go the long way around. Restoring MD5 login support is as simple as updating the pg_hba.conf file for your cluster and restarting it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Database administrative login by Unix domain socket
local   all             postgres                                peer

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

# IPv4 local connections:
host    all             all             127.0.0.1/32            scram-sha-256
# IPv6 local connections:
host    all             all             ::1/128                 scram-sha-256

# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            scram-sha-256
host    replication     all             ::1/128                 scram-sha-256

N.B. it is important that you put the md5 entries above the scram-sha-256 entries. Otherwise, logins will still be disabled and your logs will show the following message:

1
2

2022-08-17 11:52:06.725 CEST [9648] your_user@your_db DETAIL:  User "your_user" does not have a valid SCRAM secret.
    Connection matched pg_hba.conf line 99: "host    all             all             ::1/128                 scram-sha-256"

Missing OpenDMARC schema upgrade

In a previous post, I’ve talked about how to set up OpenDMARC to validate incoming mail for authenticity.. The authentication part survived the upgrade just fine, with some minor refactoring of the configuration files. The reporting bit of our Domain-based Authentication, Reporting, and Conformance did not.

DMARC has over the years gotten a new mechanism, ARC, which among other things allows for mailing lists to actually pass DMARC verification.2 Support in OpenDMARC has now landed in Ubuntu, which is great, but it also requires some schema changes in the reporting. Unfortunately, these schema changes are not currently implemented. This Debian issue tracks the missing migration but nothing has been implemented yet. This is reasonable as it’s relatively easy to work around (see below) but it does result in in error messages like this:

1
2
3
4
5

/etc/cron.daily/dmarc-reports:
opendmarc-import: failed to create table ID: Field 'repuri' doesn't have a default value
opendmarc-import: failed to insert message: Unknown column 'arc' in 'field list'
opendmarc-reports: can't extract report for domain example.org: Unknown column 'messages.arc' in 'field list'
opendmarc-expire: DELETE failed: Table 'opendmarc.arcauthresults' doesn't exist

It should be possible to manually alter the schema to reflect the new requirements, but since there is not too much value in historical data , I’ve instead:

  1. created a second database, opendmarc_jammy, from the stock schema script3,
  2. granted the existing opendmarc user access to this database, and
  3. updated the reporting script to use this database instead.

On the bright side, due to updates on the side of MariaDB, the schema no longer requires any modifications and you can simply execute the existing script as-is after changing the database name.

Minor repeat offenders

Not everything can go smoothly, but all in all most issues I had last time simply were fixed. Still, for posterity, here are the things that are still a bit annoying:

  • Apache still decides to install itself, even though you’re using Nginx. This one I saw coming, so I preemptively masked the systemd unit so it couldn’t be started and wouldn’t fight Nginx for control over the HTTP(s) ports.

  • The PHP FPM socket path still includes the PHP version, so you’ll need to update any (Nginx) configuration that refers to it. The PHP configuration in general is messy because Ubuntu installs a different set per PHP version. To me this means that I just have to bite the bullet and put my configuration in Ansible so I can quickly reinstall it.

  • Diffing changed configuration files is still more painful than it needs to be. It’s a bit jarring that half the packages handle it through a simple shell prompt while others go through a fancy curses GUI. Neither of which can hold a candle to pacdiff which actually helps you resolve the conflicts.

In conclusion

As mentioned in the introduction, I think the upgrade as smooth as it has ever been. There were some minor speed bumps, but all in all things appear to work as intended.

Really the only casualty is Selfoss. I’ve been using it as my RSS reader of choice for the past few years, and now it has finally broken. This is apparently already fixed, but it’s still waiting to be included in a release. I can probably hold my breath for that.

So it seems we will be the next two years jellyfishing. The new PHP 8.1 seems like a big performance improvement, the new systemd features look nice, OpenSSL 3.0 is a lot more secure, PostgreSQL 14 brings some minor convenience, and finally zstd can be used here and there as an upgrade over xz. This is also the first LTS release that gets the phased updates which I’m not sure about. It will probably turn our fine, and if not, there’s always room for 24.04: Nifty Nightingale.

  1. It’s not exactly new any more; it was introduced in PHP 5.4, but many projects haven’t changed for backwards compatibility. The main things is that $array = array(1, 2, 3); can be shortened to $array = [1, 2,3];. ↩︎

  2. ARC provides a mechanism where any relaying email server can securely claim “Yes, I verified this before changing it.” I really should do an article in the future that explains how SPF, DKIM, DMARC, and now ARC work together to make email slightly less insecure ↩︎

  3. The schema can be found in /usr/share/dbconfig-common/data/opendmarc/install/mysql, managed by Debian’s dbconfig. ↩︎