It’s not unusual for a service to need access to some kind of configuration, or for that configuration to contain passwords that you’d prefer not to leak. The solution is simple: create a configuration file somewhere, make sure that it’s only readable to the user the service runs at, and you’re done. And then you find out about DynamicUser services, which is where the fun begins.

Starting with the default case

Say for the sake of argument we have a very simple service, running as an unprivileged user, that depends on some password to interact with something else. We could provide that password on the command line and doing so could look something like this:

1
2
3
4
5
6
7

[Unit]
Description=Example service that depends on secrets

[Service]
Type=simple
User=something-unprivileged
ExecStart=/usr/bin/example-service --password=SECRET

Unfortunately this is a bad idea. On most systems, the command line for every running process is available to any user on the system, so finding out this password could be as simple as:

1
2
3

$ ps aux | grep example-service
somethi+     286  0.0  0.0  90076  5508 ?        Ssl  09:46   0:00 /usr/bin/example-service --password=SECRET
hacker     25285  0.0  0.0   6432  2236 pts/0    S+   15:51   0:00 grep example-service

So instead, you create a configuration file in /etc/example-service.conf that sets all the details that you care about:

1

password = "SECRET"

Then we set this file’s mode to 0600 (only readable by its owner) and then set its owner to something-unprivileged, and then we can adapt our service to read the configuration file instead, ending up with the following:

1
2
3
4
5
6
7

[Unit]
Description=Example service that depends on secrets

[Service]
Type=simple
User=something-unprivileged
ExecStart=/usr/bin/example-service --config-file=/etc/example-service.conf

This is fine, this is fairly secure, and works well enough. But we can do better.

Why you want to use DynamicUser=yes

In Unix-like systems, it’s become customary to use system users to separate privileges between different services. There’s nothing inherently different about system users, except that by convention they have a user ID < 1000. Of course you need to create a new user whenever you want to use a new service, but this is fairly simple. There just one catch: getting rid of a user is very impractical. Lennart Poettering explains it very well in his introduction to dynamic users but TL;DR: the user ID tends to stick around in every file the service touched.

The dynamic users concept in systemd addresses this shortcoming by having a user that only exists for the runtime of the service. Opting in to that is as simple as setting DynamicUser=yes in your service file. It also opts you into some other security features that are interesting but outside the scope of this article.

1
2
3
4
5
6
7

[Unit]
Description=Example service that depends on secrets

[Service]
Type=simple
DynamicUser=yes
ExecStart=/usr/bin/example-service --config-file=/etc/example-service.conf

As an added bonus you don’t even have to specify the name of the user any more. Whenever possible, a name will be generated based on the name of the service. If that’s not possible for some reason, one is generated using a hash of the service name. All files that are writeable by the service are contained in specific directories and their owner ID is patched shortly before starting up the service with whatever user ID happened to be selected.

You might see the problem coming here: how would we set the owner of the configuration file when the owner doesn’t even exist when the service isn’t running? The answer: we don’t. In the next two sections we will look at two possible workarounds.

Inserting environment variables

If we cannot make the configuration file readable for our application user, we can still make it readable for the root user and provide the secrets to the service in a different way. One way that systemd has supported this is through the EnvironmentFile directive. The trick in use there is that the environment file is read by systemd itself and therefore by root, before being passed as normal environment variables to the service itself. Working that into our service file, we get the following:

1
2
3
4
5
6
7
8

[Unit]
Description=Example service that depends on secrets

[Service]
Type=simple
DynamicUser=yes
EnvironmentFile=/etc/example-service.conf
ExecStart=/usr/bin/example-service

The environment file format is slightly different from our example above, so we will need to modify our example-service.conf from above slightly to fit the definition:

1

password="SECRET"

There, that’s it. There are caveats to using environment variables for your secrets, such as them implicitly being inherited, debug tools often automatically printing them, but in general it’s good enough assuming you can do all the configuration you need via world-readable files and environment variables.

Using systemd credentials

Not all software is flexible enough to deal with environment variables for their configuration in addition to configuration files. Unfortunately, that rules out the method described in the previous section for injecting secrets. So what can we do?

There is a weird kind of hack that you can still pull by copying the configuration into the unit’s state- or runtime directory and changing the owner using ExecStartPre commands. I call them hacks because that’s kind of what they are. But, starting with systemd v247, we no longer have to. The credentials subsystem is here to help.

The credentials subsystem hooks into lots of things, but of interest to us today is the LoadCredential directive. It allows you to mark a file as a credential to be loaded, which will then be put in some place accessible only to the service itself, in-memory only, and locked into RAM to prevent it from swapping out. Putting this into practice, that would look like this:

1
2
3
4
5
6
7
8

[Unit]
Description=Example service that depends on secrets

[Service]
Type=simple
DynamicUser=yes
LoadCredential=example-service.conf:/etc/example-service.conf
ExecStart=/usr/bin/example-service --config-file=${CREDENTIALS_DIRECTORY}/example-service.conf

Suddenly we’re almost back at our original service, but now the user the service runs as no longer matters for its access to the secrets at all. The real config file can be readable for root only and it will still work fine.

The credential is specified as [Some identifier]:[Path where it can be found]. The identifier defines where the secret will eventually be stored, relative to some magical credentials directory, and the path is simply the path to copy. This works for regular files, as with our example, but you can create more complex set ups with unix sockets to implement dynamically generated secrets. The credentials directory itself is automatically injected through a variable.

Putting this into production

I think the systemd credentials subsystem has some definite potential to allow easier implementation of dynamic users and related security measures in your service orchestration. And yet I have to be a bit of a downer. The feature was introduced in systemd v247, which was released November 2020. That is very new in terms of systemd features. In fact, when I first learned about this about a week ago the lack of articles about it prompted me to write this one. So it’s not really available in common Linux distributions. For a quick list:

  • Arch Linux: of course you can
  • CentOS: of course you cannot
  • Debian: Bullseye and up
  • Fedora: 34 and up
  • NixOS: You tell me
  • openSUSE: Tumbleweed only right now
  • Ubuntu: Hirsute Hippo and up

If you want some semblance of an LTS release, Debian is your only choice right now. Seeing as the production servers I manage largely run Ubuntu, I’ll be looking forward to Joyful Jellyfish or whatever 22.04 will be called. Until then, I’ll have to contain my excitement.