My fellow maintainer Quentin Michaud already did a nice write-up about how the malware worked, so I won’t go into detail too much about that. If you want to know more about that, read his blog. Instead, I’d like to do a crash course on how these packaging scripts work, and how you would review them yourself.

What is the AUR?

The Arch User Repository is a collection of packaging scripts, PKGBUILD files, created by users. Anyone who creates an account on aur.archlinux.org, can upload an Arch Linux packaging script, granted one doesn’t already exist for the same name. There are of course some rules around what you can submit (e.g. don’t duplicate other official or AUR packages) but generally anything goes.

Each package has one primary maintainer, by default whoever uploaded the packaging script first, but that can change over time, either by that maintainer transferring responsibility themselves, or by moderators removing the maintainer for various reasons. This is not a democracy, or even a meritocracy, but it works, and there is a lot of useful software on there.

Installing from AUR packaging scripts is not quite as simple as installing from the main packaging repos. You can run makepkg on a PKGBUILD and install the resulting package, but this gets more difficult as those PKGBUILDs often depend on other packages only found in the AUR. To make this easier, people often use AUR-helpers, which provide a pacman-like experience to manage them. That does come with some drawbacks, however.

Anyone who wants to do so can upload their PKGBUILD to the AUR. That is great to lower the barrier to entry, and it is how I got my start contributing to Arch Linux. Unfortunately, not everyone has the best intentions, and it has happened that people upload malware. Because of this, it is crucial that you vet the PKGBUILDs that you install.

What do PKGBUILDs look like?

As mentioned before, the AUR does not contain packages, but rather contains build scripts to create packages. Arch Linux PKGBUILDs are simply bash scripts that follow a certain pattern. Take for example the following example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Maintainer: John Doe <john@example.org>
pkgname=example
pkgver=1.0
pkgrel=1
pkgdesc="An example package"
arch=(x86_64)
url="https://example.org/"
license=('WTFPL')
install="example.install"
source=("https://example.org/$pkgname-$pkgver.tar.gz"
        "$pkgname-$pkgver.patch::https://git.example.org/example/pull/42.patch")
sha256sums=("de4bba005e86b4fbf0399e2cb492b1e941099b951a45137212507c2cbc8fae63"
            "b6dc933311bc2357cc5fc636a4dbe41a01b7a33b583d043a7f870f3440697e27")

prepare() {
    cd "$pkgname-$pkgver"
    patch -p1 -i "$srcdir/$pkgname-$pkgver.patch"
}

build() {
    cd "$pkgname-$pkgver"
    ./configure --prefix=/usr
    make
}

check() {
    cd "$pkgname-$pkgver"
    make -k check
}

package() {
    cd "$pkgname-$pkgver"
    make DESTDIR="$pkgdir/" install
}

It starts with a maintainer-line, which is not used programmatically but helps users of the software contact the maintainer if something doesn’t work. It is good practice to also have the emails of previous maintainers there, to recognise their contributions. Then we come to a few metadata variables.

Metadata

pkgname

defines the package we are building at the moment.

pkgver

signifies which upstream release of the software is being built. This number should only ever increase, as pacman will use this number to determine if an update is necessary.¹

pkgrel

is the package release number. This release number is incremented every time an update is done to the PKGBUILD without updating the pkgver, such as when the packaging options change. It resets to 1 every time a new upstream release is done.

pkgdesc

is a short, human-readable description of the package. This is shown in package lists and is used for searching, but doesn’t affect the built software otherwise.

arch

is a list of valid CPU architectures this package can be built for. It can also be the special array ('any'), which does not (only) denote that this package can be built for any architecture, but rather that the resulting package does not contain any architecture-specific code. This is common for packages written in languages such as Python.

url

is just link a package user might go for more information and does not otherwise affect the package.

license

is the licence under which this package is distributed. This used to be an array of all applicable licences for the project, but since the meaning of that is a bit ambiguous, these days the licence array should hold a single item which is an SPDX license expression.

install

is usually not present in a PKGBUILD. If present, it refers to the name of a script (included alongside the package file) with specific bash functions that will get executed when the package is installed, upgraded, or removed. It is run from pacman, and therefore as root.

source

is an array of sources. These can be either downloadable URLs, version control URLs such as git, or simple file names, meaning the file in question in part of the package files.

sha256sums

is another array, of the same length as sources, specifying the expected SHA-256 checksums for the package sources. There are other hashing algorithms available, from crc32sums to b2sums. Multiple sets of digests may be specified but at least one should be. The magic value SKIP may be used to skip verifying checksums for certain files, for example for external checksums, or PGP signatures.

After all that metadata, we get to the actual code that does something: the packaging functions.

Build functions

Building packages is done in 4 stages, each with their own bash function. It starts with prepare(). This function is expected to make all the necessary changes to the source code, such as applying patch files or editing out certain lines with sed. In certain ecosystems, such as Node.js and Rust, it also downloads the dependencies. Ideally, all steps following prepare() no longer need network access.

Then we get to build(), which, as the name suggests, builds the package. This is the place to call configure scripts, compilers, and whatever else produces the binaries involved. This is, generally, the part that takes time.

After packaging, there is usually a check() step. Here we want to check whether the package functions as intended, within the Arch Linux packaging. What to do here varies, but most PKGBUILDs try to run the package’s unit tests. Arch Linux generally has never versions of libraries than the ones upstream projects test with, and such tests form a reasonable smoke test that things did not break in unexpected ways.

Finally, the package() function is called. This is the only function that’s technically required, all the others can be omitted. This function should move the files, generated in the build() step, into their final positions relative to some $pkgdir. Many ecosystems provide (an equivalent of) make install for this, which makes life easier, but often you will simply have to spell this out.

What to look out for

Now that we know what the different parts of a PKGBUILD mean, we can talk about reviewing its contents. I want to go over a few of the items I find most important, but this is by no means an exhaustive list. The fact I’m writing this will probably affect the strategies people use in some way. Nevertheless, let’s fist…

Check the sources array

Regardless of what the PKGBUILD itself says, if the sources it’s going to compile are malicious, nothing else really matters. Make sure that you trust the upstream project, and that files are downloaded from a reasonable place, such as tags in official git repositories, or source releases on the official website. PGP signatures from the authors are generally even better, so you know who created, or at least signed off on the sources.

The Arch Linux project has accepted an RFC on what sources should be used, in order to have the most trustworthy sources for packaging. While such rigour isn’t necessary or, in some cases, even desirable for the AUR, you can use it as inspiration.

Patches are also sources, and malware can hide in them. You should have a look at any patches being applied to the source code, and, if possible where they come from. It’s not unusual to apply merged-but-not-released changes from upstream to deal with newer versions of libraries, but make sure it makes sense.

Check if the build steps make sense

The prepare(), build(), check(), and package() functions are inherently code that will be executed on your machine when you build a package so if any

• No downloads should happen in build(), check(), or package(), and ideally not in prepare(), though many ecosystems (Go, Node.js, Rust to name a few) require it.

• The commands run by the script should be obvious packaging commands, and any custom scripts should come from the upstream repos. Often the packaging script matches the upstream “how to install” but often it doesn’t, as package maintainers generally call the build tools directly rather than call wrapper scripts.

• Build scripts should not run sudo or anything similar. If it does that anyway, it’s wrong. At best, it’s a packaging error, as sudo shouldn’t be expected to work in a non-interactive environment like a build chroot. Sometimes a packager mistakenly tries to move package files into place instead of adding them to the package.

Scrutinise install scripts

Install scripts are, as mentioned above, rarely seen, but if when they are, they need to be scrutinised as their contents will run, as root, when a package is installed, upgraded, or removed. Most software doesn’t need this, as common use cases have safer alternatives.

Similarly, newly installed pacman hooks should also raise an eyebrow. Pacman hooks install to /usr/share/libalpm/hooks (and sometimes to /etc/pacman.d/hooks though that’s incorrect) and can run a command whenever its triggers are hit. The ones included in the main Arch repos are benign and in fact reduce the need for install scripts, by automatically regenerating font- and icon caches as needed, regenerating the initramfs, and many more. An arbitrary PKGBUILD adding one of their own is unusual, and you should pay attention to what that hook is trying to do.

If you don’t understand it, don’t use it

While we try to keep it clean, the AUR is freely accessible to anyone who figures out how to get past the CAPTCHA, and anyone can upload code to it. Even popular, well-intentioned code can have bugs sometimes and do harm to your system. The AUR is, at the best of times, maintained by volunteers, many of which are currently getting their feet wet for the first time while packaging. Mistakes can and will happen, and that’s okay.

When something might be malicious

Now that we have a hunch as to what might be malicious, we can decide what to do with it. The easiest way is to simply ask. The best place for that, in my opinion, is the #archlinux-aur IRC channel on Libera, which is the official venue for AUR discussion. The forums are also a good place to ask. If all fails, there is also the mailing list.

If the package is indeed malicious, one of the Package Maintainers can take it down and block the offending user(s). Should they try to avoid their ban, the measures will also escalate. As a user, that’s where your input ends.

That feels too simple

And it is. The AUR was made for a different kind of internet. More cooperative, less hostile. It’s based on trust that users will generally do sensible things.² The AUR is old software. It started as “just some FTP server” where people could upload files. Such a system would not be designed in 2026.

There are improvements to be had for sure. The moderation systems are somewhat archaic, the contribution workflow is not what people expect. The licensing of PKGBUILD files, especially when the original author is long gone, is tricky.³ A pull-request like system is often suggested. I think that would be a real improvement. In general, the system needs more love. That’s what you get with open source projects; if no one wants to work on it, it gets no work. And that’s okay.

Right now I believe there is no end in sight for the AUR. We can keep going as-is for a while longer. But if people were to step up to make things better, we can do a lot more.

Except the random person in Nebraska has been replaced by AWS, Google Cloud, or whoever caused the outage this week. And that kind of bothers me, because it misses the point entirely.

Cloud providers aren’t small indie companies

The original comic is a joke and expression of concern at the fact that lots of our modern technology depends on small projects that largely are maintained by a single driven developer writing code in their spare time. They are important, yet fragile. This is not comparable to the outages we’ve seen this year.

To contrast, these four cloud providers are, for better or worse, important to the web as we know it. But they’re not small. We should recognize that these are huge players, with revenues larger than the GDP of many countries.¹ Cloudflare isn’t anywhere near as big as the other three, but it still has a proportionally gigantic impact on the web due to how much data flows through them.

In addition to how important they are, they are all also among the largest and most valuable companies in the world. It’s concerning how reliant we are on just this handful of players, and when governments become more reliant on them, that is a huge risk. It is however the same, boring risk of influence and dependence it always is with large companies, rather than a risk of single individuals disappearing and taking our technology with them.

Support your guy in Nebraska

There are many people who are effectively a one-man show supporting most of modern technology in their spare time without compensation, and those deserve our support as a society, or at the very least recognition. I’ve tried to highlight a few here.

• The comic waves at ImageMagick, which was created by John Cristy while working at DuPont. This is really the only fact I can find about it. Every article delving into the history of ImageMagick mentions this and nothing else. I don’t know much about John Cristy, so I can’t comment on him. However, ImageMagick is currently maintained by Dirk Lemstra, who lives in the Netherlands. He does appear to do this all in his spare time.

• Finnish developer Lasse Collin is the main force behind the XZ compression library. Unfortunately, he is more famous for the social engineering attack on him than he is for his work. Regardless, XZ is an essential piece of software for many ecosystems that require high compression rate regardless of CPU cost. It has been the Arch Linux package compression algorithm of choice for years until it was replaced with zstd.

• cURL developer and AI slop victim Daniel Stenberg has surprisingly beaten the odds and made a career out of maintaining his open source project, but that doesn’t make him less admirable. Did I recommend his blog enough yet? He lives in Stockholm, Sweden.

• The late Bram Moolenaar is known for two things: the VIM text editor, and his support for orphaned children in Uganda. The latter of the two has long been the first message you see when opening the former. He lived in Lisse, the Netherlands, and the editor he created continues to be one of the most important tool on the belt of system administrators everywhere.

This is not meant to be an exhaustive list, though if I’ve overlooked someone significant (perhaps actually in Nebraska?) then don’t take that as me disregarding their contributions. There are many more, but the very nature of who they are prevents listing them all. I might update this list for a while, so do get in touch.²

Update November 26th

I asked and people delivered. Thank you to all of you who reached out with helpful suggestions. The following maintainers also deserve our recognition:

• Nick Wellnhofer has been maintaining libxml2 for years, while it is part of basically every functional Linux, MacOS, or even Windows installation out there. Recently he wrote about the demands put upon the project by these bigger organisations, and how it affects his work. He lives in München, Germany.

• The privilege escalation program sudo, present on almost every Linux system, has been maintained by Todd C. Miller for more than 30 years, both as part of his job and as a volunteer. His Github profile picture is the XKCD about sudo, which made me giggle. He lives in Boulder, Colorado, United States.

And finally, I’d like to give an honourable mention for the SQLite team. They don’t technically qualify under my rules, but they’ve been suggested more than once and it’s definitely an underappreciated piece of software. Curiously, they do not even take contributions, made their own version control system, and release the code as public domain rather than one of the more common licenses.

What next?

Relevant XKCD comics are nothing new to the field. When I was still in university I was educated³ by my friends on the ones that apply most often. I have since become part of the problem and have been making references to it with others. In a way it’s a rite of passage for a certain kind of geek. So here I am trying to educate the next generation on what things mean.

On the bright side: some people on Reddit do appear to “get it” and point out the sheer ridiculousness of implying the cloud providers are tiny unassuming parts. Others have instead taken the joke and ran with it, off of a cliff, to the point where it no longer makes sense. By the time I finish writing this, it’s probably already old and not funny any more. Nature is healing, I suppose.

What is an edition?

Rust has, at the time when 1.0 was being finalized, made a promise: code that compiles with any 1.x version of the compiler, should compile without hassle with a later 1.y version of the compiler. Unless behaviour was obviously a bug, or code should keep compiling. This is a beautiful ideal, but it also limits what changes you can make. This is where editions come in.

An edition in Rust is effectively a compatibility mode for the compiler. It is a way to make certain changes “not exist” for older code, while allowing improvements for newer code. Code compiled in older editions can be freely mixed with code with newer editions and vice versa. This makes it easier to upgrade to newer compiler versions, as you can be generally sure that code will continue to compile and work as expected.

As a simple example, the first ever edition, in 2018, made async a keyword. That meant that variables could no longer be called async, but that was deemed worth it in order to enable asynchronous programming. Crucially, code that existed before that time, could now use what is called “edition 2015”, could continue to exist unchanged and new code could depend on existing libraries, while using newer features itself.

Of course, that means that code compiled with edition 2015 could not use async fn, but that functionality wouldn’t be released until late 2019 anyway. Edition 2024 offers a similar selection of small changes, fixing small pain points in the language that would otherwise break the backwards compatibility promise.

Core language

Most of the changes in Edition 2024 affect the core language constructs and as such will be the most visible, as they apply regardless of what standard library features you use. Luckily, only one is likely to require manual changes to your code base, if at all. Let’s walk through them.

Return-position impl Trait lifetime capture rules

Putting impl Trait in the position of the return type can make your code easier to read, but it can cause issues when lifetimes get involved.

1
2
3
fn numbers(nums: &[i32]) -> impl Iterator<Item=i32> {
  nums.iter().copied()
}

The above will fail to compile in edition 2021, as the type impl Iterator is assumed to be 'static, that is, it doesn’t borrow anything, while it instead borrows from the nums slice.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
   Compiling playground v0.0.1 (/playground)
error[E0700]: hidden type for `impl Iterator<Item = i32>` captures lifetime that does not appear in bounds
 --> src/lib.rs:2:3
  |
1 | fn numbers(nums: &[i32]) -> impl Iterator<Item=i32> {
  |                  ------     ----------------------- opaque type defined here
  |                  |
  |                  hidden type `Copied<std::slice::Iter<'_, i32>>` captures the anonymous lifetime defined here
2 |   nums.iter().copied()
  |   ^^^^^^^^^^^^^^^^^^^^
  |
help: add a `use<...>` bound to explicitly capture `'_`
  |
1 | fn numbers(nums: &[i32]) -> impl Iterator<Item=i32> + use<'_> {
  |                                                     +++++++++

The error message even has the fix: add + use<'_> to state that the lifetime of the slice is captured into the returned iterator, so the compiler may correctly reason about its use.

The change coming in edition 2024 is a change in default captures. Instead of capturing none of the lifetimes by default, impl Trait will now by default capture all lifetimes. That means that with edition 2024, the example above will compile as-is. To revert to the previous behaviour, you can explicitly capture no lifetimes:

1
2
3
4
fn display(label: &str, ret: impl Sized) -> impl Sized + use<> {
    println!("{label}");
    ret
}

I’m not sure whether this change is an improvement; in my code it empirically introduces more unnecessary lifetimes than it removes workarounds. 'static lifetime by default seems like the more sensible choice. Nevertheless, the change has been made, so

References:

• Edition guide - RPIT capture rules

Changes to temporary scope

Edition 2024 comes with two small quality of life changes with respect to the lifetime of specific temporaries. These are situations you might have hit already, that have easy workarounds, but now they just work out of the box.

In edition 2021, when using if let, any temporaries created in the matching expression will live for the entirety of the if/else if/else branch. This can have some unexpected results:

1
2
3
4
5
6
7
8
9
fn get_cached_or_init(cache: &RefCell<Option<String>>) -> String {
    if let Some(value) = cache.borrow() {
        value.clone()
    } else {
        let value = complicated_string_gen();
        *cache.borrow_mut() = value.clone(); // PANIC! Already borrowed
        value
    }
}

Starting edition 2024, the temporary scope for the guard expression will end at the end of the if branch, and the above code will be valid as-is. If you need the guard to remain valid for the entire duration, you can use a match block instead.

Another small change has to do with the lifetime of temporaries in tail expressions. According to the 2021 temporary scope rules, a temporary created inside such a tail expression will live until the end of the next temporary scope, which is after local variables have been dropped. This can cause unexpected errors:

1
2
3
4
5
// Before 2024
fn f() -> usize {
    let c = RefCell::new("..");
    c.borrow().len() // error[E0597]: `c` does not live long enough
}

Example taken from the edition guide as I have hit this only once, ever, and couldn’t think of a simple use case. Nevertheless, the Borrow temporary generated by borrow() is here expected to live until the end of the function, but c is cleaned up before that, so it doesn’t work. In edition 2024, the temporary’s lifetime is shortened and it will compile as-is.

This may in some cases have side effects when used with temporary lifetime extension, but I expect this will be unlikely to occur in real code.

References:

• Edition guide - if let temporary scope
• Edition guide - Tail expression temporary scope
• Temporary lifetime extension

Match ergonomics reservations

There is a small change to how ref, mut, and ref mut and friends may be used. In edition 2024, it is an error to use these capture modifiers in a non-explicit pattern, that is, a pattern using match ergonomics. In practice, this looks as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
fn f(opt: &mut Option<i32>) {
    if let &mut Some(ref mut val) = opt {
        // valid, val is &mut i32
    }

    if let Some(val) = opt {
        // valid, val is &mut i32, match ergonomics used
    }

    if let Some(mut val) = opt {
        // Valid in edition 2021, invalid in edition 2024
    }
}

The stated reason for this change is that the syntax is intended to be used for match ergonomics expansion in the future. I don’t know what this expansion would look like, so I can’t comment on the usefulness, but I will say that the current compiler interpretation is not that useful or predictable, so not a lot is lost.

References:

• Edition guide - Match ergonomics reservations

Changes to unsafe

unsafe in Rust is used to denote sections of code where the compiler cannot prove all the properties of the code that are required for the code to be well-behaved. Edition 2024 requires you to mark more sections as unsafe that previously did not require it.

The first change, is that extern blocks are now required to be marked as unsafe. Function and constant definitions in an extern block must match their counterparts across the FFI bridge, but the compiler cannot ascertain that they do. Marking the block unsafe should signal the developer that they must pay attention.

Rust added the ability for attributes to be unsafe as well. A few existing attributes now require being marked unsafe. The names used for #[no_mangle] and #[export_name] need to be globally unique, otherwise unexpected linking errors may occur. As such, they are now #[unsafe(no_mangle)] and #[unsafe(export_name)]. In addition, #[link_section] must now be unsafe, as it can be used to add symbols to a section that might not work, such as adding mutable data to a read-only section.

References:

• Edition guide - Unsafe extern blocks
• Edition guide - Unsafe attributes
• Edition guide - unsafe_op_in_unsafe_fn warning

No more references to static mut data

In Rust, the existence of a reference &T presumes (barring interior mutability) no writes will happen to the data being referenced. This has always been in conflict with static mut variables, as any thread anywhere may be writing to them at any point in time. Edition 2024 therefore makes it an error to have a reference to static mut data. This was previously a warning.

The primary solution to get around this, is by not having static mut data. Make sure your global data is covered by a mutex or similar. If you must have static mut, any access to it will have to go through the pointer APIs, and you will have to take responsibility for upholding Rust’s safety guarantees.

References:

• Edition guide - Disallow references to static mut

Never type coercion

The Never type, !, is a type of which no value can ever exist. This is used in the language as the return value for functions that do not return, such as panic!(). It signals to the compiler and to a developer that a code path will never (heh) be taken. To ease type checking in these scenarios, the Never type will coerce to any other type.

Sometimes, the code doesn’t offer an unambiguous type to coerce to, in which case the fallback is used. Edition 2024 changes the fallback for this coercion from the unit type () to the Never type itself. This might cause minor issues for cases where a trait is implemented for () but not for !. The workaround is to explicitly coerce to the unit type in those situations by adding type annotations.

References:

• The never type
• Edition guide - Never type fallback change

Macro fragment specifiers

Macro fragment specifiers have, in my opinion, an unfortunate name. What it refers to is effectively the type of an argument of a macro_rules!-style macro.

1
2
3
4
5
6
macro_rules! my_print {
    ($val:expr) => { print!("{}", $val); };
}

// later
my_print!(1 + 2 + 3); // prints 6

Here, $val is captured, but only if it is an expression. my_print!(match) would not be accepted. Edition 2024 brings the :tt fragment specifier up-to-date with the new additions to the language, allowing it to match _- and const expressions as part of an :expr fragment. If you must maintain the previous behaviour, you can replace :expr with :expr_2021.

In addition, it is now an error to have macro fragments without type specifiers. Previously, this would be alright, as long as the variant with the missing fragment specifier wasn’t matched. Simply remove the variant without the specifier, or add an appropriate specifier.

1
2
3
4
5
6
7
// Compiles in edition 2021, compile error in edition 2024
macro_rules! broken_macro {
    () => { println!("Valid variant"); };
    ($without_specifier) => { unreachable!("Cannot match this"); };
}

broken_macro!();

References:

• Edition guide - Macro fragment specifiers
• Edition guide - Missing macro fragment specifiers

New reserved constructs

Edition 2024 adds a new reserved keyword, gen, in anticipation of the upcoming generators which may or may not be called coroutines. If you really want to continue naming your variables gen, you can use raw identifiers.

In addition, Rust now reserves one or more # in a row followed by a string literal, and two or more # in a row not separated by whitespace. There are no workarounds to this.

References:

• Edition guide - gen keyword
• Edition guide - Reserved syntax

Standard library

Aside from core language features, the standard library also has some backwards-incompatible changes.

Changes to the prelude

The prelude defines all the types you never have to import, like String. In edition 2024, Future and IntoFuture are added to the prelude. At this point, six years after the async keyword was introduced, this is unlikely to cause issues in regular code bases.

References:

• Edition guide - Changes to the prelude
• Prelude 2024

Add IntoIterator to boxed slices

In all editions, Box<[T]> will now implement IntoIterator<Item=T>. This could be a breaking change, as previously, my_box.into_iter() would evaluate to &[T]::into_iter, which iterates by reference rather than by value.

To mitigate this, direct calls to Box::<[T]>::into_iterator will be hidden in previous editions, which maintains the previous behaviour. You might remember this is the same workaround that was chosen back in edition 2021, when IntoIterator was added to [T; N].

Trait implementations cannot be edition specific, but this way, old code gets to keep working.

References:

• Edition guide - Add IntoIterator for Box<[T]>
• Edition 2021 guide - Add IntoIterator for arrays

Newly unsafe functions

Over the years, some functions have been added to the standard library that turned out to be problematic in certain cases. Despite attempts to make them safe and well-behaved in all circumstances, they fundamentally cannot be. Edition 2024 marks these functions as unsafe to signal that these functions require special care.

Two functions for modifying environment variables, std::env::{set_var, remove_var} have been the subject of a complicated range of CVEs I don’t want to get in to, but crucially, they cannot be used safely in a multithreaded environment. Rust tried to fix this by adding a RwLock internally that protects the internal state, but this turns out to still be unsafe when combined with the many libc functions that might be calling getenv in unrelated threads. As such, these functions are now unsafe and should only be used in single-threaded environments.

std::os::unix::process::CommandExt::before_exec is also unsafe now, in addition to being deprecated. pre_exec was introduced in Rust 1.37 as an unsafe replacement for it, but now before_exec will reflect its unsafe status.

References:

• Edition guide - Newly unsafe functions

Cargo

The package manager/build system/holy grail received some minor breaking changes that will generally make life easier.

First, cargo got a new dependency resolver that will take the current Rust version into account. In previous editions, this resolver can be opted into by setting resolver = "3". From my initial testing, it doesn’t work as well as it could, mostly because crates claim supported rust versions they very much do not. I expect this to get better in the future, and to at the very least make it easier to test for rust version support and maintain backwards compatibility.

A minor change is that the naming of options has been made more consistent. Cargo.toml (and related) now consistently use kebab-case for all keys. Previously, snake_case was allowed for some (but not all) keys.

Finally, there is some clean-up with workspace dependencies: it is now an error to disable default-features for an inherited dependency when the workspace does not disable default-features. This never worked; the dependency would always have its default features enabled regardless as features in Rust are additive. Now the compiler will simply let you know.

References:

• Edition guide - Cargo: Rust-version aware resolver
• Edition guide - Cargo: Table and key name consistency
• Edition guide - Cargo: reject unused inherited default-features

Rustdoc

While technically breaking changes, Rustdoc mainly received some nice quality-of-life improvements. In edition 2024, Rustdoc tests will be combined into a single binary as much as possible. This should generally improve compile times for the tests. It is not always possible to combine tests. compile_fail examples will always be compiled separately, and some other obvious breakage will be detected and avoided. If you need to enforce that an example is compiled separately, you can add the standalone_crate language tag to ensure that specific example will be compiled on its own.

1
2
3
4
5
/// Compiles example code
///
/// ```standalone_crate
/// println!("Hello, world!");
/// ```

A smaller, but more breaking change is that nested include!s in your documentation are now handled relative to the included document rather than to the original source location. That is, if your code uses something like #[doc=include_str!(../README.md)], any include_bytes!() that might be in your examples will now be handled relative to README.md.

Rustfmt

The biggest change to Rustfmt is that as of edition 2024, Rustfmt can have its own editions with backwards-incompatible changes now. By default, it will use the same edition for formatting as it will for compiling, but you can override this by setting style_edition in rustfmt.toml. This allows you to opt in to improved formatting, while not raising the minimum supported rust version. That said, let’s look at the breaking changes that are included.

Formatting fixes

Most of the changes to Rustfmt don’t involve sweeping changes to the resulting format, but rather fix circumstances where the formatter did not work correctly. The edition guide has examples for all changes, but I’ve tried to summarize them here.

• Trailing comments after a line-end comment will no longer be aligned as if they were related.
• Strings in comments will no longer be indented.
• Long strings will no longer cause the formatter to stop formatting the expression that contains them.
• impl block generics will no longer have a double-indent.
• Some complicated fn blocks would previously get a strange indentation and no longer do. The complex fn blocks required to trigger this are still not very readable.
• Nested tuple indexings will no longer get an extraneous space.
• return, break, and continue statements inside a match block now consistently end with a semicolon.
• Long array and slice patterns now wrap to avoid overflow.
• The last expression statement in a block will now be formatted as a one-liner most of the time.
• The formatting between a macro call and a function call is now more consistent.
• Closures containing a single loop will now get a block to wrap the closure body. This is the only change that feels like a regression to me.
• Empty lines in where blocks are now removed.
• else clauses from a let else with an attribute are no longer formatted incorrectly.
• Comment- and macro argument wrapping no longer has an off-by-one bug.
• Comments with => inside a match block are no longer formatted incorrectly.
• Having more than one inner attributes in a match block no longer causes extraneous indents.

All but one of these should only ever improve formatting in my opinion, thus working around this shouldn’t be necessary, but you can still ignore these improvements and avoid a diff in your code by opting for style edition 2021 now.

References:

• Edition guide - Rustfmt: Formatting fixes

Sort order changes

Rustfmt doesn’t sort things often in source code, but when it does, the resulting order makes sense. Edition 2024 makes two changes to this. First, raw identifiers, used when an identifier overlaps with a keyword, will now correctly sort in the position the identifier would, rather than sorting based on the raw identifier prefix r#. This should generally put the resulting list in a more “sensible” order.

In addition to that, the sort order was changed from lexicographical to an algorithm the guide refers to as “version-sort.” The main difference to this is that it will handle strings containing numbers better, so that NonZeroU8 will now sort before NonZeroU16 rather than after it, and lowercase characters will now sort after uppercase characters.

References:

• Edition guide - Rustfmt: Raw identifier sorting
• Edition guide - Rustfmt: Version sorting

To summarize

Rust 2024 is characterized not by big changes, but rather by a plethora of small ones. I expect for most code bases, very few changes will have to be made, if any at all, and most of the changes that do have to be made can be addressed by cargo fix.

To me, this seems like a healthy state for the language. Change keeps happening, but most of it can be relied on to remain recognizable to people in the future. The language isn’t stagnating, but it’s also not adding a torrent of new features and syntax any more. This kind of slow evolution should allow for greater adoption in the real world, as the world sees that you don’t have to spend half your time keeping up with all the new changes. The future of the language seems alright, and I’m here for it.

After burning myself out halfway through the puzzles last year, I decided I was going to make it easy on myself this year, and use Python, as most problems are simply solved in the standard library and most other things are easily solved with Numpy or NetworkX. Then a friend told me I was being suspiciously normal, so I opted to honour my day job in DevOps and, as a challenge, implement as many puzzle solutions as possible in Terraform.

Terraform, as we will go into more detail about soon, is not exactly a programming language. Its intended use is to define a desired set of cloud infrastructure, which you can then feed to the terraform binary and tell it to “make it so.” As such, you might be surprised that it can do programming at all. After all, it’s a configuration tool, and you would like your configuration to be simple, static, and predictable¹. While that’s partially true, years of adding convenience to the language have definitely created some structures that could be used to solve puzzles.

If you’re not interested in learning Terraform, you can skip ahead to the discussion of the puzzles.

What is Terraform?

HashiCorp Terraform, to call it by its full, nominatively fair use name, is a tool used to codify infrastructure. In its most basic form, you use provider plugins to write resource definitions, which can then be apply‘d to make reality match your configuration. For example, the following configuration would set up a server in the Netherlands in Google Cloud Platform.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
provider "google" {
  project = "my-gcp-project"
}

resource "google_compute_instance" "my_server" {
  name         = "my-server"
  machine_type = "n2-standard-2"
  zone         = "europe-west4-a"

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-12"
    }
  }
}

And this works pretty well. When your operation grows, you will however generally accumulate more and more servers, and you want to avoid repeating yourself for each one of them. Because of this, over time, several features have been added that allow you to not repeat yourself, do little calculations, and overall express yourself more concisely.

Our tools

As mentioned, Terraform is not intended for heavy computational use, but it has a few things that we might put to work as such any way. Let’s go over what we have to work with.

Data types

Terraform has a reasonably complete set of data types, though they rarely come up due to automatic coercion. We have basic primitives string, number², and boolean, as well as collection types list, set, and map. The map type only has string keys, but its value type can vary. The collections can also be composed, so list(map(number)) is a valid type. null is also a type, but for our purposes it doesn’t really come up.

Even though the documentation insists that casts are rarely necessary because everything generally coerces to the right type, this stops being true as soon as you start to do any programming. The string "0" will indeed coerce to 0 when passed to a context that expects a number, but these values are not equal.

1
2
3
4
5
6
run "type_equality" {
  assert {
    condition     = "0" != 0
    error_message = "Integers aren't equal to strings!"
  }
}

For completeness, it should be noted that there are two more data types, tuple and object, that are simply list and map but with different types of values rather than all the same. I will not be using those for our solution programs.

Structure

Our Terraform project is structured in modules. A module is effectively some directory with .tf files³. Terraform doesn’t really have an ordering between files, they should all be viewed together at once as if they were one big file. In its simplest form, a module defines some input variables and some output variables that are derived from that. For example, the module below takes a number and doubles it.

1
2
3
4
5
6
7
variable "my_num" {
    type = number
}

output "my_result" {
    value = var.my_num * 2
}

It’s often useful to compute a repeated bit once, and then use it later. You can do so with local variables:

1
2
3
4
locals {
    double = var.my_num * 2
    quadruple = local.double * 2
}

In a way, modules are like functions: they take some input, do their transformations on it, and produce an observable output. Any local variables are implementation details and are hidden from the caller. Modules can also call each other to create more complex structures. Here, they either point to a module in some well-known registry, but you can also simply point to different directory of Terraform files. You can then refer to the outputs of that module to continue your computation.

1
2
3
4
5
6
7
8
9
module "doubler_as_a_service" {
    source = "./doubler"

    my_num = var.outer_input
}

output "is_42" {
    value = module.doubler_as_a_service.my_result == 42
}

It should be noted that the order that expressions and definitions appear in the files do not matter; instead, Terraform attempts to build a directed acyclic graph between all definitions, and then computes everything starting from the leaves. It does so in parallel, which has the neat little side effect that it’s trivial to write highly multithreaded code in Terraform.

Basic arithmetic

It might seem minor given the above areas, but Terraform also has a reasonably complete set of arithmetic operators, which combine in all the ways you would expect. In addition, it also has boolean logic, though it doesn’t do short-circuiting for con- and disjunctions. This will come to bite us later, but we can make it work.

Loop structures

Terraform doesn’t have loops directly, but it does have two sets of primitives that we can use as loops in our programs. Both have roughly the same limitations:

• Iterations cannot depend on the result of a previous iteration
• The number of iterations is always bounded
• We cannot exit early
• We can skip individual iterations conditionally

This, it turns out, is good enough for many purposes.

Comprehensions

If you are familiar with Python, you will have seen list- and dictionary comprehensions, as a way of creating these collections from an iterator, without a for loop.

1
2
3
4
5
first_list = []
for n in range(10):
    first_list.append(n * 2)

second_list = [n * 2 for n in range(10)]

In Terraform, we don’t have the luxury of for loops, but we do have for expressions which are essentially list- and map comprehensions. We will be using those as our primary looping mechanism. Terraform has roughly the same functionality as Python here. We can loop over one thing, construct another, and optionally skip elements that don’t meet certain conditions.

1
2
3
4
5
6
7
8
9
10
11
locals {
    list_example = [for n in range(10) : 2 * n]
    map_example = {for n in range(10): n => n * 2}

    list_from_map = [
        for key, value in local.map_example :
        value
        # Skip keys that are divisible by three
        if key % 3 != 0
    ]
}

As previously, mentioned, keys in a map will always be strings. This is luckily mostly managed by automatic type conversion, so you can use integers as map keys just fine.

There is one more special comprehension. For maps, it is an error to have more than one value for the same key. For the cases where you actually do have more than one value per key, you can use the grouping map comprehension. This combines all values with the same key into a list, in order of appearance.

1
2
3
4
5
6
locals {
    grouped_map = {
      for n in range(10000):
      n % 7 => n...
    }
}

Repetition

Aside the earlier comprehensions, we can also repeat resources and modules using the meta arguments count and for_each. count takes a number and repeats a given resource or module that many times. for_each takes a map and instantiates a given resource or module once for each key in the map.

1
2
3
4
5
6
7
8
9
10
11
12
13
module "count_module" {
    source = "./source"
    count = 10
    # The 0-based index of the resource is available to customize the different modules
    input_var = count.index
}

module "foreach_module" {
    source = "./source"
    for_each = local.some_map
    # the key and value of the current map entry are available to use
    input_var = each.key + each.value
}

count can also be used in a conditional this way: by setting count to 0, you can conditionally turn of that particular module. This is, as of now, also the only way to have conditionally-created resources in Terraform, even when using it as intended. It feels like a hack every time you do.

Standard functions

Aside from code structures we might use, Terraform also offers a collection of functions that you can use to express complicated structures that you otherwise wouldn’t be able to. The main workhorses for the puzzles were split and the various regex operations, as they allowed me to parse the input in about the same way as I would in Python.

Unit testing

Any development experience will be infinitely more awful if you can’t rapidly iterate, and for that, you need a way to quickly test your code. Terraform offers both unit and integration tests for this, where it will run your code either in plan or in apply mode respectively, and allows you.

1
2
3
4
5
6
7
8
9
10
locals {
  two_times_two = 2 * 2
}

run "test_math" {
  assert {
    condition     = local.two_times_two == 4
    error_message = "Maths machine broke"
  }
}

I’ve used this to encode the sample inputs and their from the stories into test cases, so I could quickly check whether my code gave the right answer yet.

What we don’t have

With all the features outlined above, one might think Terraform is a pretty normal, albeit rather quirky programming language. This is wrong. It is an expressive declarative configuration language. As such, we don’t have any method to mutability. Variables are declared exactly once, with their initial value. Due to the lack of loops, there is no way to declare a variable again with a new value. Because of this, algorithms that walk through a grid following some arbitrary rules are impossible to implement. You can’t write Dijkstra’s algorithm without maintaining some to-do list. You might be able to with sufficient recursion, except you can’t.

It might seem tempting to have modules call themselves conditionally to achieve some sense of looping or smart recursion, but this doesn’t work. At the early stages of evaluation, Terraform builds a tree of your module structure to figure out in what order it should do its work. At this point, no other values have been evaluated, so it assumes that any use of a module actually happens. You can kind of achieve recursion by hard-coding your maximum depth into your folder structure, and you even can kind of avoid repeating yourself by using symlinks

1
2
3
4
5
6
7
outer_module/
└── recursive_module/
    └── main.tf
        └── recursive_module/
            └── main.tf (symlink to ../main.tf)
                └── recursive_module/
                    └── recursion_end.tf

This is of course not very fun to create, and since all recursion happens unconditionally, you will always suffer the overhead of having this entire structure, even if you don’t need it. In other words, it’s slow. This is even worse if you have some conditional branching, as the modules for both branches will be instantiated.

The puzzles

Now that we’ve established that programming in Terraform is a bad idea, let’s try to do it anyway. I will go over the various puzzles that worked out, and the workarounds that were needed to get there. With these odds, I’m happy with 18, perhaps 20 stars.

Day 1: Linked lists

Code - Puzzle

We start the puzzles easy, but we also encounter the limitations of automatic type coercion for the first time. It is simple enough to parse the lines of numbers into two lists of numbers. After that, we do have to explicitly cast the values to numbers before using the sort function.

The sort function will only sort numbers by value; any other value will be sorted lexicographically instead. That is reasonable behaviour, but can be unexpected when sorting numeric strings. This behaviour is also not documented; the documentation suggests that any form of sorting happens lexicographically regardless of type. Two stars.

Day 2: Report redaction

Code - Puzzle

On the second day, we get the first use of modules as reusable functions. We are asked to check whether a list is either ascending or descending, and whether the difference for each step is between one and three, inclusive. We can write a module that computes that for a given list.

For the second part, we are asked whether we can make the lists that do not yet match our requirements, pass the check after removing a single element from them. There exists a nice Dynamic Programming solution that can check whether a list of numbers can be fixed like that in a single pass, but since this is only the second puzzle, the lists are nice and short (up to eight elements) and simply checking whether the deletion of any one of those elements results in a valid list is good enough. Two stars.

Day 3: To multiply or not to multiply

Code - Puzzle

The third puzzle gives us garbled string like sub(2b,3)r7do()mul(3,4)don't()mul(3,10) and asked what the total for all the included multiplications mul(a,b) is. There are many ways to do this, but the easiest to me is to use a simple regular expression to find all the multiplications. Terraform’s regular expression engine proved capable of this just fine.

After solving that, the twist for the second puzzle is that there are additional do() and don't() instructions included in the string. Any mul after a don't() should be ignored until the next do().

At first, I thought this would be very hard to do in Terraform as my Python solution keeps a boolean indicating whether the last thing I’ve seen is a do() or a don't(), which is mutation. I did manage to implement this by tokenizing the string into a sequence of mul(), do(), and don't(), then for every mul(), check whether the most recent do() has a higher index than the most recent don't(). This implementation was made slightly harder by lack of a good “last matching element before position” function, though you can construct one using slice(), reverse(), and index() functions.

I was then informed by a colleague, who was attempting to complete the puzzles in an even less suitable programming tool⁴, that this is clearly the hard way to do this. You can simply use regular expression text replacements to filter out the parts you should ignore. You can first replace all don't()s until the next do() with the empty string, and finally delete any remaining don't() until the end of the string. Just be sure that you have enabled multiline matching in your regex engine, as the input spans multiple lines. After that, I updated my implementation which performed significantly better.

Originally I had thought that this approach would be impossible, as I had interpreted the problem as a matching-brackets kind of problem, where every don't() should be matched up to a do(). This is famously impossible with regular expressions due to the pumping lemma. The regular expression solution however does work, because you don’t have to match these tokens; the most recent gets to decide, so this fits perfectly into a regular language. Regardless, another two stars.

Day 4: Crazy crossword

Code - Puzzle

The fourth puzzle is a giant crossword puzzle. The puzzle is very simple, as it only contains the word “XMAS”. We have to find how often it contains this word. The way to solve this in Terraform is not that different from how it would be in any other language: look at every individual point in the crossword, look in all directions, and see if you hit anything you want to see.

The way I ended up implementing this, is with a module that is called once for every cell of the grid, then from there, call a second module that walks in a specified direction, and see what the grid spells in each of those directions. Here we hit a tiny wall.

Terraform modules turn out to have significant overhead, and evaluating these roughly $140 \times 140 \times (1 + 8) = 176400$ modules requires about 10GiB of RAM. I noticed this problem in the previous days’ puzzles, but here it became too much for my poor laptop. If we want this to run at all, we have to reduce the number of modules we expand. Luckily we can simplify the problem a little.

Horizontal instances of words are easy: we can simply do a string search for XMAS and SAMX in our input text, which we can do once for the entire input. This has immediately reduces the work we have to do by 22%, because we no longer have to search in those two directions.

The other six directions don’t have simplifications that are this nice, but we can at least reduce the amount of work by half by only searching of three of them, and searching for both XMAS and SAMX at the same time.

Together, these optimizations reduce the amount of RAM required enough so that I can run this on my laptop again.

For the second part, it turns out we weren’t looking for the word XMAS, but rather for pairs of the word MAS intersecting each other to make an X. A+ joke. This turns out to be much simpler than searching for XMAS, as the orientation is largely fixed. We simply look at all As, and see whether both diagonals running through the A spell MAS. I wrote a separate module for this, though thinking back, it could’ve been inlined as well.

This adds two more stars, though I have the code commented out in my Terraform project as the time to solve it hampers my iteration speed considerably. If you really want to, you can uncomment it to see it run.

Day 5: Directing cyclic graphs

Code - Puzzle

Today we are print lists of numbers, and we have a set of rules where one page must be printed in front of the other. For the first part, we must find how many of our lists are valid to be printed. To check this, you could try to find a topological order for the rules, and see whether the print lists follow this order. As various people on Reddit figured out, the rules are cyclic, so this topological order doesn’t exist. There is however a different way.

For every rule that says A must come before B, we can use this knowledge to see that if we ever see a B in a list, the rest of that list must not contain A. We can use the grouping map comprehension to build a blacklist for every number, then walk iteratively through our lists and for every position, check if tail starting after that position contains any numbers on the blacklist for the number at that position. This is kind of $O(n^3)$ per list, but the lists are still relatively short and it works out.

For the second part, we are required to figure out the correct order for the lists that aren’t in the correct order. I have not managed to design a sorting algorithm that works in Terraform, so I’m going to consider this one impossible. First incomplete puzzle, one star.

Day 8: Radio stars

Code - Puzzle

Our arch nemesis, the Easter Bunny, is using an array of radio antennae to convince the masses to buy cheap imitation chocolate. This is of course a completely unrealistic scenario and no real life bunny would ever do this. Nevertheless, we have to stop him.

Each of the antennae emits a certain frequency, and when two antennae with the same frequency interact, it causes antinodes, where the amplitude of the waves is maximized. That is unfortunately where the real-life accuracy ends, as the puzzle definition for antinodes works differently.

In part one, antinodes for each pair of antennae are located on all places that are twice as far from one antenna as they are from the other. The puzzle is nice, having no pairs at a distance of an integer multiple of three from each other, so every pair $a, b$ has two antinodes:

The rest of the algorithm is then fairly straightforward in Terraform: loop over all grid coordinates to locate the antennae, group by frequency, compute all pairs, compute the resulting antinodes from those pairs, check whether they land inside the grid, and finally build a set out of all of these resulting nodes.

The second part adds slightly more nodes to the puzzle by having any grid point that is collinear with the pair of nodes be an antinode. That simplifies the list of grid points to

for all integers $k$. For simplicity, I had $k$ go from -max(width, height) to max(width, height) which should cover any grid point that may be an antinode. In theory, you should compute divide the $b - a$ vector by the greatest common denominator of its components, but at least in my puzzle it appears not to matter. It is however possible so let me show you. Euclid’s algorithm for greatest common denominator is usually written recursively as:

1
2
3
4
5
6
def gcd(a: int, b: int) -> int:
    if b < a:
        return gcd(b, a)
    elif a == 0:
        return b
    return gcd(b % a, a)

Euclid’s algorithm is very efficient, and runs (worst case) in $O(\log_{\phi}(n))$ recursive calls, where $\phi$ is the golden ratio, and $n$ is the smaller of our two numbers. Since our width is bounded to our input, we can linearly write out the algorithm for enough steps to cover that. So I did.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
locals {
  dx = var.second[0] - var.first[0]
  dy = var.second[1] - var.first[1]

  # sort() doesn't work as it turns the numbers into strings
  gcd0 = abs(local.dx) < abs(local.dy) ? [abs(local.dx), abs(local.dy)] : [abs(local.dy), abs(local.dx)]

  # Do as many iterations as necessary.
  gcd1  = local.gcd0[0] == 0 ? local.gcd0 : [local.gcd0[1] % local.gcd0[0], local.gcd0[0]]
  gcd2  = local.gcd1[0] == 0 ? local.gcd1 : [local.gcd1[1] % local.gcd1[0], local.gcd1[0]]
  gcd3  = local.gcd2[0] == 0 ? local.gcd2 : [local.gcd2[1] % local.gcd2[0], local.gcd2[0]]
  gcd4  = local.gcd3[0] == 0 ? local.gcd3 : [local.gcd3[1] % local.gcd3[0], local.gcd3[0]]
  gcd5  = local.gcd4[0] == 0 ? local.gcd4 : [local.gcd4[1] % local.gcd4[0], local.gcd4[0]]
  gcd6  = local.gcd5[0] == 0 ? local.gcd5 : [local.gcd5[1] % local.gcd5[0], local.gcd5[0]]
  gcd7  = local.gcd6[0] == 0 ? local.gcd6 : [local.gcd6[1] % local.gcd6[0], local.gcd6[0]]
  gcd8  = local.gcd7[0] == 0 ? local.gcd7 : [local.gcd7[1] % local.gcd7[0], local.gcd7[0]]
  gcd9  = local.gcd8[0] == 0 ? local.gcd8 : [local.gcd8[1] % local.gcd8[0], local.gcd8[0]]
  gcd10 = local.gcd9[0] == 0 ? local.gcd9 : [local.gcd9[1] % local.gcd9[0], local.gcd9[0]]

  # 10 iterations should cover numbers up to 55, which is more than the width/height
  gcd = local.gcd10[1]
}

One final thing to mention is that I while writing the section explaining how the eighth puzzle was impossible, I realized it was indeed very much possible to do so. Another two stars.

Day 11: Collatz’ Angels

Code - Puzzle

It was going so well, but suddenly the puzzles start requiring real programming and I couldn’t get Terraform to do any of it. Luckily, this day works surprisingly okay. We are asked to simulate a set of strange stones.

The stones have numbers written on them, and every time we blink, the stones change and split up according to the following rules:

• if the stone is labelled $0$, it is replaced by a stone labelled $1$,
• else, if the number has an odd number of digits, it is multiplied by $2024$,
• else, the stone is split into two separate stones: one labelled with the first half of the digits, and one labelled with the second half. A stone labelled 2005 would split into on two labelled 20 and 5 respectively.

We are to simulate this process for 25 blinks, and then for 75 blinks. The step that’s splitting the stones in two is suspicious, as exponential growth would make our pool of stones really big really fast. There has to be a trick to this.

The rules for the problem are rather similar in their reduction to the rules from the Collatz conjecture, so it makes sense that there is some loop involved. Of course, we have to deal with an exponential number of stones, but this loop will provide the solution to that.

It turns out, that despite the number of stones expanding rather quickly, we will only every have three thousand something unique labels on those stones. We can use this fact to group stones with the same number on them, as they will evolve in the same way. We simply have to multiply the result with the number of like-labelled stones we have. In other words, our algorithm works as follows:

1. Convert the input into a map of {stone_label: number_of_stones_with_that_label}
2. Then, for every step of the simulation:
   1. For each label and count, convert them to a list of new labels and counts
   2. Group these new counts by label
   3. Sum the counts grouped by label to create a new map
3. After all steps are done, sum all the map values to arrive at the final number

This is essentially the same algorithm I used in Python too, though with some issues due to Terraform. First, how do we write out 75 steps? My original idea was to do something like the following:

1
2
3
4
5
6
module "step" {
  source = "./step"
  count = 75

  input_map = count.index == 0 ? local.input_map : step[count.index - 1].output_map
}

Unfortunately, this doesn’t work: Terraform considers all instances of repeated resources and modules to be the same thing. Despite this hypothetical step module never actually referring to itself, Terraform does consider it a cycle, and will error when you try to evaluate this. The solution of course is to copy-and-paste this step module 75 times.

The second issue encountered here happened only when I moved to optimize my code a little. Originally, the step module called a separate transform module on every label individually and transformed it into a list of different labels. This involves a fairly large number of modules, so I wanted to inline the computation, because that way the overhead would be significantly reduced. After copying the code out of the module and into the parent, the code stopped working.

I managed to track this down to the fact that the input arguments for the module had a fixed type coerced the arguments to integers. The stone labels were stored as the key in a dictionary, but this coercion ensured that by the time they encountered my logic, they were in fact numbers. After inlining, this was no longer the case, and I got to discover that 0 != "0", even though everything else still works.

Despite all that, another two stars for the collection.

Day 13: Unfair fun fair

Code - Puzzle

After another day that was clearly impossible due to the 2D maze solving required, we end up with one that is just as easy in Terraform as it is in any other language. We are in control of a series of claw machines, each with two buttons. For each machine, those buttons move the claw a different number along the X and Y axes. The question is then how many times we have to press each button to get the claw to reach some specific position, and whether it is even possible.

The problem text explicitly asks for the minimal number of presses, but this is a red herring: there is exactly one unique combination of button press numbers that will bring us to the goal. This is because the underlying problem we are solving is the solution to the following linear equation:

This results in a system of two linear equations with two unknowns, $a_{press}$ and $b_{press}$, which you can then solve for as you’ve learned in high school. To filter out the instances where the claw cannot reach the destination, simply verify that the solution you found still works after rounding to integers. Part two simply ensures that you weren’t brute-forcing button combinations by adding a huge offset to the number. Another two stars.

Day 14: Bathroom break

Code - Puzzle

Suddenly, there are a bunch of robots racing around the restroom, all with a known initial position and speed, teleporting when they hit a wall. We are asked to simulate where they end up after 100 seconds. This is simple modular multiplication, which Terraform can do easily.

The second half of the puzzle requires us to find the moment in time where all robots line up to form a picture of a Christmas tree. Unfortunately, I have not been able to come up with a heuristic for this that works in Terraform. Alas, another lone star.

Day 19: Magical towels

Code - Puzzle

My favourite joke from last year returns as we arrive at the Hot Springs of Advent of Code lore. Here, we are to see whether we can make a given pattern of five colours by sequencing together some towels with specific patterns that we have at our disposal. Both our towels’ pattern and the target are sequences of w, u, b, r, and g. We can actually build a regular expression of our towel patterns, of ^(every|single|towel|pattern)*$ and see whether the target patterns match this. This makes part 1 solvable in Terraform.

Unfortunately, for part two, the question changes to how many different ways we can construct to create each pattern. To the best of my knowledge this requires actual recursion, which we can’t do with our limited tools. One more star.

Day 25: Keys to success

Code - Puzzle

Our last day is coincidentally also the last puzzle I managed to solve. We are given a list of schematics for keys and locks, and we must find out how many combinations of keys and locks could theoretically fit each other. A key does not fit if the height of one of its teeth pin stacks overlaps with one of the matching pin stacks⁵ in the lock. This is simple enough; the challenge is in parsing these heights from the input. Our diagrams look as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#####
##.##
##.##
.#.##
.#...
.....
.....

.....
.....
#....
#....
#.##.
#.###
#####

We can tell whether an entry is a lock or a key by seeing whether the first row is full. The first item here is a lock with heights [3, 5, 1, 4, 4] and the second is a key with heights [5, 1, 3, 3, 2]. These almost fit each other, except the first pin overlaps.

The diagrams would be significantly easier to parse if they were horizontal, but with a slightly inefficient for loop, we can parse the heights, check every lock against every key, and determine the potential matches.

This is our final star. I didn’t manage to get anywhere close to getting all other stars, so the part 2 bonus star for today remains locked. That brings us to a nice and round 18 (out of 50) stars achievable in pure Terraform.

What didn’t work

So despite getting about one third of the stars available, there are a lot of things that I could not get Terraform to do. For completeness, I’d like to go through a few reasons why I deemed puzzles impossible and what might yet be possible.

• Any algorithm that requires mutating state is immediately out, as we don’t have mutable variables. Sometimes I could work around this by adding another factor $n$ to the algorithmic complexity, but this only works in specific cases.

• Any walking through a maze generally requires mutation to do efficiently, which means that any puzzle that involved walking through a maze was immediately impossible.

• Some puzzles can theoretically be phrased in terms that Terraform may compute, but anything that requires a bunch of modules is likely to be prohibitively slow. The overhead of expanding a module appears on the order of several kilobytes, even for otherwise empty modules.

This all said, I have a hunch that both parts of day 7 might be possible, by using the fake-bounded recursion that I discussed previously. It would require $O(n \cdot 2^l)$ modules for the first part and $O(n \cdot 3^l)$ for part two, with $n$ the number of samples and $l$ the length of the lists. I don’t believe my computer can do that.

Final thoughts

There exists a literary technique called constrained writing, where the writer binds themselves to an unnecessary restriction and tries to make their story work anyway. The book “Initial Instructions” by Joseph Prouser for example is a translation of the Book of Genesis that does not use the letter “E”. Instead of creating the heavens and the earth, it starts:

God’s initial act in His constantly unfolding work of cosmogony was formation of our world and its sky.

I am not a religious person, but playing with language in this way tickles me pink. The point of the exercise is to make you see your work differently. The restrictions force you to be creative to work around them, which in turn leads you to new ideas.

Attempting to problem-solve with Terraform made me think differently. Last year I wrote about being burned out on Advent of Code, but this year I had the most fun I’ve had since doing the polyglot challenge. Terraform is very much not made for programming, but finding new ways to use the primitives it does offer together has been a great way to keep me thinking.

I encourage anyone to try a similarly restrictive challenge for Advent of Code, or any other programming puzzle. If you instead send me a pull request in this style, I do reserve the right to request changes. Happy New Year.

The claim is not even unique to LinkedIn, it has been widely circulated in the news, with variations on the severity of the claims, such as on national television complete with terrible data security advice, the BBC, as well as various other outlets. That kind of attention doesn’t appear out of nothing, so let’s examine the claim a bit further.

Why it’s wrong

Houses are big, and emails are not. This is an intuition that I think we can start from. For the savings of 80 billion deleted emails to power millions of houses, they have to be expensive to save. This is not, in fact, the case. Let’s do an oversimplified comparison.

At the time of writing, there are 2587 emails in my inbox, which weigh a total of 167MB¹. This works out to an average of 61kB of data per email. More than I expected, but as we will see, not that much. Assuming all eight billion people alive² today delete ten emails, that works out to roughly five petabytes ($\approx 5 \cdot 10^{15}$ bytes) of data.

A petabyte is a lot of data, but not that much by today’s standards. You can buy an 18TB hard drive for €280 so for the low, low price of €77 thousand you too can store this much data. Alternatively, you could pay for Google Cloud Storage, and store all that data for about €92 thousand a month, going as low as €5.5 thousand a month if you opt for Archive Storage. Since we’re talking about emails we would otherwise delete and therefore can’t be that valuable, I’d argue that archive storage will do nicely.

With our calculations above, we now have a good enough picture of what the potential savings are of us deleting all of those emails. We cannot directly compare the monetary cost of storing this data to the carbon footprint of the power consumption of millions of homes, but we can make a good guess that it doesn’t compare. Considering that the average household over a thousand euros a year on energy (both power and heating) a year, multiplied by millions of households, it should be obvious that there are several orders of magnitude in between them in terms of cost.

How do these claims happen

“Why do people believe things that are false” is a big question that I am not qualified to answer. For the claim central to this article, however, we can make some educated guesses. Searching for the origin of this claim, led me to this article about the footprint of an email. In the article, it is claimed that the carbon footprint of a longer email sent from a laptop and read on a laptop causes the equivalent of 17 grams of CO2 to be emitted. There are a few caveats to this number but let’s assume that it’s true. This gives us a CO2 “compensation” of 138 thousand tonnes.

Now for our millions of households. I’m going to take the numbers for the Netherlands because I live there, but other European countries should be similar. According to the Central Bureau for Statistics, the average household uses 2640 kWh of electricity annually. Our World in Data mentions that on average, the Netherlands emits 325 grams of CO2 per kWh of energy. Putting that all together and solving for the number of households gives us:

Not quite millions of households, but at least a single million. This number is directly proportional to the carbon footprint of an email, and that’s gone down over the years. This is unfortunately where things stop lining up. The carbon footprint of an email, as written by Berners-Lee, is not in its transport, nor the electricity. It’s in the embodied carbon of your laptop. While the book doesn’t provide figures for a laptop, other than noting that it has more embodied carbon than a phone, for a phone this looks something as follows:

“Embodied in the phone” in this case means that the CO2 was emitted in the process of creating the phone, averaged out over its lifetime, for the duration of dealing with the email. In a fun twist of events, it causes more CO2 to delete the emails, as you will be spending time and energy on doing those actions. There are of course caveats to that too, and eventually, the amount of data does start to matter.

One thing that is interesting to note, is how little CO2 is involved with producing electricity these days. My recent bike trip in June back and forth to Den Haag, about 27 kilometers, had a carbon footprint comparable to 2 days of my domestic electricity use.³

Why make a point out of this

You might be asking yourself, why anyone would go through the trouble of fact-checking a claim that as posted on a social media platform largely without consequences or persistence. And the answer to that is twofold. The first part is very simple.

The second part however is more personal, so bear with me as I step on my soap box. The climate change discussion has long been poisoned by less than truthful information. Big institutions pay a lot to pretend there’s any debate at all that the climate is changing. Sweeping changes will have to be made by every single country around the world. For some, it may already be too late.

Claims like the one that sparked this article do not deny the need for change, but they are in my opinion harmful in different ways. In a small part, they give climate deniers ammunition (“Look at these ridiculous claims, who knows what else they might be lying about”) but as part of the bigger picture they maintain the illusion that climate change is a personal issue that will be addressed if we all do our part. That is not the world we live in.

Berners-Lee, in his book, champions the 5-tonne lifestyle, where everything you do in life should amount to no more than 5,000 kilograms of CO2 equivalent emissions per year. This is a noble goal and one that I respect immensely. The world would be a better place if more people lived like that. This is, for reference, about 40% of the average per capita emissions of the United Kingdom⁴. Unfortunately, it is not a structurally feasible solution in my opinion.

There is a lot of talk in politics about a carbon tax as if it’s something new, but we’re already there. Countries around the world are effectively already paying for it. My famously under-sea-level home country of the Netherlands has to invest in raising, widening, and doubling the dikes to deal with the rising sea levels. I live in a rich country and it will be fine for the foreseeable future. Many Pacific island nations will have to do similarly expensive work without those resources. A small number of countries benefited greatly from emitting greenhouse gases, boosting economies, and bringing wealth, but the costs of the warming planet are borne by the world as a whole as a worldwide flat tax. This won’t go away by itself either, due to the tragedy of the commons: everyone individually benefits by making more CO2 emissions, which inevitably makes life worse for everyone else.

The classic solution to this tragedy is to achieve consensus on how to manage the shared resources, which translates to worldwide treaties about emissions. The 2015 Paris Agreement was a step in this direction, though it does not appear to be doing nearly enough. We should as a planet agree where our priorities lie, and ensure that the environmental costs of our activities are properly priced in for those undertaking them, rather than borne by the world as a whole. Flying is the easiest example, due to plane fuel being barely taxed at all, but there are other examples.

While the tech sector is not nearly as big as that, carbon footprint wise, bitcoin still famously uses more energy than several developed countries, of which Finland is the most recent example I could find, though this number has likely changed significantly over the last three years. The “new” kid on the block, AI, is also growing significantly, and is expected to consume more energy than the entirety of the Netherlands. Going back to the classics, a lasting rumour stated that a single Google Search is roughly proportional to boiling a kettle of water, though recent estimates put that number far lower⁵. No matter what it is, its utility should be weighed against its cost.

The internet as a whole consumes a lot of power but it also brings a lot of utility. AI and recently large language models have been very powerful and can improve human productivity. Bitcoin is certainly a thing that exists. I do not mean to argue that we should abandon all that, though we should further incentivize computational efficiency, and reduce the carbon footprint of the power requirements that remain. Not just for tech, but for everything, and not just as an ideal, but as global policy.

Take away

To get back to the original topic and maybe contradict this article’s title a little: individual actions can’t solve the problem, but they can help. You could eat less meat or dairy. You could avoid flying. The concept of a carbon footprint isn’t without merit. It is a good teaching tool to make people aware of the impacts of their actions.

Even though the cost of an email is small, that doesn’t mean that you can send as many of them as you want. People send more emails today than they would’ve ever sent traditional mail, and so the carbon saved from winding down the post service is more than compensated for by the additional emissions from email.

You won’t save the planet by deleting emails, but sending fewer emails is still a good goal to have, just as any small improvement can be, as long as we don’t forget the big picture. Try to unsubscribe from all of those marketing emails. Design your web apps to not triple confirm every single thing that happens. And if you’re discussing details that need some figuring out, maybe don’t send a follow-up email every day. Phil, if you’re reading this, stop it. You’re stressing me out.

During the writing of this article I reached out to mr. Berners-Lee for comment, whose assistant thanked me for getting in touch, but declined to comment due to time constraints.

In this year’s story, there’s a problem with the amount of snow around the world¹ which (while a problem you’ve solved before) requires drastic measures such as strapping you into a trebuchet and sending you into the sky, and into a set of puzzles.

As a bonus challenge, I tried to make the program that solves all of these problems run in less than the time a Python interpreter to start up and do nothing. On my questionable laptop, this takes 37 milliseconds. I did not succeed in this; after day 16 the budget was more than spent, and I gave up on hyper-optimizing. I’ll include a few tips that I used below for reference.

The puzzles

Day 6: Joe Speedboat

You’re given a few toy boats, where you can press a button to let them gain speed, and when you release the button the boat travels at that speed $v$ proportional to how long you held the button, which needs to cover some known distance $d$ within a given time $t$. In other words, we’re looking for all the values $v: v (t - v) > d$.

The values given are disappointingly small so you can just brute force all possible $0 < v < t$, there’s room to be smart here. The solution that I should’ve thought of first is to realize that $-v^2 + vt > d$ is a simple quadratic equation that you can just solve using the quadratic formula. This is also the fastest way I know to do this, and I’ve revised my initial solution to reflect this. However, I’ve been doing too much Leetcode lately and I’m not thinking of maths, I’m thinking of algorithmic tricks.

This is a problem that can also be solved using binary search to find the minimum value that goes further than $d$. You can trivially prove that the distance covered is maximized when you spend half the allotted time holding the button, and use that as your upper bound. You can set your lower bound at 0 or 1, whichever you prefer.

Using these bounds, you can check in the middle between the two bounds. If that speed makes it further than $d$, it is your new upper bound. On the other hand, if it doesn’t, your lower bound is the middle you tried, plus one, since you know the middle doesn’t work. You can repeat this until the lower bound and the upper bound are equal.

What surprised me is that for sufficiently small numbers, as present in part 1 of the puzzle, this algorithm is very nearly as fast as solving the quadratic equation directly. My only guess is that the square root needs to do a couple of rounds of the Newton-Raphson-method, or whatever estimate my CPU is using to compute roots. For larger numbers, the quadratic formula still decisively wins.

Day 8: Spooky scary camel paths

In today’s puzzle, we are stranded in a desert, and you have to follow a set of instructions telling you to go left or right, along a map that tells you where you end up as you go left or right at a given node. For the first part, there is no actual pathfinding involved; you only need to follow the instructions and accept that the map doesn’t make sense in Euclidean space. The only “smart” thing I managed to do here was to interpret all node names as base 26 numbers, alleviating the need for expensive hash maps. More on that later.

For the second part, our problem gets more interesting. We don’t start at node AAA, we start simultaneously at all nodes that end in A, and we’re only done once all of our parallel realities arrive at a node that ends in Z at the same time. Of course, you can try to simulate it all, but for my input, it requires $\approx 10^{13}$ steps, which with my implementation takes approximately forever.

Our only way forward then must be to find a pattern of when the camels arrive at “end” nodes, then use the Chinese Remainder Theorem² to figure out the period of the whole cycle and determine when things align. There are various ways in which this can be made harder; there could be several end nodes on every cycle, which would blow the problem up.

It turns out the input is not only not evil, it’s kind. There is only one end node in every loop, and the period of each loop is equal to the offset (the time it takes to reach the first end state). Because of that, our final answer is simply the lowest common multiple of all individual offsets. This is hard to see in the visualizations, but you can already see that for the most part, the path is a line around a circle without random jumps.

We can simplify the graph above a bit by removing duplicate edges. This reveals the paths for what they truly are: two circles, where you “randomly” switch between the inner and outer circles. At the end of one of the circles is your exit node, and then you start right back in, roughly at the same place your starting node put you. You can see that in the insert below.

In the making of these visualizations, I managed to crash both Graphviz and Imagemagick, and I figured out that Graphviz does in fact have a line length limit. Due to that, making this explanation turned out to be harder than solving the actual problem. Oh well.

Day 14: Rolling stones

Today we are simulating a bunch of rolling rocks lying on top of a mirror. We tilt the mirror in one of the cardinal directions and then observe the rolling rocks O slide into either the edges or into the stationary rocks #.

The first part of the puzzle verifies that you’ve understood the mechanics of the rolling boulders, and then the second part asks you to do this a ridiculous number of times. Luckily, at some point, the system will start to repeat a pattern and you can take a shortcut. You can use any cycle-finding algorithm to do so (I prefer a modified version of Brent’s Algorithm) to find this shortcut, and then simulate a much smaller number of iterations.

For me, this was the obvious solution, but I’ve been spoiled by 9 years of Advent of Code and other competitive programming interests. I was not going to write about this puzzle in detail, but a colleague who is not as familiar with all that asked me how he was supposed to know that it would repeat, and I didn’t have a clear answer for that. It feels like it should repeat, and even rather quickly, and while I cannot pinpoint why, I can show my clues:

• The number of rolling rocks doesn’t change
• The rocks will (after one cycle) always end up somewhere close to the bottom-left corner
• There is a limited amount of space to put the boulders there, and therefore a limited number of configurations.

That same colleague asked me where that intuition came from. For me, it came from a childhood toy. The very same toy that caused me to fear the second part today after I was reading part one. Instead of “what happens when you shake this thing a bunch of times” which is what we got, I feared the question: how do you have to shake the plate to end up with all the stones in one particular corner? Luckily it wasn’t that, and the toy below was not necessary to solve today’s puzzle.

Day 24: Linear Algebra 101

Welcome to the lecture, please take a seat. We are inside a hail storm, and we want to know if and where the hailstones collide. Every hailstone $h_i$ is defined by its three-dimensional current position $d_i$ and velocity $v_i$ vectors.

Part one

For the first half of the problem, we want to know which hailstones $h_i, h_j$ collide with each other in just the first two dimensions, and whether that happens in some limited bounding box. The first idea I came up with was to start with $\vec{d_i} + t \cdot \vec{v_i} = d_j + u \cdot \vec{v_j}$. Since we’re talking about two dimensions, this gives us the following system of linear equations:

This is a linear system of two equations with two unknowns, so this has either zero solutions or one solution³ in most meaningful cases, or infinitely many if the initial position and velocities happen to be equal. You can solve this with various techniques and I tried implementing the substitution method as it allows you to work in integers without losing precision. Unfortunately, this is hard and I got demotivated.

Luckily the puzzle is not as I’d feared, and our bounding is small enough to fit within the contiguous integer region of a double-precision float⁴ so you can solve the problem with them just fine. This allows you to rewrite the equations into the form of $y = a_i \cdot x + b_i$ by computing the slope⁵ $a$ from the velocity vector and using this instead. When we set these equal as in the equation below, we only have one variable, which is trivial to solve. You can then work backwards to discover whether $t$ and $u$ are indeed in the future.

Part two

Unfortunately, the second half is not nearly as easy. None of the hailstones seem to collide when looking at all three dimensions. Instead, we’ll throw a rock into the storm at some known speed and make it hit every hailstone. This already gives us six unknowns: $x$, $y$, and $z$ for both our velocity and starting position.

At our disposal is a rather infinite supply of equations: each individual hailstone. But we only need three. This gives us three additional unknowns (the time of impact for each of the stones) but also nine equations, one for every dimension of every hailstone.

This number makes intuitive sense as well for me. When you consider only 1 hailstone, you can hit it from basically any direction at any speed. With two hailstones, there are still a lot of options that work out, and they form a kind of plane. This still makes sense, since you have two unbound variables in this case, which do indeed form a plane. Only with three hailstones, there is one path that uniquely intersects all of them. You can of course add more hailstones, but they will only complicate your system of equations.

It turns out you don’t need nine equations, you can make it work with six. Remember, we’re solving the following equations:

That last step may seem weird (it gets rid of the time factor) but it’s actually a property of the cross product $×$. Two parallel vectors have a cross product of zero and since the vector $t_{i} (\vec{v}_i - \vec{v}_{rock})$ is equal to the vector $\vec{d}_{rock} - \vec{d}_i$, it is also by definition parallel to it, and a scalar multiplier such as $t_i$ can be divided out without changing the direction. This is a bilinear equation in both $\vec{v}_{rock}$ and $\vec{d}_{rock}$, which you can then set equal between different pairs of hailstones and solve it.

Anyway, since solving large systems of equations math is not my forte I enlisted the help of a general linear algebra crate, ndarray. By itself, it cannot solve $A \cdot x = b$ unfortunately, so I had to add a separate crate that can do it. The supported way appears to be ndarray-linalg which wraps the classical Fortran lapack library. I briefly tried to get that to work, but I couldn’t get it to pick up on the openblas libraries in my system, and then it wanted to compile them itself. I instead found the linfa-linalg crate that offers the same functionality in pure rust. It doesn’t appear very maintained, but I can attest that it works perfectly fine. That just leaves the equation to solve. I put it below.

Other days

Not every puzzle has a lot of detail to it, but I still want to share my notes for them as they might be of use to someone else. In order:

• Day 1 I used the aho-corasick crate to detect overlapping matches, as regular regex tends not to find overlapping matches. You need overlapping matches to correctly identify the value of fiveight as 58 rather than 55. This is an optimal solution on paper, but a colleague of mine showed that since the words we’re searching for are relatively slow, just hardcoding the matching code from the beginning or end is even faster.

• Day 2: You can ignore the individual grabs of marbles and just look at the largest number of marbles present for any particular colour. To my surprise, this still worked for part two.

• Day 3: Part 1 was relatively straightforward, for part 2 the key insight is that you want to store the values that are connected to cogs per cog. Also small assumption that the numbers are only connected to a single cog, but this happens to be true for my input.

• Day 4: One observation: lottery tickets are integers below 100 (since they’re at most two digits) so you can represent the tickets in a set as bits in a 128-bit integer. This makes determining winners a simple bitwise AND operation.

• Day 5: All mappings are the same, so you can just make them the same. I initially thought of using a BTree for look-ups per section, but a sorted array has the same time complexity and better cache locality.

  For part two, Rust happens to be fast enough that my part 1 solution could solve it by brute force in under a minute, but by transforming entire ranges of seeds at once you can speed it up considerably.

• Day 7: Fun puzzle, only one snag. You should read the text well. I completely missed that the J moves down in the priority order of the second part, which cost me an unreasonable amount of time.

• Day 9: This was not too difficult, but it was fun to think about how to implement this efficiently. For me the, crucial observation was that you are computing the backwards diagonals as shown in the examples. Then after finishing the puzzle, I realized that you don’t have to special-case the initial values and that if you treat those as derivatives as well the whole system still works.

• Day 10: Part one is not interesting, but for part two I learned something new about geometry. You can use a kind of ray-casting to count the number of times you cross an edge of a polygon, and if that’s odd, you’re on the inside. I’ve seen this called the crossing lines theorem but I can no longer find the sources where I got it from and now I wonder whether it was all a dream.

• Day 11: Classic Advent of Code puzzle. The first half shows you a simple problem, and the second verifies you didn’t pick the hardest way of solving it.

• Day 12: “Hot Springs” referring to actual metal springs is quite possibly the best pun in the event’s history. Eric, if you’re reading this, it’s okay that you’re not sorry. I’m proud. Also, the problem is a good introduction to Dynamic Programming, should you need one.

• Day 15: The puzzle was designed to teach you about hash maps, and then make you use them. The only tricky part is that for part 2 you need to remember the insertion order of the items. In Python, the dict already does that, in other languages, you need to opt into it. The data structure you’re looking for is called the linked hash map. It combines the traditional hash map with a linked list going through the items, which maintains the insertion order.

• Day 16: Just do it, still looking for a way to be smarter than “just try every individual position.”

• Day 17: Dijkstra and A* are still interesting algorithms. No other work is required.

• Day 18: Carrying on from day 10, we’re looking for the area of a polygon. The crossing lines theorem I used back then might still work, but there’s also the shoelace formula, which elegantly computes the area as long as you know the coordinates of the corners.

• Day 20: The start of the final stretch, and for me the first truly hard challenge. I didn’t like that part twp requires a deeper investigation of the network. I always feel coding specifically for my input instead of for the problem is doing it “wrong.” I made a visualisation to see whether the structure would make more sense. It does not.

• Day 21: Second unreasonably hard puzzle. In the first half, you learn the magic of parity. Because you cannot move diagonally, and therefore cannot waste moves, you can only ever reach squares that are 1. reachable in fewer than 64 steps, and 2. have an even distance.

  Part two, I don’t quite know. For reasons that are caused by the shape of the real input but not the sample input, there’s a repeating pattern. The number of garden plots you can reach whenever you reach the edge of a map tile (after 65 steps, and then every 131 steps) fits a quadratic expression. You can simulate until you reach the first three, and then extrapolate from there. Conveniently, the number of steps you have to simulate is $65 + 2023 \cdot 131$.

• Day 22: Still a lot of work, but in the end, it was a fun, messy simulation. Nothing is fundamentally hard about it. You can do a topological sort to avoid a repeated simulation in part two, but not doing it is still reasonably fast.

• Day 23: It took me embarrassingly long to figure out my graph simplification code. The main trick I went with in the end is to distinguish between “junction I need to do something with” and “path I need to do something with.”

  The longest path problem is fundamentally NP-hard, though the slopes remove any potential cycles in part 1. For part 2, this is no longer true, and you get to brute force all of the potential routes. Takes a while, even in Rust, but it’s not hard assuming you programmed part 1 right.

• Day 25: Finally, I have relevant expertise. The puzzle tries to teach you about the Minimum cut problem, and once upon a time, I wrote a thesis around the related maximum flow problem. The minimum cut is the minimum amount of (weighted) connections that need to be cut to be disjoint. In this case, we’re not looking for that, since we’re given the answer: three. Instead, we’re interested in where to cut.

  I tried to implement Karger’s algorithm but I failed to correctly detect the cases where it doesn’t find the cut correctly. Instead, I opted to repeatedly find the shortest path between two random nodes, and count how often each edge is used. The most used edges are maybe probably the right ones.

And that’s all the notes I had on the puzzles.

Performance tricks

During the first half of the event, I worked on over-optimizing my puzzle solutions in hopes of solving all puzzles in less than the time it takes for a Python interpreter to start up. I got the idea from a blog post I read a few years ago but I’m not able to find any more. Nevertheless, here are what few tips I have based on my experience trying to optimize my runtime.

• Don’t use hashmaps if you can avoid it. While your access is $O(1)$, your constant factor is important, and Rust’s hash function is somewhat inefficient by design, for security reasons. Many times, if you can use a large array and map your keys to integers yourself, you should.

• Avoid allocations. Try to reuse heap-allocated collections by passing buffers to inner functions. You also want to avoid nested collections, as you’ll easily start getting bottlenecked on the allocator. On Day 9 for example, I managed to reduce the number of lists in my input to two, by storing all of the numbers in one list and then in a second list store ranges of indices that belong to each line. Admittedly, you could implement this puzzle with streaming which might even be faster.

• Avoid large types. Rust written naturally has a bunch of “moves”, where the compiler has to effectively memcopy a variable from one place to another. In most cases, the optimizer can make this efficient by constructing things in their final location, but this does not always happen. To help, you can instead use indirect storage. For example, instead of an [u8; 1024], you can instead use a Box<[u8]>, or even a Box<[u8; 1024]>, but do note that constructing the latter will sometimes run into stack overflow issues.

• Be mindful of your iteration order. DFS is generally better than BFS by a surprising margin, even with an explicit stack. The improved locality of your data access is surprisingly important for your cache hit rate. The same goes for iterating over grids; if you can mostly read the grid in the order that it is stored in memory, it will likely be faster.

All of these tricks combined helped me achieve a total runtime of slightly under 0.4 seconds. Much worse than I’d originally set out to, but I’m pretty content with the result.

In conclusion

This year’s edition of Advent of Code was pretty hard. According to the subjective opinion of some guy on Reddit, it is the second hardest edition so far. To me, it was the first year that I truly got burned out near the end.

The last five days required a lot of work to solve, either because programming them was somewhat tedious, complicated math was involved, or because you shouldn’t try to solve the generic problem and should look at your input. They were not fundamentally wrong, but the lack of rest days in between was a bit much. The Reddit threads also show a pattern of “I just threw Z3 at it and it solved it” which is the opposite of fun to me.

I know I’m doing the puzzles wrong by trying to over-optimize on time, but even with that in mind, I don’t think this was particularly fun. You might also notice that I didn’t have any fun and elegant algorithms to talk about in this review.

Should there be a next edition⁶ I’ll probably still participate, but in a language that is more suitable for bodging together a quick and dirty solution. Python is always nice, though if I find something new along the way that’s also cool.

Let me not discourage you; Advent of Code was still fun. It’s kept me working hard throughout the month, and finishing my 450th star was still satisfying.

Happy New Year, and let’s meet back up for the puzzles in December.

To be precise, it wants to run as a Node.js process on your server. Unfortunately, I have a bit of an aversion to running node processes on my server¹ so I wanted to avoid that. Luckily, this is actually a somewhat supported process. Today I’d like to elaborate on the “somewhat” bit, and hopefully help someone else looking to solve the same problem.

Creating a static export

Let’s start with the good part: Next.JS actually has a pretty nice static export feature that allows you to generate a static version of a website, and ship it as just a bunch of files. This is great because it avoids running an additional web server process just to serve an ostensibly static website.

In previous versions of Next.JS, the export was made through the next export command. You’d still need to next build first, making it somewhat inconvenient. In version 13.3 however, the static export became a configuration setting. To be clear, this article explains how to set this up for Next.JS 13.3 and up.

To start, we need to update our next.config.js file to create a static export as its build result. This s simple enough:

1
2
3
4
5
6
7
8
/**
 * @type {import('next').NextConfig}
 */
const nextConfig = {
  output: 'export',
};
 
module.exports = nextConfig;

That is actually all. An out folder should not contain the static export of your website.

Image optimization

Except it might not be. If you use any images on your website at all. You will be greeted with the following message instead:

Image Optimization using the default loader is not compatible with export.

By default, Next.JS tries to serve images that are optimized for the specific resolution and quality preferences of the user. This, in theory, saves data, but it also has the potential to slow things down due to reducing cache hit rates and generally introduces complexity.

Next.JS provides support for external optimization services which you can use, but in most cases, it offers a very minor performance improvement. My recommendation is therefore to make an appropriately sized WebP] export of your images, either lossy or lossless, and forego optimization altogether.

The error message already explains how to disable image optimization, but for completeness, the final configuration file looks like this:

1
2
3
4
5
6
7
8
9
10
11
/**
 * @type {import('next').NextConfig}
 */
const nextConfig = {
  output: 'export',
  images: {
    unoptimized: true,
  },
};
 
module.exports = nextConfig;

Other unsupported features

Even with the issues regarding images fixed, there are still some things that don’t work. In general, features that require configuration of the response beyond its contents won’t work, be it headers or status code. You also cannot run server side code, because there is no server.

Configuring Nginx

With our archive of files in hand, we can now try to configure Nginx to host it. This is almost straightforward, except that, while Next.JS likes to have URLs without file extensions, but the export does generate files with file extensions. The file for a URL like /page will be /page.html, and occasionally for reasons I don’t quite understand /page/index.html.

It’s possible to fix this with Nginx configuration, by using the try_files feature. The only other thing that needs configuring is the 404 page, which is conveniently generated as 404.html. With that in mind, our (simplified) configuration looks as follows:²

1
2
3
4
5
6
7
8
9
10
11
12
server {
    listen 80;

    root /your/web/root/;
    index index.html;

    error_page 404 /404.html;

    location / {
        try_files $uri $uri/ $uri.html =404;
    }
}

Now we can upload the files generated before to our web root, and the website should be served the way we expect it to.

In conclusion

While it’s a shame that the default way to host Next.JS is a dedicated Node process, it’s pleasantly simple to statically host it statically on a shared web server. Not just that, but ever since I started using the feature in January of this year, the list of unsupported features has steadily shrunk.

As of now, it seems that most features that could reasonably be supported by some static files is actually supported. Things that require setting custom headers, status codes, or even doing server side work obviously isn’t there, but if your website is actually static, most things will actually work.

So with that in mind, I encourage anyone to try to ditch their Node.js servers and run everything through a nice and boring Nginx server.

The very quick version is that asynchronous functions in Rust are just syntactical sugar for regular functions, but instead of directly returning the value it gives you a complex state machine that implements the Future trait.

1
2
3
4
5
6
7
8
9
// The following function
async fn foo() -> i32 {
    // …
}

// Actually desugars to
fn foo() -> impl Future<Output = i32> {
    // …
}

The compiler then generates an appropriate type and implementation. That is helpful, but it’s still overly complicated. I will focus on what I think are the two most important components, the Future trait and the executor, and finally show my minimal executor implementation that can be used to run your async code. Let’s start with the trait first.

The Future trait

The Future trait, while fundamental to how async Rust works, is a rather simple one. It’s so simple, here it is in its entirety:

1
2
3
4
5
pub trait Future {
    type Output;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

The Output associated type denotes what type the future will eventually return, and the poll method will check whether the asynchronous process is complete yet and will return either Poll::Ready(Self::Output) if it was complete or Poll::Pending if it’s not. That Pin<&mut Self> has complicated reasons for its existence that we will largely ignore in this article.

Crucially, a Future by itself does nothing. It should return quickly with either a Poll::Pending or Poll::Ready so that the program can either continue to check other futures or go back to sleep. The actual work of the future should happen elsewhere.

The only other interesting thing about the poll method is that it takes a Context method. The context one purpose as of the time of writing: to provide you with (a reference to) a Waker that can later wake the Executor. We will talk about that next.

An example Future

So async functions get turned into impl Future types, but something has to be async in the end. To illustrate this, let’s look at a simple future that returns nothing, but spawns a thread and completes when the thread it spawned is done.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};

struct ThreadedFuture {
    started: bool,
    completed: Arc<AtomicBool>
};

impl std::future::Future for ThreadedFuture {
    type Output = ();

    fn poll(mut self: std::pin::Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        // 1. Check if the completion was already done
        if !self.completed.load(Ordering::Acquire) {
            if !self.started {
                // 2. If we have not completed and haven't started the thread, start a background thread.
                self.started = true;

                let completer = Arc::clone(&self.completed);
                let waker = cx.waker().clone();

                thread::spawn(move || {
                    // 3. First mark the future as completed
                    completer.store(true, Ordering::Release);
                    // 4. Then wake up the executor
                    waker.wake();
                });
            }

            // 5. Now we've set up the async completion but haven't finished yet, so return Pending.
            Poll::Pending
        } else {
            // 6. Already completed, so return.
            Poll::Ready(())
        }
    }
}

There is a bunch of boilerplate here, but the basics are relatively simple:

1. Check if we have completed the future. If not, check if we’ve started the async computation
2. If not, spawn a thread in the background that will eventually complete the computation
3. In that thread, mark ourselves as completed
4. And wake up
5. Then return a Pending state
6. Finally, return the Ready state if we turned out to have completed.

The Executor

The other piece of the async puzzle is the executor. Executors primarily need to poll futures, go to sleep when there’s nothing to do, and provide a suitable Waker to be woken up with. Many implementations, such as tokio, provide a ton of additional functionality, but deep down all it needs to do is complete the following steps:

When the user provides a Future to the executor, the executor will start polling it until it finally returns a value. This polling is not a busy loop; instead, the executor will wait until it receives a wake-up.¹ This signal can come from any thread, or even a simple C-style signal handler, and will generally be specific to the asynchronous process underpinning the operation. Alternatively, in flow chart form:

For the implementation of the executor, it doesn’t matter where the signal comes from. As long as it causes a wake-up, the future should be polled again. With this knowledge, we can construct a basic executor. Except we don’t have to; in the documentation of the standard library you can find an intentionally somewhat buggy executor implementation² that implements just this.

I could write a correct sample executor like I did an example future, but it so happens I already did and published it as a crate. Let’s have a look at that now.

Introducing Beul

The main reason for this article is the 1.0.0 release of Beul. Beul is a safe, minimalistic futures executor based on the ideas above. It is so minimalistic, it is only 84 lines of (commented) code and its entire public API looks like this:

1
pub fn execute<T>(f: impl Future<Output = T>) -> T

The code is relatively straightforward from there and if you’re still trying to grasp the idea of async execution I recommend you check it out. The obvious question is then, what would one use this for, instead of a fully featured async framework? The main reasons are writing tests of executor-agnostic async code, and working with async libraries in largely synchronous code.

It is possible to nest calls to beul::execute, where you call some sync code from your async code which in turn uses Beul to call async code. In many cases, it is still better for performance to not do this, and make sure to .await any asynchronous function calls whenever possible.

Prior art

Beul is hardly the first minimal futures executor. Many crates implement the same thing with similar ideas. It seems appropriate to acknowledge the ones I am aware of, and why I think Beul might be a better choice.

• Pollster provides largely the same implementation as this crate, with one minor use of unsafe code that can be removed as of Rust 1.68. It uses an extension trait to provide its blocking functionality. It has recently also gained a set of (optional) procedural macros that allow you to annotate async main() or tests in a way where they can be executed normally. Beul is slightly more minimal and uses dynamic dispatch to reduce code size.

• futures-executor provides several executors as well as utilities to make working with futures simpler. Its block_on executor is similar to Beul’s API, though calls to it can intentionally not be nested. It’s also quite a large crate.

• extreme has the same API as Beul but predates the Wake trait and as such has to use unsafe Rust for its executor implementation. extreme is also licensed under the GNU Public License which may make it unsuitable for many applications. The versioning scheme is somewhat peculiar though fine.

• Yet Another Async Runtime (or yaar) and safina-executor both provide more of a general executor framework rather than simply something that executes futures. They can make sense if you need more, but for simple future execution, Beul should suffice.

And if that doesn’t convince you, Beul extreme are the only ones among these who have moved to or past 1.0.0, which should make the fine and sarcastic people of ZeroVer happy.

The future

With the current state of standard-library asynchronous Rust, I consider Beul “finished.” It executes futures, it does so efficiently, and there are (as of now) no major improvements possible to how it works.

Of course, that can all still change. Future Rust will surely have a more extensive async standard library and I’d like Beul to support it. For now, this is all.

In conclusion

Asynchronous Rust consists of executors that repeatedly poll futures until they are complete. When a future is still pending, the executor will go do something else, usually sleep, before it polls the future again. Repeat until completion.

With that, I hope I’ve provided an intuition for what is happening under the hood. Ideally, you will never need to look into the details, but when the abstraction gets too leaky, now you know. Also, I would appreciate you checking out Beul. And that’s it.

Background

Like many techie people, I get asked for my opinion on (and preferably a solution to) my family’s computer woes. In this case, my mother had bought a refurbished desktop PC, which after about one year of use had become excruciatingly slow. IO would frequently pause for multiple seconds while the disk reset. All credit to my mother here: this is a valid problem and she had correctly diagnosed it was caused by the disk being terrible. Now that we know that, we can move on to the thing itself.

Looking at the case, you might notice the giant QR code on the label. The link was dead when I started writing this article, but now points to a website displaying ads for shady apps and “connection opportunities” so I will not link it. I can’t find for certain when the disk was a released to the market, but Tweakers pricewatch suggests December 2017. That is suspiciously recent for a website to be this dead.

While erasing the disk for disposal, I found that this pattern of bad performance was pretty reproducible: I could average around 6 MB/s write¹ speed, but this was because it alternated between 30 MB/s bursts and multiple seconds of nothing. This behaviour seemed familiar to me: it is roughly the performance I get from the SD cards my Raspberry Pi eats. That made me wonder whether it was an SSD at all, and not just a collection of SD cards. Time to find out.

Software robustness

Most disks will support S.M.A.R.T., a technology where disks can self-assess their health and track certain potentially harmful or ominous events happening over their lifetime. Think of the number of bad sectors, but also simply the number of hours the device has been powered on. To see whether this disk was actually just very dead somehow, I looked up the S.M.A.R.T. data. This started out okay:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
=== START OF INFORMATION SECTION ===
Device Model:     KingFast
Serial Number:    REDACTED
Firmware Version: R0831B0
User Capacity:    240,057,409,536 bytes [240 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        Not in smartctl database 7.3/5319
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Nov 10 22:05:38 2022 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

But somehow accessing the health metrics does not quite work

1
2
Error -4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.   

The “error -4” in this message is not an unusual error code, it should be an index into the error log. However, the error log is a beautiful sea of nothingness. Alright, so the error reporting doesn’t work, and the metrics are probably rubbish. I had one more trick: S.M.A.R.T. self-tests. Sufficiently broken drives will generally report something of use there.

1
2
3
4
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1639         -
# 2  Short offline       Completed without error       00%      1639         -

Yet again, all clean. So let’s cast aside the software analysis and try to understand what makes it tick on the inside.

Taking it apart

After asking the disk nicely why it was so slow failed, we might as well get it open. The case had a kind of warranty seal on it, but it was already broken. It did prevent me from opening the case regardless. This is a great deal for the manufacturer, as the warranty is void before I even begin.²

After breaking the seal with a little more conviction, we can now open the case. There are no screws or anything; you simply slide a screwdriver in between one of the gaps on the corners and pry the case open.

Finally, with the case open we can start to identify the components and see what we can learn.

I have highlighted the different components in the picture. The first bit to talk about is the SSD controller.

SSD controller

An SSD controller or flash memory controller exists because flash is actually a fickle technology that needs some help to actually be functional. Naively implemented SSDs would have a very short life cycle since any individual bit can only have a limited number of writes. The controller fixes this with various techniques that are out of scope for this article. The thing to note is that they are important and that they need to be good.

The controller included in the KingFast F6 Pro is a SiliconMotion SM2258XT, which as far as I can tell is a pretty normal chip. I even found its fact sheet which is actually interesting, though it also contains a stock photo featuring a woman more commonly known as the jealous girlfriend in the distracted boyfriend meme.³ This is not technically relevant, I just thought it was funny.

One other interesting thing to note is that if you look up this chip, most of the image results are from other people who’ve taken apart their KingFast F6.

Flash memory

So if the SSD controller is, generally speaking, an okay component, the actual storage might be iffy. For that we are looking at the flash memory. These hold the reprogrammable NAND gates that store the data.

Here I am less sure on exactly what components we have. The best lead I’ve got is the product number 29F32B2ALCMG2, which is probably made by Intel, though I cannot find a definitive source on that. There are a lot of complaints about it (sometimes in relation with KingFast) on Chinese internet forums and some tweets from Chinese Twitter users are not that positive either.⁴ I will refrain from linking them, but searching for the model should bring it all up.

That was of course before a friend recommended I ask the Electrical Engineering Stackexchange, where someone provided a plausible description of the chip within minutes. This suggests that the chips hold 32 GiB of data. With the five visible chips, that would space for 160 GiB of data. That begs the question of how this drive stores 240 GiB.

Normally, SSDs have a small bit of extra capacity, called over-provisioning, which should deal with particular data blocks being worn out and needing to be replaced. For example, the Western Digital NVMe SSD that powers my laptop has a true capacity of 512 GiB but only provides 476.9 GiB for actual use, meaning it has an over-provisioning of $\frac{512 - 476.9}{476.9} \approx 7.4\%$. This is somewhat consistent with this paper from Samsung which states they use $6.7\%$ over-provisioning, and highlight the effects of leaving even more space.

Compare this to our patient. It advertises 240057409536 bytes. Curiously, this is neither a round number in binary gigabytes (1 GiB is $2^{30}$ bytes) nor SI gigabytes (1 GB = $10^9$ bytes). SSDs are generally measured in the former, while spinning platters are measured in the latter because they sound bigger that way. Instead, the drive claims to be $29303883 \cdot 2^{13}$ bytes, which you can factor into $2^{13} \cdot 3^3 \cdot 7 \cdot 155047$ if you really want to.

Weird sizing aside, the KingFast F6 Pro also advertises itself as having that exact same number of bytes. That gives us a whopping $0\%$ of over-provisioning. However, assuming what we know from the components on the board is correct, the drive doesn’t even have that number of bytes. Instead, we are left with $\frac{5 \cdot 2^{35} - 240057409536}{240057409536} \approx -28\%$ over-provisioning. In other words, it wouldn’t be over-provisioned, it would be under-provisioned by roughly $\frac{2}{7}$ of its capacity!

In conclusion

There is no real conclusion, at least not yet. Without the official data sheet for the flash memory, we cannot be sure that the drive indeed is missing ±63 GiB of storage. The drives might be 64 GiB a piece, which would be plenty. It would be odd to have so much extra space, but it’s possible.

Even if my suspicions about the missing capacity turn out to be correct, it would still be interesting to know what kind of black magic the SSD controller is doing to pretend there’s more storage than there actually is. It could be compression, it could silently drop data⁵ or it could do something more complicated. My mother somehow hasn’t lost data that we noticed, but that might just be luck combined with a helping of mostly cloud-based document storage.

With my (lack of) tools and expertise, this is as far as I can dig into the problem for now. Please do get in touch if you can help me fill in some of the remaining blanks. I can say one thing for certain, though: this is a terrible SSD and you should buy a different one. That’s what I did, and my mother has been happy ever since.

My website is built on Bootstrap. Bootstrap is very nice because it makes it much easier to get an acceptable design. It does use a bit more javascript than I’d like to The former is nice because it does most of the design for me, but the latter is open for debate. Also, the colour scheme could use some work.

So in no particular order, let’s see what I actually discovered along the way and what things turned out to be useful.

Updating Bootstrap 4.6 to 5.2

Bootstrap has both changed a lot and not changed at all, which made the upgrade somewhat tedious. You can read the Bootstrap changelog for more details. For the most the most part, things were alright. A non-exhaustive list of things that bit me is

• The official Sass compiler was changed from libsass to Dart Sass. You can still compile Bootstrap with libsass, but now you will notice the bugs in the compiler that Bootstrap no longer works around. For example, libsass handles negative numbers incorrectly causing them to sometimes not be equal to themselves. It only causes minor issues like the pagination number boxes being rounded when they shouldn’t, so if you want to avoid Dart you can still go for it.

• Many CSS classes have moved from outer element to the inner one. For example: items in the navigation bar in 4.6 would apply the .active class to the outer .nav-item element, but Bootstrap 5 requires you to apply it to the .nav-link element instead.

Actually putting this all together is not too hard. Most things actually translated quite naturally. As part of updating the styling, I found that it was easier to remove my custom components and replace them with modified Bootstrap components.

Speaking of modifications: previously this site used the Superhero Bootstrap theme which worked well enough but requires a bunch of scaffolding code and doesn’t really integrate into Bootrap, but rather forces itself through. This has funny results like $blue not being blue at all, though most of the time it results in redundant code. Theming is now completely done with variable overrides, and just 30 lines of overrides (including comments and whitespace) is enough to somewhat consistently override default colours and “work around” assumptions of contrast between them.

The basics for this theming can be as simple as the snippet below. The only hard thing about it was choosing a proper color scheme. I found Coolors helpful to compensate for my less-than-stellar design skills. The linked colour scheme is not coincidentally a colour scheme I liked but didn’t actually go for. By all means use it.

1
2
3
4
5
6
7
8
9
10
@import "bootstrap/functions";

// Don't actually use these colors
$body-bg: purple;
$body-color: azure;
$primary: ivory;
$secondary: navy;
$dark: thistle;

// Import the rest of Bootstrap

Moving the build to Github Actions

While I was working on the the big refactor, Github first teased, then finally announced a solution to the much requested Jekyll 4 support in Github Pages. Seeing as I’ve been waiting for this feature ever since Jekyll came out¹ I wasn’t really expecting anything, but it is now possible to manually build your site using Github Actions. That means that we can now have full control over the build process and can do anything we want.

The example workflows Github has provided are actually very simple and you can mostly drop them in. I had one issue with them, which could be resolved by removing the --baseurl override, but this has since been resolved. Great issue response, guys!

It turns out that I don’t really want all that much. The only immediate thorn in my side that I could fix was to update the default Sass compiler. Jekyll ships with sassc by default, which doesn’t handle negative numbers properly and causes a few miscompilations with Bootstrap. Luckily, it also supports using the reference Dart Sass compiler. All that’s needed is adding the sass-embedded gem and overriding the compiler preference:

1
2
3
# _config.yml
sass:
  implementation: sass-embedded

Replacing the math renderer

My posts occasionally contain some maths and because I have been properly taught at some point in my life, I like to typeset that in $\LaTeX$.² Regrettably, support for Latex in HTML is still not really a thing so you need to render it one way or another.

Kramdown by default comes with “MathJax support” but that mostly means that instances of $$some maths$$ are replaced with \(some maths\). No actual rendering is done while building the site. To add the actual rendering, you are required to load quite some additional JavaScript files, which then in turn marks up the text and loads some more fonts for the true Latex look and feel. This is kind of slow and changes the page layout, so you get that much maligned layout shift effect.

You are familiar with this effect, but you might not know it was called like that. For a refresher, try clicking the green button in the demo below.

All in all, I didn’t really like what MathJax did to the site. Luckily, thanks to moving to Github Actions, I could now add my own dependencies to the build. Which brings me to KaTeX. KaTeX is also a JavaScript-based TeX renderer, but unlike MathJax, it can render the math server-side. The final page then only requires some CSS and webfonts to render correctly. The webfonts are technically even optional, though everything looks slightly off without them.

There is an official plugin for Kramdown that makes it use KaTex. You can simply add the gem to your build, and enable with by adding the following to your _config.yml:

1
2
kramdown:
  math: katex

Unfortunately that doesn’t include the KaTeX CSS yet, so we still need to add that to the site. The recommended method appears to be to use the jsDelivr CDN, but public CDNs aren’t that great of an idea so I didn’t want to do that. Instead I wrote my own Jekyll plugin to automatically copy the relevant files from the KaTeX gem. This plugin can probably be done better by someone who actually knows how to write Ruby, but it gets the job done:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
require 'katex'
require 'find'
require 'pathname'

module Jekyll
    module KatexBundle
        class KatexResource < Jekyll::StaticFile
            def destination(dest)
                File.join(dest, 'assets', 'katex', @dir, @name)
            end
        end

        class Generator < Jekyll::Generator
            def generate(site)
                @site = site

                @site.static_files << css_file
                @site.static_files.concat font_files
            end

            private

            def css_file()
                path = Pathname.new File.join(Katex.gem_path, 'vendor', 'katex', 'stylesheets', 'katex.css')

                source = path.dirname()
                file = path.basename()

                KatexResource.new @site, source, '', file
            end

            def font_files()
                assets_dir = File.join(Katex.gem_path, 'vendor', 'katex')
                fonts_dir = File.join(assets_dir, 'fonts')

                fonts = []

                Find.find(fonts_dir).each do |path|
                    next if File.directory?(path)

                    path = path.sub(fonts_dir, 'fonts')
                    dir = File.dirname(path)
                    file = File.basename(path)

                    static_file = KatexResource.new(@site, assets_dir, dir, file)

                    fonts << static_file
                end

                fonts
            end
        end
    end
end

In the future I may decide to release this as a standalone plugin. The KaTeX gem also appears to support sprockets, so maybe I’ll try to get that to work instead. The Jekyll Assets plugin looks promising.

Moving on

The main update is now a few months behind me and things work well enough. Now back to non-radical, incremental change. Or maybe I’ll rewrite it in Cobalt or Zola in a few months. Who knows? For now, the site can look like this for a while.