Way back in September I apologized for the gap since my last update. I am, once again, sorry it’s been so long since I posted an update. I recently presented at talk at Flock 2026 - if you prefer consuming updates via video where live demos fail, you can find that on YouTube.

This post is something of a summary of that talk, but the short version is Siguldry has all the features Sigul has, plus some. That means we can start deploying it in the coming weeks.

PKCS#11

In the last post, I noted I had implemented server-side support for PGP signing. That’s gone now, I’m pleased to say, because I came across a much neater approach. I implemented a small PKCS#11 module inventively called siguldry-pkcs11.

I can’t take credit for any of the ideas I’m about to describe, since I stole them from multiple other projects. Our friends over in Flatcar have to sign things as well, and they use a key stored in Azure Key Vault. The way they use it is that they implemented a minimal PKCS#11 module. That was my primary inspiration, but I also recently spent some time building a drop-in replacement for pesign-daemon and the idea of exposing the signing interface over a Unix socket that can be mounted into isolated build environments was another concept I borrowed.

For those who aren’t familiar, PKCS#11 is a standard that is made up of a C interface. To implement it, you create a shared object file with a few well-known C functions used to discover your implementations of the interface. Users load your shared library, call a well-known function to get a table of functions where you provide your implementations for the various features. What’s nice about this is that common libraries like OpenSSL can speak to PKCS#11 modules, so you can make any tool that uses those common cryptography libraries use your implementations.

PKCS#11 is a fairly broad specification that covers not just signing, but also encryption/decryption, digest calculation, random number generation, and key management (creating, modifying, deleting, etc). What I realized while poking around Flatcar’s implementation is that you actually don’t need to implement everything. You can, instead, stub out all the functions you don’t want to implement and have them return the handy “function not supported” error code. The list of functions I opted to not support is extensive.

In fact, all I implemented was the functions to authenticate, list keys, and sign. However, that’s enough to make it possible to use the Siguldry server with tools including (but definitely not limited to): rpmsign, ostree gpg-sign, cosign (if built with PKCS#11 support), gpg2 (with the gnupg-pkcs11-scd service), sq (once it merges its PKCS#11 support), systemd-measure, systemd-sbsign, pesign, and perhaps most humorously, ssh. All these tools know how to speak to PKCS#11 modules so they “just work” with Siguldry. Any new tool people make will also likely just work.

One thing that’s neat about this approach is that we sit between the tool requesting the signature and the signing server. This means we can hash the content client-side and requests to the server are very small: just a hash, the algorithm used for the hash, and the key name. No more sending a 4GiB ISO over the network only to have the server hash it, that all is done before the request starts.

The last thing that’s really cool about this approach is that if Fedora decides it needs a new signing server and spends some big bucks on a fancy hardware security module, it will definitely ship with a PKCS#11 module of its own so we can start using it with our existing tooling, no development required. If Siguldry stops being the right choice for Fedora, it’s really easy to swap it out.

Unix Socket API

The other thing I did was to implement a Siguldry client that binds a Unix socket and proxies requests from the local system to the remote Siguldry server. I did this because you can expose the socket into locked down environments, such as RPM build environments, install the PKCS#11 module into that environment, and sign things. Right now Fedora uses this general approach for SecureBoot, except it’s specific to the pesign utility. While I’d prefer to disconnect the signing event from the building of a package, it’s important to support the current workflows and this was a neat way to do that which is generic across signing tools.

It has another advantage. Since the Unix socket is set up using a systemd socket unit, we can sandbox the client proxy, and also configure it with the necessary credentials to connect to the Siguldry server and unlock signing keys. This lets you, if necessary, allow anyone to sign content without needing to manage secrets themselves. The secrets can be encrypted with systemd-creds and bound to hardware. All we have to do is ensure the keys that are configured that way get a special flag when queried via PKCS#11: the “protected authentication path” flag.

siguldry-fedora-autopen

The primary way Fedora uses its signing server is via an AMQP consumer that automatically signs things. Robosignatory is the current tool used, but it needs some significant changes to work with this new workflow. In particular, we don’t want to sign content serially, but the fedora-messaging Python library commonly used to interact with the AMQP broker doesn’t handle concurrency well. And, since we now call out to the particular tool used to sign content rather than sending it to the server, the consumer is primarily a mapping between message topics and those various tools.

All those reasons led me to re-implement it, and since I’ve got no imagination with names, it’s called siguldry-fedora-autopen. It uses the lapin AMQP client with Tokio to manage hundreds or thousands of RPMs/ISOs/containers/etc signing operations at the same time. This means builds should no longer languish in the signing queue for many hours.

IMA

All this probably (hopefully) sounds great, but as with all things there are a few problems. Well, really just one big problem. Back in the Fedora 37 days, we enabled IMA signing for RPMs. The short version of how that works is that each file inside an RPM gets signed. And, unfortunately, the PKCS#11 interface is synchronous. This means that rpmsign requests a signature when it creates an OpenPGP signature on the RPM header. It then requests a signature for each file, waiting for each signature request to finish before starting the next one. Now, when the RPM only has dozens or a couple hundred files that happens fairly quickly. A signature might only take a couple milliseconds and the request/response latency might be a couple dozen more milliseconds (or a lot less depending on how close the server is from the client).

Of course, not all RPMs only have a few hundred files. Some have many, many thousand. For these, a simple OpenGPG signature takes less than a second (even if the RPM is, say, 10GiB). The IMA signing can take many minutes, depending on how many files it has. I’ve been running a client locally, with a server in a nearby datacenter, and the largest time I saw for a signing operation was around 40 minutes. This is really unfortunate, but it is important to remember that we can now do thousands of signing operations at the same time, so even when these slow builds are signed, other builds can be serviced in a timely manner.

There’s a couple ways we can make this better, but the “easiest” involve a custom signing implementation that avoids using the PKCS#11 path.

What’s Next?

In the next few weeks I’m hopeful we’ll have enough time to sit down and deploy Siguldry to Fedora’s staging environment. I want to really kick the tires and ensure it’s rock solid before we move it to production, but I think it’s reasonable to hope for that in the next couple of months.

While that’s happening, I’m going to add support for ML-DSA signing. That will allow us to produce signatures that are safe in a post-quantum cryptography world. After that, I’ll investigate OpenPGP’s hybrid signature schemes, although given recent developments I wonder if we want to bother with those. I’ll defer to what the professional cryptographers have to say about that.

Sometime in the next few Fedora releases, though, you should expect to see new signature algorithms in use (pending FESCo approval etc etc).

Comments and Feedback

Thoughts, comments, or feedback greatly welcomed on Mastodon