Every machine image you launch in the cloud is a trust decision you inherited from whoever made it. Public AMIs are a grab bag of defaults, some sensible, some not, and the one you pick becomes the floor that everything above it stands on. Hardening one by hand is tedious, and worse, it drifts: you tighten a box, you touch it three weeks later, and the careful work has quietly rotted.

So I built a factory. You hand it a product, it hands you back a reproducible, CIS-hardened image. This is how it works, and the bits that turned out to be more interesting than I expected.

The shape of it

One repository, HashiCorp Packer, and EC2 spot instances. It covers around eighteen products, the usual server-room suspects: nginx, PostgreSQL, Redis, k3s, Keycloak, Harbor, Kafka and the rest, across a few OS families. One command builds the lot.

Each build launches a throwaway spot instance, provisions it, snapshots an AMI, and destroys the instance. A full run is dozens of images, five to fifteen minutes each, and a few dollars in spot fees rather than a few hundred. Every build writes its AMI IDs to a manifest so you know exactly what came out.

The design rule underneath all of it: the security baseline is shared, and the software is not.

Hardening is one script, not eighteen

The temptation is to harden each product as you build it. That is how you end up with eighteen slightly different ideas of what “secure” means, and a baseline nobody can audit. So the hardening is a single script that every image runs, whatever it is going to become. The product scripts only add their software and open the specific ports they need, through one helper.

The baseline is CIS Level 1 and 2: uncommon filesystems disabled, /tmp and /dev/shm mounted nodev,nosuid,noexec, a locked-down SSH config, a PAM password policy, and a sysctl pass that turns off the old network foot-guns and closes the easy information leaks:

kernel.kptr_restrict = 2
kernel.yama.ptrace_scope = 1
kernel.randomize_va_space = 2
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1

SSH gets the same treatment: root login off, modern ciphers and key exchange only, a short login grace, a warning banner, forwarding disabled.

Keeping this in one place is the only reason it stays honest across everything the factory makes.

Hardening breaks things, and you find out late

Here is the part the guides skip. A correct hardening step will break a working application, and it will do it quietly, at the worst moment. Mounting /tmp with noexec is exactly right and it also breaks any installer that extracts a binary into /tmp and runs it. You harden the box, the app refuses to start, and nothing tells you why except a permission error three layers down.

The fix is not to loosen the mount. It is to carve a narrow, deliberate exception for the one thing that needs it and leave the baseline intact. Security you cannot see until it bites is the tax you pay for having a baseline at all, and it is worth paying. You just have to know the bill is coming.

”Latest” is a lie you have to design around

The thing I underestimated most: what does “the newest PostgreSQL” actually mean, reproducibly?

Every product answers differently. Distro repositories give you whatever is current at build time. Some projects need the GitHub Releases API to find the newest tag. Kafka means scraping an Apache mirror. A couple ship installer scripts that default to latest stable. So “rebuild the image” quietly means “pick up whatever is current right now,” which is precisely what you want for a security patch and precisely what you do not want as a surprise.

PostgreSQL is the honest example. Its major version is pinned by hand, because letting a routine rebuild jump a major version underneath somebody’s data is how you ruin a Tuesday. The lesson generalises: you can automate the fetch, but the one decision that actually matters, the major-version bump, is the one you should not automate. The factory rebuilds patches on a schedule and leaves the dangerous jumps to a human on purpose.

The boring lessons that still cost a day

Two of them. Spot capacity evaporates, so the build has to retry across several instance types and treat a failure as noise rather than an error, otherwise a full run dies halfway on a bad afternoon. And AMIs are regional: build in the wrong region and the image exists, just nowhere you can use it. Everybody learns that one exactly once.

The factory was the easy part

I built this expecting the engineering to be the hard bit. It was the most enjoyable bit, and it is essentially done: the images build, they are hardened, and the path to publishing them is open.

What shelved it was not technical. A hardened image is only worth something if someone trusts it, and a storefront from a name nobody has heard of sits on the shelf beside established vendors that buyers already trust. The pitch, “take my word for it, this one is locked down”, is a hard sell from an unknown, however careful the work underneath. That is a distribution and trust problem, and no amount of Packer solves it.

So the lights are off, not because the machine does not work, but because I could not answer the only question that actually matters: why would anyone pick this one? The build never asks you that. It should. A production line with the lights off is a fair description of a lot of good infrastructure, and of most of the reason good infrastructure never ships. Turning it on is one command. Earning the trust to bother is the real work, and it always was.