Skip to main content

Postmortem: when a routine backup freezes an entire server

· 10 min read
BENE Maël
System Administrator

A business database server (proprietary DBMS engine, on Ubuntu 22.04) froze one afternoon during a backup that had been running for months without a single incident. Eight minutes of downtime, no data lost, but I eventually traced the freeze back to a memory saturation event the previous night, through a seemingly completely different mechanism.

Diagram showing the DBMS engine's internal backup job as the common cause behind three symptoms: RAM climbing to 60 GB, local archives piling up, and the NAS backup saturating

Never expose the Docker socket: Traefik up front, an SSH tunnel for all access

· 5 min read
BENE Maël
System Administrator

The Docker socket grants de facto root access to whatever machine exposes it. This post describes the pattern I set up to let a reverse proxy discover the containers on a remote Docker host without ever exposing that socket on the network: a point-to-point SSH tunnel, paired with a proxy that filters the API itself.

Diagram: incoming traffic reaches Traefik in the DMZ zone, crosses a restricted SSH tunnel to a filtering socket-proxy in the backend zone, the only thing allowed to talk to the Docker socket

What infrastructure-as-code doesn't cover: the machines configured by hand

· 4 min read
BENE Maël
System Administrator

An entire fleet of VMs versioned, with a pipeline that verifies signatures and redeploys itself every 30 minutes. And then two or three machines, outside that system, whose state only lives on themselves. This post is about a case I see regularly, and rarely documented honestly: the technical debt that piles up on the exceptions to the very system you built yourself.

SSH without static keys: 10-hour certificates backed by Entra ID

· 7 min read
BENE Maël
System Administrator

Replacing an admin team's static SSH keys with ephemeral certificates, issued on demand through OIDC authentication (Entra ID + MFA), valid for 10 hours. This post describes the architecture set up on a fleet of about ten VMs, and above all the two or three surprises you won't find in the docs until you've actually put it in production.

Flow diagram: an admin authenticates via OIDC against Entra ID, step-ca signs an ephemeral SSH certificate from the token's claims, which is then presented to the target VM

GitOps without Vault: reducing blast radius by verifying instead of storing

· 12 min read
BENE Maël
System Administrator

Deploying a fleet of VMs with no central server pushing configuration, no secrets vault permanently exposed, and a simple question asked on every cycle: "was what I'm about to execute signed by someone authorized?" This post describes the pull-based GitOps architecture I built around that question, and why it mechanically shrinks the attack surface compared to a classic push model.

Diagram of the ansible-pull pipeline: a VM fetches the Git repository, verifies the tag signature before any execution, runs the playbook only if verification passes, otherwise aborts without executing anything