Mutual TLS with a Private CA

Running an internal certificate authority to support mTLS and zero trust.

Concepts

Why nothing inside the network is trusted without proof, and how to run the CA that makes it possible.

This book is a working reference for putting mutual TLS (mTLS) between every internal component of a self-hosted environment, backed by a private certificate authority you run yourself. It opens with the case for doing it, then walks through standing up the CA, issuing and installing certificates, enforcing mTLS at the proxy layer, and the part that makes this sustainable, keeping renewal boring. The specific use-case I provide as an example is Web Application Firewall (WAF)-to-host, but the same pattern can be applied to other services.

Why internal traffic shouldn't be trusted

For a long time, internal networks ran on an assumption of trust by location. A request coming from inside the perimeter was treated as trustworthy simply because of where it came from. But location is not identity. Anything that gains ingress to the internal network including a compromised workstation, a rogue device, or a service host an attacker has gained a foothold on, inherits the trust the perimeter was supposed to gate. On a flat network where internal callers are trusted by default, one compromised device can reach every service that assumes internal traffic is safe, because nothing along the way asks it to prove who it is. The blast radius of a single foothold is everything.

Mutual TLS (mTLS) breaks that assumption. It does two things. It encrypts every internal connection, so a device that manages to intercept traffic cannot read it, and it requires both ends of every connection to present a certificate proving who they are, issued by an authority both sides trust. A backend configured for mTLS does not just encrypt, it refuses connections that cannot present a valid certificate. Being on the network is no longer enough; you have to hold a key.

This does not make a compromised host harmless, and it's worth being precise about what it does and doesn't do. An attacker who compromises a host can use whatever certificate that host legitimately holds, to reach whatever that certificate was authorized to reach. What mTLS changes is the scope of a compromise. On a network based on location trust, one foothold reaches everything. With mTLS, one foothold reaches only what that specific identity was permitted to reach, and a device holding no valid certificate cannot open an authenticated connection to a protected service at all. The boundary moves from "inside the network" to "holds a valid identity," and the blast radius shrinks from everything to one identity's authorized reach.

What "identity" means here: When a person logs in, they prove identity with something they know or hold, a password, a second factor, a passkey. A machine can't do that, so it proves identity a different way; with a certificate. A certificate is a small signed document that says "this host is service.int.example," countersigned by an authority both sides trust. When a service presents its certificate, the other end can verify that signature and know it is talking to the real holder of that name, not an impostor. That is all a machine identity is, a name, bound to a cryptographic key, vouched for by a trusted authority. Where a user identity answers "which person is this," a machine identity answers "which host or service is this," and mTLS is simply both ends asking that question of each other before exchanging any traffic.

Why a private certificate authority

Certificates need an issuer both ends trust. For public websites that's a public CA like Let's Encrypt. For internal service identity, a public CA is the wrong tool:

A private CA is, in effect, the identity system for machines, the same role an IdP plays for users. Users authenticate to an identity provider; services authenticate to each other with certificates from the CA. Both answer the same underlying question: prove you are who you claim to be before I trust you.

Why step-ca

The CA in this environment is step-ca from Smallstep. The reasons it fit:

Architecture

Before the configuration, the model. Three ideas make the rest of this book make sense: where the CA lives, what the trust chain looks like, and the fact that every internal connection uses two certificates, not one.

Where the CA lives

The certificate authority runs on an internal control-plane host, reachable only from inside the private network. It is never exposed publicly. Its only job is to answer certificate requests from hosts that have already proven they belong in the environment, and to hold the signing key that every host trusts. Because that key is the root of all internal trust, the CA host is treated as one of the most sensitive systems in the environment, on par with the identity provider and the secrets manager.

The trust chain

Step-ca is initialized with a root certificate and an intermediate certificate. The root signs the intermediate; the intermediate signs the certificates issued to hosts. Every host in the environment is configured to trust the root. From then on, a certificate is trusted if it chains back to that root; root signed the intermediate, intermediate signed the host cert. This is why nginx is configured with ssl_verify_depth 2 later on: the chain is two links deep, and verification has to be allowed to trace both.

Trust is established once, by pinning the root's fingerprint when a host bootstraps (covered on the next page). After that, no host ever has to be told about individual certificates, it trusts anything the CA signs, and distrusts everything else.

The two-certificate model

A single mutually-authenticated connection involves two certificates, one presented by each end.

Each end verifies the other's certificate against the shared CA root. If either certificate is missing, expired, or not signed by the CA, the connection is refused. That mutual check is the whole point. The backend won't serve just anyone who can reach it, and the service won't forward to just any host claiming to be the backend.

Standing up the CA

This is the identity root for every machine in the environment. Everything else in the book depends on it existing and being trusted. Treat its keys with the same care as any other root secret.

Initialize step-ca

On the control-plane host, initialize the CA:

sudo curl -fsSL https://packages.smallstep.com/keys/apt/repo-signing-key.gpg -o /etc/apt/trusted.gpg.d/smallstep.asc && echo 'deb [signed-by=/etc/apt/trusted.gpg.d/smallstep.asc] https://packages.smallstep.com/stable/debian debs main' | sudo tee /etc/apt/sources.list.d/smallstep.list

sudo apt update && sudo install step-cli step-ca

sudo step ca init

Make sure port 9000 is open on your firewall, this is the default port step-ca listens on.

The prompts and the choices made here:

Prompt Value Why
Deployment type Standalone A single self-contained CA; no need for the more complex modes at this scale.
PKI name (your CA name) Labels the root and intermediate.
DNS / IP for the CA (internal CA host IP) The address hosts will reach the CA at, inside the private network only.
CA bind address (internal IP):9000 step-ca listens here for issuance requests.
First provisioner name admin-provisioner The identity allowed to request certificates. Referenced later when issuing.
Password for the CA keys (strong secret) Encrypts the CA's private keys at rest. Store it in the password manager.

step ca init produces the root certificate, the intermediate certificate, and the signing keys. The root is the thing every host will be told to trust.

The CA key password goes in a file the service reads at startup:

sudo nano /root/.step/secrets/password.txt

This file, and the key material it unlocks, are the crown jewels for your CA. Anyone who can read both can mint certificates the entire environment trusts. Lock down the host accordingly.

Set the issuance policy

Short-lived certificates are one of the main arguments for a private CA, and the issuance policy is where you enforce them. Edit the CA config:

sudo nano /root/.step/config/ca.json

Add a claims block setting the minimum, maximum, and default certificate lifetimes:

"claims": {
  "minTLSCertDuration":     "1h",
  "maxTLSCertDuration":     "2160h",
  "defaultTLSCertDuration": "720h"
}

What this says: a certificate can live no less than an hour, no more than 2160 hours (90 days), and defaults to 720 hours (30 days) if a duration isn't specified. The 90-day ceiling is the important one, it means a stolen certificate is worthless within a quarter no matter what, and in practice renewal happens far more often than that. Setting a maximum is what stops a convenient long-lived cert from quietly becoming a long-lived liability.

Capture the root fingerprint

Every host that joins the trust domain will pin the root by its fingerprint. Get it:

sudo step certificate fingerprint /root/.step/certs/root_ca.crt

Record this. It's used on every host bootstrap on the next page, and pinning by fingerprint means a host checks the CA against a value you already know, so a substituted or malicious root is rejected instead of trusted.

Run it as a service

Run step-ca under systemd so it starts on boot and restarts on failure:

[Unit]
Description=Smallstep Certificate Authority
After=network-online.target

[Service]
User=root
Environment=STEPPATH=/root/.step
ExecStart=/usr/bin/step-ca /root/.step/config/ca.json --password-file /root/.step/secrets/password.txt
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The --password-file is what lets it start unattended, it reads the key password from the file rather than prompting. That convenience is also why that file's permissions matter so much: it is the thing standing between "a service that restarts cleanly" and "anyone who reads this file owns the CA."

Issuing and Installing Certificates

Every host follows the same bootstrap: make the internal names resolve, trust the CA root, then request the certificates it needs.

First: make the internal names resolve

Internal services are addressed by names like service.int.example. These names deliberately don't exist in public DNS, they're internal-only, so each host needs to resolve them locally. Rather than run an internal DNS server, my environment is small enough to just use /etc/hosts entries, which is a right-sized choice for a fixed, small set of hosts.

On each host, /etc/hosts maps every internal service name to its private IP:

10.x.x.2   jump.int.example
10.x.x.3  tools.int.example
10.x.x.4   pass.int.example
10.x.x.6   files.int.example
10.x.x.7   applications.int.example
10.x.x.8  devbox.int.example
10.x.x.9   auth.int.example
10.x.x.10  waf.int.example
# ...one line per host

This matters to mTLS directly, and it's a subtle failure mode worth understanding: mTLS verifies the certificate against the name being connected to. When the WAF connects to service.int.example, it checks the backend's certificate against that name. If /etc/hosts resolves the name to the wrong IP, or doesn't resolve it at all, you get failures that look like certificate problems, such as verification errors and connection refusals, but are actually name-resolution problems. When an mTLS connection misbehaves, confirm the name resolves to the right host before you start suspecting the certificate.

A gotcha while you're in here

Make sure you map the host's own short hostname to the loopback range:

127.0.1.1   thishostname

Without it, sudo incurs a noticeable delay on every invocation, because it tries and fails to resolve the machine's own hostname before proceeding. The symptom (slow sudo) looks nothing like the cause (a missing hosts entry). It isn't strictly an mTLS concern, but every host should get this line as part of base configuration and it's the kind of thing you only debug once if you've seen it before.

Second: trust the CA root

Bootstrap the host's trust in the CA, pinning the root by the fingerprint captured when the CA was initialized:

sudo step ca bootstrap \
  --ca-url https://10.x.x.2:9000 \
  --fingerprint <CA_ROOT_FINGERPRINT>

Pinning by fingerprint is what makes this first trust decision safe. The host isn't blindly accepting whatever the CA presents; it's accepting only the specific root whose fingerprint you supplied out of band. Then install the root cert where the system and nginx will look for it:

sudo install -m 644 -D /root/.step/certs/root_ca.crt /etc/ssl/certs/example-ca.crt

From here on, this host trusts anything the CA signed and nothing it didn't.

Third: request the certificates

Recall the two-certificate model. Each internal connection uses a server cert (held by the backend) and a client cert (held by the caller). For a backend host that serves an application, both get issued.

The host's server certificate, identifies the backend as itself:

sudo step ca certificate \
  --san service.int.example \
  --not-after 2160h --kty EC --curve P-256 \
  service.int.example \
  /etc/ssl/certs/service.crt \
  /etc/ssl/private/service.key

The WAF's client certificate for reaching this host, issued on the WAF, identifies the WAF as the authorized caller:

sudo step ca certificate \
  --not-after 2160h --kty EC --curve P-256 \
  waf-to-service \
  /etc/ssl/certs/waf-to-service.crt \
  /etc/ssl/private/waf-to-service.key

A few things about these commands worth noting:

Enforcing mTLS

This is where the policy becomes real. The two configurations below, one on the calling side, one on the serving side, are what turn "both ends should authenticate" into "no valid certificate, no connection." Everything before this page was setup, this is the enforcement.

Recall the two ends of an internal connection: We'll use a WAF as an example, but this can apply to any communication. The WAF must present its client certificate and verify the backend's server certificate, and the backend must present its server certificate and require the WAF's client certificate. Both halves have to be configured.

The calling side (WAF)

On the WAF, inside the location block that proxies to the backend:

location / {
    proxy_pass https://service.int.example;

    # Verify the backend's server certificate against the CA
    proxy_ssl_server_name on;
    proxy_ssl_name service.int.example;
    proxy_ssl_trusted_certificate /etc/ssl/certs/example-ca.crt;
    proxy_ssl_verify on;
    proxy_ssl_verify_depth 2;

    # Present the WAF's own client certificate (this is the "mutual" half)
    proxy_ssl_certificate     /etc/ssl/certs/waf-to-service.crt;
    proxy_ssl_certificate_key /etc/ssl/private/waf-to-service.key;
}

Line by line, what each directive enforces:

The serving side (backend)

On the backend host, the internal gateway server block:

server {
  listen 443 ssl;
  server_name service.int.example;

  # This host's server certificate
  ssl_certificate     /etc/ssl/certs/service.crt;
  ssl_certificate_key /etc/ssl/private/service.key;

  # Require and verify the caller's client certificate
  ssl_client_certificate /etc/ssl/certs/example-ca.crt;
  ssl_verify_client on;
  ssl_verify_depth 2;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!aNULL:!MD5;

  location / {
    # hand off to the local application
    proxy_pass http://127.0.0.1:80;
    proxy_set_header Host $host;
    # ...forwarding headers
  }
}

The three directives that do the enforcing:

The backend terminates the mutually-authenticated TLS, then hands the request to the local application over plain localhost HTTP (127.0.0.1:80). The application itself doesn't need to know anything about certificates. The gateway enforces mTLS in front of it. That keeps the application simple and puts all the certificate logic in one place per host.

Gotchas

Adapt this for…

Any internal service-to-service connection, not just WAF-to-backend. Anywhere two internal components talk and you want that traffic authenticated rather than merely encrypted, this is the shape, and ssl_verify_client on (or its equivalent on the serving side) is the line that enforces it.

Keeping Renewal Boring

Short-lived certificates are only a good idea if renewal is reliably automated. The price of short lifetimes is that expiry must be a non-event, and the only way to get there is automation.

Certificates in this setup presented live 90 days at most, usually less. Across a dozen hosts, each with a server certificate and one or more client certificates, manual renewal is tedious, and risks that something will expire unnoticed, and take a service down. The goal is a system where certificates are renewed before expiry, and the only time a human is involved is when renewal fails.

Per-host renewal

Each host runs the step-ca renewal service that watches its own certificates and renews them as they approach expiry. As a systemd unit:

[Unit]
Description=Auto-renew internal host certificate
After=network.target

[Service]
ExecStart=/usr/bin/step ca renew /etc/ssl/certs/service.crt /etc/ssl/private/service.key --daemon --exec "systemctl reload nginx"
Restart=always
RestartSec=10
User=root
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=step-renew

[Install]
WantedBy=multi-user.target

The important pieces:

The WAF's client certificates

The WAF holds a client certificate for every backend it talks to (waf-to-service, one per host). These are renewed on a weekly timer which executes a script to check and renew the certificates approaching expiry:

The timer runs it weekly. This is enabled in systemd, not the service file. The timer is the scheduler; it holds no logic beyond when.

[Unit]
Description=Run WAF mTLS renewal weekly

[Timer]
# Run every Monday at 02:00
OnCalendar=Mon *-*-* 02:00:00
Persistent=true
AccuracySec=1h
Unit=waf-mtls-renew.service

[Install]
WantedBy=timers.target

The service file performs the renewal pass, reissuing any waf-to-<host> certificate that expires within the next 21 days. The service is a oneshot: it runs the renewal script once and exits, which is the correct shape for timer-driven work rather than a long-running daemon.

[Unit]
Description=Renew WAF mTLS client certificates (waf-to-*.crt)
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/renew-waf-mtls.sh
User=root
PrivateTmp=yes
ProtectSystem=full
ProtectHome=read-only
NoNewPrivileges=yes

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=waf-mtls-renew

The script walks every waf-to-*.crt in the certificate directory, inspects each one's expiry with step certificate inspect, and renews only those inside the threshold window. Certificates with time to spare are logged as healthy and skipped, so a run that renews nothing still records the full certificate inventory and its expiry dates. When any certificate is renewed, nginx is reloaded once at the end so the WAF picks up the new certificates. When nothing is renewed, the reload is skipped to avoid needless churn.

#!/bin/bash
set -euo pipefail

STEP_BIN="${STEP_BIN:-/usr/bin/step}"
CERT_DIR="${CERT_DIR:-/etc/ssl/certs}"
KEY_DIR="${KEY_DIR:-/etc/ssl/private}"
CA_URL="${CA_URL:-https://10.x.x.2:9000}"
RELOAD_CMD="${RELOAD_CMD:-systemctl reload nginx}"
RENEW_THRESHOLD_HOURS=504  # 21 days

log() {
    logger -p local5.info -t waf-mtls-renew "$1"
    echo "$(date -Is) [waf-mtls-renew] $1"
}

RENEWED_ANY=0
SKIPPED=()
NOW_EPOCH=$(date +%s)
THRESHOLD_SECONDS=$((RENEW_THRESHOLD_HOURS * 3600))

for CRT_PATH in "$CERT_DIR"/waf-to-*.crt; do
    [[ -e "$CRT_PATH" ]] || continue

    BASENAME=$(basename "$CRT_PATH" .crt)
    KEY_PATH="$KEY_DIR/${BASENAME}.key"

    EXPIRES_TS=$("$STEP_BIN" certificate inspect --format json "$CRT_PATH" \
        | jq -r '.validity.end')
    EXPIRES_EPOCH=$(date -d "$EXPIRES_TS" +%s)
    EXPIRES_HUMAN=$(date -d "$EXPIRES_TS" '+%Y-%m-%d %H:%M %Z')
    DAYS_LEFT=$(( (EXPIRES_EPOCH - NOW_EPOCH) / 86400 ))

    if (( EXPIRES_EPOCH - NOW_EPOCH < THRESHOLD_SECONDS )); then
        log "Renewing $BASENAME (expires $EXPIRES_HUMAN, ${DAYS_LEFT}d remaining)"

        "$STEP_BIN" ca renew \
            --ca-url "$CA_URL" \
            --force \
            "$CRT_PATH" "$KEY_PATH"

        log "Renewed $BASENAME"
        RENEWED_ANY=1
    else
        SKIPPED+=("$BASENAME|$EXPIRES_HUMAN|${DAYS_LEFT}d")
    fi
done

if [[ $RENEWED_ANY -eq 1 ]]; then
    log "Reloading nginx"
    $RELOAD_CMD
else
    log "No renewals needed. Current certificate status:"
    for ENTRY in "${SKIPPED[@]}"; do
        IFS='|' read -r NAME EXPIRY DAYS <<< "$ENTRY"
        log "  OK  $NAME — expires $EXPIRY ($DAYS remaining)"
    done
fi

Three pieces divide the work cleanly. The timer decides when (weekly, with catch-up for missed runs), the service defines how it runs (once, sandboxed, logged), and the script does the actual work (inspect every client certificate, renew what's near expiry, reload nginx only if something changed). The design goal behind all three is that certificate renewal is observable and self-correcting. It logs what it did and what it skipped, it recovers from a missed run, and it renews with enough margin that no single failure reaches an outage. Expiry becomes a routine log entry rather than an incident.