Lessons Learned

The lessons I've learned from running a single-edge WAF in a self-hosted environment.

• Run learning mode longer than feels necessary. The instinct is to turn on blocking as soon as it is installed, because an inspecting WAF that isn't blocking feels pointless. Resist it. The model needs weeks, if not months, of real traffic to learn what your applications legitimately do, and blocking before it has that baseline means block legitimate requests. The learning period is not a delay before the real work, it is the work that makes blocking safe.

• A single edge means the edge's own hardening and uptime are now critical. Concentrating all traffic through one host is what makes it defensible by one person, and it is also what makes that host a single point of failure. In a small ecosystem that is an acceptable trade, but it raises the stakes on that one host. For an ecosystem larger than a dozen applications and services with a few dozen users, a high-availability WAF tier stops being optional. Run two WAFs behind a load balancer.

• Tune false positives before they train you to ignore alerts. A WAF that cries wolf useless, because a stream of false positives teaches you to dismiss its alerts, with the risk that you miss a real one. Keeping the block threshold conservative, reviewing what gets blocked, and fixing false positives with narrow exceptions is what keeps the alerts meaningful enough to act on.

• Get the forwarded-protocol header right, or chase phantom auth bugs. The single most disproportionate source of pain at the edge I encountered is the X-Forwarded-Proto header. If an application backend doesn't know the original request was HTTPS, it builds http:// redirects, browsers refuse them, and you get login loops that look like an application bug but are actually a one-line proxy header. Set it explicitly, and remember it exists the next time a login mysteriously loops.

• Keep every inspection exception narrow and documented. Sometimes a legitimate source genuinely trips the WAF and needs an exception. Every exception is a hole in the inspection, so make each one as specific as possible. A single source rather than a range, to a URL, and include any relevant conditions. Write down why they exist, comments in the yaml is appropriate, and revisit it periodically. An undocumented exception whose purpose no one remembers is a liability sitting in your security control.

• Remember nginx is pinned. Because open-appsec attaches to specific nginx versions, an automatic nginx upgrade can silently break inspection. Hold nginx, check open-appsec's compatibility before updating, and move the two together. A WAF that has stopped inspecting because nginx moved out from under it looks exactly like a WAF finding nothing, which is the most dangerous failure mode there is.