Skip to main content

Adding Inspection

Attaching application-layer inspection to the reverse proxy. Standing up the edge gave you a proxy that routes and terminates TLS. This page adds the part that examines requests and refuses those identified as malicious.

Installing open-appsec

Open-appsec attaches to nginx as a module plus a set of agent services. With nginx already installed from its official repository and pinned, install open-appsec:

cd /tmp
wget https://downloads.openappsec.io/open-appsec-install
chmod +x open-appsec-install
sudo ./open-appsec-install --auto --no-email
sudo ./open-appsec-install --download

The --download step prints the exact commands to install the nginx attachment module for your specific nginx version. Run the printed commands rather than assuming paths, because they are version-specific. They copy the attachment libraries into place and the nginx module itself:

# example — use the paths from your own --download output
sudo cp /tmp/open-appsec/ngx_module_<ver>/libosrc_shmem_ipc.so            /usr/lib/
sudo cp /tmp/open-appsec/ngx_module_<ver>/libosrc_compression_utils.so    /usr/lib/
sudo cp /tmp/open-appsec/ngx_module_<ver>/libosrc_nginx_attachment_util.so /usr/lib/
sudo cp /tmp/open-appsec/ngx_module_<ver>/ngx_cp_attachment_module.so     /usr/lib/nginx/modules/

Load the module by adding this at the very top of nginx.conf:

load_module /usr/lib/nginx/modules/ngx_cp_attachment_module.so;

Then install the agent services that do the actual inspection work behind the nginx module:

sudo /tmp/open-appsec/openappsec/install-cp-nano-agent.sh --install --hybrid_mode --server 'NGINX Server'
sudo /tmp/open-appsec/openappsec/install-cp-nano-service-http-transaction-handler.sh --install
sudo /tmp/open-appsec/openappsec/install-cp-nano-attachment-registration-manager.sh --install
sudo systemctl restart nginx

--hybrid_mode runs open-appsec in standalone mode, managed by the local policy file rather than a cloud console. That keeps management local and self-hosted, consistent with the rest of the environment.

The learn-then-prevent lifecycle

This is the single most important operational fact about running open-appsec: the machine-learning model must observe real traffic before it can be trusted to block. Open-appsec scores requests by how anomalous they are relative to what it has learned normal traffic looks like. Before it has learned, it does not have a reliable baseline, and turning on blocking prematurely means legitimate but unusual requests get refused.

So the lifecycle is:

  1. Install and run in a learning posture. Let it watch real traffic for a period, weeks, not hours, so it sees the normal range of what your applications actually do: large uploads, automation calls, the odd-looking-but-legitimate request patterns each app has.
  2. Review what it would have blocked. During learning, the logs show what the model flags. This is where you find the false positives before they become outages.
  3. Switch to prevention. Once the baseline is solid, enable blocking. From then on, flagged requests are actually refused.

Skipping the learn phase, or switching to prevention too soon, is the path breaking legitimate traffic and ripping out open-appsec in frustration. The learning period is the cost of the model-based approach, and it is what makes the approach work. Budget time to do it right.

Open-appsec expresses this as modes on each protection, and the naming is worth understanding because it appears throughout the policy file:

  • detect-learn: observe and learn, log what would be blocked, block nothing.
  • prevent-learn: block, and keep learning to refine the model.

You move a protection from detect-learn to prevent-learn when you trust its baseline.

The policy file

The policy lives in /etc/cp/conf/local_policy.yaml. Here is the working policy, annotated. It has three parts: the default policy applied to traffic, the practice that defines what protections run and in what mode, and the log trigger.

policies:
  default:
    mode: prevent-learn                 # overall posture: blocking, still learning
    practices: [appsec-best-practice]   # which practice (protection set) applies
    triggers: [appsec-log-trigger]      # how events are logged
    custom-response: 403-forbidden      # what a blocked client receives
    source-identifiers: ""
    trusted-sources: ""
    exceptions:
      - name: allow-backup-sync
        condition:
          source-ip: "203.0.113.10"     # placeholder — a specific trusted source IP
        action: allow                   # exempt this source from inspection
  specific-rules: []

The exception is worth dwelling on, because it is a real-world necessity and a real-world risk. A specific trusted source, here a placeholder for a known backup sync endpoint, is exempted from inspection because its legitimate traffic pattern was tripping the WAF. That is sometimes unavoidable, but every exception is a hole in the inspection, so each one should be as narrow as possible (a specific source IP, not a broad range), documented as to why it exists, and revisited periodically to confirm it is still needed. An exception you have forgotten is a liability.

practices:
  - name: appsec-best-practice
    openapi-schema-validation:
      configmap: []
      override-mode: detect-learn       # schema validation: learning only
    snort-signatures:
      configmap: []
      override-mode: detect-learn       # signature matching: learning only
    web-attacks:
      max-body-size-kb: 1000000
      max-header-size-bytes: 102400
      max-object-depth: 40
      max-url-size-bytes: 32768
      minimum-confidence: critical      # only block when confidence is highest (see note)
      override-mode: prevent-detect-learn       # web attacks: actually blocking
      protections:
        csrf-protection: prevent-detect-learn
        error-disclosure: prevent-detect-learn
        non-valid-http-methods: true
        open-redirect: prevent-detect-learn
    anti-bot:
      injected-URIs: []
      validated-URIs: []
      override-mode: detect-learn       # anti-bot: learning only

Log analysis and adding exceptions

Once you have completed several weeks in detect-learn, What the mode choices say, and why they differ:

  • web-attacks is in prevent-learn: the core protection (injection, XSS, traversal, and the rest) is actively blocking. This is the protection you most want enforcing, and the one whose baseline stabilises soonest.
  • minimum-confidence: critical: a deliberately conservative setting. Open-appsec only blocks a web-attack when its confidence is at the highest level, which minimises false positives at the cost of letting lower-confidence suspicious requests through to be logged rather than blocked. For a small environment where a false block means a real person is locked out and there is no one else to fix it, erring toward fewer false positives is the right call. A higher-security posture would lower this threshold, accept more false positives, and have a team to perform continuous tuning.
  • snort-signatures, openapi-schema-validation, and anti-bot are in detect-learn: these are observing and logging but not blocking. They are the protections still building confidence, or ones whose blocking behaviour needs more tuning against this environment's real traffic before being trusted to refuse requests.
  • Leaving them in learning mode is the honest position: they add visibility now and can be promoted to blocking later, deliberately, once their baseline is proven.

The mixed posture, some protections blocking, some still learning, is not a half-finished configuration. It is the correct steady state for a model-based WAF: you promote each protection to blocking on its own timeline, as its baseline earns your trust, rather than flipping everything to prevent at once and hoping.

Applying policy changes

After editing the policy, apply it:

open-appsec-ctl --apply-policy

Adapt this for…

Any WAF with a learning or baseline mode, which is most modern ones. The specifics are open-appsec's, but the discipline is universal: run in detection/learning long enough to build a real baseline, review what it flags before you let it block, promote protections to blocking one at a time on their own schedules, and keep every inspection exception narrow and documented. The failure mode this avoids, deploying a WAF that immediately blocks legitimate traffic and gets torn out, is the same regardless of product.

Back to top