Concepts

Meaningful security monitoring without a SIEM product, and how to reason about what to watch.

This book is a working reference for building basic security monitoring across a self-hosted (linux) environment using tools you already have: the system logger, a few scripts, and an automation layer. It is not a guide to deploying an enterprise SIEM product. It opens with what a real SIEM does and why a small environment does not need most of it, then covers centralizing logs, deciding what is worth watching, turning logs into signals, and getting alerts to a human.

A note on what this book deliberately omits. A monitoring system's exact thresholds and detection logic are kept private, because publishing where the lines are just tells someone how to stay under them. So this book is generous with architecture and reasoning and deliberately quiet on specifics. You will find the shape of how detection works and the categories worth monitoring, but not the exact numbers, timings, or parsing rules. Those are particular to an environment anyway, and adapting the approach to your own is more useful than copying someone else's tripwires.

What a SIEM actually is

SIEM stands for Security Information and Event Management. Underneath the acronym it names its function: gather logs and events, put them in one place, analyze them for signs of trouble, and alert a human when something warrants attention. Collect, centralize, analyze, alert.

The products that carry the SIEM label are another matter. A commercial or enterprise SIEM typically offers a large set of capabilities:

• Log aggregation at scale from hundreds or thousands of sources, with parsing and normalization into a common schema.
• Long-term retention and fast search across enormous volumes of historical events, for investigation and compliance, and potentially forensic evidence.
• Correlation engines that tie together events from different systems to spot multi-step attacks no single log would reveal.
• Threat intelligence integration, matching observed activity against feeds of known-bad indicators.
• User and entity behaviour analytics, modelling normal behaviour and flagging deviations.
• Prebuilt detection rules mapped to known attack techniques, maintained by the vendor.
• Case management and workflow, so a security team can triage, assign, and track incidents.
• Dashboards and compliance reporting for auditors and management.

That is a serious toolset, and for an organization with a security team, meaningful traffic, and regulatory obligations, it earns its cost. It is built for scale, for teams, and for threat models that a small ecosystem doesn't necessarily have.

Why a small ecosystem does not need most of that

In a self-hosted environment serving a few dozen people, most of that capability is overkill, and paying for it, in money or in operational weight, would be spending on a threat model you do not have. You are not correlating attacks across thousands of endpoints. You do not have a compliance regime demanding a year of searchable retention. You do not have a team to staff a case-management workflow. You have less than a dozen hosts, a small set of services, and one person who needs to know when something is wrong.

What that person actually needs is narrow. Know when someone is trying to break in, know when a host is unhealthy, know when a security control has fired, and get told about it without having to watch logs by hand. That is the collect, centralize, analyze, alert function, and every piece of it can be built from tools already present on the systems. The system logger for collection, a central host to receive the logs, small scripts to analyze them, and an automation layer to send alerts.

A homebrew SIEM is an essentially free basic monitoring kit, not a full-fledged SIEM. It gives you the core function, detection and alerting appropriate to a small environment, without the product, the cost, or the operational weight. It will not do behaviour analytics or correlate a sophisticated multi-stage intrusion without a lot of planning and scripting. It will tell you that something is hammering your login endpoint, that a host is running out of disk, that a firewall is dropping a flood of traffic, or that a service is throwing critical errors, which, for a small environment, is most of what monitoring is for.

The honest limits

• This is detection and alerting, not a security operations centre. It watches for the categories of trouble worth watching and tells you about them. It does not investigate, correlate broadly, or hunt for threats you did not think to look for.
• Coverage is a deliberate, bounded choice. You monitor the things most worth monitoring and accept that you are not watching everything. That is the correct tradeoff at this scale, but it is a tradeoff, and pretending otherwise would be dishonest.
• The monitoring is itself infrastructure that must be secured. The central log collector and the automation layer that sends alerts are high-value. The collector holds everyone's logs, and the automation layer can reach the environment and send mail. Monitoring infrastructure needs the same scrutiny as what it monitors, a point the operating page returns to.

What the following pages cover

• The architecture: the collect, centralize, analyze, alert pipeline.
• Collection: forwarding logs from every host to one place.
• What to monitor, and why: the categories worth watching, and the reasoning, without the tripwire specifics.
• Analysis: turning collected logs into signals, with one worked example and the rest in general terms.
• Alerting and scheduling: getting signals to a human, by cron or by an automation layer, and how often.
• Operating it: tuning out noise and watching the monitor itself.
• What I'd tell someone starting out: the lessons learned.