How a security data pipeline compares to other tools
We often hear questions from SOCs about the difference between security data pipelines and other tools, so here’s a simple definition.
vs SIEM · vs data lake · vs log forwarder · vs observability pipeline
| Tool | Main job | Where it sits | Changes the data? |
|---|---|---|---|
| Security data pipeline | Collects, processes and routes security data | Between sources and tools | Yes, in transit |
| SIEM | Analyzes data, runs detections, supports investigations | Destination | Parses on arrival |
| Data lake | Stores large volumes cheaply for search and analytics | Destination | Rarely |
| Log forwarder | Moves logs from a source to a destination | On or near the source | Little or none |
| ETL tool | Moves and transforms business data, often in batches | Between databases and warehouses | Yes, usually in batches |
| Observability pipeline | Collects and routes logs, metrics and traces for engineering | Between apps and monitoring tools | Yes, in transit |
What’s the difference between a security data pipeline and a SIEM?
A SIEM and a security data pipeline do different jobs. Most teams that run a pipeline still run a SIEM.
A SIEM analyzes security data to find threats. A security data pipeline decides what data reaches the SIEM, and in what shape. They work together. The pipeline sits in front, and the SIEM does detection, search and investigation.
| Security data pipeline | SIEM | |
|---|---|---|
| Main job | Collect, process and route data | Detect threats and support investigations |
| Where it sits | Upstream, between sources and tools | Downstream, as a destination |
| What it stores | Varies by product. Some keep nothing, while others keep a searchable copy of all data | Indexed data for search and detections |
| How it is usually priced | Varies by volume, by source or by deployment | Usually by data ingested or compute used |
| Who uses it day to day | SOC teams on managed platforms, engineers on self-hosted ones | SOC analysts and detection engineers |
| What it cannot do alone | Run detections or manage cases | Control data before it arrives |
How they work together
A pipeline sits in front of the SIEM. The security data pipeline sends the SIEM the events and fields your detections need, in a consistent format, and sends everything else to cheaper storage. The SIEM then gets to do what it is good at, which is detection, search and investigation.
Do you need both?
If your SIEM costs are stable, you have few sources and no plans to add tools, the SIEM alone may be enough. A pipeline starts to pay off when ingest costs climb, when you add a second destination such as a data lake, or when you plan to change SIEMs.
Realm example
Realm is a SOC-aware security data pipeline that sits in front of SIEMs such as Splunk, Microsoft Sentinel, Cortex XSIAM, CrowdStrike NG-SIEM and Sumo Logic.
Common questions
Does a security data pipeline replace a SIEM?
No. The pipeline controls data before it reaches the SIEM. Detection, search and case management stay in the SIEM.
Can a SIEM do what a pipeline does?
Some SIEMs now offer native tools that filter data before ingest. These tools serve only that one SIEM, and they rarely check a cut against your detections. A security data pipeline serves every destination you send data to.
Which should I buy first?
Most teams already have a SIEM. A pipeline is usually added when SIEM cost or complexity becomes a problem.
What’s the difference between a security data pipeline and a data lake?
A data lake stores data. A security data pipeline decides what goes into it. SOC teams often use a pipeline to route data to a security data lake.
A data lake stores large amounts of data cheaply for later search and analysis. A security data pipeline decides what goes into the lake, what goes to the SIEM and in what format. Teams often use a pipeline to route data to a security data lake.
| Security data pipeline | Data lake | |
|---|---|---|
| Main job | Collect, process and route data | Store data at low cost for search and analytics |
| Where it sits | Between sources and destinations | Destination |
| Speed | Near real time, in transit | Queried on demand, often slower than a SIEM |
| Typical use | Control volume, format and routing | Long-term retention, threat hunting, compliance |
| Main cost driver | Throughput or deployment size | Storage and query compute |
How they work together
A common pattern is to keep a full copy of all security data in low-cost retention and send a trimmed stream to the SIEM. That copy can live in a security data lake or in a searchable retention layer. The pipeline normalizes the data first, so the stored copy is easy to search later.
Which do you need?
If you want to keep more data for longer without paying SIEM prices, you need low-cost retention. A security data lake is one option, where your team builds the schemas and runs the platform. A searchable retention layer is another. If you want control over what goes to each place, you need the pipeline.
Common questions
Can a data lake replace a SIEM?
Not on its own. A data lake stores and searches data, and does not alert or run detections by itself. Teams that build detections on a lake take on that engineering work. Most keep a SIEM for real-time detection and case management.
Do I need a pipeline to use a data lake?
You can load data directly. A pipeline makes the data consistent and lets you send different subsets to the lake and the SIEM.
What’s the difference between a security data pipeline and a log forwarder?
A log forwarder moves logs. A security data pipeline also shapes them and sends them to many places.
A log forwarder moves logs from a source to a destination with little or no processing. A security data pipeline also shapes the data in transit and can send it to many destinations. Forwarders often feed a pipeline.
| Security data pipeline | Log forwarder | |
|---|---|---|
| Main job | Collect, process and route | Ship logs from A to B |
| Processing | Filter, normalize, enrich, redact | Little or none, sometimes basic filtering |
| Destinations | Many at once, each shaped differently | Usually one per stream |
| Where it runs | Central service or cluster | On or near the source |
| Typical use | Cost control, migration, governance | Getting logs off a host |
How they work together
Many teams keep their existing forwarders or agents and point them at the pipeline. The forwarder handles collection on the host. The pipeline handles everything after that.
Which do you need?
If you only need to get logs from a host into one tool, a forwarder is enough. Once you want to change the data or send it to several tools, you need a pipeline.
Common questions
Is syslog a pipeline?
Syslog is a protocol for sending log messages. Syslog servers and relays forward logs. They do not usually transform or route them by policy.
Can I keep my forwarders?
Usually, yes. Most pipelines accept data from common forwarders and agents.
What’s the difference between a security data pipeline and an observability pipeline?
One is specifically designed for SOCs, the other is for general engineering workflows.
Observability pipelines and security data pipelines use similar mechanics. Observability pipelines serve engineering teams working with logs, metrics and traces of all kinds. Security data pipelines are built around SOC needs such as detections, investigations and security schemas.
| Security data pipeline | Observability pipeline | |
|---|---|---|
| Main users | SOC, security and detection engineers | Platform, SRE and engineering teams |
| Data | Security logs and events | Logs, metrics and traces |
| Main goal | Detection coverage, SIEM cost, governance | Reliability, performance, monitoring cost |
| Typical destinations | SIEM, security data lake, XDR, AI SOC | Monitoring and APM tools |
| Context added | Users, assets, threat intelligence | Services, hosts, deployments |
How they work together
Some companies run one pipeline for both. That can work, as long as the security team can see which fields its detections depend on and controls what reaches the SIEM.
Which do you need?
If the goal is SIEM cost, detection coverage or security governance, look for a tool built around security data. If the goal is application monitoring outside of security then it is an observability pipeline.
Common questions
Can an observability pipeline handle security data?
Yes, it can move and transform it but it is not likely to do as efficiently when it comes to understanding your detections and security schemas.
Curious? Start a free 7-Day Data Assessment
Try a pipeline on real data from your environment. Realm’s free 7-Day Data Assessment starts with one of your log sources, usually firewall logs. Realm checks every cut against the detections you run. In about a week you get a report on what is safe to cut and what it would save, with detection coverage shown before and after. There is no commitment.