Security data pipeline FAQs, myths and glossary
Short answers to common questions and myths, plus plain-English definitions of the terms you will meet.
Six myths about security data pipelines
A pipeline means throwing security logs away.
Reducing volume is one option. Data kept out of the SIEM can stay in searchable, low-cost storage and go back to the SIEM when needed.
My SIEM already does this.
A SIEM can filter data after it arrives, and by then you have usually paid to ingest it. A pipeline acts before the data reaches the SIEM.
More data always means better security.
Detection coverage depends on having the fields your detections use. Paying to index every field of every event is a separate cost decision.
A security data pipeline is just a log forwarder.
A forwarder moves data from one place to another. A pipeline also changes the data in transit and can send it to many destinations.
Pipelines are only for large enterprises.
Any team that pays by the gigabyte or runs more than one security tool can use one. Managed options mean you do not need a data engineering team.
Setting one up means reconfiguring every source.
Most pipelines accept the formats your sources already send, such as syslog, agents and APIs. You point each source at the pipeline once.
The basics
What is a security data pipeline?
Software that sits between security data sources and the tools that use the data. A security data pipeline collects the data, processes it in transit and routes it to destinations such as a SIEM, data lake or AI SOC.
Does a security data pipeline replace a SIEM?
No. A security data pipeline sits in front of the SIEM and controls what the SIEM receives. Detection and alerting stay in the SIEM, and older data can be searched in a separate retention layer or data lake.
Where is a pipeline deployed?
In your cloud, on-premises or as a managed service, depending on the product.
What data sources does a security data pipeline handle?
Most handle firewalls, endpoint tools, identity providers, cloud audit logs and SaaS apps. Check the source list for any product you evaluate.
Cost and the SIEM
How does a pipeline reduce SIEM costs?
A security data pipeline removes events and fields your detections do not use before they reach the SIEM, and sends them to cheaper storage instead.
Will reducing data hurt detections?
It can if you remove a field a detection needs. Map detections to the fields they use before cutting anything.
What happens to data not sent to the SIEM?
The data can go to a data lake or a searchable retention layer. Check how each one is searched, since some need a restore before a query and others charge per search. When an investigation needs events, you resupply them to the SIEM or another tool.
Can I use a pipeline during a SIEM migration?
Yes. A security data pipeline can send data to the old and new SIEM at the same time, which makes a staged move simpler.
Running a pipeline
Who manages a security data pipeline?
That depends on the product. With a managed platform, the SOC team usually runs the pipeline day to day and the vendor runs the infrastructure. Self-hosted pipelines usually need security or data engineers to build and maintain them.
Does a pipeline add latency?
Pipelines usually process events in milliseconds to seconds. Check the figure for any product you evaluate.
Can a security data pipeline send data to more than one tool?
Yes. Routing to several destinations, each with its own format, is one of the main reasons teams use one.
About this site
Who runs SecurityDataPipeline.com?
Realm Security, Inc. owns and operates this site. General explanations are vendor-neutral. Where we show Realm, the section is labeled as a Realm example.
How are figures on this site sourced?
Figures from Realm customers are attributed to their source. Calculator results and demo volumes are modeled or illustrative, and each tool says so.
Which privacy policy applies?
This site follows Realm Security’s privacy policy and terms of service.
Glossary
A
Agent
Software installed on a host that collects logs and sends them on.
AI SOC
Security operations tools that use AI models or agents to triage alerts and investigate. They depend on clean, well-labeled data.
D
Data lake
Low-cost storage built to hold and search large volumes of data over long periods. A data lake works best when data arrives parsed and normalized, and does not run detections on its own.
Destination
Any tool or store that receives data from the pipeline.
Detection coverage
How much of the attack behavior you care about your detection rules can see. Often mapped to MITRE ATT&CK.
Detection rule
Logic in a SIEM or XDR that looks for signs of an attack in incoming data and raises an alert.
E
Enrichment
Adding context to an event, such as user, asset or threat-intelligence details.
ETL
Extract, transform, load. A pattern for moving business data between systems, usually in batches.
F
Filtering
Removing events or fields that are not needed downstream.
I
Ingest
The data a tool takes in. Many SIEMs price by daily ingest volume.
L
Log forwarder
A lightweight tool that ships logs from a source to a destination with little or no processing.
Log source
Any system that produces security data, such as a firewall or identity provider.
M
MDR
Managed detection and response. An outside team that monitors and responds to threats for you.
N
Normalization
Converting events from different sources into one shared set of field names and formats.
O
Observability pipeline
A pipeline for operational data such as logs, metrics and traces, used by engineering teams.
OCSF
Open Cybersecurity Schema Framework. An open, vendor-neutral schema for security events.
P
Parsing
Splitting a raw log line into named fields.
PII
Personally identifiable information, such as names, email addresses or ID numbers.
R
Redaction
Masking or removing sensitive values from data before it is stored or shared.
Rehydration
Sending selected events from low-cost storage back to a SIEM or another tool when an investigation needs them. Also called resupply or replay.
Retention
How long data is kept, and where.
Routing
Deciding which destination receives each event.
S
Schema
The agreed structure and field names for a type of data.
Security data pipeline
Software that collects security data from sources, processes it in transit and routes it to destinations such as a SIEM, data lake or AI SOC.
SIEM
Security information and event management. The tool that collects security data, runs detections and supports investigations.
SOC
Security operations center. The team that monitors, investigates and responds to threats.
Syslog
A long-standing standard for sending log messages over a network.
T
Telemetry
Data that systems emit about what they are doing. In security, mostly logs and events.
X
XDR
Extended detection and response. A platform that correlates endpoint, network, identity and cloud data to detect threats.
Curious? Start a free 7-Day Data Assessment
Try a pipeline on real data from your environment. Realm’s free 7-Day Data Assessment starts with one of your log sources, usually firewall logs. Realm checks every cut against the detections you run. In about a week you get a report on what is safe to cut and what it would save, with detection coverage shown before and after. There is no commitment.