Concepts
SIEM streaming formats
Enterprise customers do not want to log in to your product to read your audit log. They want your events in the tool where everything else already is, correlated with their VPN logs and their endpoint alerts. Getting that right is mostly a matter of speaking a format their pipeline already parses.
Why a stream and not a download
A security team's working assumption is that anything not in their central system does not exist. An alert that fires in your product, in your interface, reaches somebody who is not watching — and during an incident nobody has time to open five vendors' dashboards in sequence.
Continuous streaming changes the question from can we get your logs to your logs are already here. It also removes a support burden: the ticket asking for last Tuesday's sign-ins for one user stops arriving, because they can answer it themselves in the tool they already know.
The cost is that you are now part of somebody else's pipeline, and pipelines have opinions about format, ordering and volume. Meeting those opinions is easier than it sounds, because there are only a handful in common use.
The formats that matter
| Format | Shape | Where it is expected |
|---|---|---|
| JSON | One object per event | Modern log platforms and generic HTTP collectors. |
| NDJSON | One JSON object per line, newline delimited | The default for bulk ingestion. Streams and splits cleanly. |
| CEF | Key-value pairs in a fixed prefix format | Long-established security platforms with existing parsers. |
| Syslog (RFC 5424) | A structured line with facility, severity and timestamp | Anything that predates HTTP collectors, and plenty that does not. |
NDJSON is the sensible default when a customer has no preference: it preserves structure, streams without buffering, and every log platform reads it. Choose CEF or syslog when the customer's tooling asks for them by name, which the customer's security engineer will do without prompting.
Mapping an event into a security format
The mapping is mechanical but the choices matter. Security tools index a small number of fields — actor, action, target, source address, outcome, time — and anything not mapped to one of those becomes an unsearchable blob. The example below is one audit event in NDJSON and the same event as CEF.
1# NDJSON — her satirda bir olay2{"ts":"2026-06-18T08:14:03.220Z","id":"event_01HQ8ZK3M4N5P6R7S8T9V0W1X2","action":"invoice.exported","actor":{"id":"user_01HQ8ZK3M4N5P6R7S8T9V0W1X2","type":"user","name":"Avery Lindqvist"},"target":{"id":"invoice_2026_06","type":"invoice"},"org":"org_01HQ8ZK3M4N5P6R7S8T9V0W1X2","src":"203.0.113.24","outcome":"success"}34# CEF — ayni olay, sabit onek ve anahtar-deger5CEF:0|Paycux|AuditLogs|1.0|invoice.exported|Invoice exported|3|rt=1781683243220 suser=avery@foo-corp.example suid=user_01HQ8ZK3M4N5P6R7S8T9V0W1X2 src=203.0.113.24 outcome=success cs1Label=organization cs1=org_01HQ8ZK3M4N5P6R7S8T9V0W1X2
Delivering it reliably
A stream that drops events under load is worse than no stream, because it looks complete. The properties below are what a security team will assume and rarely ask about explicitly — right up until an investigation finds a gap.
- 1Deliver at least once, and include a stable event id so the receiving side can deduplicate. Exactly-once delivery over a network is a promise nobody should make.
- 2Preserve ordering within a tenant where you can, and include a monotonic timestamp so the receiver can order events itself where you cannot.
- 3Buffer and retry with a backoff when the destination is unavailable. A collector being down for an hour should produce a catch-up, not a hole.
- 4Batch by size and by time, so a quiet tenant still sees events promptly and a busy one does not send a request per event.
- 5Expose delivery health to the customer. A stream that silently stopped three weeks ago is the failure mode that undermines the whole feature.
The event stream is the backstop for all of this. Events are readable through the API for 30 days, so a customer whose collector was misconfigured can be caught up from the source rather than told the data is gone.
Who configures it
Streaming destinations are the customer's, not yours: their collector, their credentials, their format preference. Configuring it on their behalf means holding a credential to their security system, which most security teams will decline for exactly the reason you would.
So the configuration belongs in a screen the customer's own admin can reach, with the format choice, the endpoint and the secret supplied by them. Your side of the contract is the events, the delivery guarantees, and a visible indication of whether it is currently working.