How we built LogSeam, part 1: the lake and the pipeline

We spent years building incident response and MDR businesses on top of other people's SIEMs and security products. When we started LogSeam, the first decision wasn't about AI or detections. It was the one every log platform makes first: how to store the logs, which comes down to what you build as they arrive and whether storage and compute share a node.

That choice matters more for a SIEM than for almost any other data system, because a SIEM has to do several things at once that usually pull in opposite directions:

• Take in everything, without stopping. Endpoints, identity providers, firewalls and cloud services write logs around the clock, from hundreds of sources in each vendor's own format, and the formats change between releases.
• Make it searchable in minutes. A detection is only useful while there's still time to act, so data has to be queryable soon after it lands.
• Find one needle, and count everything. An analyst needs one IP across every log in seconds; a baseline or a hunt needs to scan months. Both run against the same data.
• Keep it for a year or more, unchanged. Attackers dwell, investigations reach back and regulators ask. The logs are evidence, so nothing may alter them.
• Absorb bursts. Load spikes during an incident, exactly when the system can least afford to slow down.
• Serve AI as well as people. AI is changing how a SIEM gets used. A person runs a few searches and reads the results; an agent runs hundreds of small queries in parallel, around the clock, across all the data, and needs answers compact enough to reason over, not pages of rows. A system sized and priced for human search strains under that load.
• Stay affordable as it grows. Log volume only goes up. When cost grows with it, teams start dropping the data they'll need later.
• Stay yours. The data should outlive any one tool, in a format other tools can read.

Most data systems handle two or three of these well. A SIEM has to handle all of them, on the same data, at the same time. That's what makes it hard to build, and it's what the choice of how to store the logs decides.

Cost, search speed, retention and whether you own your data all follow from that choice. This is the first of five posts. It covers how we made the choice, the architecture that came out of it and the pipeline that fills the lake. Part 2 covers the compute, part 3 what it costs to run, and parts 4 and 5 the AI and the work.

General approaches to storing logs

There are three typical ways to store logs, and most SIEMs on the market are variations on one of them.

• Document store (Splunk, Elasticsearch, OpenSearch). Every event is kept as a document, plus an inverted index of its terms. Indexers own shards of the data, and a search head merges what they find.
• Columnar index (ClickHouse, Kusto, Druid). Sorted column parts sit on local disk and merge in the background. Each node owns a shard, and a coordinator merges the partial results.
• Data lake (Parquet and Iceberg on object storage, Amazon Security Lake). Open columnar files live in a bucket, and a catalog tracks files, schema and statistics. Search is the part you bring: any engine that speaks the format plans from the catalog's statistics, then reads only the files that survive from the bucket. The common choices are Trino and Starburst, Amazon Athena, Spark and Databricks SQL, Snowflake external tables, Dremio, with Cribl Search and Scanner on the security side. How well that search works decides how useful the lake is.

The first two keep storage and compute on the same node. The lake separates them. Scored against SOC workloads, the typical trade-offs look like this:

We chose the data lake

A document store's index can cost more than the data it indexes, and its nodes are stateful. A columnar engine's format belongs to the engine. Those weaknesses are structural: they fix compute and storage shapes up front and add management overhead every time you scale.

The lake's weaknesses are different. Slow needles, slow small reads, schema mapping, memory-hungry aggregation, compute sized in advance and data that lands minutes behind are engineering problems. So we kept what the lake gives you structurally (one open copy in your bucket, object-storage retention, compute separate from storage) and spent our engineering on its six gaps.

Why not Iceberg

Iceberg is the obvious way to build an open lake, so it's worth saying why we didn't. It's a real standard: Spark, Trino, Athena and Snowflake can all read one table, and you get atomic commits, time travel, rollback and schema evolution without rewriting data.

It charges for that on every query. Catalog, metadata file, manifest list, manifests, then data files: that's a chain of sequential object reads before the first row. Its stats are file-level, so a needle lookup still opens many files. Schemas are declared per table, and small files and snapshots need ongoing compaction.

Our path is shorter: every node looks files up in one in-memory catalog, then reads them straight from the bucket.

What we gave up is real. LogSeam has no transactions: the lake is append-only, with no ACID updates, deletes or rollback. We took that trade on purpose. A log is a record of something that already happened, and it is only worth something as evidence if it never changes. Once a file's final write lands, it is immutable: nothing in LogSeam edits or rewrites it, so the log you search is the log that was written. The files are still standard, open Parquet in your bucket, readable by any engine.

The architecture

Before going deeper, here's the whole system. Logs come in through the pipeline, are written once as open Parquet in your bucket, and every layer above reads that one copy. This post walks through the pipeline and how a search reads the lake; the compute and the cost each get a post of their own.

The AI and the work layers get posts of their own.

The pipeline

LogSeam doesn't need data to arrive in any particular shape. If a source lands as OCSF, fine. If it lands in the vendor's own format, also fine. Either way, it keeps its own columns, and the system works out what they hold. Data typically arrives as JSON, read from a bucket, pushed to an API endpoint or pulled from the source. The JSON is converted to Parquet Variant columns, so nested JSON that drifts from one release to the next stays queryable and columnar without anyone writing a schema for it.

Storage starts with compression, and the shape of the data is what makes it work. Logs repeat themselves: the same hosts, users, event types and field values appear millions of times. Written as columns and sorted by time, each column becomes long runs of near-identical values, and dictionary encoding and compression collapse those runs to almost nothing.

We consistently see 90 to 97% compression on real customers' raw JSON logs. At that ratio a terabyte of raw logs is 30 to 100 GB in storage, which changes what it costs to keep a year of data rather than a month.

Then data moves through bronze, silver and gold stages constantly, not as a nightly job, so small fresh batches are folded into large, well-formed files.

By gold, every file is the right size and shape: sorted by time, with its bloom filters, statistics and page indexes written in. Getting that right once, at ingest, is what lets every query after it skip most of the work. That doesn't mean rules and queries wait for the optimization to finish: jobs run continuously as files flow through the stages.

We catalog every file in the lake and run Rosetta, our file context engine. It looks at each column as it lands and maps between sources, so data in different shapes can be searched together. It works on two layers:

• File catalog. Columns, types, ranges, min and max for every file written, the mechanical bits.
• Semantic layer. The values themselves. A column called source may hold an IP, a region or a direction; Rosetta decides which, maps it to an entity type, and adds context to the vendor's own definitions of the data. That context is reviewed on every pass and added to what the system and its AI know.

Schema drift stops being a project. New columns layer in on their own, with no mapping to write or maintain, and a rule written last quarter still matches. More than 140 log definitions ship in the catalog today, and unknown sources are learned from their values.

Only do the work you need to do

A lake is slow at needles because it doesn't know where anything is until it opens files. We moved that work to ingest: everything a query needs to skip data is written on the way in.

A hunt for one entity works through the lake like this:

1. Catalog and context building (Rosetta). The catalog and Rosetta map every column to the entity types it holds, so the catalog knows which columns to read, even when every source names that column differently.
2. Time-sorted files. Data is written in time order, so whole files fall out of the time range first.
3. Entity index. Before any node is assigned work, the entity index tells the master where this specific user, host or IP appears.
4. Cached footer. Schema, row groups and column stats are read once, then remembered.
5. Column pruning. A log can have hundreds of columns; the query reads the three it needs.
6. Row-group stats. Min and max per column, so whole groups are skipped.
7. Bloom filters. If the value isn't in a group, the group is never fetched.
8. Dictionary and page index. Only the pages that match are decoded.
9. Parallel row groups. Each fetched group decodes on its own core, all at once.

The result: most of every file is never read, only the matching rows cross the network, and the answer is cached.

Where object storage pushes back, and what we do about it

Object storage is cheap and durable, but it isn't a disk. Four of its limits matter for a SOC, and each one shaped a design decision. The latency and request-rate figures come from AWS's own performance guidelines.

Next: the compute

That's the lake and how data gets into it: written once, in your bucket, in a shape a search can skip through. Part 2 covers the compute that reads it: always-on and dynamic nodes, where the searching actually happens, the four shapes a query takes and how the rest of the lake's six gaps close.