(by Anton Chuvakin & Usman Chaudhary)
Prologue: Three Years After “The End Is Nigh”
Back in 2023, one of us wrote “Log Centralization: The End Is Nigh?””Š”, “Šan admittedly incomplete-thought blog with a scary premise: after 20+ years of yelling “centralize your logs!” (the earliest surviving deck is from 2003), we may be running out of places where centralizing all the logs is feasible, workable, or even worth the pain. The conclusion then was cautious: centralize as long as you can, in as many places as you can, and augment with some form of centrally defined, lightly managed, highly distributed collection. By 2025 the position got more contrarian: the SIEM of 2027 will be roughly “90% centralized / 10% federated,” and anybody promising you the inverse is selling a demo, not an architecture. This post is the practical sequel. Not “is federation the future?” (it is a future, not the future), but the far more useful question: when, precisely, does the federated SIEM actually work”Š”, “Šand when does it blow up in your face at 3AM?
The Breaking Point of the Centralized Vault
For more than two decades, SIEM tools ran on a simple covenant: collect all telemetry into one repository, pay for the ingest and the storage, normalize everything upfront into one grand schema, and query from one console. In the era of predictable on-premises networks this worked”Š”, “Šand, frankly, it still works for a lot of organizations. But multi-cloud sprawl, ephemeral infrastructure, hundreds of SaaS applications and now AI agents have strained it in four specific ways: The ingest and duplication burden. If you are present in multiple public clouds at scale, you are very likely not collecting logs into one place in one cloud. Egress fees, redundant storage, and pipeline sprawl make that a questionable decision. Add a few hundred SaaS apps, and “one vault” becomes a very expensive fantasy. The fragile taxonomy tax. Forcing thousands of log sources into one rigid, deeply nested data model creates brittle pipelines. This was largely known since the mid 2000s when the first schema on-read vendors appeared. A minor upstream vendor format change silently breaks parser mappings, and your detection rules go blind without so much as a warning. The volume-to-value problem. Some log types are hugely useful in bulk and for investigations but almost never trigger a detection on their own”Š”, “ŠDHCP leases, VPC flow logs, long-tail of DNS resolution logs. Many organizations simply stop collecting them because they are “too costly to centralize” (especially when the SIEM vendor charges per EPS or per GB). That is not a data decision; that is a billing decision masquerading as one (Making that tradeoff an explicit engineering discipline”Š”, “Šcost per detection, which telemetry earns full-fidelity treatment”Š”, “Šis something we’ve written about in FinOps for SecOps.) The volume curve just bent. Log growth was already relentless; AI made it vertical. A recent State of Log Management 2026 research found AI workloads drove a 93% increase in log and telemetry volume in a single year, with 1 in 5 organizations seeing growth above 150%”Š”, “Šand organizations now exclude an average of 86% of their log data just to manage cost. Read that again: most enterprises are already discarding the vast majority of their telemetry, not by security design, but by budget necessity. Agents generating machine-speed telemetry will not slow this down. Here is the honest framing, unchanged since 2023: the problem isn’t that the distributed approach is easy. The problem is that the centralized approach is getting harder as volumes, source counts, and geographic sprawl go up. And, as we keep saying in the output-driven SIEM context: if you collect, you pay. Somebody has to own the hard drives. Note that there is also another driver that is neither cost nor architecture: data sovereignty. For multi-jurisdiction and sovereign-cloud organizations, some telemetry legally cannot cross borders”Š”, “Šresidency mandates make centralizing certain logs not expensive but impossible. For that class of organization, federation is not a temptation to resist; it is a compliance requirement to engineer for.
The Federated Temptation
Into this gap stepped two families of technology alternatives:
- Federated query platforms. Leave the telemetry where it lives”Š”, “Šcloud object stores, SaaS vendor event stores, edge repositories”¦ your uncle’s flooded basement ;-)”Š”, “Šand push the compute to the data via distributed indexing and schema-on-read. The open security lakehouse. Decouple storage from analytics: keep structured logs in open formats (Apache Iceberg, Parquet, etc) on cheap”Š”, “Šrelatively”Š”, “Šcloud storage and query them through engines you already have (BigQuery, Snowflake, Databricks, and friends).
Plus the classic third option that predates both”Š”, “Štiering: dump the “less useful” logs into cheap storage and pray to the security gods you never have to search them at speed. The reality may look different from a marketing glossy or an RSA demo. Specifically:
The pitch is intoxicating: stop paying egress! stop duplicating data! just federate the search! It is also, in specific and bounded conditions, correct. The trouble starts when “specific and bounded” quietly becomes “default.”
The Dark Side of Federation (Read This Before You Sign)
Federated search sounds magical until you are investigating a breach at 2 AM. Here are the costs and risks you actually have to swallow:
1. It is slower than centralized”Š”, “Šunless you architect specifically for speed (and then pay for it)
A federated query across three clouds and forty SaaS APIs is bounded by the slowest source, the tightest rate limit, and the coldest object-storage tier. It looks fast on a demo dataset sitting in one bucket. Cross-source joins on read are expensive by construction. The vendors know this, which is why the serious ones build distributed indexes at the source”Š”, “Šbut indexes must be built, refreshed, stored, and paid for. And here is the trap from the 2023 post: if you deploy big indexers in every cloud, you haven’t eliminated centralization”Š”, “Šyou’ve just created several smaller islands of it. That isn’t inherently bad, but let’s be honest about what you’re doing: you aren’t escaping the architectural tax of centralization, you’re just trading one big central bill and management domain for three smaller ones that will each grow their own operational headaches over time.
2. You trade cost for resilience
This one is rarely on the slide. A centralized SIEM is one thing to harden, one SLA, one on-call rotation. A federated platform is a query engine whose answer depends on N independent sources being up, reachable, authenticated, and under quota”Š”, “Šat the exact moment you need them. You pay less for storage, and in exchange the overall resilience of your detection-and-response platform goes down. Yes, we really do mean it! Every added source is an added dependency, and dependencies fail at the least convenient time, by definition. Naturally, centralized platforms fail too”Š”, “Šbut that risk is priced, contractually owned, and covered by one SLA. In federation, you self-insure across N sources. In theory, people assume that “distributed systems” are somehow more resilient. In practice and in this case, they are clearly less so.
3. No assurance the logs are even there
If you simply hope the logs will be there when your magical decentralized query tool reaches for them, you will be disappointed a lot. Sources get compromised, and attackers delete local logs. SaaS retention windows expire. A well-meaning admin “cleans up” a bucket. Then your IR consultant finishes the engagement and says: “Sorry, not sure what happened here”Š”, “Šthere were no logs”Š”, “Šbut here is the $100K bill for all the things we tried.” Centralization has a cost, but once you pay it, you reliably own the logs. Federation gives you a pointer, not a possession.
4. Compliance did not get the memo
Many mandates directly require collection and centralization. PCI DSS v4 Requirement 10.3.3, for one, expects audit logs to be promptly backed up to a secure and central log server (or other media that is difficult to modify). Security people love to mock regulations as outdated for the cloud era; in this case they are a stabilizing force, perhaps. Yes, you can mitigate this in a federated model”Š”, “Šobject lock, versioning, WORM buckets, immutable retention policies, documented evidence that every source enforces them. But note who does that work: you, the client. The federated search vendor gives you a query layer; it does not give you an audit trail your QSA will accept, at least not without a stressful argument. Budget the engineering time”Š”, “Šand the assessor’s skepticism”Š”, “Šaccordingly.
5. Federated search is workable; federated analytics mostly isn’t
Detection is not the same as search. Continuous complex event processing”Š”, “Šstateful detection windows, multi-event sequences, streaming IoC matches at line rate”Š”, “Šneeds data flowing through one high-speed engine, normalized to something. Mapping blast radius and lateral movement across users, assets, and service accounts needs a persistent entity graph, not a multi-table join fired off on read. If your algorithms rely on normalized logs, you will wait a very long time for all logs to be normalized “naturally” wherever they sit (OCSF or no OCSF). We have barely made centralized analytics work well; decentralized analytics is a research project, not a product category. To detect real-world threats, you may need a separate tool that sits on a stream of pre-normalized data and allows for fast detections. In the age of AI-speed attacks, speed matters again.
6. Operational toil, and nobody to scream at
A natively designed, integrated SIEM is simpler to run than a multi-component stack you assemble at home. A DIY lakehouse-plus-federated-search-plus-detection-layer is a data platform, and data platforms come with data platform engineers. If you do not employ them, you are not building a federated SIEM; you are building a science project with a SIEM logo. And when it breaks, you lose the underrated benefit of a “single face to scream at.”
7. AI agents do not make it less messy (enough)
AI agents genuinely help here in one specific way: they are patient. An agent can fan out slow federated queries in the background without a human staring at a spinner. But “I didn’t save any logs from X”Š”, “Šhey agent, go get me the logs from X” does not work in real life. Worse, watch for the nastiest failure mode: an agent that reports “nothing found” when the truth is “source unreachable.” In a centralized system that distinction is obvious. In a federated one it is a silent false negative, and automation bias will make sure nobody questions it. This is a big deal, folks! Always require explicit status reporting from agents so that absence of evidence does not become evidence of absence.
The Architectural Spectrum: Is There a Middle Path?
Yes”Š”, “Šbut the middle is much closer to the centralized end than the vendor decks suggest. Here is the full spectrum, honestly labeled:
Two observations. First, the “classic tiering” row is where many organizations already live comfortably and should probably stay. Second, the jump from “hybrid” to “federation-first” is not a matter of degree”Š”, “Šit flips who bears the assurance, compliance, and resilience burden from the platform to your engineering team.
When Can Federated Actually Work? The Criteria
Federation is not a wand; it is a tool for specific conditions. The discipline that matters is deciding”Š”, “Šin writing, ahead of time”Š”, “Šwhich bucket each source falls into, rather than discovering the answer mid-incident. For a given log source, federated/decentralized handling works well when all of the following hold:
- The use is largely asynchronous. Deep-dive forensics, threat hunting, post-incident review”Š”, “Šsituations where a query that takes 20 minutes costs you patience, not the company. If an active attacker is moving laterally, you cannot afford an hourglass spinner. The source is reliable, managed, and tamper-resistant. A robust SaaS platform with documented retention, or a cloud store with versioning, object lock, and retention policies you control and can prove. If the attacker who compromised the host can delete the log, that log is not federated”Š”, “Šit is gone. The volume-to-value ratio is terrible. Petabytes of flows and DNS queries you rarely touch but desperately need when a specific IP shows up in an alert. The queries are targeted, not fishing. “All DHCP leases for MAC X on date Y””Š”, “Šyes. “Show me anything weird across everything””Š”, “Šno, that is what your hot core is for. No compliance mandate requires a central, immutable copy of this data”Š”, “Šor you have already built and evidenced the equivalent controls at the source. Nothing in your real-time detection depends on it. Federated data is for context and investigation. The moment a detection rule needs it, it belongs in the core. Somebody owns the “is it still there?” question. A central catalog of what lives where, with retention, ownership, and health checks”Š”, “Šcentrally defined, lightly managed, highly distributed. Without this, you have not federated your logs; you have misplaced them.
If any criterion fails for a given source, the pragmatic answer is boring: centralize that source.
The Pragmatic Hybrid: Mapping Telemetry to Tiers
Rather than an all-or-nothing choice, modern architectures converge on an integrated high-speed core for continuous detection and graph correlation, coupled with open lakehouse federation for on-demand investigation. Concretely:
Strategic Takeaways for Security Leaders
Avoid dogmatic extremes”Š”, “Šbut anchor on centralized. Pure centralization creates cost and schema bottlenecks; a fully disconnected DIY federated stack trades those for operational complexity, lost resilience, performance surprises, and toil. Look for platforms that integrate fast streaming detection with flexible storage options, with the center of gravity firmly in the integrated core. Separate hot detections from cold investigations, and expect AI to widen the gap. Route identity, endpoint, and control-plane data through real-time detection; stow voluminous low-signal telemetry in cost-effective open lakehouses. AI-driven detection will increase the pull toward the hot core”Š”, “Šmodels correlating across identity, endpoint, and cloud events need the data in one place, fresh, and normalized. The federated tier is where AI agents go for context, at their own pace, with a hard rule that “unreachable” is never reported as “nothing found.” Inventory before you federate. Know which logs exist, where, for how long, under whose control, and with what immutability guarantees. If you cannot answer those questions for a source, you are not ready to leave it there. Build on open standards”Š”, “Šbut “open” is not “free.” Open formats preserve agility and keep security telemetry aligned with the enterprise data architecture. Somebody still runs the lakehouse, and that somebody works for you. Test the 2 AM query. Before production, run your worst realistic investigation query across every federated source during a simulated outage of one of them. The result tells you more than any vendor benchmark.
The Verdict
The centralized approach to logs will work as long as it can and in as many places as it can”Š”, “Šthat sentence has survived three years and two blog posts unchanged, and we see no reason to retire it. The physics of cloud-scale data means we will augment the centralized brain with centrally defined, lightly managed, highly distributed collection and federated analysis. Fine. Just remember what you are buying: cheaper storage in exchange for assurance, resilience, speed, and compliance work that lands on your desk. Choose the 10% wisely. Or prepare to explain either your cloud storage bill to the CFO, or your missing logs to the regulator”Š”, “Šand only one of those conversations ends with a budget adjustment. We aren’t going back to the 1980s where you need to telnet to see logs. But we are entering an era where every log has to earn its place in the center.
Related posts
Log Centralization: The End Is Nigh? (2023) Decoupled SIEM: Where I Think We Are Now? (2025) Why Your Security Data Lake Project Will “¦ Well, Actually “¦ Output-driven SIEM”Š”, “Š13 Years Later Stop Building a 2003 SOC with AI: Triage Must Die (Part 2) FinOps for SecOps: How to optimize the agentic SOC for value
SIEM: Centralize Like You Mean It, Federate Like You Have To was originally published in Anton on Security on Medium, where people are continuing the conversation by highlighting and responding to this story.
First seen on securityboulevard.com
Jump to article: securityboulevard.com/2026/08/siem-centralize-like-you-mean-it-federate-like-you-have-to/
