Apache Kafka
Open SourceThe open-source distributed event streaming platform.
Published 29 May 2026 · Last updated 27 September 2026
Scores
Popularity4/5
De facto standard for event streaming used at LinkedIn, Netflix, and Uber — but most applications never need streaming infrastructure, so its relevance is category-high rather than universal.
Learning Curve2/5
Broker configuration, partition sizing, consumer group semantics, offset management, and schema registries all require significant hands-on experience before production readiness.
Flexibility5/5
Handles any event type at any scale; 200+ Kafka Connect connectors cover every major source and sink; Streams API enables stateful real-time processing in the same ecosystem.
Performance4/5
Proven at petabyte scale with sub-2ms publish latency, but JVM GC pauses and ZooKeeper overhead mean newer C++ engines like RedPanda benchmark with lower tail latency under comparable hardware.
Portability4/5
Apache 2.0 licensed; the Kafka API is an industry standard — Redpanda, Confluent, and MSK are all wire-compatible, meaning producers and consumers are reusable.
About Apache Kafka
Apache Kafka is a distributed event streaming platform originally developed at LinkedIn and open-sourced in 2011. It is designed to handle trillions of events per day with latencies as low as 2ms, making it the de facto standard for real-time data pipelines, event-driven microservices, and streaming analytics.
Kafka's core model is a distributed commit log: producers write events (messages) to topics, and consumers read from those topics at their own pace. Topics are partitioned for parallelism and replicated across brokers for fault tolerance. This decouples producers from consumers and guarantees durable, ordered, replayable event streams.
Key APIs include the Producer API (write events), Consumer API (read events), Streams API (stateful stream processing in Java/Python), and Kafka Connect (plug-and-play connectors to databases, file systems, and cloud services). As of Kafka 3.x, ZooKeeper has been replaced by KRaft (Kafka's own consensus protocol), simplifying operations significantly.
In production data engineering, Kafka sits between data sources and sinks — capturing CDC streams from databases, feeding real-time ML pipelines, or triggering downstream processing with tools like Flink or Spark Streaming. Confluent provides a managed Kafka cloud service and enterprise additions.
Key Features
- Distributed commit log with configurable retention (time or size-based)
- Sub-2ms publish latency at petabyte scale
- Kafka Streams API for stateful, exactly-once stream processing
- Kafka Connect with 200+ pre-built source/sink connectors
- KRaft mode: no ZooKeeper dependency since Kafka 3.x
- Consumer groups for parallel, fault-tolerant event consumption
- Topic compaction for changelog and event sourcing patterns
Pros
- Industry standard — massive ecosystem, tooling, and community
- Extremely high throughput and durability at scale
- Flexible retention — replay historical events, not just latest state
- Kafka Connect makes integration with databases and cloud services plug-and-play
- Apache 2.0 licensed: fully open source, no licence concerns
Cons
- Steep operational learning curve — tuning brokers, partitions, and replication requires expertise
- Resource-heavy for small-scale use cases — overkill for simple task queues
- KRaft mode still maturing for some edge cases
- Java/JVM-based, which adds overhead compared to lighter message brokers
- Confluent's managed cloud offering can be expensive at volume
Apache Kafka Pricing
Open SourceTech Stacks with Apache Kafka
Streaming Analytics Pipeline
ProjectEvent-driven pipeline for real-time analytics: Kafka ingests millions of events per second from producers; ClickHouse stores and queries the stream at sub-second latency; dbt runs incremental transformation models; Grafana displays live dashboards and fires alerts. Docker containerises all components.
PostHog Self-Hosted
InfrastructureSelf-host PostHog when product data must stay on infrastructure you control. The official Docker Compose deployment runs the PostHog services with PostgreSQL, ClickHouse, Kafka, and Redis on one large server, and it's built for modest volumes: PostHog recommends it below about 300,000 events a month.
Sentry Self-Hosted
InfrastructureSelf-host Sentry error tracking and performance monitoring on your own server with the official Docker Compose setup. PostgreSQL stores projects, issues, and users, ClickHouse stores event data, Redis handles caching and queues, and Kafka moves events between services. It's the heaviest stack in the family, and it keeps every error and trace on infrastructure you control.
Tools Related to Apache Kafka
Works well with Apache Kafka(2)
PostgreSQL logical replication via Debezium publishes row-level change events as Kafka topics — the standard CDC pattern for capturing database changes in real time.
Databricks Structured Streaming reads from Kafka topics natively via the built-in Kafka source connector — the primary pattern for real-time Delta Lake ingestion.
Integrates with Apache Kafka(3)
Kafka is the primary streaming source into ClickHouse — events flow from Kafka topics into ClickHouse tables via the built-in Kafka table engine.
Airbyte has an official Kafka destination connector; publishes each synced record as a Kafka event, enabling downstream stream processing of ingested data.
Airflow's official Kafka provider lets DAGs produce to and consume from Kafka topics, or wait for a specific event before running a batch job; Kafka keeps the streaming, Airflow the schedule.
Alternatives to Apache Kafka(1)
Redpanda speaks the Kafka protocol as a single C++ binary with no JVM, so Kafka clients work unchanged; Kafka has the larger ecosystem and a fully open Apache 2.0 license.