Apache Kafka
Open SourceThe open-source distributed event streaming platform.
Scores
About
Apache Kafka is a distributed event streaming platform originally developed at LinkedIn and open-sourced in 2011. It is designed to handle trillions of events per day with latencies as low as 2ms, making it the de facto standard for real-time data pipelines, event-driven microservices, and streaming analytics.
Kafka's core model is a distributed commit log: producers write events (messages) to topics, and consumers read from those topics at their own pace. Topics are partitioned for parallelism and replicated across brokers for fault tolerance. This decouples producers from consumers and guarantees durable, ordered, replayable event streams.
Key APIs include the Producer API (write events), Consumer API (read events), Streams API (stateful stream processing in Java/Python), and Kafka Connect (plug-and-play connectors to databases, file systems, and cloud services). As of Kafka 3.x, ZooKeeper has been replaced by KRaft (Kafka's own consensus protocol), simplifying operations significantly.
In production data engineering, Kafka sits between data sources and sinks — capturing CDC streams from databases, feeding real-time ML pipelines, or triggering downstream processing with tools like Flink or Spark Streaming. Confluent provides a managed Kafka cloud service and enterprise additions.
Key Features
- Distributed commit log with configurable retention (time or size-based)
- Sub-2ms publish latency at petabyte scale
- Kafka Streams API for stateful, exactly-once stream processing
- Kafka Connect with 200+ pre-built source/sink connectors
- KRaft mode — no ZooKeeper dependency since Kafka 3.x
- Consumer groups for parallel, fault-tolerant event consumption
- Topic compaction for changelog and event sourcing patterns
Pros
- Industry standard — massive ecosystem, tooling, and community
- Extremely high throughput and durability at scale
- Flexible retention — replay historical events, not just latest state
- Kafka Connect makes integration with databases and cloud services plug-and-play
- Apache 2.0 licensed — fully open source, no licence concerns
Cons
- Steep operational learning curve — tuning brokers, partitions, and replication requires expertise
- Resource-heavy for small-scale use cases — overkill for simple task queues
- KRaft mode still maturing for some edge cases
- Java/JVM-based — adds overhead compared to lighter message brokers
- Confluent's managed cloud offering can be expensive at volume
Pricing
Open SourcePossible Stacks
Streaming Analytics Pipeline
ProjectEvent-driven pipeline for real-time analytics: Kafka ingests millions of events per second from producers; ClickHouse stores and queries the stream at sub-second latency; dbt runs incremental transformation models; Grafana displays live dashboards and fires alerts. Docker containerises all components.
PostHog Self-Hosted
InfrastructureSelf-host PostHog for full-stack product analytics with complete data ownership. PostgreSQL handles transactional storage for user and event metadata; ClickHouse powers fast analytical queries over large event volumes. Kafka buffers the event ingestion pipeline between capture and ClickHouse; Redis provides the in-memory caching layer. Docker Compose orchestrates the full multi-service deployment.
Related Tools
Works well with (4)
Learning Resources
No resources yet — check back soon.