Apache Kafka

Apache Kafka

Open Source

The open-source distributed event streaming platform.

Data Engineering & ETL
Streaming

Published 29 May 2026Last updated 31 October 2026

Scores

Popularity4/5

Widely used

De facto standard for event streaming used at LinkedIn, Netflix, and Uber — but most applications never need streaming infrastructure, so its relevance is category-high rather than universal.

Learning Curve2/5

Easy

Broker configuration, partition sizing, consumer group semantics, offset management, and schema registries all require significant hands-on experience before production readiness.

Flexibility5/5

Unopinionated

Handles any event type at any scale; 200+ Kafka Connect connectors cover every major source and sink; Streams API enables stateful real-time processing in the same ecosystem.

Performance4/5

Fast

Proven at petabyte scale with sub-2ms publish latency, but JVM GC pauses and ZooKeeper overhead mean newer C++ engines like RedPanda benchmark with lower tail latency under comparable hardware.

Portability4/5

Portable

Apache 2.0 licensed; the Kafka API is an industry standard — Redpanda, Confluent, and MSK are all wire-compatible, meaning producers and consumers are reusable.

About Apache Kafka

Apache Kafka is a distributed event streaming platform originally developed at LinkedIn and open-sourced in 2011. It is built for very high event volumes at low latency, which makes it a standard choice for real-time data pipelines, event-driven microservices, and streaming analytics.

Kafka's core model is a distributed commit log: producers write events to topics, and consumers read from those topics at their own pace. Topics are partitioned for parallelism and replicated across brokers for fault tolerance. This decouples producers from consumers and gives durable, ordered, replayable event streams.

Key APIs are the Producer and Consumer APIs, the Kafka Streams library for stateful stream processing in Java and Scala, and Kafka Connect for moving data between Kafka and databases, file systems, and cloud services. Clusters coordinate through KRaft, Kafka's built-in consensus protocol; ZooKeeper was removed in Kafka 4.0, so a cluster is just brokers and controllers.

In data engineering, Kafka sits between sources and sinks: capturing change streams from databases, feeding real-time ML pipelines, or triggering downstream processing in Flink or Spark. You can run it yourself or use a managed service such as Confluent Cloud, or the Kafka offerings from the major cloud providers.

Key Features

  • Distributed commit log with configurable retention by time or size
  • Partitioned, replicated topics for parallelism and fault tolerance
  • Kafka Streams library for stateful, exactly-once stream processing
  • Kafka Connect with a large ecosystem of source and sink connectors
  • KRaft mode: built-in consensus with no ZooKeeper dependency
  • Consumer groups for parallel, fault-tolerant event consumption
  • Topic compaction for changelog and event-sourcing patterns

Pros

  • Industry standard with a very large ecosystem, tooling, and community
  • High throughput and durability at scale
  • Flexible retention lets you replay historical events, not just read the latest state
  • Kafka Connect makes integration with databases and cloud services largely configuration
  • Apache 2.0 licensed with no licence concerns

Cons

  • Steep operational learning curve: tuning brokers, partitions, and replication takes expertise
  • Resource-heavy for small workloads, and overkill for a simple task queue
  • Runs on the JVM, which adds overhead compared with lighter message brokers
  • Managed services such as Confluent Cloud can get expensive at high volume
  • Kafka Streams is a Java/Scala library, so other languages need other stream processors

Apache Kafka Pricing

Open Source

Tech Stacks with Apache Kafka

PostHog Self-Hosted

Infrastructure

Self-host PostHog when product data must stay on infrastructure you control. The official Docker Compose deployment runs the PostHog services with PostgreSQL, ClickHouse, Kafka, and Redis on one large server, and it's built for modest volumes: PostHog recommends it below about 300,000 events a month.

Deploy on:
Reverse Proxy:
Self-Hosted PaaS:
Tunnel add-on:

Sentry Self-Hosted

Infrastructure

Self-host Sentry error tracking and performance monitoring on your own server with the official Docker Compose setup. PostgreSQL stores projects, issues, and users, ClickHouse stores event data, Redis handles caching and queues, and Kafka moves events between services. It's the heaviest stack in the family, and it keeps every error and trace on infrastructure you control.

Deploy on:
Reverse Proxy:
Self-Hosted PaaS:
Tunnel add-on:

Streaming Analytics Pipeline

Project

Event-driven pipeline for real-time analytics: Kafka ingests millions of events per second from producers; ClickHouse stores and queries the stream at sub-second latency; dbt runs incremental transformation models; Grafana displays live dashboards and fires alerts. Docker containerises all components.

CI/CD add-on:
Containerization add-on:

Tools Related to Apache Kafka

Works well with Apache Kafka(2)

PostgreSQL logical replication via Debezium publishes row-level change events as Kafka topics — the standard CDC pattern for capturing database changes in real time.

Databricks Structured Streaming reads from Kafka topics natively via the built-in Kafka source connector — the primary pattern for real-time Delta Lake ingestion.

Integrates with Apache Kafka(3)

Kafka is the primary streaming source into ClickHouse — events flow from Kafka topics into ClickHouse tables via the built-in Kafka table engine.

Airbyte has an official Kafka destination connector; publishes each synced record as a Kafka event, enabling downstream stream processing of ingested data.

Airflow's official Kafka provider lets DAGs produce to and consume from Kafka topics, or wait for a specific event before running a batch job; Kafka keeps the streaming, Airflow the schedule.

Alternatives to Apache Kafka(1)

Redpanda speaks the Kafka protocol as a single C++ binary with no JVM, so Kafka clients work unchanged; Kafka has the larger ecosystem and a fully open Apache 2.0 license.

Tags

Open SourceSelf-hostableWeb

Details

Maintained
Yes
Tool type
Streaming
Primary language
Java
Hosting
Cloud & Self-hosted
Open source
Yes
GitHub stars
34.1k
Stars updated
2026-10-31