Real-time data pipelines have become the default architecture for data-intensive applications, often regardless of whether the use case actually requires real-time processing. The appeal is clear: immediate availability of fresh data, event-driven processing, and the ability to react to conditions as they occur rather than hours later. The cost is significant: streaming infrastructure is more complex to operate than batch pipelines, failure modes are less intuitive, and the debugging experience is materially worse.

This article covers Apache Kafka as the foundation for production real-time data pipelines — when it is the right choice, how to structure producers and consumers correctly, the operational patterns that keep it healthy, and the mistakes that cause production incidents that are preventable with the right design decisions.

When Streaming Is the Right Choice

Streaming is the right choice for three scenarios. First, when downstream consumers need data within seconds rather than minutes — fraud detection that needs to act on a transaction before the user's session ends, dashboards that need to reflect current state for operational decisions, alerting systems that need to notify operators before a condition becomes a problem. Second, when event volumes are too high for polling-based integration — ten thousand events per second cannot be queried efficiently from a relational database. Third, when processing logic needs to join or aggregate events across time windows — a user's behaviour over the last five minutes, a machine's error rate over the last hour.

Streaming is the wrong choice when batch processing is sufficient. Most analytics use cases do not need data fresher than an hourly batch. Most ML feature pipelines do not need sub-minute freshness. Adding Kafka to a system that does not need real-time processing adds operational complexity and a new failure domain for no benefit. Evaluate the actual latency requirement before committing to streaming architecture.

A useful question for evaluating the streaming requirement: what business decision requires data within N seconds that could not be made with data from N minutes ago? If the answer is "none that we have identified yet," the use case probably does not require streaming. If the answer is specific and consequential — detect the fraud, alert the operator, personalise the response — streaming is justified.

Kafka Fundamentals for Data Engineers

Kafka topics are append-only, time-ordered logs. Producers write events to topics; consumers read events from topics by maintaining an offset — a position in the log. This architecture provides three properties that make Kafka valuable for data pipelines: events are retained for a configurable period (days to weeks), multiple consumer groups can read the same topic independently, and consumers can replay events from any offset. These properties make Kafka fundamentally different from a traditional message queue where a message is consumed and deleted.

Partition count is the most consequential design decision for a Kafka topic. Partitions are the unit of parallelism — a topic with one partition can only be consumed by one consumer simultaneously; a topic with ten partitions can be consumed by up to ten consumers in parallel. The throughput of a consumer group scales with the number of partitions, up to the number of consumers. Set partition count based on your maximum expected throughput divided by the single-partition throughput of your consumer logic. Increasing partition count after the fact is operationally complex — set it correctly at topic creation.

Consumer group offsets represent a significant operational responsibility. When a consumer crashes and a new instance starts, it reads from the committed offset — the last position successfully committed by the consumer group. If the consumer crashes after processing an event but before committing the offset, the event will be reprocessed. Consumer logic must be idempotent — processing the same event twice should produce the same result as processing it once. This is not optional in Kafka architectures; it is a fundamental guarantee of the at-least-once delivery model.

 

Kafka Design Decisions

  • Partition count: throughput target / single-partition throughput (cannot easily reduce later)
  • Replication factor: 3 for production (tolerates 2 broker failures)
  • Retention: match to your replay and debugging window requirement
  • Consumer idempotency: required — at-least-once delivery guarantees reprocessing
  • Offset commit: commit after successful downstream write, not on message receipt

Pipeline Patterns

The fan-out pattern routes a single source topic to multiple downstream consumers with independent processing logic. An order events topic fans out to: an inventory service (deduct stock), a fulfilment service (trigger warehouse pick), a notifications service (send confirmation email), and an analytics pipeline (update revenue metrics). Each service maintains its own consumer group and processes events at its own pace without affecting the others. This decoupling is one of Kafka's most valuable architectural properties.

The enrichment pattern adds context to events before they reach downstream systems. Raw events from sensors or application logs often contain minimal data — a device ID and a value. An enrichment consumer joins the raw event with a reference data store (device metadata, user profile, product catalogue) and produces an enriched event to a downstream topic. This moves enrichment logic from the consuming application into the pipeline, reducing duplication and ensuring consistency across consumers.

Kafka Streams and ksqlDB provide stream processing capabilities — stateful aggregations, time-window joins, and per-event transformations — without requiring a separate stream processing cluster. For moderate processing complexity (aggregating event counts over rolling windows, joining two event streams on a common key), Kafka Streams is the right tool. For complex analytical transformations at very high volume, Apache Flink provides more powerful semantics and better performance at the cost of significantly higher operational complexity.

Production Operations

Consumer lag is the primary operational health metric for Kafka consumers — the number of events in a topic that have not yet been consumed by a specific consumer group. A consumer group that keeps up with production throughput will have near-zero lag. A consumer group that processes slower than events arrive will accumulate lag. Monitor consumer lag per consumer group per partition, alert when lag exceeds a threshold that represents acceptable processing delay, and investigate before lag becomes a data freshness problem for downstream systems.

Schema management becomes critical when multiple teams produce and consume from the same topics. A producer that changes the schema of events it produces will break consumers that depend on the old schema without warning. A schema registry — Confluent Schema Registry is standard — enforces schema compatibility rules: new schemas must be backward-compatible (consumers can still read old events) or forward-compatible (old consumers can read new events) depending on the policy. Schema evolution discipline is a prerequisite for Kafka deployments with multiple teams.

Kafka cluster maintenance — broker upgrades, partition rebalancing, scaling — requires careful sequencing to avoid consumer disruption. Rolling upgrades of broker nodes maintain availability but require that all consumers and producers support the minimum protocol version across the broker version transition. Test upgrade procedures in a staging environment that mirrors production topology before executing in production.

Common Mistakes in Kafka Data Pipelines

Non-idempotent consumers are the source of the majority of Kafka data correctness incidents. A consumer that writes to a database without deduplication logic will write duplicate records when it reprocesses events after a crash. Use message keys or event IDs as idempotency keys in downstream systems, and add an explicit deduplication check before processing when duplicate events would cause incorrect state.

Unbounded topic growth from setting retention too high or from consumer groups that stop consuming but keep their topic subscription active. A consumer group that is no longer needed should have its offset deleted — otherwise it holds the Kafka offset tracking indefinitely and the topic's log compaction cannot clean up consumed events. Periodically audit consumer groups and remove stale ones.

Using Kafka as a database is a persistent anti-pattern. Kafka is a log, not a store — querying it for the current state of an entity requires replaying the entire event history, which becomes impractical as topics grow. If you need queryable state derived from a Kafka topic, materialise it to a database using a consumer that maintains current state. Kafka Streams and ksqlDB provide change data capture patterns for this purpose.