Kafka or Redis Streams: How to Choose

Fifteen posts ago, I began a comparison of systems for price ticks and threshold alerts. I built the ticker twice, with Kafka and Redis Streams running in Docker. Those experiments gave me concrete behavior to compare.

This final post connects the sixteen articles to a choice of architecture.

Every claim links back to the post where I watched it happen against a live broker instead of asserting it from memory. The post stands on its own, with links to each experiment for the evidence.

Both call themselves "a log with consumer groups"

Kafka and Redis Streams share a surface vocabulary: append only log, consumer group, offsets, acknowledgment. Underneath, they are built for different failure modes and different budgets, so the shared words are what make the choice confusing.

The table is a qualitative guide. Its latency ranges are not measurements from these experiments or limits guaranteed by either system. Delivery and durability also depend on client behavior and configuration.

Durability and retention

Kafka retains a log on disk. Its survival of failures depends on replication, acknowledgment, and storage configuration. Redis Streams use memory, with persistence through AOF or RDB when configured.

Retention is a separate decision. Trimming with MAXLEN, as in the fundamentals, limits available history. Define the required history and memory budget before using a stream as the authoritative record.

Ask whether you need to replay history, audit what happened last week, or backfill a new consumer from three days ago. Kafka answers that structurally. Redis answers it only as far back as whatever you chose to keep in RAM.

Throughput and fan out

Kafka can retain data volumes that exceed RAM, while Redis Streams keep retained entries in memory. Throughput and latency need workload measurements. The benchmark article explains the local harness and its limits. Test representative hardware and payloads before a deployment decision.

To send updates to many clients, Redis offers Streams and Pub/Sub. Streams retain entries for later reads through commands such as XRANGE. Pub/Sub delivers only to connected subscribers and has no retained history. Redis Pub/Sub vs Streams compares these behaviors.

Kafka consumer groups read retained records independently. Each group tracks its own position. A client that needs only the current state might not need retained event history.

Ordering and partitioning

The ticker uses symbol as the Kafka key and 6 partitions in market-updates. BBCA ticks map consistently while the partition count and partitioner remain unchanged. Kafka Partitions, Keys, and Ordering explains the murmur2 mapping. A partition count increase can change the destination of future records. Plan that change before relying on continuous ordering for a key.

One Redis stream is one ordered log. To distribute data across nodes, an application can use several stream keys. Redis Cluster uses CRC16 to select their slots. Hash tags can put related keys in the same slot. Several consumers can process one stream concurrently, but their completion order can differ from entry order.

Kafka Consumer Groups and Redis Streams Consumer Groups and the PEL explain ownership and delivery. Co Partitioning aligns matching keys in market-updates and notifications. Local joins also require matching consumer assignments and initialized local state.

Delivery guarantees

Both systems can repeat delivery, so consumers need idempotent effects. In Kafka, I commit after successful handling and route persistent failures to a DLQ. The retry article describes that policy.

Kafka transactions can commit output and source offsets together when both remain in Kafka. I tested that boundary in the transaction article. It does not include external effects.

A Redis consumer group retains an entry in the PEL until XACK. The application can use XAUTOCLAIM and delivery counts for recovery and retry limits. The recovery article describes that process. Redis can combine output and acknowledgment commands atomically within its key constraints, but the application still owns the processing and deduplication protocol.

Operational weight

A Kafka deployment needs brokers and metadata management. Replica and ISR settings determine which broker failures it can tolerate. I tested leader failure in Kafka ISR vs Redis Cluster Sharding.

Redis Cluster distributes 16384 hash slots across nodes. A single Redis instance can also hold streams. An existing Redis deployment may reduce setup work, but queue durability and capacity still need review. The operational cost depends on what the team already runs and can support.

Team experience affects operating cost. Existing runbooks, monitoring, and upgrade procedures reduce the work needed to support a system. I include those costs when a new platform appears to fit the workload better.

When Redis Streams is enough

• You already run Redis and do not want to add infrastructure.
• Volume is moderate and you are comfortable trimming with MAXLEN.
• The shape of the problem is a work queue: competing consumers, acks, retries, no strict per key order.
• Lowest latency matters more than long retention or replay.
• You need real time fan out and can choose deliberately between Streams (durable) and Pub/Sub (ephemeral).

When Kafka earns its complexity

• You need durable, replayable history: audits, reprocessing, backfilling a new consumer from days ago.
• You need per key ordering preserved across many consumers at once, beyond a single worker.
• The pipeline needs exactly once read process write, beyond careful idempotency.
• Throughput is high and sustained, or data volume outgrowing RAM is a design constraint rather than an edge case.
• You want the surrounding ecosystem: connectors, stream processing frameworks, schema registries, all the tooling built around a durable commit log as source of truth.

The pragmatic hybrid

My ticker uses both systems. Redis handles immediate alert delivery, while Kafka retains the event history for later processing. This division lets each consumer use the history it needs.

graph LR
    T[Price tick] --> MU["Kafka: market-updates (durable log)"]
    MU --> S["Derived state: OHLC windows + alert evaluation"]
    S -->|"alert crosses threshold"| N["Redis: notification fan-out"]
    N --> D1[User device]
    N --> D2[User device]

In my design, Kafka retains market data for replay. Redis supports alert delivery to connected clients. This choice reflects the role of each component, rather than a measured latency advantage.

Stateful Stream Processing covers OHLC windows and alert evaluation. The SSE dashboard article covers delivery to the browser.

A decision checklist

The whole series, compressed into one flow:

graph TD
    A[Pick the event backbone] --> B{"Need replay, audit, or backfill?"}
    B -->|Yes| K1[Kafka]
    B -->|No| C{"Order preserved per key, across many consumers?"}
    C -->|Yes| K2[Kafka]
    C -->|No| D{"Need exactly-once read-process-write?"}
    D -->|Yes| K3[Kafka]
    D -->|No| E{"Throughput huge and sustained, data will outgrow RAM?"}
    E -->|Yes| K4[Kafka]
    E -->|No| F{"Already running Redis for something else?"}
    F -->|Yes| R1[Redis Streams]
    F -->|No| G{"Team comfortable operating a Kafka cluster?"}
    G -->|No| R2[Redis Streams]
    G -->|Yes| H["Either works, pick the one your team knows"]

The flowchart uses replay, ordering, transaction boundaries, memory limits, and existing operational experience. These requirements help explain the choice for a specific workload.

Closing the series

The experiments took longer than a comparison table, but they helped me understand the behavior. I observed key ordering, rebalances, DLQ handling, and broker failure. I also tested a custom partition assigner that keeps related topic partitions on one process.

I would choose from the retention, ordering, and recovery requirements first. The experiments made those requirements easier for me to connect to a system.

If you have run either at a scale I have not, I would like to know where this framework breaks. That is the part I could not learn from a laptop.