Kafka or Redis Streams: How to Choose

Fifteen posts ago I started with a question no comparison table could settle for me: if I needed a real-time system keyed by symbol, carrying both a firehose of price ticks and a thinner stream of crossed-threshold alerts, would I reach for Kafka or Redis Streams? The only way I trusted to find out was to build the thing twice, against real brokers in Docker, and write down what broke.

So no new mechanism in this one. Sixteen posts of broken things, turned into a decision.

If you read along, what follows should feel like a recap with receipts. Every claim links back to the post where I watched it happen against a live broker instead of asserting it from memory. If you dropped in here first, it stands on its own, and the links are waiting whenever you want proof rather than my word.

Both Call Themselves "a Log With Consumer Groups"

Kafka and Redis Streams share a surface vocabulary: append-only log, consumer group, offsets, acknowledgment. Underneath, they are built for different failure modes and different budgets, so the shared words are what make the choice confusing.

Dimension	Kafka	Redis Streams
Storage	Durable log on disk, retained by time or size	In memory, trimmed by MAXLEN, AOF/RDB optional
Parallelism unit	Partitions, order preserved per partition	One stream is one ordered log, you shard keys yourself
Consumer group model	Partition ownership, sticky, preserves per-key order	Competing consumers, work queue, no per-key order
Delivery default	At-least-once, exactly-once via transactions	At-least-once via the PEL, XACK, XAUTOCLAIM
Typical latency	Low single digit ms	Sub-millisecond
Retention and replay	First class, rewind to any offset	History until trimmed, replay via XRANGE
Ops footprint	A cluster: brokers, KRaft, replication	A Redis you probably already run

Every question below is that table applied to a workload you have, rather than a generic one.

Durability and Retention

Kafka's log lives on disk and survives broker restarts by design. Redis Streams live in RAM, and whatever durability you get comes from AOF or RDB persistence layered on top, which is still weaker than a replicated disk log. Trim aggressively with MAXLEN, as I did when I first went through the fundamentals, and you are explicitly deciding you do not need long retention. Fine for a work queue. Much less fine if you have quietly started using a stream as your system of record without ever agreeing to the memory bound that comes with it.

The question worth asking out loud: do you need to replay history, audit what happened last week, or backfill a new consumer from three days ago? Kafka answers that structurally. Redis answers it only as far back as whatever you chose to keep in RAM.

Throughput and Fan-out

Kafka is built for high, sustained throughput and for data volumes that outgrow RAM. Redis Streams are built for low latency and comfortable, bounded volume. I am not putting numbers here, for the reason I spent a whole post on: any figure I quote is a fact about my laptop, and your hardware and payload shape will move it anyway. Measure on the machines you plan to deploy on.

Fan-out is related but separate. If you need to push updates to many live clients cheaply, both systems can, and they do it differently. Redis hands you an explicit choice between Streams (durable, replayable via XRANGE) and Pub/Sub (ephemeral, zero persistence). I picked that split apart in Redis Pub/Sub vs Streams: Pub/Sub drops messages when nobody is listening, so never point it at anything you cannot afford to lose. Kafka has no Pub/Sub mode at all. Every consumer group gets the durable log, which is more guarantee than plenty of fan-out use cases asked for.

Ordering and Partitioning

Most of my ticker's design fell out of this one row. In Kafka you key by whatever must stay ordered. Mine was the symbol: market-updates has 6 partitions, and every tick for BBCA lands on the same partition every time, so consumers see BBCA's ticks in order. I walked through the mechanics, murmur2 hashing included, in Kafka Partitions, Keys, and Ordering. The catch is that partition count has to be chosen ahead of need, because adding partitions later reshuffles the key to partition mapping and quietly breaks the ordering guarantee you were relying on.

Redis Streams have no partitions. One stream is one ordered log. If you want more parallelism, or per-key order across workers, you shard yourself into multiple stream keys and use hash tags to co-locate the related ones, the same idea as Kafka's key based routing, hashed with CRC16 instead of murmur2. Competing consumers reading a single stream will reorder work across keys, which is correct for a work queue and wrong the moment you assumed per-symbol order. All three angles get the full treatment in Kafka Consumer Groups, Redis Streams Consumer Groups and the PEL, and Co-Partitioning, where the same key lands on the same partition number in both market-updates and notifications so a join can happen locally instead of over the network.

Delivery Guarantees

Both systems default to at-least-once, and both push the real work onto you: making the consumer idempotent. Kafka does it by committing the offset after processing and dead-lettering poison messages instead of retrying them forever, which I built out in At-Least-Once in Kafka. When at-least-once falls short, Kafka has a real answer, idempotent producers plus transactions for exactly-once read-process-write, and I ran that end to end in Exactly-Once in Kafka.

Redis Streams get you at-least-once through the pending entries list. A consumer reads an entry, the entry sits in the PEL until XACK, and XAUTOCLAIM recovers whatever a dead consumer was holding, with a delivery counter you can use to dead-letter poison entries. That is the entire mechanism, laid out in Recovering Stuck Messages in Redis Streams. What Redis lacks is a transactional exactly-once primitive for read-process-write. If your pipeline needs that guarantee end to end, that gap alone points you to Kafka.

Operational Weight

Kafka needs a cluster: brokers, KRaft for metadata, replication factor and in-sync replicas doing the quiet work of surviving a broker loss without dropping committed data. I ran all of it for real in Kafka ISR vs Redis Cluster Sharding, including killing the leader broker mid-run to watch failover happen. Redis Cluster has its own operational shape, 16384 hash slots spread across nodes, but if you are not clustering, Redis Streams live inside a Redis you were probably already running for caching or sessions. A real cost difference, not a stylistic one: standing up and operating a Kafka cluster is infrastructure you do not have today, while adding a stream to an existing Redis is a data structure you do not have today.

Team familiarity compounds it. Whichever system your team already operates comes with its learning curve already paid: on-call runbooks, failure modes people recognize at 3am, upgrade paths, monitoring somebody already trusts. A team fluent in Redis will ship a correct, boring Streams consumer faster than a team standing up its first Kafka cluster, even where Kafka is the better theoretical fit. I would rather run the boring correct thing.

When Redis Streams Is Enough

• You already run Redis and do not want to add infrastructure.
• Volume is moderate and you are comfortable trimming with MAXLEN.
• The shape of the problem is a work queue: competing consumers, acks, retries, no strict per-key order.
• Lowest latency matters more than long retention or replay.
• You need real-time fan-out and can choose deliberately between Streams (durable) and Pub/Sub (ephemeral).

When Kafka Earns Its Complexity

• You need durable, replayable history: audits, reprocessing, backfilling a new consumer from days ago.
• You need per-key ordering preserved across many consumers at once, beyond a single worker.
• The pipeline needs exactly-once read-process-write, beyond careful idempotency.
• Throughput is high and sustained, or data volume outgrowing RAM is a design constraint rather than an edge case.
• You want the surrounding ecosystem: connectors, stream processing frameworks, schema registries, all the tooling built around a durable commit log as source of truth.

The Pragmatic Hybrid

Plenty of real systems refuse to pick. They run Redis at the hot edge for alert delivery, presence, rate limiting, caches, and Kafka as the durable backbone: the immutable event log everything else gets derived from, replayable when a new feature needs backfilling. My ticker split exactly that way.

graph LR
    T[Price tick] --> MU["Kafka: market-updates (durable log)"]
    MU --> S["Derived state: OHLC windows + alert evaluation"]
    S -->|"alert crosses threshold"| N["Redis: notification fan-out"]
    N --> D1[User device]
    N --> D2[User device]

Market data stays in Kafka because it is the source of truth and has to be replayable. The instant an alert fires, though, the problem changes into one-shot, low-latency delivery, and Redis is the better tool for getting it onto a device fast. Neither system has to do the other's job. I built the state-holding half, the OHLC windows and edge-versus-level alert evaluation, in Stateful Stream Processing, and the delivery half in Building a Live Market Dashboard with SSE and Next.js.

A Decision Checklist

The whole series, compressed into one flow:

graph TD
    A[Pick the event backbone] --> B{"Need replay, audit, or backfill?"}
    B -->|Yes| K1[Kafka]
    B -->|No| C{"Order preserved per key, across many consumers?"}
    C -->|Yes| K2[Kafka]
    C -->|No| D{"Need exactly-once read-process-write?"}
    D -->|Yes| K3[Kafka]
    D -->|No| E{"Throughput huge and sustained, data will outgrow RAM?"}
    E -->|Yes| K4[Kafka]
    E -->|No| F{"Already running Redis for something else?"}
    F -->|Yes| R1[Redis Streams]
    F -->|No| G{"Team comfortable operating a Kafka cluster?"}
    G -->|No| R2[Redis Streams]
    G -->|Yes| H["Either works, pick the one your team knows"]

The flowchart leaves out feature counts, benchmark bragging rights, and whichever technology is trendier this year, because none of that predicts whether the system survives its first bad week in production. Replay, ordering, exactly-once, RAM limits, and what you already operate do.

Closing the Series

Building the same system twice, on real brokers, with nothing mocked, took far longer than reading a comparison table would have. It was also the only way I came to trust the answer. Every claim above traces back to something I watched happen on my own machine: a partition staying strictly ordered because of a key, a rebalance draining lag, a poison message getting dead-lettered instead of wedging a partition, the custom assigner I wrote turning a remote join into a local one, a cluster shrugging off a broker I killed on purpose.

If sixteen posts collapse into one line, it is that Kafka and Redis Streams are built for different jobs. Asking which one is better never got me anywhere. Which one my problem is asking for turned out to be answerable in about five minutes, once I had built both and knew where each one hurts.

If you have run either at a scale I have not, I would like to know where this framework breaks. That is the part I could not learn from a laptop.