Kafka vs Redis Streams: Building a Stock Ticker Twice

Kafka and Redis Streams get compared constantly: in blog posts, in interview questions, in the Slack thread where someone asks whether we could just use Redis for this instead of standing up Kafka. Most answers collapse into one sentence: Kafka is durable and high throughput, Redis Streams is fast and simple, pick based on your workload. That sentence is true and close to useless, because it says nothing about what happens when a partition rebalances, when a consumer crashes mid batch, or when you need the same key to land in the same place across two different channels.

I got tired of repeating that sentence without having earned it. So I built the same system twice as a side project: once on Kafka, once on Redis Streams, against real brokers running in Docker, nothing mocked. This post opens a series about what building it taught me.

Why Compare Two Systems With Different Jobs

Both are append-only logs. Both support consumer groups. That is roughly where the surface similarity ends, and it is exactly why the comparison is worth doing properly instead of from memory: the shared vocabulary (log, offset, consumer group, partition or stream) hides two very different engines, and picking the wrong one for a given job shows up later as either wasted infrastructure spend or, worse, silently lost data.

The only way I know to feel that difference is to build something non-trivial on both and watch where the engines disagree. That meant a pipeline with ordering guarantees, failure recovery, and a join across two streams of data, rather than a toy produce-a-message-consume-a-message demo.

The Running Example: A Ticker With Price Alerts

My little ticker system tracks 10 IDX symbols such as BBCA and BBRI, the kind of feed behind any Indonesian trading app, with synthetic seeded prices so every run is reproducible. Two channels carry all the traffic:

• market-updates: a firehose of price ticks, one per symbol change, keyed by symbol, 6 partitions on the Kafka side.
• notifications: fired when a user's price alert (<, <=, =, >=, >) crosses, also keyed by symbol so it lands on the same partition as the price ticks for that symbol.

That last detail, both channels keyed by symbol, is deliberate. It sets up co-partitioning, one of the things I get to much later on: if partition p of market-updates and partition p of notifications are owned by the same consumer, a notification can be enriched with the market data that triggered it without a network hop. Get the keying wrong and every join becomes a remote lookup instead.

graph LR
    S["Price Simulator"] -->|"key: symbol"| MU["market-updates (6 partitions)"]
    MU --> AE["Alert Evaluator"]
    AE -->|"price crosses alert"| N["notifications (co-partitioned)"]
    N --> D["Live Dashboard (SSE)"]

A price simulator publishes ticks for all 10 symbols. An alert evaluator watches the stream per symbol, holding the last known price so it can detect a crossing rather than a threshold breach: someone who sets an alert at 9,000 wants exactly one notification the moment BBCA first goes above it, not one on every subsequent tick that happens to still be above 9,000. Getting that right forces the evaluator to hold state, which is why I ended up with a whole post on stateful stream processing. The dashboard consumes notifications over Server-Sent Events.

Rebuild the same pipeline on Redis Streams and the shape of it stays identical. Only the mechanics underneath change, and those mechanics are what I wanted to compare.

Two Different Logs

Kafka is a durable, partitioned, replicated commit log. Every topic is split into partitions, each partition is an ordered, immutable sequence of records on disk, and each partition is replicated across brokers (my optional cluster setup runs with a replication factor of 3) so a single broker dying does not lose data. Consumer groups do not compete for individual messages; they get exclusive ownership of a set of partitions, which is precisely what preserves per-key ordering. As long as BBCA always hashes to the same partition, whichever consumer owns that partition sees every BBCA tick in order, no matter how many other consumers are in the group.

Redis Streams is an in-memory radix-tree log living inside a general-purpose data store. Internally it is a radix tree of entries, each with an ID that encodes the millisecond timestamp it was appended plus a sequence number, kept in memory with AOF or RDB persistence as an option rather than a given. No partitioning is built in: one stream is one ordered log, and if you want to shard, you create multiple streams and route keys yourself, which is how I ended up solving the market-updates/notifications co-partitioning problem on the Redis side, with hash tags instead of Kafka partition keys. Consumer groups in Redis Streams are competing consumers: whichever consumer reads next gets the next entry, so ordering across consumers is not guaranteed the way it is in Kafka, only ordering within the stream itself.

The short version: Kafka spends complexity on durability, replication, and ordered parallelism at scale. Redis Streams spends simplicity on being an in-memory data structure you probably already know how to operate, because it lives right next to the Redis you are likely already running for caching or rate limiting.

Why I Mocked Nothing

Every claim above is easy to state and surprisingly easy to get subtly wrong in practice. So I ran actual Kafka (in KRaft mode) and actual Redis in Docker, and before trusting a word of my own notes I proved a real produce-and-consume round trip on both.

From there I kept coming at each concept from more than one direction: conceptual notes to force myself to explain it, small command line programs written against KafkaJS and ioredis and pointed at the live brokers, browser simulations for the parts that are easier to watch than to read, and a live dashboard streaming pipeline state over SSE next to a throughput and latency benchmark. I also reimplemented the Kafka partitioner (murmur2) and the Redis Cluster hash function (CRC16) in TypeScript, so the exact same hashing functions run inside the browser simulations and in front of the real brokers, each one checked against the other. What you watch animate in a simulation is the same math that decides where your keys land.

That matters because reading "a consumer group rebalance can pause processing" is a different kind of understanding than watching consumer lag spike on a dashboard while you kill a container yourself, then watching the group rebalance and drain the backlog in front of you. The second kind is what I was after.

Where This Goes Next

Over the next few weeks I want to write up the parts that surprised me, roughly in the order I hit them.

The primitives come first: how Kafka uses keys and partitions to keep each symbol strictly ordered, and what XADD, entry IDs, and MAXLEN do in place of partitioning on a system that has none of it. Then consumer groups on both sides, which sound like the same feature and are not. Kafka hands out exclusive partition ownership. Redis Streams hands you whatever entry is next and quietly remembers everything you have not acknowledged.

After that, the failure modes, which is where I spent most of my time. At-least-once delivery and where exactly the commit goes, retries, dead letters, Kafka's exactly-once transactions, and why pub/sub keeps getting mistaken for a stream. Then the part I enjoyed most: co-partitioning, and the custom KafkaJS assigner I wrote after discovering that the round-robin assignment you get out of the box breaks local joins without telling you.

Near the end, the operational half. Kafka's in-sync replica model against Redis Cluster's hash-slot sharding, including what I saw when I killed the leader broker mid-run. Stateful processing with edge versus level alerts and OHLC windows. The dashboard itself. A benchmark run the same way against both so the numbers are comparable. And then the question everyone asks first, answered last: Kafka or Redis Streams, how do you choose.

Every claim in this series came out of something I ran, watched break, and ran again.

I am starting with the mechanic the rest of Kafka rests on: keys, murmur2, and how six partitions keep every symbol in line.