Kafka and Redis Streams both store event sequences and support consumer groups. Those similarities do not explain how either system handles a consumer crash, ordering, or related data across streams. I wanted to compare those behaviors with a working application.
I built the same system twice as a personal project, once with Kafka and once with Redis Streams. Both used brokers running in Docker without mocks. This series records what I learned from the implementations.
Why compare two systems with different jobs
Both systems expose logs and consumer groups, but their ownership and recovery models differ. The shared vocabulary can hide these differences. A useful comparison needs to identify who receives each record and what happens when processing fails.
I built the same system with both brokers to compare their behavior. The pipeline needs ordered processing, failure recovery, and a join between two data streams.
The running example: a ticker with price alerts
The ticker tracks 10 IDX symbols, including BBCA and BBRI. Seeded synthetic prices make each run repeatable. Two channels carry the traffic:
- market updates: a firehose of price ticks, one per symbol change, keyed by symbol, 6 partitions on the Kafka side.
- notifications: price alerts use comparisons such as
<,<=,=,>=, and>. Notifications use the symbol as their key to align with price tick partitions.
Both channels use the symbol key and matching partition counts. This supports matching partition numbers across topics. If one consumer owns partition p of both market-updates and notifications, it can keep related state locally. Local state still needs initialization before a notification can use it.
graph LR
S["Price Simulator"] -->|"key: symbol"| MU["market-updates (6 partitions)"]
MU --> AE["Alert Evaluator"]
AE -->|"price crosses alert"| N["notifications (co-partitioned)"]
N --> D["Live Dashboard (SSE)"]
A simulator publishes ticks for all 10 symbols. The alert evaluator retains the previous price to detect a threshold crossing. For an alert at 9,000, it should notify once when BBCA crosses above 9,000. Further ticks above the threshold should not repeat that crossing alert. The dashboard receives notifications through Server Sent Events.
Rebuild the same pipeline on Redis Streams and the shape of it stays identical. Only the mechanics underneath change, and those mechanics are what I wanted to compare.
Two different logs
Kafka stores topic records in ordered partitions on disk. My optional cluster uses replication factor 3, with partition copies on different brokers. Within a consumer group, each partition has one owner at a time.
A stable key keeps BBCA records in one partition. The consumer reads those records in append order. Replication and acknowledgment settings determine how the log handles broker failure.
Redis Streams stores entries in memory with a radix tree representation. Generated entry IDs combine a millisecond component and a sequence number. AOF or RDB persistence is optional.
One stream is one ordered sequence. I create multiple streams when I need sharding and use hash tags for related keys. Consumer group members compete for entries, so concurrent processing does not preserve completion order across consumers.
Kafka provides partitioned logs, replication, and independent consumer positions. Redis Streams adds an ordered log to Redis. Teams that already operate Redis can reuse that experience, while still checking persistence, capacity, and recovery requirements.
Why I mocked nothing
I ran Kafka in KRaft mode and Redis in Docker. First, I verified that each system could produce and consume a message.
I used KafkaJS and ioredis programs against the brokers, browser simulations, and an SSE dashboard. I also implemented murmur2 and CRC16 in TypeScript to compare routing predictions with broker behavior. The browser and test harness share those hash functions.
I wanted to observe lag and recovery during a failure. Stopping a container let me see the group rebalance and process the remaining backlog.
Where this goes next
Over the next few weeks I want to write up the parts in roughly the order I hit them.
The series starts with Kafka keys and partitions, then Redis commands such as XADD and MAXLEN. It also covers Redis entry IDs. Kafka groups assign partitions to consumers. Redis groups distribute entries and track those that need acknowledgement.
I spent most of my time on failures: commit timing, retries, dead letters, and transactions. I also compared Pub/Sub delivery with retained streams. The part I enjoyed most was the custom KafkaJS assigner. It keeps matching partitions together so the ticker can join their data locally.
Near the end, the operational half. Kafka's in sync replica model against Redis Cluster's hash slot sharding, including what I saw when I killed the leader broker mid run. Stateful processing with edge versus level alerts and OHLC windows. The dashboard itself. A benchmark run the same way against both so the numbers are comparable. And then the question everyone asks first, answered last: Kafka or Redis Streams, how do you choose.
Every claim in this series came out of something I ran, watched break, and ran again.
I am starting with the mechanic the rest of Kafka rests on: keys, murmur2, and how six partitions keep every symbol in line.
Loading comments...