Redis Streams Fundamentals: XADD, Entry IDs, and MAXLEN

Most people know Redis as the cache: SET, GET, an EXPIRE, done. Redis Streams, added in Redis 5.0, is a different data type entirely. It is an append only log that lives inside the same server process as your cache, deliberately borrows Kafka's vocabulary (producers, consumer groups, acknowledgements), and asks you to think about ordering and retention the way Kafka does, with wildly different mechanics underneath.

The borrowed vocabulary is what pulled me in. I have been building the same stock price ticker twice, once on Kafka and once on Redis Streams, both running against real brokers in Docker on my laptop. My little ticker fans out price updates and fires user price alerts, so per-symbol ordering matters and a dropped tick shows up immediately. Below is the Redis half of that plumbing, plus the two spots where my Kafka instincts stopped transferring: how a stream gets identified and ordered, and what "retention" means when the log lives in memory.

The Log Inside the Cache

You append to a stream with XADD. Each entry gets an ID Redis assigns for you, and the entry itself is just a flat list of field/value pairs (everything is a string):

XADD market-updates * symbol BBCA price 9530 seq 1
=> "1718000000123-0"

The * tells Redis "give me the next ID." A tick from my market simulator goes in the same way, through a small ioredis wrapper:

import { createRedis } from "../../lib/redis/client";
import { MarketSimulator, tickToRedis } from "../../lib/domain";

const redis = createRedis();
const sim = new MarketSimulator({ seed: 3, volatilityScale: 5 });

for (const tick of sim.batch(8)) {
  // '*' tells Redis to assign the next time-ordered ID.
  const id = await redis.xadd(KEY, "*", ...tickToRedis(tick));
}

tickToRedis flattens the tick into the field/value pairs XADD expects: symbol, price, prevPrice, volume, ts, seq. No schema, no key/value split like a Kafka record, just a flat row appended to a log.

Entry IDs Are Not Offsets

Every entry ID has the form <millisecondsTime>-<sequence>, and it always increases.

A Kafka offset is an opaque, monotonically increasing integer per partition. It tells you where a record sits, nothing about when it arrived, that's a separate timestamp field on the record. A Redis Streams entry ID collapses those two ideas into one identifier: the millisecond component is real wall clock time from the Redis server, and the sequence component only exists to break ties between entries appended in the same millisecond. Append two ticks in the same millisecond and you get ...123-0 then ...123-1; the next millisecond resets the sequence to 0 again.

The practical consequence is that a Redis Streams ID is the Redis analog of a Kafka offset, your cursor into the log, but it's a cursor that also happens to be a timestamp. That's why XRANGE can take a millisecond value directly as a range boundary and why you don't need a separate "give me the offset for this timestamp" lookup the way you would with Kafka.

Reading It Back: XRANGE and XREAD

Three commands cover most of what you need:

• XRANGE key - + reads a range by ID (- and + mean the whole stream). Good for replay and inspection.
• XREAD tails the stream, like tail -f, waiting for new entries past a given ID.
• XLEN returns the entry count.

const range = await redis.xrange(KEY, "-", "+");
for (const [id, flat] of range.slice(0, 4)) {
  const f: Record<string, string> = {};
  for (let i = 0; i + 1 < flat.length; i += 2) f[flat[i]] = flat[i + 1];
  console.log(`${id}  ${f.symbol} price=${f.price} seq=${f.seq}`);
}

Redis hands back entries as [id, flatFieldArray] pairs, so reading the wire format back into an object is a small manual step: no schema registry does it for you.

One Stream, One Partition

One stream key is a single, totally ordered log, the direct equivalent of one Kafka partition. Redis has no built in concept of partitioning a single key.

That is a real divergence from what happens on the Kafka side when you key a record. Kafka's market-updates topic has a fixed 6 partitions, and every symbol gets hashed into one of them with murmur2(key) & 0x7fffffff % numberOfPartitions. Ten symbols sharing 6 buckets means collisions are guaranteed, and when I printed the assignment for my own symbol list, four of the ten landed on partition 0. Ordering per symbol is preserved, but capacity per partition is not evenly shared.

Redis sidesteps the hashing problem entirely, at the cost of pushing the sharding decision onto you. To scale a Redis Streams design past one key, you don't add partitions to an existing stream, you create more streams: market:{BBCA}, market:{BBRI}, one per symbol. Each stream is its own fully ordered log with no risk of two symbols colliding into the same log, but now you own N streams instead of a fixed, bounded number of partitions.

graph TD
    A["market-updates (Kafka topic)"] --> B["6 partitions, murmur2(key) picks one"]
    B --> C["10 symbols hash into 6 buckets, collisions guaranteed"]
    D["Redis: no topic, just keys"] --> E["market:{BBCA}"]
    D --> F["market:{BBRI}"]
    D --> G["market:{...}, one stream per symbol"]

Neither approach is strictly better. A fixed partition count bounds how many things you're operating, at the cost of hash collisions you have to design around. Per-symbol streams remove the collision problem entirely, at the cost of an unbounded, data-dependent number of logs to create, monitor, and eventually shard across Redis Cluster nodes, which is a rabbit hole I want to go down on its own another time.

It Lives in RAM, So You Trim It

The RAM tradeoff is the operational difference that bites people. A Kafka topic spools to disk and is retained by a time or size policy you configure once. A Redis stream just grows, in memory, until you tell it to stop.

await redis.xadd(KEY, "MAXLEN", "~", "10", "*", ...tickToRedis(tick));

MAXLEN ~ N keeps roughly N entries. The ~ matters: an exact trim (MAXLEN N without the tilde) has to walk and remove entries precisely on every write, which is expensive at high throughput. The approximate form lets Redis trim in whole macro-nodes of its internal structure instead, so it only trims occasionally rather than on every single XADD, at the cost of the stream sometimes holding a bit more than N entries.

graph LR
    P["Producer: XADD"] --> S["Stream in RAM"]
    S --> M{"MAXLEN set?"}
    M -->|"No"| G["Memory grows unbounded"]
    M -->|"Yes, MAXLEN ~ N"| T["Approximate trim, cheap"]

Durability follows the same in-memory story. Redis persistence is configurable but still memory-bound: RDB snapshots periodically, so a crash can lose the seconds or minutes since the last snapshot; AOF appends every write, and with appendfsync everysec you risk about a second of data on crash, while always fsyncs every write for safety at a latency cost. My Redis container runs with --appendonly yes --appendfsync everysec, a reasonable default. Even so, that is generally weaker durability than Kafka's replicated, disk-backed log, and it's a deliberate trade Redis makes for speed and simplicity, not an oversight.

Real Calls, Real Client

One detail from the client setup worth calling out: I ended up keeping two separate Redis connection factories.

export function createRedis(opts: RedisOptions = {}): Redis {
  return new Redis(env.REDIS_URL, opts);
}

export function createBlockingRedis(opts: RedisOptions = {}): Redis {
  return new Redis(env.REDIS_URL, { maxRetriesPerRequest: null, ...opts });
}

createRedis is for ordinary, non-blocking commands: XADD, XLEN, pipelines. createBlockingRedis exists specifically for XREAD/XREADGROUP calls that pass BLOCK, because while a connection is blocked waiting for new entries, nothing else can run on that same socket. maxRetriesPerRequest: null stops ioredis from treating a long block as a stuck, failing command. It's a small detail, but it's the kind of thing that only shows up once you run blocking reads against a real broker instead of reading the docs.

The Short Version

• A Redis Stream is an append only log with the same basic shape as a Kafka partition, except it's a Redis data structure you poke with plain commands, not a separate distributed system.
• Entry IDs are <millis>-<seq>, always increasing. Unlike a Kafka offset, the ID itself carries real wall clock time, which is why XRANGE can query by time directly.
• One stream key equals one Kafka partition in ordering terms, but Redis has no built-in partitioning. Scaling means sharding into more stream keys yourself, one per symbol rather than one topic with a fixed partition count.
• Kafka retains on disk by policy. A Redis stream grows in memory until you trim it with MAXLEN, and the ~ approximate form trades a slightly fuzzy count for cheap trimming.
• Redis persistence (RDB, AOF) is configurable but still fundamentally memory-bound, and weaker by default than Kafka's replicated disk log.

The question I keep circling back to is what happens when the consumer holding a batch of entries falls over mid-flight. Before I get to the Redis answer, I want the Kafka one in front of me first: consumer groups, lag, and what a rebalance does to work already in progress.