Redis Streams Fundamentals: XADD, Entry IDs, and MAXLEN

Redis Streams, introduced in Redis 5.0, stores an ordered sequence of entries inside a Redis key. The data type supports retained history and consumer groups. Its ordering and retention model differs from Kafka partitions.

I compared both systems while building the same ticker twice, with brokers in Docker on my laptop. The ticker publishes price updates and evaluates alerts. This post explains Redis entry IDs, reads, and retention.

The log inside the cache

Append entries with XADD. Redis can assign the entry ID automatically. Each entry contains field and value pairs, represented here as strings:

XADD market-updates * symbol BBCA price 9530 seq 1
=> "1718000000123-0"

The * tells Redis "give me the next ID." A tick from my market simulator goes in the same way, through a small ioredis wrapper:

import { createRedis } from "../../lib/redis/client";
import { MarketSimulator, tickToRedis } from "../../lib/domain";

const redis = createRedis();
const sim = new MarketSimulator({ seed: 3, volatilityScale: 5 });

for (const tick of sim.batch(8)) {
  // '*' tells Redis to assign the next time-ordered ID.
  const id = await redis.xadd(KEY, "*", ...tickToRedis(tick));
}

tickToRedis converts a tick into the field and value pairs that XADD accepts: symbol, price, prevPrice, volume, ts, and seq. The application defines the meaning of these fields.

Entry IDs are not offsets

Every entry ID has the form <millisecondsTime>-<sequence>, and it always increases.

A Kafka offset identifies a position within a partition. Record timestamps are separate. Redis generated IDs combine a millisecond component with a sequence component.

If two entries use the same millisecond component, Redis increments the sequence, for example ...123-0 and ...123-1. When time advances, the sequence can restart at 0. If the clock moves backward, Redis preserves increasing IDs using the previous time component. Explicit IDs can also differ from wall clock time.

A Redis entry ID is a cursor into the stream. For generated IDs, its millisecond component also supports time range queries with XRANGE. Do not treat every possible entry ID as an exact arrival timestamp.

Reading it back: XRANGE and XREAD

Three commands cover most of what you need:

• XRANGE key - + reads a range by ID (- and + mean the whole stream). Good for replay and inspection.
• XREAD tails the stream, like tail -f, waiting for new entries past a given ID.
• XLEN returns the entry count.

const range = await redis.xrange(KEY, "-", "+");
for (const [id, flat] of range.slice(0, 4)) {
  const f: Record<string, string> = {};
  for (let i = 0; i + 1 < flat.length; i += 2) f[flat[i]] = flat[i + 1];
  console.log(`${id}  ${f.symbol} price=${f.price} seq=${f.seq}`);
}

Redis returns entries as [id, flatFieldArray] pairs. The application converts each field array into an object. The command does not supply a schema registry.

One stream, one partition

One stream key is a single, totally ordered log, the direct equivalent of one Kafka partition. Redis has no built in concept of partitioning a single key.

Kafka key routing selects from a fixed set of partitions. My market-updates topic has six. It routes symbols with (murmur2(key) & 0x7fffffff) % numberOfPartitions.

Ten symbols must share those six partitions. Four of my symbols selected partition 0. Records for each symbol remain together, but the distribution does not divide load evenly.

To divide Redis stream data, the application creates several stream keys. For example, market:{BBCA} and market:{BBRI} each hold one symbol. Each stream preserves its own entry order. This design manages one stream per symbol instead of a fixed number of Kafka partitions. Redis Cluster still hashes those stream keys into slots.

graph TD
    A["market-updates (Kafka topic)"] --> B["6 partitions, murmur2(key) picks one"]
    B --> C["10 symbols hash into 6 buckets, collisions guaranteed"]
    D["Redis: no topic, just keys"] --> E["market:{BBCA}"]
    D --> F["market:{BBRI}"]
    D --> G["market:{...}, one stream per symbol"]

A fixed partition count limits the number of logs to manage, but keys can share partitions. A stream per symbol separates their entries into different logs. The number of streams then grows with the number of symbols. Redis Cluster can distribute these streams, although different keys can still share a slot or node.

It lives in RAM, so you trim it

A Kafka topic stores log segments on disk and applies configured retention rules. Redis keeps stream entries in memory. Without trimming or another removal policy, the stream grows as entries arrive.

await redis.xadd(KEY, "MAXLEN", "~", "10", "*", ...tickToRedis(tick));

MAXLEN ~ N sets an approximate length limit. Exact trimming with MAXLEN N removes enough entries to meet the limit precisely. Approximate trimming removes whole internal macro nodes, which can reduce work. The stream can thus retain more than N entries.

graph LR
    P["Producer: XADD"] --> S["Stream in RAM"]
    S --> M{"MAXLEN set?"}
    M -->|"No"| G["Memory grows unbounded"]
    M -->|"Yes, MAXLEN ~ N"| T["Approximate trim, cheap"]

Persistence depends on configuration. RDB recovery can lose writes since the last snapshot. With AOF and appendfsync everysec, a crash can lose about a second of writes. The always policy synchronizes writes more frequently at additional cost.

My container uses --appendonly yes --appendfsync everysec. Compare persistence and replication settings before making a durability claim about either Redis or Kafka.

Client calls and connections

The client setup uses two separate Redis connection factories.

export function createRedis(opts: RedisOptions = {}): Redis {
  return new Redis(env.REDIS_URL, opts);
}

export function createBlockingRedis(opts: RedisOptions = {}): Redis {
  return new Redis(env.REDIS_URL, { maxRetriesPerRequest: null, ...opts });
}

The createRedis factory handles ordinary commands such as XADD, XLEN, and pipelines. The createBlockingRedis factory provides separate connections for XREAD or XREADGROUP with BLOCK. Other commands on a blocked connection must wait.

The maxRetriesPerRequest: null setting removes the request retry limit in ioredis. It controls retry behavior, not the Redis blocking timeout. The BLOCK argument controls that wait.

The Redis model in brief

• A Redis Stream retains an ordered sequence of entries within a key.
• Entry IDs use <millis>-<seq> and increase within the stream. Generated IDs usually reflect server time, subject to monotonicity rules.
• One stream has no internal partitioning. More streams permit separate routing and workers.
• The MAXLEN option limits retained entries. Approximate trimming with ~ can leave more entries than the target.
• AOF, RDB, replication, and retention settings determine recovery limits.

The next question is what happens when the consumer holding a batch of entries falls over mid flight. I will start with Kafka's answer, then turn to Redis: consumer groups, lag, and what a rebalance does to work already in progress.