Redis Streams Consumer Groups: Competing Consumers and the PEL

Kafka and Redis Streams both let you call a group of workers a "consumer group," and both use that group to share out messages. Under the hood, they don't agree on what sharing means. In Kafka, sharing means partition ownership: whole partitions get assigned to one consumer each, and every key inside that partition arrives at that consumer, in order, for as long as it holds the partition. In Redis Streams, sharing means something closer to a work queue: any consumer that calls XREADGROUP can pick up the next available entry, whoever asks first. Same vocabulary, different machine underneath it.

I spent the previous post on how a Kafka group splits partitions between its members and reshuffles them when membership changes. Redis answers the same question with a different set of primitives, and once you look closely the resemblance stops: XREADGROUP, the Pending Entries List, and XACK. I only noticed how far apart they are because I built the same stock ticker twice, once per broker, and had to make the two versions behave the same way under load.

From Partitions to a Work Queue

In Kafka, six partitions across two consumers means three each, and a key always lands in the same partition, so the same consumer handles every tick for BBCA. Order is a side effect of ownership.

Redis consumer groups have no concept of ownership over a key or a range of entries. XREADGROUP with the special ID > just hands out the next unclaimed entries in the stream to whichever consumer asked. No partition count caps how many consumers can usefully compete for work: start one, start ten, they will all pull from the same stream. The tradeoff is exactly the flip side: two ticks for the same symbol can be picked up by two different consumers and processed concurrently, so per-key order is not preserved across the group. If you need that guarantee in Redis, you have to shard it yourself, one stream per symbol, one consumer per stream, which just rebuilds Kafka's partition model by hand.

Creating a Group: XGROUP CREATE

A Redis consumer group is created explicitly against a stream key, the same key I had been pushing ticks into with a plain XADD:

XGROUP CREATE market-updates evaluators 0 MKSTREAM

The last argument before MKSTREAM is the starting ID. $ means the group only sees entries added after it is created. 0 means the group starts from the very beginning of the stream, so anything already sitting there counts as unread backlog. MKSTREAM creates the stream if it doesn't exist yet, which avoids a NOGROUP error when you're setting up a group on a stream that's still empty.

I wrapped it in a helper that swallows the "group already exists" error, so my setup code can call it on every run without thinking about it:

export async function ensureGroup(
  redis: Redis,
  key: string,
  group: string,
  start = "$",
): Promise<void> {
  try {
    await redis.xgroup("CREATE", key, group, start, "MKSTREAM");
  } catch (e) {
    if (!(e instanceof Error) || !e.message.includes("BUSYGROUP")) throw e;
  }
}

For the run I'm about to describe I passed start = "0" before writing a single entry, so the group's backlog and the 200 ticks I generated afterward were the same set.

XREADGROUP: First Consumer to Ask, Wins

Reading as part of a group looks similar to a plain XREAD, but it needs a group name and a consumer name, and it changes what Redis does internally:

XREADGROUP GROUP evaluators pod-1 COUNT 10 STREAMS market-updates >

pod-1 is just a label this particular connection chose for itself; Redis doesn't validate it against anything. Whichever consumer sends this command next gets the next batch of unclaimed entries. Nothing assigns work ahead of time: no rebalance, no coordination between pod-1 and pod-2 beyond both hitting the same group. That's the entire mechanism for spreading load: start more consumers calling XREADGROUP > against the same group.

Read Is Not Acknowledged: the Pending Entries List

The moment Redis hands entries to a consumer via XREADGROUP >, it doesn't forget about them. Each entry gets recorded in the group's Pending Entries List (PEL): which entry ID, which consumer currently owns it, how long it's been idle since delivery, and how many times it's been delivered. The entry sits in the PEL whether or not the consumer that received it ever finishes processing.

The core thing to keep straight: read and acknowledged are different states. Getting an entry back from XREADGROUP only means Redis marked it as delivered to you. It says nothing about whether your code did anything with it.

You can inspect the PEL directly:

XPENDING market-updates evaluators

which returns a summary: total pending count, the lowest and highest pending IDs, and a per-consumer breakdown. An extended form also lists individual entries with their idle time and delivery count, which is the information you'd need to decide an entry has been stuck long enough to reclaim. That reclaiming mechanism, XAUTOCLAIM, turned into a post of its own, so I'll leave it alone here.

XACK: Telling Redis You're Done

The only thing that moves an entry out of the PEL is an explicit acknowledgment:

XACK market-updates evaluators 1720000000000-0 1720000000001-0

My drain loop puts the whole read-then-ack cycle in one place:

async function drain(
  conn: Redis,
  consumer: string,
  counters: Record<string, number>,
): Promise<void> {
  for (;;) {
    const entries = await readGroup(conn, { key: KEY, group: GROUP, consumer, count: 10 });
    if (entries.length === 0) break;
    counters[consumer] = (counters[consumer] ?? 0) + entries.length;
    await conn.xack(KEY, GROUP, ...entries.map((e) => e.id));
  }
}

Every batch that comes back from readGroup gets acknowledged right after it's counted, entry by entry, using the IDs Redis handed back. If the process died between the readGroup call and the xack call, those entries would stay in the PEL, owned by a consumer that's no longer running, until something explicitly reclaims them.

graph TD
    A["XADD market-updates"] --> B[("Stream entries")]
    B -->|"XREADGROUP GROUP evaluators pod-1 >"| C["pod-1 processes"]
    B -->|"XREADGROUP GROUP evaluators pod-2 >"| D["pod-2 processes"]
    C --> E[("PEL entry, owner=pod-1")]
    D --> F[("PEL entry, owner=pod-2")]
    C -->|"XACK"| G["Removed from PEL"]
    D -->|"XACK"| H["Removed from PEL"]

Both consumers pull from the same stream node, not from separate partitions assigned to each of them, which is where the model diverges from Kafka.

Two Consumers, No Ownership

My test harness for this is small: write 200 simulated ticks into the stream, create a group called evaluators, open two separate ioredis connections named pod-1 and pod-2, then run the drain loop above on both at once through Promise.all.

sequenceDiagram
    participant P1 as pod-1
    participant R as Redis
    participant P2 as pod-2
    P1->>R: "XREADGROUP GROUP evaluators pod-1 COUNT 10 STREAMS market-updates >"
    R-->>P1: "10 entries, PEL += 10, owner=pod-1"
    P2->>R: "XREADGROUP GROUP evaluators pod-2 COUNT 10 STREAMS market-updates >"
    R-->>P2: "next 10 entries, PEL += 10, owner=pod-2"
    Note over R: "PEL now holds 20 unacked entries"
    P1->>R: "XACK market-updates evaluators ...ids"
    P2->>R: "XACK market-updates evaluators ...ids"
    Note over R: "PEL empty, all 20 acknowledged"

Each call to readGroup pulls up to 10 entries at a time, and both consumers race for the same 200-entry pool until it's drained. The split between pod-1 and pod-2 ends up roughly even, but it's not deterministic like Kafka's partition assignment: it depends on which consumer's XREADGROUP call lands first each round. After both loops finish, pendingSummary on the group reports zero pending entries, because every batch was acknowledged right after it was counted.

The detail worth sitting with, and the one I left myself a comment about while writing the loop: no per-symbol ownership exists here. A BBCA tick and the next BBCA tick can be handed to different consumers and processed out of order relative to each other. Harmless when the consumers only count entries, as mine do in that run. Less harmless in the ticker itself, where alerts fire off price movements and the order they arrive in changes the answer. My market-updates stream keys entries by symbol, and to keep a competing-consumer group safe for that workload I'd need per-symbol streams, or Kafka.

When a Work Queue Beats Partition Ownership

None of this makes Redis groups worse, just different. Scaling out is trivial: no partition count to plan around, no rebalance protocol to trigger, no limit on useful parallelism other than Redis itself becoming the bottleneck. For work where order across items doesn't matter (fan-out notifications, independent background jobs, anything idempotent per item) a Redis consumer group gets you a horizontally scalable queue with a lot less ceremony than Kafka's partition machinery.

	Kafka group	Redis group
Assignment	Partitions pinned to members	Entries handed to whoever asks
Per-key order across consumers	Preserved	Not preserved
Max useful parallelism	Equal to partition count	Unbounded, until Redis is the bottleneck
Add capacity	Add partitions and consumers, triggers a rebalance	Just start more consumers
Read vs. done	Committed offset	PEL entry cleared by XACK
Backpressure signal	Consumer lag	PEL size and stream length

The Short Version

Kafka consumer groups divide partitions among members, so per-key order survives the whole group at the cost of a hard parallelism ceiling and rebalances when membership changes. Redis consumer groups divide entries among whoever asks first, so scaling is trivial but per-key order is not preserved unless you shard streams yourself. And within a Redis group, XREADGROUP and XACK are two separate steps on purpose: a delivered entry sits in the Pending Entries List, owned by whichever consumer read it, until that consumer explicitly says it's done.

If it never says so, the entry stays in the PEL until something else reclaims it, which leaves the question of what to do about the messages nobody finishes. Kafka has a more opinionated answer than Redis does, so I want to start there next time, with commit timing, bounded retries, and a dead letter queue.