Redis Streams Consumer Groups: Competing Consumers and PEL

Kafka and Redis Streams both provide a "consumer group," but they assign different units of work. Kafka assigns partitions to group members. Redis Streams assigns available entries to consumers that request them. This difference affects ordering and recovery.

Kafka consumer groups assign partitions to members and change assignments when membership changes. Redis uses XREADGROUP, the Pending Entries List, and XACK to distribute and track entries. I compared them while building the same stock ticker twice. The two versions needed similar behavior under load despite their different delivery models.

From partitions to a work queue

In this Kafka example, two consumers each receive three of the six partitions. Stable key routing puts BBCA ticks in the same partition. Its assigned consumer handles those ticks until a rebalance changes ownership.

Redis XREADGROUP with ID > returns entries that the group has not delivered before. Any member can request the next batch. Two entries for the same symbol can thus reach different consumers and finish out of order.

More consumers can share the work without a partition count limit, but Redis capacity and the workload still limit useful concurrency. Ordering for a symbol requires a separate design, such as one stream and one active worker per symbol.

Creating a group: XGROUP CREATE

I create the group for the stream key that receives ticks through XADD:

XGROUP CREATE market-updates evaluators 0 MKSTREAM

The argument before MKSTREAM sets the initial group position. $ starts after the entries that exist when the group is created. 0 starts before the first retained entry. MKSTREAM creates the stream if it does not exist. Without it, group creation requires an existing stream.

I wrapped it in a helper that swallows the "group already exists" error, so my setup code can call it on every run without thinking about it:

export async function ensureGroup(
  redis: Redis,
  key: string,
  group: string,
  start = "$",
): Promise<void> {
  try {
    await redis.xgroup("CREATE", key, group, start, "MKSTREAM");
  } catch (e) {
    if (!(e instanceof Error) || !e.message.includes("BUSYGROUP")) throw e;
  }
}

I passed start = "0" before writing any entries. The 200 ticks generated afterward formed the group's unread backlog.

XREADGROUP: first consumer to ask, wins

A group read needs both a group name and a consumer name. Unlike a plain XREAD, it also changes the group's delivery state:

XREADGROUP GROUP evaluators pod-1 COUNT 10 STREAMS market-updates >

The consumer name pod-1 identifies a group member. Redis does not verify that it corresponds to one unique process, so use distinct names for independent consumers. Each XREADGROUP > call requests the next available batch. No partition rebalance precedes delivery.

Read is not acknowledged: the pending entries list

For a normal XREADGROUP > delivery without NOACK, Redis records the entry in the Pending Entries List (PEL). Pending metadata includes the entry ID, consumer name, delivery time, and delivery count. Successful application processing alone does not remove this pending state.

Read and acknowledged are different states. Getting an entry back from XREADGROUP only means Redis marked it as delivered to you. It says nothing about whether your code did anything with it.

You can inspect the PEL directly:

XPENDING market-updates evaluators

The summary includes the pending count, the lowest and highest pending IDs, and counts for each consumer. The extended form lists each entry's idle time and delivery count. These values help a recovery process select entries to reclaim. The recovery post explains XAUTOCLAIM.

XACK: telling Redis you are done

After successful handling, use XACK to remove the pending record. This acknowledgment does not delete the stream entry:

XACK market-updates evaluators 1720000000000-0 1720000000001-0

My drain loop puts the whole read then ack cycle in one place:

async function drain(
  conn: Redis,
  consumer: string,
  counters: Record<string, number>,
): Promise<void> {
  for (;;) {
    const entries = await readGroup(conn, { key: KEY, group: GROUP, consumer, count: 10 });
    if (entries.length === 0) break;
    counters[consumer] = (counters[consumer] ?? 0) + entries.length;
    await conn.xack(KEY, GROUP, ...entries.map((e) => e.id));
  }
}

After readGroup, the consumer counts the entries and acknowledges each ID through xack. A crash between these calls leaves the entries in the PEL. Another consumer can reclaim them, or the original consumer can read its pending entries after recovery.

graph TD
    A["XADD market-updates"] --> B[("Stream entries")]
    B -->|"XREADGROUP GROUP evaluators pod-1 >"| C["pod-1 processes"]
    B -->|"XREADGROUP GROUP evaluators pod-2 >"| D["pod-2 processes"]
    C --> E[("PEL entry, owner=pod-1")]
    D --> F[("PEL entry, owner=pod-2")]
    C -->|"XACK"| G["Removed from PEL"]
    D -->|"XACK"| H["Removed from PEL"]

Both consumers pull from the same stream node, not from separate partitions assigned to each of them, which is where the model diverges from Kafka.

Two consumers, no ownership

The test uses a group called evaluators and 200 simulated ticks. Two ioredis connections identify their consumers as pod-1 and pod-2. Both run the drain loop through Promise.all:

sequenceDiagram
    participant P1 as pod-1
    participant R as Redis
    participant P2 as pod-2
    P1->>R: "XREADGROUP GROUP evaluators pod-1 COUNT 10 STREAMS market-updates >"
    R-->>P1: "10 entries, PEL += 10, owner=pod-1"
    P2->>R: "XREADGROUP GROUP evaluators pod-2 COUNT 10 STREAMS market-updates >"
    R-->>P2: "next 10 entries, PEL += 10, owner=pod-2"
    Note over R: "PEL now holds 20 unacked entries"
    P1->>R: "XACK market-updates evaluators ...ids"
    P2->>R: "XACK market-updates evaluators ...ids"
    Note over R: "PEL empty, all 20 acknowledged"

Each readGroup call requests up to 10 entries. The two consumers compete for the 200 entries, so the allocation depends on request timing. After both loops finish, pendingSummary reports zero pending entries because the loops acknowledged every counted entry.

A Redis consumer group does not assign symbols to consumers. Different consumers can process consecutive BBCA ticks concurrently and finish in a different order. My test only counts entries, so this does not change its result.

The ticker evaluates price changes, where order affects the result. A symbol field in market-updates does not enforce consumer ownership. Separate streams with controlled consumer assignment, or Kafka partitions, can provide the required processing order.

When a work queue beats partition ownership

Redis groups let additional workers request entries without partition reassignment. This fits independent jobs that can complete in any order. Throughput still depends on Redis capacity, handler cost, and other shared resources. More consumers do not guarantee more throughput.

The assignment tradeoff

Kafka groups divide partitions among members, with ordering inside each partition. Redis groups assign individual entries, so concurrent workers can complete related entries out of order. Redis records delivered entries in the PEL until acknowledgment or another operation removes their pending state.

An entry that needs acknowledgement remains pending until the group removes that pending record. Recovery code must decide when to retry it or send it to a dead letter stream. The next post compares commit timing, bounded retries, and dead letters on Kafka.