Redis Streams: Stuck Messages, XAUTOCLAIM, and Dead Letters

After a Kafka consumer fails, another member can resume its partition from the committed offset. Redis Streams recovery works per entry. A consumer that fails after XREADGROUP can leave work in the pending entries list. The application must use a recovery process, such as XAUTOCLAIM, to retrieve that unfinished work.

The previous post tested bounded Kafka retries. A poison tick on market-updates eventually moves to a dead letter topic. I tested the same failure on Redis. Its recovery process uses pending entries and delivery counts instead of partition offsets.

The pending entries list

A Redis Streams consumer group tracks in flight work with a structure called the PEL (pending entries list), one per stream and group. The mechanics are simple:

1. XREADGROUP GROUP g c ... STREAMS key > delivers new entries to consumer c and, in the same step, records each one in the PEL as owned by c.
2. The consumer processes the entry and calls XACK, and the entry leaves the PEL.
3. Or the consumer never calls XACK, because it crashed, hung, or got killed by a deploy. The entry remains pending under that consumer's name.

I initially expected a visibility timeout like SQS. Redis does not automatically make a pending entry available to a new reader after a timeout. A recovery process must request pending entries or reclaim them.

I created the evaluators group on a stream of market ticks. The pod-crash consumer reads entries but acknowledges none:

await ensureGroup(redis, KEY, GROUP, "0");

const sim = new MarketSimulator({ seed: 33, volatilityScale: 3 });
for (const tick of sim.batch(TOTAL)) await redis.xadd(KEY, "*", ...tickToRedis(tick));

// "pod-crash" reads everything but ACKs nothing (simulating a crash).
const taken = await readGroup(redis, {
  key: KEY,
  group: GROUP,
  consumer: "pod-crash",
  count: TOTAL,
});

ensureGroup is a thin wrapper over XGROUP CREATE ... MKSTREAM, and readGroup builds the variadic XREADGROUP call. With TOTAL at 50 ticks across 10 symbols, all 50 entries are now in the PEL, all owned by pod-crash, and pod-crash never comes back.

XPENDING: who is holding what

Before you can reclaim anything, you need to see it. XPENDING has two shapes, and I ended up wrapping both.

The summary form (no id range) gives you the total pending count and a breakdown per consumer:

export async function pendingSummary(redis: Redis, key: string, group: string) {
  const res = await redis.call("XPENDING", key, group) as [
    number,
    string | null,
    string | null,
    Array<[string, string]> | null,
  ];
  const perConsumer: Record<string, number> = {};
  for (const [c, n] of res?.[3] ?? []) perConsumer[c] = Number(n);
  return { total: Number(res?.[0] ?? 0), perConsumer };
}

Right after pod-crash read and abandoned its 50 entries, this printed now PENDING: 50 (owned by pod-crash). That shape is your crash signal in the wild: a consumer name that shows up in perConsumer and never shrinks.

The extended form accepts an ID range and a count. Each result contains the entry ID, owner, idle time, and delivery count:

export async function pendingDetails(redis: Redis, key: string, group: string, count = 100) {
  const res = await redis.call("XPENDING", key, group, "-", "+", String(count)) as Array<
    [string, string, number, number]
  >;
  return (res ?? []).map(([id, consumer, idleMs, deliveries]) => ({
    id,
    consumer,
    idleMs: Number(idleMs),
    deliveries: Number(deliveries),
  }));
}

That deliveries field is the important one. It is Redis's built in answer to "how many times has this specific entry been handed to a consumer," and it is what a dead letter decision should be based on.

XAUTOCLAIM: reclaiming idle work

XAUTOCLAIM is the tool that moves ownership. It takes a minimum idle time and selects pending entries whose idle time meets that limit:

export async function autoClaim(
  redis: Redis,
  p: { key: string; group: string; consumer: string; minIdleMs: number; start?: string; count?: number },
) {
  const res = await redis.call(
    "XAUTOCLAIM",
    p.key,
    p.group,
    p.consumer,
    String(p.minIdleMs),
    p.start ?? "0-0",
    "COUNT",
    String(p.count ?? 100),
  ) as [string, Array<[string, string[]]>, string[]?];
  return { cursor: res[0], claimed: parseEntries(res[1]) };
}

The minimum idle time selects entries that have remained pending long enough to investigate. It cannot prove that their consumer has failed. A slow consumer may still process an entry after another consumer claims it. Choose the threshold from expected processing time and make repeated effects safe.

I let the entries sit for a moment, then reclaimed them with a threshold shorter than the wait:

await sleep(150); // let entries accrue idle time

const { claimed } = await autoClaim(redis, {
  key: KEY,
  group: GROUP,
  consumer: "pod-rescue",
  minIdleMs: 100,
  count: TOTAL,
});

After 150ms, the entries exceed the 100ms idle threshold. The test retrieves all 50 entries under pod-rescue. Ownership changes atomically within Redis, but the previous consumer is not fenced from external work. Processing must thus tolerate overlapping attempts.

sequenceDiagram
    participant PC as pod-crash
    participant R as Redis
    participant PR as pod-rescue
    PC->>R: XREADGROUP GROUP evaluators pod-crash COUNT 50
    Note over R: 50 entries enter the PEL, owned by pod-crash
    Note over PC: pod-crash dies before XACK
    Note over R: entries sit idle, still owned by pod-crash
    PR->>R: XAUTOCLAIM key evaluators pod-rescue 100 0-0 COUNT 50
    Note over R: idle >= 100ms, so ownership moves to pod-rescue
    R-->>PR: 50 claimed entries, delivery count incremented

That last note matters: claiming an entry through XAUTOCLAIM counts as a delivery, the same as the original XREADGROUP did. After this one reclaim, every one of those 50 entries already shows a delivery count of 2.

From delivery count to dead letter

I use the delivery count from XPENDING to select entries for the DLQ. After N deliveries, the handler copies the entry to a DLQ stream and calls XACK for the original.

The Kafka example counts attempts inside one consumer loop. This Redis version counts recorded deliveries across consumers. A delivery does not prove that the handler completed an attempt.

A recovery loop claims idle entries, checks deliveries through pendingDetails, and processes entries below the retry limit. Entries at the limit go to a DLQ. Acknowledgment follows successful processing or a successful DLQ write.

This test performs one recovery cycle, so each claimed entry has a delivery count of 2. That count cannot distinguish persistent failures from entries abandoned by pod-crash. I thus use seq as a deterministic poison marker for the demonstration:

let acked = 0;
let deadLettered = 0;
for (const entry of claimed) {
  const isPoison = Number(entry.fields.seq) === 3; // ~1 per symbol -> ~10 poison
  if (isPoison) {
    await redis.xadd(DLQ, "*", ...Object.entries(entry.fields).flat(), "deadLetteredFrom", entry.id);
    await redis.xack(KEY, GROUP, entry.id); // ack so it leaves the PEL
    deadLettered++;
  } else {
    await redis.xack(KEY, GROUP, entry.id);
    acked++;
  }
}

The DLQ write copies the original fields and adds deadLetteredFrom with the original entry ID. Both successful processing and a successful DLQ write are followed by XACK.

The separate DLQ write and acknowledgment have a crash gap. A restart can write another DLQ entry before acknowledgment succeeds. Use idempotency or an atomic Redis operation where the required keys permit it.

The run uses 50 entries across 10 symbols, with roughly one poison tick per symbol. It finishes with about 40 processed, about 10 sent to the DLQ, and PENDING at 0. The final pendingDetails check prints any remaining pending entries. It prints none when the PEL is empty.

graph TD
    A["XAUTOCLAIM reclaims an idle entry"] --> B["XPENDING: check deliveries"]
    B -->|"deliveries <= threshold"| C["process, then XACK"]
    B -->|"deliveries > threshold"| D["XADD to DLQ stream"]
    D --> E["XACK original id"]
    C --> F["entry leaves the PEL"]
    E --> F

One poison entry does not block anyone else

A Kafka group assigns each partition to one member. If its handler stops at a poison record, later records in that partition wait. A Redis group tracks each pending entry separately. XREADGROUP > can continue delivering new entries while an earlier entry remains pending.

I observed the ordering tradeoff when I first tested Redis consumer groups. Two consumers can process ticks for the same symbol concurrently. The later tick can finish first. Independent entry delivery avoids blocking the entire stream, but the application must enforce any required processing order.

Repeated delivery remains possible

Redis Streams can support repeated delivery when the application reclaims pending work. Neither a PEL entry nor XACK automatically runs recovery. Retention and persistence must also preserve the entry.

A Lua script or MULTI/EXEC can combine output and acknowledgment commands. Redis Cluster requires the involved keys to share a slot. External effects need separate coordination, and Redis command errors do not provide transactional rollback.

Make the handler idempotent. pod-crash might have completed part of the work before another consumer reclaimed its entry. pod-rescue must tolerate repeated processing. Use a stable identity, such as (symbol, seq) for a tick or the notification ID.

What I'd watch in production

Monitor pendingSummary for entries that remain pending. A rising PENDING count can indicate slow, failed, or inactive consumers. It does not prove a crash. Inspect idle times and consumer health before recovery.

Set the XAUTOCLAIM idle threshold above expected processing time, but assume that a slow consumer can still overlap with recovery. Use delivery counts and the failure context to bound retries. Monitor the DLQ and provide an inspection and replay process.

Both ticker implementations need idempotent handlers when work can repeat. The next post examines Kafka producer idempotence and the scope of Kafka transactions.