Recovering Stuck Messages in Redis Streams: XAUTOCLAIM and Dead Letters

When a Kafka consumer dies mid message, the fix is almost administrative: the group rebalances, another consumer inherits the partition, and processing resumes from the last committed offset. Redis Streams has no partitions to reassign and no rebalance protocol. A consumer that reads an entry with XREADGROUP and then crashes just leaves that entry sitting in a list called the pending entries list, still marked as owned by a consumer that is never coming back. Redis will not notice on its own. Recovering it is on you, and that is exactly what XAUTOCLAIM and a delivery counter are for.

Last post I walked through the retry-then-dead-letter path on the Kafka side of my little ticker system: a poison tick gets a bounded number of attempts on market-updates, then goes to a side topic. I wanted the Redis half of that story. Same domain, same failure, but no partitions to reassign and no committed offset to rewind. What I got instead was a pending list and a per-entry delivery count that between them do roughly the same job.

The Pending Entries List

A Redis Streams consumer group tracks in-flight work with a structure called the PEL (pending entries list), one per stream and group. The mechanics are simple:

1. XREADGROUP GROUP g c ... STREAMS key > delivers new entries to consumer c and, in the same step, records each one in the PEL as owned by c.
2. The consumer processes the entry and calls XACK, and the entry leaves the PEL.
3. Or the consumer never calls XACK, because it crashed, hung, or got killed by a deploy. The entry just sits there, still attributed to a consumer that is not coming back.

Nothing times it out automatically. I went in half expecting a visibility timeout, the way SQS hands a message back to the queue on its own after a while, and Redis has no such thing. A dead consumer's entries stay pending forever unless another consumer goes looking for them.

So I built the situation on purpose. I created a group called evaluators on a stream of market ticks, then pointed a consumer named pod-crash at it that reads everything and acknowledges nothing:

await ensureGroup(redis, KEY, GROUP, "0");

const sim = new MarketSimulator({ seed: 33, volatilityScale: 3 });
for (const tick of sim.batch(TOTAL)) await redis.xadd(KEY, "*", ...tickToRedis(tick));

// "pod-crash" reads everything but ACKs nothing (simulating a crash).
const taken = await readGroup(redis, {
  key: KEY,
  group: GROUP,
  consumer: "pod-crash",
  count: TOTAL,
});

ensureGroup is a thin wrapper over XGROUP CREATE ... MKSTREAM, and readGroup builds the variadic XREADGROUP call. With TOTAL at 50 ticks across 10 symbols, all 50 entries are now in the PEL, all owned by pod-crash, and pod-crash never comes back.

XPENDING: Who Is Holding What

Before you can reclaim anything, you need to see it. XPENDING has two shapes, and I ended up wrapping both.

The summary form (no id range) gives you the total pending count and a breakdown per consumer:

export async function pendingSummary(redis: Redis, key: string, group: string) {
  const res = await redis.call("XPENDING", key, group) as [
    number,
    string | null,
    string | null,
    Array<[string, string]> | null,
  ];
  const perConsumer: Record<string, number> = {};
  for (const [c, n] of res?.[3] ?? []) perConsumer[c] = Number(n);
  return { total: Number(res?.[0] ?? 0), perConsumer };
}

Right after pod-crash read and abandoned its 50 entries, this printed now PENDING: 50 (owned by pod-crash). That shape is your crash signal in the wild: a consumer name that shows up in perConsumer and never shrinks.

The extended form takes an id range and a count, and gives you one row per entry: its id, its current owner, how long it has been idle, and how many times it has been delivered:

export async function pendingDetails(redis: Redis, key: string, group: string, count = 100) {
  const res = await redis.call("XPENDING", key, group, "-", "+", String(count)) as Array<
    [string, string, number, number]
  >;
  return (res ?? []).map(([id, consumer, idleMs, deliveries]) => ({
    id,
    consumer,
    idleMs: Number(idleMs),
    deliveries: Number(deliveries),
  }));
}

That deliveries field is the important one. It is Redis's built-in answer to "how many times has this specific entry been handed to a consumer," and it is what a dead-letter decision should be based on.

XAUTOCLAIM: Reclaiming Idle Work

XAUTOCLAIM is the tool that moves ownership. It takes a minimum idle time, and only touches entries that have been sitting unacknowledged for at least that long:

export async function autoClaim(
  redis: Redis,
  p: { key: string; group: string; consumer: string; minIdleMs: number; start?: string; count?: number },
) {
  const res = await redis.call(
    "XAUTOCLAIM",
    p.key,
    p.group,
    p.consumer,
    String(p.minIdleMs),
    p.start ?? "0-0",
    "COUNT",
    String(p.count ?? 100),
  ) as [string, Array<[string, string[]]>, string[]?];
  return { cursor: res[0], claimed: parseEntries(res[1]) };
}

The min-idle-time argument matters more than it looks. It is the difference between "this consumer crashed" and "this consumer is just slow." If you reclaim entries the moment they are pending, you will steal work from a consumer that is still legitimately in the middle of processing it, and now two consumers might act on the same entry. Set the threshold comfortably above your worst-case processing time and you only catch the ones that are stuck.

I let the entries sit for a moment, then reclaimed them with a threshold shorter than the wait:

await sleep(150); // let entries accrue idle time

const { claimed } = await autoClaim(redis, {
  key: KEY,
  group: GROUP,
  consumer: "pod-rescue",
  minIdleMs: 100,
  count: TOTAL,
});

150ms of idle time clears the 100ms bar, so all 50 entries come back, now owned by pod-rescue instead of the dead pod-crash. The call is atomic: it reassigns ownership and hands back the entries' current field data in a single round trip, so two rescuer consumers racing to claim the same idle entries cannot both walk away thinking they own it.

sequenceDiagram
    participant PC as pod-crash
    participant R as Redis
    participant PR as pod-rescue
    PC->>R: XREADGROUP GROUP evaluators pod-crash COUNT 50
    Note over R: 50 entries enter the PEL, owned by pod-crash
    Note over PC: pod-crash dies before XACK
    Note over R: entries sit idle, still owned by pod-crash
    PR->>R: XAUTOCLAIM key evaluators pod-rescue 100 0-0 COUNT 50
    Note over R: idle >= 100ms, so ownership moves to pod-rescue
    R-->>PR: 50 claimed entries, delivery count incremented

That last note matters: claiming an entry through XAUTOCLAIM counts as a delivery, the same as the original XREADGROUP did. After this one reclaim, every one of those 50 entries already shows a delivery count of 2.

From Delivery Count to Dead Letter

The rule I settled on: the per-entry delivery counter from XPENDING is the DLQ trigger. After N deliveries, copy the entry to a DLQ stream and XACK the original. That is the Redis equivalent of Kafka's bounded local retry count, just measured differently: instead of counting attempts inside one consumer's loop, you are counting deliveries across however many consumers and crashes an entry has passed through.

In production that decision loop is naturally cyclical: reclaim with XAUTOCLAIM, check deliveries for each claimed entry via pendingDetails, process the ones under the threshold and let them get acked, and for the ones over the threshold, dead-letter them instead of processing.

My run only did one crash-then-rescue cycle, though, so every claimed entry came back with the same delivery count of 2, which leaves nothing to threshold on; a real poison message and a message that merely happened to be read by pod-crash look identical by count alone at that point. So I swapped in a direct poison marker instead, using the tick's own seq field to stand for "this entry would never succeed no matter how many more times it gets redelivered":

let acked = 0;
let deadLettered = 0;
for (const entry of claimed) {
  const isPoison = Number(entry.fields.seq) === 3; // ~1 per symbol -> ~10 poison
  if (isPoison) {
    await redis.xadd(DLQ, "*", ...Object.entries(entry.fields).flat(), "deadLetteredFrom", entry.id);
    await redis.xack(KEY, GROUP, entry.id); // ack so it leaves the PEL
    deadLettered++;
  } else {
    await redis.xack(KEY, GROUP, entry.id);
    acked++;
  }
}

Two things worth noticing here. First, the dead letter write copies every original field over, plus a deadLetteredFrom pointer back to the entry's original id, so the DLQ stream is self-describing: you can look at an entry in it and know exactly which entry in market-updates it came from. Second, XACK runs in both branches. Whether an entry was processed successfully or dead-lettered, it gets acknowledged and leaves the PEL either way. Skipping the ack on the dead-lettered path would leave you exactly where you started: an entry stuck in the PEL, just now duplicated into a DLQ stream too.

With 50 entries across 10 symbols and roughly one poison tick per symbol, the run ended with about 40 acked, about 10 dead-lettered, PENDING back at 0, and the DLQ holding the dead-lettered entries. I left a spot check on pendingDetails at the end to print any survivors; when the PEL clears, it prints nothing at all, which is the boring outcome you want.

graph TD
    A["XAUTOCLAIM reclaims an idle entry"] --> B["XPENDING: check deliveries"]
    B -->|"deliveries <= threshold"| C["process, then XACK"]
    B -->|"deliveries > threshold"| D["XADD to DLQ stream"]
    D --> E["XACK original id"]
    C --> F["entry leaves the PEL"]
    E --> F

One Poison Entry Doesn't Block Anyone Else

The Redis version diverges from Kafka's right here, beyond mechanics alone. A Kafka consumer group hands each partition to exactly one member, and that partition is an ordered log: if you refuse to advance past a poison message, everything behind it on that partition waits too. A Redis consumer group has no partitions and no per-key ownership. XREADGROUP > hands the next entries to whichever consumer asks first, so one stuck entry just occupies its own row in the PEL. Every other entry keeps flowing to whichever consumer is free to take it.

	Kafka	Redis Streams
In-flight tracking	one committed offset per partition	a PEL entry per message
What a crash blocks	the whole partition, until reassigned	nothing else; other entries keep moving
Retry mechanism	bounded attempts inside one consumer's loop	redelivery across XAUTOCLAIM sweeps
DLQ trigger	local attempt counter	delivery count from XPENDING
Dead letters land in	a side topic	a side stream

That is a real tradeoff, more than a syntax difference. Kafka's partition-wide blocking is also what gives you strict per-key ordering. Redis's per-entry independence is also why, as I found when I first put a consumer group on one of these streams, order across consumers is not preserved at all: two ticks for the same symbol can be picked up and processed concurrently by two different workers. You get resilience to a poison entry for free; you give up the ordering guarantee to get it.

Still Just At-Least-Once

None of this turns Redis Streams into something stronger than at-least-once delivery. The PEL and XACK are exactly how Redis gives you that guarantee: an entry is either acknowledged, or it eventually gets reclaimed and delivered again. Kafka's transactional exactly-once has no equivalent here. Redis has no built-in multi-key transaction across process-and-output the way Kafka's producer transactions tie the output write to the offset commit. MULTI/EXEC or a Lua script gets you atomicity for operations on the same key, but stitching "did the side effect happen" and "did the entry get acked" into one atomic unit is not something the broker does for you.

Which means the same lesson from the Kafka side of this series applies here too: make processing idempotent. A reclaimed entry might have partially run on pod-crash before it died, so the handler on pod-rescue should be safe to run again, keyed on something stable like (symbol, seq) for ticks or the notification's own id, not on the assumption that this is the first time this entry has ever been seen.

What I'd Watch in Production

Watch pendingSummary the way you would watch consumer lag in Kafka. A PENDING count that only grows, especially concentrated under one consumer name that stopped renewing anything, is your crash signal, and it will not fix itself.

Pick a min-idle-time for XAUTOCLAIM that is comfortably above your worst realistic processing time, so a slow-but-alive consumer never gets its work stolen out from under it. Base the dead-letter decision on the delivery count from XPENDING, not a single pass, since one XAUTOCLAIM sweep alone cannot distinguish a message that is merely unlucky from one that is poison. And once an entry lands in the DLQ stream, treat it the same way you would treat a Kafka DLQ topic: something inspectable, with alerting on it, not a place messages quietly pile up forever.

Both sides of the ticker have now landed on the same answer, at-least-once plus idempotent handlers, so the obvious next thing to poke at is the one guarantee that claims to be stronger. Next post I go back to Kafka for idempotent producers and transactions, and see how much of "exactly-once" survives contact with a pipeline.