At-Least-Once in Kafka: Retries and Dead Letter Queues

"At-least-once delivery" depends on offset commits after successful handling. A consumer can still repeat work if it crashes after processing but before the commit. The handler must make repeated effects safe.

Redis keeps delivered entries in the Pending Entries List until acknowledgment. Kafka records a committed offset per group and partition. The application decides when successful handling permits that offset to advance.

I tested consumer crashes, rebalances during a batch, and messages that fail on every attempt. This post describes the Kafka retry and dead letter behavior.

Delivery semantics are a commit decision

Kafka does not track "did the consumer finish processing this." It tracks one number per partition: the committed offset. Everything about delivery semantics falls out of when you move that number.

A commit before processing can skip unfinished work after a crash. A commit after processing permits replay from the earlier offset. The next consumer may then repeat an effect, such as an alert notification.

Kafka transactions can commit Kafka output and source offsets together, as described in the transaction post. The table assumes retained input and correctly configured consumers. Its transaction guarantee covers Kafka records and offsets, not external effects.

Commit after, not before

In KafkaJS, turn off auto commit and commit manually after the message is handled.

await consumer.run({
  autoCommit: false,
  eachMessage: async ({ topic, partition, message }) => {
    await handle(message);
    await consumer.commitOffsets([
      { topic, partition, offset: (Number(message.offset) + 1).toString() },
    ]);
  },
});

The position of commitOffsets relative to handle determines the recovery point. If the commit occurs first, normal recovery starts after that record even if handling failed. A commit after successful handling permits replay when the consumer fails before the commit. The application must tolerate repeated handling.

Retries without blocking the partition

Offset commits after processing support crash recovery, but a message can fail on every attempt. If the consumer never advances beyond it, later records in that partition cannot proceed. A single bad record can delay every symbol in that partition.

I made the failures deterministic so I could test the retry behavior. The consumer classifies ticks on market-updates by their sequence number for each symbol. Most succeed immediately. Some fail the first few attempts, then succeed. Poison ticks fail every attempt:

function classify(t: Tick): Kind {
  if (t.seq % 25 === 0) return "poison"; // permanent failure -> belongs in the DLQ
  if (t.seq % 7 === 0) return "flaky"; // transient -> succeeds once retried
  return "ok";
}

Inside eachMessage, a message gets a bounded number of local attempts, with a short backoff between them, before the consumer gives up on it:

let ok = false;
let lastErr: unknown;
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
  try {
    processTick(tick, attempt);
    ok = true;
    break;
  } catch (e) {
    lastErr = e;
    await sleep(15); // backoff before retry
  }
}

These are local retries: the partition remains at the same record during each attempt. A temporary downstream failure may resolve within the retry limit. The MAX_ATTEMPTS bound prevents a message that always fails from blocking the partition indefinitely.

The dead letter queue

After the retry limit, the consumer writes the failed record to a dead letter queue (DLQ) topic. The record includes enough context for later investigation. After a successful DLQ write, the consumer can advance its source offset.

graph TD
    M["Message arrives"] --> P["processTick(tick, attempt)"]
    P -->|success| C["commitOffsets"]
    P -->|failure, attempt < MAX| R["backoff, retry"]
    R --> P
    P -->|failure, attempt = MAX| D["dlqProducer.send to DLQ topic"]
    D --> C

The DLQ write carries the original key and value plus headers describing why it failed and where it came from:

await dlqProducer.send({
  topic: DLQ,
  messages: [
    {
      key: message.key,
      value: message.value,
      headers: {
        error: lastErr instanceof Error ? lastErr.message : String(lastErr),
        origin: `${topic}/${partition}/${message.offset}`,
      },
    },
  ],
});

What happens right after matters: the consumer commits the original offset regardless of whether the message succeeded or got dead lettered.

// Commit AFTER handling => at-least-once.
await consumer.commitOffsets([
  { topic, partition, offset: (Number(message.offset) + 1).toString() },
]);

The source offset must advance after a successful DLQ write, or the consumer will read the same failed record again. I use createIdempotentProducer() for the DLQ producer. Producer idempotence handles retries of its requests. It does not prevent another DLQ record after a crash between the DLQ write and the source offset commit.

I gave my DLQ topic a single partition. This experiment does not need parallel processing of dead letters. The topic retains failed ticks for inspection. After a fix, an operator can decide whether to replay or discard them.

Idempotent consumers: the practical answer

At least once delivery can repeat processing despite bounded retries and a DLQ. A rebalance or a crash after handle() but before commitOffsets can cause a duplicate. A local retry can also repeat part of the work. The consumer must make its effects idempotent.

Use a stable message identifier, such as symbol plus sequence number. For a database effect, enforce uniqueness on (symbol, seq) and use ON CONFLICT DO NOTHING where it matches the operation. Store the deduplication decision and database effect in the same transaction.

An external notification needs its own idempotency support or delivery protocol. A separate check followed by a send can race or fail between those operations. Repeated processing must preserve the intended result.

I used the same word for two kinds of idempotence in my code. createIdempotentProducer() prevents duplicate records from producer retries. Kafka uses a producer ID and a sequence number for each partition to recognize repeated sends.

A late offset commit can still cause a consumer to process the same record again. Consumer logic must handle that separate source of duplicates.

What I'd ship

I use delivery with possible retries and idempotent consumers as the default. Crashes and rebalances can repeat work, so repeated effects must be safe. Kafka transactions suit pipelines whose output and offsets both remain in Kafka. They cannot make an external payment or order API execute exactly once.

Monitor the DLQ and alert when records arrive. An operator needs a way to inspect failures and decide whether to replay or discard each record. This recovery process keeps failed work visible after the main partition advances.

Redis Streams exposes the PEL, delivery counts, and XAUTOCLAIM. The application must combine them with retry limits and a dead letter policy. The next post describes that implementation.