At-Least-Once in Kafka: Retries and Dead Letter Queues

"At-least-once delivery" sounds like a guarantee Kafka gives you for free, but it is a consequence of one small decision you make in your consumer: when you commit the offset relative to when you process the message. Get that ordering wrong and you either lose messages or duplicate them. Get it right and you still duplicate them, just less often, and now you need a plan for that too.

Redis is refreshingly blunt about this. An entry sits in the Pending Entries List until somebody acknowledges it, so "delivered" and "done" are two states you can query separately. Kafka folds both into one number per partition, which means the entire distinction lives in where you put the commit call. I put the same failure cases through both halves of my little ticker system to see how differently they land: a crash mid-processing, a consumer rebalanced away mid-batch, a message that throws every single time. The Kafka half is the one I want to walk through, poison messages and all.

Delivery Semantics Are a Commit Decision

Kafka doesn't track "did the consumer finish processing this." It tracks one number per partition: the committed offset. Everything about delivery semantics falls out of when you move that number.

Semantic	How	Failure mode
At-most-once	Commit the offset before processing	Crash after commit, before the work finishes: message is lost
At-least-once	Commit the offset after processing	Crash after the work finishes, before the commit: message is reprocessed (duplicate)
Exactly-once	Output and offset commit in a single transaction	Neither loss nor duplication, no partial effects

At-most-once is rarely what you want; it trades correctness for the appearance of speed. Exactly-once exists in Kafka via transactions (producer idempotence plus sendOffsets), which I pulled apart separately, but it costs throughput and isn't the default. At-least-once is the pragmatic middle: it never silently drops a message, and the price is that duplicates can happen. Where do those duplicates come from? A worker processes a message, for example sends an alert notification, and then crashes or gets rebalanced away before it commits the offset. The next consumer that picks up the partition starts from the last committed offset, which is behind where the work got to, so it reprocesses that message. Nothing is corrupted. The message is just handled twice.

Commit After, Not Before

In KafkaJS this is one flag: turn off auto-commit and commit manually, after the message is handled.

await consumer.run({
  autoCommit: false,
  eachMessage: async ({ topic, partition, message }) => {
    await handle(message);
    await consumer.commitOffsets([
      { topic, partition, offset: (Number(message.offset) + 1).toString() },
    ]);
  },
});

That's the entire difference between at-most-once and at-least-once: whether commitOffsets runs before handle or after it. Commit-before-process is tempting because it feels safer, you've "acknowledged" the message right away, but it means a crash mid-processing loses that message forever since the broker will never redeliver an offset it already considers committed. Commit-after-process means a crash mid-processing just replays the message when the partition gets picked up again. Replay is recoverable in a way that loss is not, which is why at-least-once is the sane default.

Retries Without Blocking the Partition

Committing after processing solves the crash case, but it doesn't solve the case where processing itself keeps failing. Kafka guarantees ordered delivery within a partition, which means if you don't advance past a failing message, nothing behind it on that partition gets processed either. A single bad record can wedge every symbol sharing that partition.

To watch it happen I made the failures deterministic instead of waiting for a real one. My consumer sorts ticks on the market-updates topic into three kinds by their per-symbol sequence number: most are fine, some are flaky (fail the first couple of attempts, then succeed), and a few are poison (always fail):

function classify(t: Tick): Kind {
  if (t.seq % 25 === 0) return "poison"; // permanent failure -> belongs in the DLQ
  if (t.seq % 7 === 0) return "flaky"; // transient -> succeeds once retried
  return "ok";
}

Inside eachMessage, a message gets a bounded number of local attempts, with a short backoff between them, before the consumer gives up on it:

let ok = false;
let lastErr: unknown;
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
  try {
    processTick(tick, attempt);
    ok = true;
    break;
  } catch (e) {
    lastErr = e;
    await sleep(15); // backoff before retry
  }
}

That's retry-in-place: the message stays where it is in the loop, and the partition doesn't advance while it's being retried. It's the right move for transient failures, a flaky downstream call, a momentary lock contention, because most of them resolve within a few attempts. It is the wrong move for a message that will never succeed no matter how many times you retry it. That's where the bound matters: MAX_ATTEMPTS exists specifically so a poison message doesn't retry forever and take the whole partition down with it.

The Dead Letter Queue

Once retries are exhausted, the message doesn't get dropped and it doesn't get retried indefinitely either. It gets parked in a side topic, a dead letter queue (DLQ), with enough context to debug it later, and the consumer moves on.

graph TD
    M["Message arrives"] --> P["processTick(tick, attempt)"]
    P -->|success| C["commitOffsets"]
    P -->|failure, attempt < MAX| R["backoff, retry"]
    R --> P
    P -->|failure, attempt = MAX| D["dlqProducer.send to DLQ topic"]
    D --> C

The DLQ write carries the original key and value plus headers describing why it failed and where it came from:

await dlqProducer.send({
  topic: DLQ,
  messages: [
    {
      key: message.key,
      value: message.value,
      headers: {
        error: lastErr instanceof Error ? lastErr.message : String(lastErr),
        origin: `${topic}/${partition}/${message.offset}`,
      },
    },
  ],
});

The crucial detail is what happens right after: the consumer commits the original offset regardless of whether the message succeeded or got dead-lettered.

// Commit AFTER handling => at-least-once.
await consumer.commitOffsets([
  { topic, partition, offset: (Number(message.offset) + 1).toString() },
]);

That's what unblocks the partition. Dead-lettering without committing would leave you back where you started, stuck on the same offset. Committing is the acknowledgment that this message has been handled, even though "handled" here means "given up on and archived for a human, not processed successfully." I create the DLQ producer with the same createIdempotentProducer() helper I use for the main topic, so even the act of writing a dead letter doesn't introduce its own duplicate risk if that write has to retry over the network.

I gave my DLQ topic a single partition. Dead letters don't need the ordering or parallelism the main topic does, they need to sit somewhere inspectable until someone looks at them, fixes the underlying issue, and decides whether to replay them back into the main topic or discard them.

Idempotent Consumers: The Practical Answer

None of this, commit-after-process, bounded retries, a DLQ, removes the fact that at-least-once means a message can be delivered and processed more than once. A rebalance mid-batch, a consumer crash after handle() but before commitOffsets, even a retry within the same run, can all result in the same tick being processed twice. Kafka is not going to solve that for you at the consumer level. Idempotent consumers do.

The pattern is standard: use a natural key from the message (symbol plus sequence number, or an explicit message id) and make the write an upsert rather than an insert, or check a "have I seen this id" table before applying an effect. If the processing step is "insert a row into a price_ticks table," key it on (symbol, seq) and use ON CONFLICT DO NOTHING. If it's "fire a push notification," dedupe against a small table or cache of recently-sent notification ids before sending. Processing the same tick twice should be a no-op the second time, not a double-write or a double-notification.

It's worth being precise about which idempotence I mean, because I ended up using the word for two different things in my own code. createIdempotentProducer() gives you produce-side idempotence: Kafka assigns the producer a PID and a per-partition sequence number, so a record that gets retried after a network blip is deduplicated by the broker instead of written twice. That protects you from duplicate writes into market-updates caused by producer retries. It says nothing about the consumer processing the same successfully-written message twice because of a late commit. That's a separate problem, and idempotent consumer logic is the only thing that solves it.

What I'd Ship

Default to at-least-once with idempotent consumers. It's the boring choice and it survives almost everything: crashes, rebalances, retries, redeploys. Reserve Kafka's transactional exactly-once semantics for cases where a duplicate is unacceptable, money moving twice, an order placed twice, and where you can afford the throughput cost that comes with it.

And always put a DLQ behind the retry loop, with alerting on it. A DLQ that grows without anyone watching it is a queue of problems nobody is looking at. The point of dead-lettering a poison message is to keep the pipeline flowing while surfacing the failure to a human, not to make the failure invisible.

Redis Streams ships none of this. No retry loop, no DLQ topic, no commit call to move. What it hands you instead is the PEL, a delivery counter on every entry, and XAUTOCLAIM, and assembling those into the same safety net took me a while to get right. That's the post I want to write next.