Exactly-Once in Kafka: Idempotent Producers and Transactions

"Exactly-once" is one of those phrases that sounds like a promise the universe can't keep, and mostly it can't. What Kafka gives you is narrower and more precise than the marketing name suggests: a way to make producing records and committing consumer offsets happen as a single atomic unit. That is a useful guarantee. It is also easy to misapply if you read it as "this side effect will happen exactly once," no qualifiers.

I ran into the question while wiring price alerts into my little ticker system: a tick crosses a user's threshold, a notification goes out, the offset moves past the tick. Duplicating that notification is the sort of bug a user notices immediately, and a phantom one is worse. So I built the transactional version to see what it covers and, more usefully, what it does not.

What exactly-once means

Delivery semantics come down to when you commit relative to when you do the work:

Semantic	How	Failure mode
At-most-once	Commit before processing	Crash after commit, before work: message lost
At-least-once	Commit after processing	Crash after work, before commit: message reprocessed
Exactly-once	Output and offset commit in one transaction	Neither happens, or both do: no partial effects

At-least-once, which I took apart in the post on retries and DLQs, is the sane default: it never loses data, and you pair it with idempotent handling on your side (upserts, dedupe keys) to deal with the occasional duplicate. Exactly-once semantics (EOS) is what Kafka calls the case where the framework itself removes the duplicate window, for a specific and fairly narrow kind of pipeline: read from a Kafka topic, transform, write to a Kafka topic, and advance the consumer offset. All of that happens inside one transaction, so a crash midway aborts the whole thing rather than leaving you with output but no offset advance (duplicate downstream) or an offset advance with no output (silent data loss).

The part worth being precise about: EOS covers Kafka-to-Kafka. If your "process" step calls a payment API, writes to Postgres, or sends an email, none of that is inside the transaction. The transaction can abort after your side effect has already fired, or commit after it failed. Kafka has no way to roll back an HTTP call. So EOS is for pipelines that consume from Kafka and produce to Kafka, like turning market-updates ticks into notifications when a price alert crosses. It is not a general answer to "how do I make my write-to-two-systems problem atomic."

The idempotent producer: PID and sequence numbers

A smaller, cheaper guarantee comes before transactions: the idempotent producer. It fixes one specific failure mode: a produce request times out, the client retries, and the broker ends up with the same batch written twice because the first attempt succeeded, it just never got acknowledged in time.

export function createIdempotentProducer(): Producer {
  return kafka.producer({
    idempotent: true, // forces acks=all and dedup on retry
    maxInFlightRequests: 5, // max allowed while keeping idempotence/ordering
    retry: { retries: 10 },
  });
}

Turning on idempotent: true makes the client request a Producer ID (PID) from the broker when it initializes. From then on, every batch it sends to a given partition carries that PID plus a monotonically increasing sequence number. The broker tracks the last sequence number it accepted per PID per partition. If a retried batch shows up with a sequence number it has already seen, the broker discards it instead of appending it again. That is the whole mechanism: no distributed locks, no coordination, just a sequence number the broker checks.

Two side effects of turning this on: it forces acks=all (the leader must have the write acknowledged by its in-sync replicas before it counts), and maxInFlightRequests is capped at 5, the largest value the broker can use while still detecting out-of-order or duplicate sequence numbers across pipelined requests.

What idempotence does not do: it does not dedupe two calls from your application code that both decided to produce the same logical event. It does not span multiple partitions or topics as one unit. And it says nothing about the consumer side, a consumer can still crash after processing a message but before committing its offset, and get it redelivered. Idempotent produce and exactly-once consume-produce are different guarantees, which is why transactions exist as a separate, opt-in layer on top.

Transactions: read, process, write, atomically

A transactional producer needs a stable transactionalId so the broker can recognize the same producer instance across restarts and fence off any zombie instance still holding an old one. Idempotence is implied.

const producer = kafka.producer({
  transactionalId: "s04-eos",
  idempotent: true,
  maxInFlightRequests: 1,
});

const txn = await producer.transaction();
try {
  await txn.send({ topic: "notifications", messages });
  // The defining feature of EOS: move the input group's offsets *inside*
  // the same transaction as the output. Crash here and neither side commits.
  await txn.sendOffsets({
    consumerGroupId: "s04-eos-consumer",
    topics: [{ topic: "market-updates", partitions: [/* ... */] }],
  });
  await txn.commit();
} catch (e) {
  await txn.abort();
  throw e;
}

sendOffsets is the piece that makes this exactly-once rather than just "atomic multi-topic produce." It writes the consumer group's new offsets to Kafka's internal __consumer_offsets topic as part of the same transaction as the notification records. Commit both or neither. No window exists where you have produced the notifications but not yet advanced past the source messages that generated them, and no window where you have advanced the offset but the notifications never made it out.

Commit vs abort: watching a batch disappear

Reading about a guarantee and watching it hold are different things, so I built two batches of five notifications each, from crossing a price threshold on five different symbols:

Transaction A sends its five notifications, calls sendOffsets to advance the source consumer group's offsets, and commits.

Transaction B sends its five notifications and then explicitly aborts. Nothing else, just abort.

sequenceDiagram
    participant P as Producer
    participant TC as Coordinator
    participant N as notifications topic
    participant C as read_committed Consumer
    P->>TC: begin transaction
    P->>N: send 5 notifications
    P->>TC: sendOffsets for consumer group
    P->>TC: commit
    TC->>N: write commit marker
    C->>N: poll
    N-->>C: 5 notifications visible

Then I read the same notifications topic twice, once as a read_committed consumer (Kafka's default isolation level) and once as read_uncommitted. The results:

• read_committed: 5 notifications. Only transaction A's batch.
• read_uncommitted: 10 notifications. Both batches, including the aborted one.

graph LR
    TB["Transaction B: 5 notifications"] --> Abort
    Abort --> RC["read_committed: invisible"]
    Abort --> RU["read_uncommitted: visible"]

The gap between those two counts is the guarantee in practice. The aborted batch was physically written to the partition, Kafka doesn't magically un-write bytes, but it is followed by an abort marker, and a read_committed consumer uses that marker to skip past every record from that transaction as if it never happened. A read_uncommitted consumer ignores the markers and sees raw log order, ghosts included. Nobody downstream running the default isolation level ever observes a half-finished transaction or a rolled-back one. That is what "exactly-once" is buying you here: not the absence of writes, but the absence of visible partial writes.

It costs something, naturally. Transactional writes add coordinator round trips and, in the producer config above, maxInFlightRequests: 1 to keep ordering simple to reason about. EOS is the right default for financial or derived-state pipelines where a duplicate or a phantom notification is a real problem. It is overkill for a firehose of price ticks where the consumer is already idempotent.

Redis Streams has no equivalent

The two systems diverge here, not merely in naming things differently. Redis has no built-in mechanism for atomically committing "output written" and "input acknowledged" as one unit spanning process-and-output the way Kafka's transactions do. You can get atomicity for operations touching keys in the same hash slot with MULTI/EXEC or a Lua script, and the hash-tag keys I used on the Redis side of the ticker (market:{BBCA}) are exactly what makes that possible for related keys. But that is atomicity for a batch of Redis commands, not a read-process-write guarantee that spans a consumer group's PEL and a downstream stream write.

In practice, achieving effectively-once behavior on Redis Streams means leaning harder on idempotent consumers: process the entry, XACK it, and make the processing itself safe to repeat if XACK never lands and another consumer reclaims the entry with XAUTOCLAIM later. Same discipline as at-least-once, just without a broker-level trick to shrink the duplicate window. It is a reasonable trade: Redis stays simpler operationally, and idempotent processing is a technique you need anyway, since even Kafka's EOS can't cover you the moment your pipeline touches anything outside Kafka.

Where this leaves you

Idempotent producers are close to free, turn them on for anything writing to Kafka. Transactions are a real design decision: they buy you atomicity between what you produce and what you've consumed, strictly within Kafka, at the cost of coordinator overhead and reduced throughput. Reach for them when duplicates or phantom output are unacceptable, like a notification pipeline where a duplicated alert or a resurrected aborted one erodes trust. For everything touching the outside world, whether that's a database write or an external API call, exactly-once still means at-least-once plus an idempotency key, because no broker can make an HTTP request transactional.

Enough durability for one post. Redis has a second fan-out primitive that forgets a message the instant it delivers it, and I keep going back and forth on whether that is a flaw or the entire point. Pub/Sub versus Streams is where I am headed next.