Exactly-Once in Kafka: Idempotent Producers and Transactions

"Exactly-once" in Kafka has a specific transaction boundary. Kafka can commit output records and consumer offsets as one atomic operation. This guarantee does not include an external database write, payment call, or email.

I investigated this while adding price alerts to my ticker. A tick crosses a threshold, the consumer produces a notification record, and its source offset advances. I built a transactional version to test whether the record and offset could commit together.

What exactly once means

Delivery semantics come down to when you commit relative to when you do the work:

With commits after processing, a crash can cause repeated handling. Idempotent effects are thus necessary. Kafka exactly once semantics (EOS) apply to a narrower pipeline: consume Kafka records, produce Kafka records, and commit source offsets in one transaction.

If that transaction aborts, committed readers see neither its output nor its offset advance. The guarantee depends on correct producer identity, transaction handling, and consumer isolation.

EOS covers Kafka to Kafka. If your "process" step calls a payment API, writes to Postgres, or sends an email, none of that is inside the transaction. The transaction can abort after your side effect has already fired, or commit after it failed. Kafka has no way to roll back an HTTP call. EOS fits pipelines that consume from Kafka and produce to Kafka, like turning market-updates ticks into notifications when a price alert crosses. It cannot make a write to two external systems atomic.

The idempotent producer: PID and sequence numbers

An idempotent producer handles repeated produce requests. A request may succeed at the broker, but its acknowledgment may arrive after the client timeout. The client then retries the same batch. Producer idempotence prevents that retry from appending the batch twice.

export function createIdempotentProducer(): Producer {
  return kafka.producer({
    idempotent: true, // forces acks=all and dedup on retry
    maxInFlightRequests: 5, // max allowed while keeping idempotence/ordering
    retry: { retries: 10 },
  });
}

With idempotent: true, KafkaJS requires acks=all and the producer gets a Producer ID (PID). Each batch also carries a sequence number for its partition. The broker uses this identity and sequence information to recognize repeated sends. It can reject a retry that would otherwise append the same batch again.

Producer idempotence requires compatible acknowledgment, retry, and request concurrency settings. Do not assume that enabling it automatically sets maxInFlightRequests to 5 in KafkaJS. The KafkaJS transaction guide specifies maxInFlightRequests: 1, idempotent: true, and a transactionalId for its EOS configuration.

Idempotence covers retries of one producer's requests. Application code can still issue duplicate logical events, and the guarantee spans neither multiple partitions nor multiple topics as one unit. Consumer processing can also repeat after a crash before the offset commit. Idempotent produce and exactly once consume produce are different guarantees, which is why transactions exist as a separate, opt in layer on top.

Transactions: read, process, write, atomically

A transactional producer needs a stable transactionalId for its logical input assignment. This lets Kafka reject an older producer instance after a replacement starts. Active producers for different assignments must not share that identity.

const producer = kafka.producer({
  transactionalId: "s04-eos",
  idempotent: true,
  maxInFlightRequests: 1,
});

const txn = await producer.transaction();
try {
  await txn.send({ topic: "notifications", messages });
  // The defining feature of EOS: move the input group's offsets *inside*
  // the same transaction as the output. Crash here and neither side commits.
  await txn.sendOffsets({
    consumerGroupId: "s04-eos-consumer",
    topics: [{ topic: "market-updates", partitions: [/* ... */] }],
  });
  await txn.commit();
} catch (e) {
  await txn.abort();
  throw e;
}

The sendOffsets call includes the new consumer offsets in the output transaction. Kafka stores group offsets in __consumer_offsets. The transaction commits its output and offsets together, or commits neither. This removes the gap between successful Kafka output and the corresponding source offset advance.

Commit vs abort: watching a batch disappear

To test transaction visibility, I created two batches of five notifications each. Each batch covers price threshold crossings for five different symbols:

Transaction A sends its five notifications, calls sendOffsets to advance the source consumer group's offsets, and commits.

Transaction B sends five notifications, then aborts.

sequenceDiagram
    participant P as Producer
    participant TC as Coordinator
    participant N as notifications topic
    participant C as read_committed Consumer
    P->>TC: begin transaction
    P->>N: send 5 notifications
    P->>TC: sendOffsets for consumer group
    P->>TC: commit
    TC->>N: write commit marker
    C->>N: poll
    N-->>C: 5 notifications visible

I then read notifications with read_committed and read_uncommitted consumers. KafkaJS defaults to readUncommitted: false, but defaults vary between clients. Set the isolation mode explicitly when another client is used. The results were:

• read_committed: 5 notifications. Only transaction A's batch.
• read_uncommitted: 10 notifications. Both batches, including the aborted one.

graph LR
    TB["Transaction B: 5 notifications"] --> Abort
    Abort --> RC["read_committed: invisible"]
    Abort --> RU["read_uncommitted: visible"]

Kafka writes aborted records to the log, followed by transaction markers. A consumer with read_committed excludes aborted records, while a consumer with read_uncommitted can return them. The 5 versus 10 result shows this visibility difference. It does not mean Kafka removed the aborted bytes from storage.

Transactions add coordinator requests. The producer above also uses maxInFlightRequests: 1, which limits concurrent requests. Use this design when Kafka output and source offsets must commit together. External effects still need separate idempotency or transaction coordination.

Redis command atomicity has different limits

Redis can execute an output XADD and input XACK in one Lua script or MULTI/EXEC transaction. In Redis Cluster, the relevant keys must share a slot. Hash tags such as market:{BBCA} can provide that placement.

This is atomic Redis command execution, with different error and recovery rules from Kafka transactions. The application must handle deduplication, processing, and durability explicitly. Redis transactions do not roll back commands after execution errors. See the Redis transaction documentation.

With separate processing and acknowledgment, a crash before XACK can cause repeated work. Another consumer may reclaim the entry with XAUTOCLAIM. The handler must tolerate that repetition. Both Redis and Kafka need additional coordination when processing changes an external system.

Where this leaves you

Enable producer idempotence with the settings required by the client. Use Kafka transactions when output records and source offsets need one commit boundary. Measure the coordinator and concurrency costs for the workload. For database writes and external API calls, design idempotency or explicit coordination at that system boundary.

The next post compares Redis Pub/Sub with Streams, including what happens when a subscriber disconnects.