Kafka Partitions and Keys: Keeping Symbols in Order

Kafka guarantees ordering within each partition. A topic can contain different order across partitions, which matters because out of order BBCA ticks would make the ticker show a wrong last price.

When I built the ticker with Kafka and Redis Streams, I first checked record routing. I needed to understand topics, partitions, offsets, and keys.

The log, sliced into partitions

A Kafka topic is a named log, market-updates in my case. A topic is split into partitions, and each partition is its own ordered, immutable, append only sequence of records. A record's position inside its partition is its offset, a number that only ever goes up.

Kafka preserves append order within each partition. It does not establish order between record 100 in partition 0 and record 50 in partition 3. More partitions allow more consumers in a group to read independently. This increases parallelism without creating a shared order across partitions.

The record's key decides its partition at produce time.

The key decides the partition

When a producer sends a record, it can attach a key. If it does, Kafka does not pick a partition at random or round robin, it computes one deterministically:

partition = (murmur2(key) & 0x7fffffff) % numberOfPartitions

The same encoded key maps to the same partition while the partition count and partitioner remain unchanged. I use the symbol as the key. All BBCA ticks then enter one partition in broker append order.

This does not establish event time order across independent producers. If the key is absent, KafkaJS can distribute the symbol across partitions, which removes this common ordering scope.

My keyed codec is one function. A Tick has a symbol, a price, and a seq (a per symbol counter I use purely to detect ordering violations later):

export function tickToKafka(tick: Tick): { key: string; value: string } {
  return { key: tick.symbol, value: JSON.stringify(tick) };
}

The key is the symbol string, nothing fancier, and that is what the partition routing depends on.

graph LR
    BBCA --> P0["Partition 0"]
    BBRI --> P0
    BMRI --> P0
    BBNI --> P0
    ANTM --> P1["Partition 1"]
    ASII --> P2["Partition 2"]
    GOTO --> P2
    ICBP --> P2
    UNVR --> P3["Partition 3"]
    TLKM --> P5["Partition 5"]

Reimplementing murmur2 in JavaScript

KafkaJS already computes the hash during production. I also implemented murmur2 in JavaScript for the browser simulations and test harness. That function predicts a partition without a broker. I can then compare its result with the partition used by KafkaJS.

const SEED = 0x9747b28c;
const M = 0x5bd1e995;
const R = 24;

export function murmur2(key) {
  const data = typeof key === "string" ? new TextEncoder().encode(key) : key;
  const length = data.length;
  let h = SEED ^ length;
  const blocks = Math.floor(length / 4);

  for (let i = 0; i < blocks; i++) {
    const i4 = i * 4;
    let k =
      (data[i4] & 0xff) |
      ((data[i4 + 1] & 0xff) << 8) |
      ((data[i4 + 2] & 0xff) << 16) |
      ((data[i4 + 3] & 0xff) << 24);
    k = Math.imul(k, M);
    k ^= k >>> R;
    k = Math.imul(k, M);
    h = Math.imul(h, M);
    h ^= k;
  }

  // These cases intentionally fall through, matching Kafka's Java client.
  const tail = blocks * 4;
  switch (length - tail) {
    case 3:
      h ^= (data[tail + 2] & 0xff) << 16;
      // Deliberate fall through to fold the next byte into the same mix.
    case 2:
      h ^= (data[tail + 1] & 0xff) << 8;
      // Deliberate fall through to fold the final byte into the same mix.
    case 1:
      h ^= data[tail] & 0xff;
      h = Math.imul(h, M);
  }

  h ^= h >>> 13;
  h = Math.imul(h, M);
  h ^= h >>> 15;
  return h | 0;
}

export function partitionForKey(key, numPartitions) {
  const positiveHash = murmur2(key) & 0x7fffffff;
  return positiveHash % numPartitions;
}

The fiddly part is Math.imul. JavaScript numbers are floats, and murmur2 depends on signed 32 bit integer multiplication overflowing the way it does in Java or C. Math.imul gives you that wraparound behavior. TextEncoder turns string keys into UTF 8 bytes before the hash runs, while a Uint8Array lets the caller provide the exact bytes directly. The tail switch and final mix follow Apache Kafka 4.3.1's Utils.murmur2(byte[]), and partitionForKey applies Kafka's positive hash mask before taking the modulus.

I checked the extracted function against an independent Python implementation of the same byte level contract. These fixed vectors exercise one, two, and three remaining bytes, plus a multibyte UTF 8 key. Hashes are signed 32 bit values, and partitions use six partitions:

Six partitions, one hot spot

I provisioned market-updates with 6 partitions, enough parallelism for 10 symbols without every symbol landing on its own. Running partitionForKey over the symbol list shows exactly where each one goes:

Four of ten symbols map to partition 0, while partition 4 receives none. This small key set produces an uneven distribution. A busy symbol would add load to its assigned partition.

More partitions can redistribute keys, but they cannot split one key automatically. A composite key or custom partitioner may change load distribution at the cost of a different ordering scope. Check the mapping against expected traffic before choosing a strategy.

Proving it against a live broker

Reading the docs is one thing, trusting them against a live broker is another. So I ran the question end to end against Kafka in Docker, in four steps.

1. Predict first: Call partitionForKey for all 10 symbols. Print the expected partition for each symbol. This calculation does not contact the broker.
2. Produce keyed: generate 60 simulated ticks, send them to market-updates, each keyed by tick.symbol.
3. Consume and check: Read all 60 records. Compare each partition with the prediction. For each symbol, check that seq strictly increases. A decreasing value indicates a sequence error.
4. Then break it on purpose: Repeat the simulation with a second topic. Send the same ticks without keys. Now each symbol's ticks scatter across multiple partitions, and no consumer can see them in order anymore.

sequenceDiagram
    participant Prod as Producer
    participant B as Broker "partition 0"
    participant Cons as Consumer
    Prod->>B: send "key=BBCA, seq=1"
    Prod->>B: send "key=BBCA, seq=2"
    Prod->>B: send "key=BBCA, seq=3"
    Note over B: same key, same partition, every time
    B->>Cons: seq=1
    B->>Cons: seq=2
    B->>Cons: seq=3
    Note over Cons: seq strictly increasing, ordering held

In the keyed run, each symbol mapped to one partition and seq never decreased. In the unkeyed run, symbols with enough ticks appeared in two or more partitions. The test reported those distributions automatically.

Back to the ticker

The ticker currently tracks 10 symbols and may need more. Partitions allow separate consumers to process different groups of symbols. A stable symbol key keeps that symbol in one partition, where Kafka preserves append order. The producer and handler must also preserve any required application sequence.

It also sets up something I come back to later. notifications has the same 6 partitions and the same symbol key, rather than userId. Deliberately so: BBCA's ticks and BBCA's alerts hash to the same partition number, which matters once you start joining the two streams.

The routing rule in one view

Kafka preserves order within each partition. The expression (murmur2(key) & 0x7fffffff) % numPartitions maps an encoded key to a partition. My independent function uses Math.imul for 32 bit multiplication. It lets me compare predicted routing with broker results.

A small set of keys can produce an uneven distribution. My ten symbols across six partitions put four symbols in one partition and none in another. The test made this imbalance visible before deployment.

Redis Streams has no partitions to hand out. Ordering is held together differently, which I cover next in XADD, entry IDs, and MAXLEN, along with where that story stops resembling Kafka's.