Kafka Partitions, Keys, and Ordering: Keeping Every Symbol in Line

Because Kafka only guarantees ordering within a partition, not within a topic, this becomes an interesting problem for me: if BBCA's ticks arrive out of order, the ticker shows a wrong last price.

Which is why, when I set out to build the same ticker on Kafka and on Redis Streams, routing was the first thing I had to get right. Topics, partitions, offsets, and the key that decides where a record lands.

The Log, Sliced Into Partitions

A Kafka topic is a named log, market-updates in my case. A topic is split into partitions, and each partition is its own ordered, immutable, append-only sequence of records. A record's position inside its partition is its offset, a number that only ever goes up.

The part that trips people up: ordering is a per-partition property, not a per-topic one. Kafka never promises that record 100 in partition 0 happened before record 50 in partition 3. It promises that within partition 0, records are read back in exactly the order they were written. That is the entire ordering contract, and it is also why partitions are the unit of parallelism: more partitions means more consumers can read independently without stepping on each other.

So the question becomes: which partition does a given record land on? That is decided at produce time, by the record's key.

The Key Decides the Partition

When a producer sends a record, it can attach a key. If it does, Kafka does not pick a partition at random or round-robin, it computes one deterministically:

partition = (murmur2(key) & 0x7fffffff) % numberOfPartitions

Same key, same hash, same partition, every time (as long as the partition count does not change). That determinism is what I rely on: key every market tick by its symbol, and every BBCA tick, no matter which producer instance sent it or when, hashes to the same partition and lines up behind the BBCA tick before it. Drop the key, and KafkaJS falls back to round-robining records across partitions for throughput, which scatters a single symbol across the topic and throws the ordering guarantee away.

My keyed codec is one function. A Tick has a symbol, a price, and a seq (a per-symbol counter I use purely to detect ordering violations later):

export function tickToKafka(tick: Tick): { key: string; value: string } {
  return { key: tick.symbol, value: JSON.stringify(tick) };
}

The key is the symbol string, nothing fancier, and that is what the partition routing depends on.

graph LR
    BBCA --> P0["Partition 0"]
    BBRI --> P0
    BMRI --> P0
    BBNI --> P0
    ANTM --> P1["Partition 1"]
    ASII --> P2["Partition 2"]
    GOTO --> P2
    ICBP --> P2
    UNVR --> P3["Partition 3"]
    TLKM --> P5["Partition 5"]

Reimplementing murmur2 in TypeScript

KafkaJS computes that hash internally when you produce, so I did not strictly need my own copy. I wrote one anyway, the same murmur2 variant, in TypeScript. Two reasons: it lets my browser simulations and my test harness predict a partition without ever touching a broker, and it lets me assert that the prediction matches what the real broker did.

const SEED = 0x9747b28c;
const M = 0x5bd1e995;
const R = 24;

export function murmur2(key: string | Uint8Array): number {
  const data = typeof key === "string" ? new TextEncoder().encode(key) : key;
  const length = data.length;
  let h = SEED ^ length;
  const blocks = Math.floor(length / 4);

  for (let i = 0; i < blocks; i++) {
    const i4 = i * 4;
    let k =
      (data[i4] & 0xff) |
      ((data[i4 + 1] & 0xff) << 8) |
      ((data[i4 + 2] & 0xff) << 16) |
      ((data[i4 + 3] & 0xff) << 24);
    k = Math.imul(k, M);
    k ^= k >>> R;
    k = Math.imul(k, M);
    h = Math.imul(h, M);
    h ^= k;
  }
  // ...tail bytes handled the same way the Java client does them, then a final mix
  h ^= h >>> 13;
  h = Math.imul(h, M);
  h ^= h >>> 15;
  return h | 0;
}

export function partitionForKey(key: string, numPartitions: number): number {
  return (murmur2(key) & 0x7fffffff) % numPartitions;
}

The fiddly part is Math.imul. JavaScript numbers are floats, and murmur2 depends on 32 bit integer multiplication overflowing the way it does in Java or C. Math.imul gives you that exact wraparound behavior, so the hash comes out bit for bit identical to what the Java client (and therefore the real broker's partitioner) computes. Get that wrong and your predicted partition silently disagrees with reality.

Six Partitions, One Hot Spot

I provisioned market-updates with 6 partitions, enough parallelism for 10 symbols without every symbol landing on its own. Running partitionForKey over the symbol list shows exactly where each one goes:

Symbol	Partition
BBCA	0
BBRI	0
BMRI	0
BBNI	0
ANTM	1
ASII	2
GOTO	2
ICBP	2
UNVR	3
TLKM	5

Four of ten symbols hash to partition 0, and partition 4 gets nothing at all. That is what hashing a small key space looks like. With real trading volume, BBCA being the most liquid IDX symbol would make partition 0 a hot partition, doing several times the work of partition 4 while every partition still has the same single consumer capacity. The fixes are the obvious ones: more partitions to spread the hash space thinner, a composite key if you can afford to split a single symbol's ordering guarantee, or a custom partitioner that actively balances load instead of trusting the hash. Kafka does not solve this for you. It gives you a deterministic, inspectable function so you can see the imbalance coming.

Proving It Against a Live Broker

Reading the docs is one thing, trusting them against a live broker is another. So I ran the question end to end against Kafka in Docker, in four steps.

1. Predict first. Before producing anything, call partitionForKey for all 10 symbols and print where each one should land, purely from the pure function, no broker involved.
2. Produce keyed. Generate 60 simulated ticks, send them to market-updates, each keyed by tick.symbol.
3. Consume and check. Read all 60 back and check two things per message: does the partition it landed on match the prediction, and for each symbol, is seq still strictly increasing? A seq going backwards would mean ordering broke.
4. Then break it on purpose. Rerun the same simulation against a second topic, sending the identical ticks with no key at all. Now each symbol's ticks scatter across multiple partitions, and no consumer can see them in order anymore.

sequenceDiagram
    participant Prod as Producer
    participant B as Broker "partition 0"
    participant Cons as Consumer
    Prod->>B: send "key=BBCA, seq=1"
    Prod->>B: send "key=BBCA, seq=2"
    Prod->>B: send "key=BBCA, seq=3"
    Note over B: same key, same partition, every time
    B->>Cons: seq=1
    B->>Cons: seq=2
    B->>Cons: seq=3
    Note over Cons: seq strictly increasing, ordering held

The output made the tradeoff concrete in a way the docs never did. In the keyed run, every symbol mapped to exactly one partition and seq never regressed. In the unkeyed run, symbols that produced enough ticks showed up spread across two or more partitions, and my harness flagged it rather than leaving me to eyeball a log.

Back to the Ticker

The whole reason to bother with any of this is that market-updates needs to scale (10 symbols today, more tomorrow, and a consumer group that can add workers) while a single symbol's price history must never reorder. Partitioning gives you the first property. Keying by symbol gives you the second, because it pins one symbol's entire history to one partition, and one partition is where Kafka's ordering guarantee lives. The choice of key gets you both properties at once: parallelism across symbols, strict order within each one.

It also sets up something I come back to later. notifications has the same 6 partitions and the same symbol key, rather than userId. Deliberately so: BBCA's ticks and BBCA's alerts hash to the same partition number, which matters once you start joining the two streams.

The Short Version

Ordering in Kafka lives inside a partition and nowhere else. The key is what pins a symbol to one of them, through (murmur2(key) & 0x7fffffff) % numPartitions, and having that function outside the client (with Math.imul doing the 32 bit overflow correctly) is what let me predict routing and then verify the prediction against a real broker. Key by the field whose order you care about and "many producers, many partitions" turns into "still strictly ordered per symbol."

The wrinkle worth carrying around is that hashing a small key space is lopsided by nature. Ten symbols over six partitions gave me one partition holding four of them and another holding none, and that is the kind of thing better discovered on a laptop than in production.

Redis Streams has no partitions to hand out at all, which raises the obvious question: what holds ordering together over there? Next time I get into XADD, entry IDs, and MAXLEN, and where that story stops resembling Kafka's.