Co-Partitioning: Same Key, Same Partition, Both Topics

"I want partition 3 of notifications consumed on the same pod as partition 3 of market-updates." That sentence describes what my little ticker system needs, and it hides two separate problems that I kept mashing together for weeks.

Problem one: does partition 3 of one topic hold the same symbols as partition 3 of the other? Problem two: does a single consumer process end up owning both of them? Only the first has a tidy answer, and that answer rests on nothing more exotic than a hash function being deterministic.

One Topic, One Key, One Partition

I covered the mechanics in the ordering post, so only the part that matters here: every record keyed by symbol gets routed to a partition by (murmur2(key) & 0x7fffffff) % numPartitions. Same key, same partition, every time, as long as the partition count does not change. market-updates uses this to keep every BBCA tick strictly ordered behind the BBCA tick before it.

That is a statement about one topic. Co-partitioning is what happens when you apply the identical recipe to two topics at once.

Defining Co-Partitioning

Two topics are co-partitioned when they share:

• the same partition count, and
• the same key, routed by the same partitioner.

In the ticker I built twice, market-updates and notifications are both provisioned with 6 partitions, and both are keyed by symbol. Feed "BBCA" and 6 into partitionForKey and the topic never enters the conversation. The function has no idea which topic you are about to produce to, and no reason to care. It hashes the key, mods by the partition count, returns a number. Call it for market-updates and you get a partition. Call it again for notifications with the same key and the same partition count and you get the identical number.

export function partitionForKey(key: string, numPartitions: number): number {
  return (murmur2(key) & 0x7fffffff) % numPartitions;
}

That is the mechanism. Kafka has no co-partitioning switch to flip: the property falls out of provisioning two topics with matching partition counts and pointing the same deterministic function at the same key for both.

Where Every Symbol Lands, in Both Topics

Because the inputs are identical, the output table is identical too. Whatever partition partitionForKey assigns BBCA in market-updates, it assigns BBCA the same number in notifications:

graph LR
    subgraph MU["market-updates (6 partitions)"]
        MP0["Partition 0"]
        MP1["Partition 1"]
        MP2["Partition 2"]
        MP3["Partition 3"]
        MP5["Partition 5"]
    end
    subgraph NT["notifications (6 partitions)"]
        NP0["Partition 0"]
        NP1["Partition 1"]
        NP2["Partition 2"]
        NP3["Partition 3"]
        NP5["Partition 5"]
    end
    BBCA --> MP0
    BBCA --> NP0
    ANTM --> MP1
    ANTM --> NP1
    ASII --> MP2
    ASII --> NP2
    UNVR --> MP3
    UNVR --> NP3
    TLKM --> MP5
    TLKM --> NP5

BBCA's price ticks and BBCA's alert notifications both land on partition 0, in their respective topics, every single time. ANTM's land on partition 1 in both. Nothing coordinates that: no lookup table, no config linking the two topics together. It falls out of running one pure function against the same key twice.

What Breaks It

Every piece of that definition is load bearing, and dropping any one of them breaks co-partitioning.

Different partition counts. The modulus is part of the formula, so changing it changes the answer for every key, including the ones already in use. Provision notifications with 8 partitions instead of 6, and BBCA's alerts hash to a different number than BBCA's ticks:

graph TD
    K["key: BBCA"] --> M6["mod 6, market-updates: partition 0"]
    K --> M8["mod 8, notifications: partition 2"]
    M6 -.->|"misaligned"| M8

I checked instead of assuming: partitionForKey("BBCA", 6) returns 0, and partitionForKey("BBCA", 8) returns 2. Same key, different modulus, different partition, so co-partitioning is already gone before anything else comes into play.

Different keys. Key notifications by userId instead of symbol, and a user's alert routes on who set it rather than on which stock it is about. Two alerts on BBCA from two different users scatter to wherever their respective user IDs happen to hash to, with no relationship at all to where BBCA's price ticks live.

Different partitioners. Both topics have to route through the same hashing logic. If one producer falls back to round-robin because a message went out unkeyed, or a custom partitioner is wired up on one side only, the deterministic mapping that co-partitioning depends on is gone for those records, even though the topics themselves still have matching partition counts.

Co-partitioning is a precondition. Easy to satisfy, and just as easy to break quietly by changing one number in a topic config months later.

The Redis Analog: Hash Tags and CRC16 Slots

Redis Cluster has no partitions and no numPartitions you set per key. Every key hashes to one of 16384 fixed slots using CRC16 (the XMODEM variant, polynomial 0x1021), and the slots get distributed across nodes:

export function crc16(input: string): number {
  const bytes = new TextEncoder().encode(input);
  let crc = 0;
  for (const byte of bytes) {
    crc ^= byte << 8;
    for (let i = 0; i < 8; i++) {
      crc = (crc & 0x8000) ? ((crc << 1) ^ 0x1021) & 0xffff : (crc << 1) & 0xffff;
    }
  }
  return crc & 0xffff;
}

export function keySlot(key: string): number {
  const open = key.indexOf("{");
  if (open !== -1) {
    const close = key.indexOf("}", open + 1);
    if (close > open + 1) {
      key = key.slice(open + 1, close); // only the tag is hashed
    }
  }
  return crc16(key) % 16384;
}

By default, two related keys like market:BBCA and notif:BBCA hash independently, because CRC16 sees two different strings. No shared key exists the way both Kafka topics share symbol as a field name, so nothing pushes them onto the same slot. The hash tag is the fix: wrap the shared part in braces, market:{BBCA} and notif:{BBCA}, and keySlot hashes only the substring inside the braces. Both keys collapse to the same input string for the hash function, so they land on the same slot, and therefore the same node.

I ran that over all 10 symbols my ticker trades and counted co-location. With the hash tag: 10 out of 10 symbol pairs on the same slot, every time, by construction. Without it: 0 out of 10, because plain concatenation gives CRC16 no reason to agree.

graph LR
    A["market:{BBCA}"] --> S1["slot"]
    B["notif:{BBCA}"] --> S1
    C["market:BBCA"] --> S2["slot"]
    D["notif:BBCA"] --> S3["different slot"]

Same-slot keys can be touched together in one MULTI/EXEC or a single Lua script, which is the Redis way of doing an atomic per-symbol operation across two logically separate streams. Kafka gets there with a matching partition number. Redis Cluster gets there with a matching hash slot. Different mechanism, same goal: force everything related to one key onto one machine.

Layout Is Not Placement

Co-partitioning guarantees that partition 3 of market-updates and partition 3 of notifications contain the same symbols. It says nothing about which consumer, which pod, reads partition 3 of either topic. Two different questions, decided by two different mechanisms, and Kafka's consumer group protocol does not automatically hand matching partition numbers to the same member. KafkaJS's built-in assigner, which only does round-robin, actively breaks that alignment for some pod counts even when the topics are perfectly co-partitioned underneath.

Knowing that a symbol always maps to the same partition number in both topics is necessary but not sufficient. The runtime still has to be told, explicitly, to put matching partition numbers on the same process.

The Short Version

• Co-partitioning means two topics share the same partition count and the same key, routed through the same deterministic partitioner, so a given key lands on the same partition number in both.
• The mechanism is nothing more than calling partitionForKey(key, numPartitions) with identical arguments against two topics. No linking config, no coordination step.
• Changing the partition count on either topic breaks it for every key, old and new alike, because the modulus is part of the formula. A real check: partitionForKey("BBCA", 6) is 0, partitionForKey("BBCA", 8) is 2.
• Redis Cluster's equivalent is the hash tag: only the substring inside {...} gets hashed by CRC16, so market:{BBCA} and notif:{BBCA} land on the same one of 16384 slots. Across my ticker's 10 symbols, tagged pairs co-locate 10 out of 10 times; plain pairs, 0 out of 10.
• Co-partitioning (or a matching hash slot) is a precondition for a local join, not the join itself. It says where the data lives, not where the consumer runs.

So how do you get partition 3 of both topics onto the same pod? Not by asking KafkaJS nicely. I ended up writing my own partition assigner, and that is the next thing I want to walk through.