Co-Partitioning: Same Key, Same Partition, Both Topics

"I want partition 3 of notifications consumed on the same pod as partition 3 of market-updates." That sentence describes what my little ticker system needs, and it hides two separate problems that I kept mashing together for weeks.

The first requirement concerns data: matching partition numbers must contain the same symbols. The second concerns assignment: one consumer process must own both matching partitions. A deterministic hash function can satisfy the first requirement.

One topic, one key, one partition

The ordering post explains the routing function: (murmur2(key) & 0x7fffffff) % numPartitions. With unchanged key bytes, partitioner, and partition count, the same key selects the same partition. market-updates uses the symbol as its key to keep BBCA ticks in one ordered partition.

That is a statement about one topic. Co partitioning is what happens when you apply the identical recipe to two topics at once.

Defining co partitioning

Two topics are co partitioned when they share:

• the same partition count, and
• the same key, routed by the same partitioner.

In the ticker I built twice, both market-updates and notifications have 6 partitions and use symbol as the key. The partitionForKey function receives the key and partition count. It does not receive the topic name. For the same encoded key and partition count, it returns the same number for both topics.

export function partitionForKey(key: string, numPartitions: number): number {
  return (murmur2(key) & 0x7fffffff) % numPartitions;
}

Kafka needs no separate setting for this property. Both topics must use matching partition counts, key encoding, and deterministic partition logic.

Where every symbol lands, in both topics

Because the inputs are identical, the output table is identical too. Whatever partition partitionForKey assigns BBCA in market-updates, it assigns BBCA the same number in notifications:

graph LR
    subgraph MU["market-updates (6 partitions)"]
        MP0["Partition 0"]
        MP1["Partition 1"]
        MP2["Partition 2"]
        MP3["Partition 3"]
        MP5["Partition 5"]
    end
    subgraph NT["notifications (6 partitions)"]
        NP0["Partition 0"]
        NP1["Partition 1"]
        NP2["Partition 2"]
        NP3["Partition 3"]
        NP5["Partition 5"]
    end
    BBCA --> MP0
    BBCA --> NP0
    ANTM --> MP1
    ANTM --> NP1
    ASII --> MP2
    ASII --> NP2
    UNVR --> MP3
    UNVR --> NP3
    TLKM --> MP5
    TLKM --> NP5

BBCA maps to partition 0 in both topics, and ANTM maps to partition 1 in both. The matching inputs to the hash function produce these results. No separate lookup table or topic link is necessary.

What breaks it

Every piece of that definition is essential, and dropping any one of them breaks co partitioning.

Different partition counts: the modulus is part of the formula. A count change can change the partition for a key. For example, BBCA maps differently with 8 partitions than with 6:

graph TD
    K["key: BBCA"] --> M6["mod 6, market-updates: partition 0"]
    K --> M8["mod 8, notifications: partition 2"]
    M6 -.->|"misaligned"| M8

I checked instead of assuming: partitionForKey("BBCA", 6) returns 0, and partitionForKey("BBCA", 8) returns 2. Same key, different modulus, different partition, so co partitioning is already gone before anything else comes into play.

Different keys: if notifications uses userId instead of symbol, the user determines the partition. Alerts for BBCA from different users can then map to different partitions from the BBCA ticks.

Different partitioners: both producers must use the same hash logic and key encoding. An unkeyed message or a different custom partitioner removes the matching partition guarantee. Equal partition counts alone are insufficient.

Co partitioning requires matching routing rules. A later change to a partition count can break that alignment for new records.

The Redis analog: hash tags and CRC16 slots

Redis Cluster has no partitions and no numPartitions you set per key. Every key hashes to one of 16384 fixed slots using CRC16 (the XMODEM variant, polynomial 0x1021), and the slots get distributed across nodes:

export function crc16(input: string): number {
  const bytes = new TextEncoder().encode(input);
  let crc = 0;
  for (const byte of bytes) {
    crc ^= byte << 8;
    for (let i = 0; i < 8; i++) {
      crc = (crc & 0x8000) ? ((crc << 1) ^ 0x1021) & 0xffff : (crc << 1) & 0xffff;
    }
  }
  return crc & 0xffff;
}

export function keySlot(key: string): number {
  const open = key.indexOf("{");
  if (open !== -1) {
    const close = key.indexOf("}", open + 1);
    if (close > open + 1) {
      key = key.slice(open + 1, close); // only the tag is hashed
    }
  }
  return crc16(key) % 16384;
}

The keys market:BBCA and notif:BBCA normally hash independently because their complete strings differ. The keySlot helper hashes only the text inside {...} when it finds a valid hash tag. With market:{BBCA} and notif:{BBCA}, both hash inputs are "BBCA". Thus, both keys use the same slot and node.

I ran that over all 10 symbols my ticker trades and counted co location. With the hash tag: 10 out of 10 symbol pairs on the same slot, every time, by construction. Without it: 0 out of 10, because plain concatenation gives CRC16 no reason to agree.

graph LR
    A["market:{BBCA}"] --> S1["slot"]
    B["notif:{BBCA}"] --> S1
    C["market:BBCA"] --> S2["slot"]
    D["notif:BBCA"] --> S3["different slot"]

Keys in the same slot can participate in one MULTI/EXEC operation or a Lua script. This permits atomic Redis operations on related streams. Kafka partition numbers identify data groups, but matching numbers alone do not place partitions on the same broker or consumer.

Layout is not placement

Matching partition counts and key mapping place the same symbols in matching partitions of market-updates and notifications. Consumer assignment is separate. KafkaJS round robin assignment can place matching partitions on different members for some group sizes.

Knowing that a symbol always maps to the same partition number in both topics is necessary but not sufficient. The runtime still has to be told, explicitly, to put matching partition numbers on the same process.

What co partitioning guarantees

• Both topics must use the same partition count, key encoding, and deterministic partitioner. A given key then maps to the same partition number.
• partitionForKey(key, numPartitions) receives identical inputs for both topics. No separate topic link is necessary.
• A partition count change can alter the mapping for future records. Existing records remain in their original partitions. For BBCA, partitionForKey("BBCA", 6) is 0 and partitionForKey("BBCA", 8) is 2.
• Redis Cluster uses hash tags to place related keys in the same slot. The market:{BBCA} and notif:{BBCA} keys use one of 16384 slots. In my 10 symbol test, tagged pairs matched 10 out of 10 times and plain pairs matched 0 out of 10.
• A local Kafka join also needs matching consumer assignments. Partition mapping alone does not establish process ownership.

I wrote a custom KafkaJS partition assigner to put matching partitions on the same process. The next post explains that implementation.