Kafka ISR vs Redis Cluster: Replication and Failover

The ticker initially used one Kafka broker and one Redis instance. That setup supported tests of consumer groups and the PEL, but not node failures. I added a cluster with three Kafka brokers and a Redis Cluster with three nodes. Both ran in the same Docker Compose file.

When one node is not enough

A second compose profile brings up six containers: kvr-kc1/2/3 for Kafka (KRaft mode, no ZooKeeper, ports 19092/19094/19096), and kvr-rc1/2/3 for Redis (cluster mode enabled, ports 7000/7001/7002). Two clusters, same shape on paper: three nodes each. What they do with those three nodes is where the two systems part ways.

In this test, Kafka keeps copies of each partition across three brokers. Redis divides keys between three primary nodes with no replicas. The Kafka configuration tests replication and failover. The Redis configuration tests sharding. Both products can use replication and data distribution together.

Kafka replication: leaders, followers, and the ISR

The broker configuration uses KAFKA_DEFAULT_REPLICATION_FACTOR: "3" and KAFKA_MIN_INSYNC_REPLICAS: "2". Internal offset and transaction logs also use replication factor 3. Each partition has a leader and two followers on different brokers. In this setup, clients use the leader, and followers copy its records.

The ISR (in sync replica set) contains replicas that meet Kafka's synchronization requirements. A healthy partition includes all its replicas in the ISR. Kafka removes replicas that fall too far behind or go offline. Reads need an available leader. Writes also depend on the acknowledgement and minimum ISR settings.

My test harness reads exactly this state off the admin client:

interface PartInfo {
  partition: number;
  leader: number;
  replicas: number[];
  isr: number[];
}

async function partitions(admin: Admin): Promise<PartInfo[]> {
  const meta = await admin.fetchTopicMetadata({ topics: [TOPIC] });
  return meta.topics[0].partitions
    .map((p) => ({ partition: p.partitionId, leader: p.leader, replicas: p.replicas, isr: p.isr }))
    .sort((a, b) => a.partition - b.partition);
}

A healthy replicationFactor: 3 topic has three replicas in each partition ISR. With acks=all, the leader waits for the current ISR to acknowledge the write. The min.insync.replicas setting defines the minimum ISR size required to accept that write.

With min.insync.replicas=2 and RF=3, writes can continue after one broker fails. If only one replica remains in the ISR, these writes fail.

Killing a broker and watching failover happen

I forced a broker failure after producing 30 keyed records. I used the idempotent producer from the transaction example. The test finds the leader of partition 0 and stops its container:

const victim = before[0].leader;
execSync(`docker stop kvr-kc${victim}`, { stdio: "ignore" });

let after = before;
for (let i = 0; i < 15; i++) {
  await sleep(2000);
  try {
    after = await partitions(admin);
  } catch {
    continue; // metadata fetch may blip while the broker drops
  }
  if (after[0].leader !== victim) break;
}

The test polls fetchTopicMetadata every two seconds until partition 0 has a new leader. The controller selects a remaining ISR member. In this clean failover, the ISR falls from three members to two. A further record with acks=all succeeds because the ISR still satisfies min.insync.replicas=2.

sequenceDiagram
    participant Admin
    participant B1 as Broker 1
    participant B2 as Broker 2
    participant B3 as Broker 3
    Admin->>B1: describeCluster shows leader p0 is broker 1, ISR is 1 2 3
    Note over B1: docker stop kvr-kc1
    Admin->>B2: fetchTopicMetadata polling
    Note over B2,B3: controller elects new leader from the remaining ISR
    Admin->>B2: leader p0 is now broker 2, ISR is 2 3
    Note over Admin: producer.send still succeeds with acks=all since ISR 2 meets min.insync.replicas 2
    Note over B1: docker start kvr-kc1
    B1->>B2: rejoin and replicate to catch up
    Admin->>B2: leader p0 is broker 2, ISR healed back to 1 2 3

Then I restart the container and poll until each partition has three replicas in the ISR again. This shows the restarted broker catching up. With acks=all, min.insync.replicas determines the minimum ISR size required for a successful write.

Redis Cluster: sharding the keyspace into 16384 slots

This Redis Cluster has three master nodes. Each owns a separate range of the 16384 hash slots. I use redis-cli --cluster create with the three announce IPs and no replicas. The setup demonstrates sharding, with no replica available to replace a failed master.

Every key hashes to a slot with CRC16, and that slot decides which node owns it:

/** CRC16/XMODEM, exactly as Redis computes it (poly 0x1021, init 0). */
export function crc16(input: string): number {
  const bytes = new TextEncoder().encode(input);
  let crc = 0;
  for (const byte of bytes) {
    crc ^= byte << 8;
    for (let i = 0; i < 8; i++) {
      crc = (crc & 0x8000) ? ((crc << 1) ^ 0x1021) & 0xffff : (crc << 1) & 0xffff;
    }
  }
  return crc & 0xffff;
}

/** The cluster slot for a key, honoring hash tags {...}. */
export function keySlot(key: string): number {
  const open = key.indexOf("{");
  if (open !== -1) {
    const close = key.indexOf("}", open + 1);
    if (close > open + 1) key = key.slice(open + 1, close);
  }
  return crc16(key) % 16384;
}

The ioredis Cluster client uses CLUSTER SLOTS to read slot ownership. I then call keySlot for each of the ten symbols. The containers advertise internal Docker addresses (172.30.0.1x:6379) that the host cannot reach.

I spent part of an evening on this connection problem. A natMap maps each internal address to its published localhost port. Without it, the first connection succeeds but redirected requests fail.

Hash tags: the Redis answer to co partitioning

An atomic operation on several Redis Cluster keys requires those keys to share a slot. Without a common hash tag, market:{BBCA} and notif:{BBCA} could select different slots. Redis hashes only the substring inside {...} when the key has a valid hash tag. Both keys thus share a slot and its node. A Lua script or MULTI transaction can access them together.

await cluster.xadd("market:{BBCA}", "*", "p", "9500");
await cluster.xadd("notif:{BBCA}", "*", "p", "alert");
const lua = "return { redis.call('XLEN', KEYS[1]), redis.call('XLEN', KEYS[2]) }";
const ok = await cluster.eval(lua, 2, "market:{BBCA}", "notif:{BBCA}");
// -> one node, one atomic call

Cross the tags and the cluster refuses outright:

try {
  await cluster.eval(lua, 2, "market:{BBCA}", "notif:{BBRI}");
} catch (e) {
  // rejected: CROSSSLOT Keys in request don't hash to the same slot
}

The keys market:{BBCA} and notif:{BBRI} use different hash tags and can map to different slots. A script that requires both keys in one slot then fails with CROSSSLOT.

The custom KafkaJS assigner solves a related process placement problem. It assigns matching topic partitions to one consumer pod. Redis hash tags instead control data placement in a slot. These are separate mechanisms with different guarantees.

graph TD
    subgraph Slots["16384 hash slots split across 3 masters"]
        N1[rc1 master]
        N2[rc2 master]
        N3[rc3 master]
    end
    K1["market:{BBCA}"] -->|hash tag BBCA| N2
    K2["notif:{BBCA}"] -->|hash tag BBCA| N2
    K3["notif:{BBRI}"] -->|hash tag BBRI| N3

Same Docker Compose, two different reasons to add nodes

Put the two clusters next to each other and the philosophies are almost opposites:

Kafka replication lets another replica serve a partition after a leader failure. Redis sharding distributes keys across nodes to increase capacity. I configured these tests separately to observe each behavior. A Redis Cluster with replicas can provide sharding and failover, subject to its persistence and replication limits.

Replication and sharding in one view

• Kafka's replication factor specifies the number of partition replicas. The ISR contains the replicas that meet its synchronization requirements.
• acks=all waits for the current ISR. min.insync.replicas sets the minimum ISR size required for the write to succeed.
• In this RF=3 test, another ISR member replaces the stopped leader. Writes resume if the available ISR meets the configured minimum. The restarted broker catches up and rejoins the ISR.
• Redis Cluster splits a fixed 16384 slots across master nodes using CRC16 of the key, or of the hash tag inside {...} if present.
• Hash tags put related keys in one slot. A Lua script can access those keys atomically. An operation that requires one slot fails with CROSSSLOT if its keys span slots.
• Sharding divides data across nodes. Replication creates copies. Redis Cluster can use both, although this test has no replicas.

The test now includes broker failure. The next post adds state to the ticker through OHLC windows and price alerts that respond to threshold crossings or current levels.