Replication and Failover: Kafka ISR vs Redis Cluster Sharding

Everything I have built for this ticker so far ran against a single Kafka broker and a single Redis instance. Fine for learning partitions, consumer groups and rebalancing, and the PEL, but it dodges the question that matters once something is in production: what happens when a machine dies, or when your data no longer fits on one machine. So I brought up a three broker Kafka cluster and a three node Redis Cluster, side by side, in the same docker compose file, and started killing things on purpose.

When One Node Is Not Enough

A second compose profile brings up six containers: kvr-kc1/2/3 for Kafka (KRaft mode, no ZooKeeper, ports 19092/19094/19096), and kvr-rc1/2/3 for Redis (cluster mode enabled, ports 7000/7001/7002). Two clusters, same shape on paper: three nodes each. What they do with those three nodes is where the two systems part ways.

Kafka's three brokers exist to keep multiple copies of the same data. Redis Cluster's three nodes exist to split the data into disjoint pieces. One is about surviving loss. The other is about surviving size. Confusing the two is an easy mistake to make when you only look at the node count.

Kafka Replication: Leaders, Followers, and the ISR

The shared broker config sets KAFKA_DEFAULT_REPLICATION_FACTOR: "3" and KAFKA_MIN_INSYNC_REPLICAS: "2", along with replication factor 3 for the internal offsets and transaction logs too. A replication factor of 3 means every partition has three replicas, placed on different brokers. One of those replicas is the leader; every produce and fetch for that partition goes through it. The other two are followers, pulling from the leader to stay caught up.

The ISR (in-sync replica set) is the list of replicas that are currently caught up enough to be trusted. A healthy partition has an ISR equal to all its replicas. When a broker falls behind or goes offline, Kafka drops it out of the ISR, but the topic keeps serving reads and writes as long as enough replicas remain.

My test harness reads exactly this state off the admin client:

interface PartInfo {
  partition: number;
  leader: number;
  replicas: number[];
  isr: number[];
}

async function partitions(admin: Admin): Promise<PartInfo[]> {
  const meta = await admin.fetchTopicMetadata({ topics: [TOPIC] });
  return meta.topics[0].partitions
    .map((p) => ({ partition: p.partitionId, leader: p.leader, replicas: p.replicas, isr: p.isr }))
    .sort((a, b) => a.partition - b.partition);
}

Right after creating a replicationFactor: 3 topic, every partition looks the same: leader set, three broker ids in replicas, ISR matching replicas exactly. min.insync.replicas is what makes the ISR matter for correctness rather than mere observability: paired with acks=all on the producer, Kafka will not acknowledge a write until it has been replicated to at least that many in-sync replicas. With min.insync.replicas=2 and RF=3, you can lose one broker and keep accepting durable writes. Lose two, and produces start failing loudly instead of silently losing acknowledged data.

Killing a Broker and Watching Failover Happen

Describing failover is cheap, so I forced it to happen. After producing 30 keyed records with the same idempotent producer I set up for exactly-once semantics, my run grabs the current leader of partition 0 and stops that broker's container directly:

const victim = before[0].leader;
execSync(`docker stop kvr-kc${victim}`, { stdio: "ignore" });

let after = before;
for (let i = 0; i < 15; i++) {
  await sleep(2000);
  try {
    after = await partitions(admin);
  } catch {
    continue; // metadata fetch may blip while the broker drops
  }
  if (after[0].leader !== victim) break;
}

It polls fetchTopicMetadata every two seconds until the leader for partition 0 changes. Once the controller notices the broker is gone, it elects a new leader from the remaining ISR members, no data loss, because it only ever promotes a replica that was already caught up. The ISR for that partition shrinks from three brokers to two. Then I produce one more record with acks=all, and it succeeds, because two in-sync replicas still satisfies min.insync.replicas=2.

sequenceDiagram
    participant Admin
    participant B1 as Broker 1
    participant B2 as Broker 2
    participant B3 as Broker 3
    Admin->>B1: describeCluster shows leader p0 is broker 1, ISR is 1 2 3
    Note over B1: docker stop kvr-kc1
    Admin->>B2: fetchTopicMetadata polling
    Note over B2,B3: controller elects new leader from the remaining ISR
    Admin->>B2: leader p0 is now broker 2, ISR is 2 3
    Note over Admin: producer.send still succeeds with acks=all since ISR 2 meets min.insync.replicas 2
    Note over B1: docker start kvr-kc1
    B1->>B2: rejoin and replicate to catch up
    Admin->>B2: leader p0 is broker 2, ISR healed back to 1 2 3

Then I restart the stopped container and poll again, this time waiting for the ISR to climb back to three replicas on every partition as the rejoined broker catches up on what it missed. Watching that loop finish is the clearest way I have found to understand min.insync.replicas: it is a gate the broker checks before acknowledging every write, rather than a health metric you read after the fact.

Redis Cluster: Sharding the Keyspace Into 16384 Slots

Redis Cluster solves a different problem. Its three master nodes each own a disjoint slice of a fixed 16384 hash slots. I form the cluster with redis-cli --cluster create against the three announce IPs and pass no replica count, so it comes out as three masters with zero replicas each. That detail matters: unlike the Kafka side, this setup demonstrates sharding only, with no high availability.

Every key hashes to a slot with CRC16, and that slot decides which node owns it:

/** CRC16/XMODEM, exactly as Redis computes it (poly 0x1021, init 0). */
export function crc16(input: string): number {
  const bytes = new TextEncoder().encode(input);
  let crc = 0;
  for (const byte of bytes) {
    crc ^= byte << 8;
    for (let i = 0; i < 8; i++) {
      crc = (crc & 0x8000) ? ((crc << 1) ^ 0x1021) & 0xffff : (crc << 1) & 0xffff;
    }
  }
  return crc & 0xffff;
}

/** The cluster slot for a key, honoring hash tags {...}. */
export function keySlot(key: string): number {
  const open = key.indexOf("{");
  if (open !== -1) {
    const close = key.indexOf("}", open + 1);
    if (close > open + 1) key = key.slice(open + 1, close);
  }
  return crc16(key) % 16384;
}

On the Redis side I connect with ioredis's Cluster client, ask any master for CLUSTER SLOTS, and print which node owns which slot range, then compute keySlot for each of the ten symbols to watch them land on different nodes. One wrinkle cost me a good chunk of an evening: the containers gossip their internal Docker IPs (172.30.0.1x:6379), unreachable from the host, so I give ioredis a natMap that rewrites each internal address to its published localhost port. Without it, the client connects fine to the first node and then fails every redirect it gets told to follow.

Hash Tags: The Redis Answer to Co-Partitioning

Splitting the keyspace across nodes creates a new problem the moment you need a multi-key operation. If market:{BBCA} and notif:{BBCA} hashed independently, they could land on different nodes, and there would be no way to touch both in one atomic Lua script or MULTI block. Redis's answer is the hash tag: only the substring inside {...} gets hashed, so any keys sharing the same tag land on the same slot, and therefore the same node.

await cluster.xadd("market:{BBCA}", "*", "p", "9500");
await cluster.xadd("notif:{BBCA}", "*", "p", "alert");
const lua = "return { redis.call('XLEN', KEYS[1]), redis.call('XLEN', KEYS[2]) }";
const ok = await cluster.eval(lua, 2, "market:{BBCA}", "notif:{BBCA}");
// -> one node, one atomic call

Cross the tags and the cluster refuses outright:

try {
  await cluster.eval(lua, 2, "market:{BBCA}", "notif:{BBRI}");
} catch (e) {
  // rejected: CROSSSLOT Keys in request don't hash to the same slot
}

market:{BBCA} and notif:{BBRI} almost certainly hash to different slots on different nodes, so a Lua script needing both keys locally cannot run. Same problem the custom KafkaJS co-partition assigner exists to solve, wearing different clothes. Kafka solves it at the partition assignment layer, pinning the same partition number of both topics to the same consumer pod. Redis solves it at the key naming layer, with braces. Neither is automatic: forget the assigner and Kafka round robins your partitions apart; forget the braces and Redis scatters your keys across nodes and hands you CROSSSLOT the moment you touch two of them together.

graph TD
    subgraph Slots["16384 hash slots split across 3 masters"]
        N1[rc1 master]
        N2[rc2 master]
        N3[rc3 master]
    end
    K1["market:{BBCA}"] -->|hash tag BBCA| N2
    K2["notif:{BBCA}"] -->|hash tag BBCA| N2
    K3["notif:{BBRI}"] -->|hash tag BBRI| N3

Same Docker Compose, Two Different Reasons to Add Nodes

Put the two clusters next to each other and the philosophies are almost opposites:

	Kafka (RF=3 cluster)	Redis Cluster (3 masters)
What each node holds	A full copy of its assigned partitions	A disjoint slice of the 16384 slots
Why you add nodes	Durability and availability	Capacity and throughput
What happens if a node dies	Leader fails over to an in-sync replica, no acknowledged data lost	That node's slots become unavailable until a replica takes over, or are lost if it had none
The safety mechanism	ISR plus min.insync.replicas plus acks=all	Per-shard replicas (not configured in mine)
The multi-key coordination problem	Co-partitioning: keep related keys on the same partition	Hash tags: keep related keys in the same slot

Kafka replicates the log: the same bytes exist on multiple brokers, so losing one changes nothing about what data exists, only which broker currently serves it. Redis Cluster shards the keyspace: different keys live on different nodes, so the cluster's total capacity is the sum of all nodes, not the size of the smallest one. I kept the two lessons apart on purpose. Kafka gets real replication so the failover run demonstrates it cleanly; Redis gets zero replicas so the sharding run has no failover story muddying the picture. A production Redis Cluster would normally add a replica per shard to get both properties at once, but that is a second, separate axis on top of sharding, not something sharding gives you for free.

The Short Version

• Kafka's replication factor is how many full copies of a partition exist; the ISR is the subset currently caught up enough to trust.
• acks=all plus min.insync.replicas turns "we have replicas" into a real durability guarantee: a write is not acknowledged until enough in-sync replicas have it.
• Killing the leader broker in a healthy RF=3 topic causes a clean failover to another ISR member; the ISR shrinks, writes keep flowing as long as the floor is met, and the ISR heals once the broker rejoins.
• Redis Cluster splits a fixed 16384 slots across master nodes using CRC16 of the key, or of the hash tag inside {...} if present.
• Hash tags co-locate related keys on one slot, and therefore one node, which is what makes atomic multi-key Lua scripts possible; crossing tags in one operation gets rejected with CROSSSLOT.
• Sharding and replication are different axes: I isolated them here on purpose, and a real production Redis Cluster would layer replicas on top of the sharding shown.

So the stream survives a dead broker now. What it still cannot do is remember anything: every tick arrives, gets handled, and is forgotten. What changes when I ask it to hold state, build OHLC candles from a rolling window, and tell an edge-triggered price alert apart from a level-triggered one?