Kafka Consumer Groups: Lag, Draining, and Rebalancing

Once ticks are landing in market-updates with per-symbol order guaranteed by the key, the next question is who reads them. A single consumer process can drain a 6-partition topic just fine, until volume grows or that process dies mid-shift. Kafka's answer to both problems is the same primitive: the consumer group.

The steady state was never the hard part for me. What took longest to get comfortable with was the moment group membership changes and Kafka has to renegotiate who owns what while data keeps arriving. So I made the Kafka half of my stock ticker experiment do it on purpose: pile up a backlog, watch it drain, then push a third consumer into the group mid-run and see what moves.

Partitions, not messages, get divided

A Kafka consumer group divides a topic's partitions among its members. Each partition is owned by exactly one consumer in the group at a time. With market-updates at 6 partitions:

• 2 consumers in the group: 3 partitions each.
• 3 consumers: 2 each.
• 6 consumers: 1 each. A 7th consumer has nothing left to own and sits idle. Partition count is the hard ceiling on useful parallelism.

That single-owner rule is why per-key order survives having multiple consumers at all. A given key, say BBCA, always hashes to the same partition, and that partition has exactly one owner at any moment. Every BBCA tick is handled by the same consumer, in the order it was produced. Ownership can move between consumers, but a partition's internal order never gets rearranged.

My test harness makes this observable by listening to KafkaJS's GROUP_JOIN event, which fires on every member each time the group settles on an assignment:

function makeConsumer(label: string, assignments: Assignments): Consumer {
  const consumer = kafka.consumer({
    groupId: GROUP,
    sessionTimeout: 10000,
    heartbeatInterval: 3000,
  });
  consumer.on(consumer.events.GROUP_JOIN, (e) => {
    assignments.set(label, e.payload.memberAssignment);
  });
  return consumer;
}

heartbeatInterval is how often a consumer pings the group coordinator to say "I'm alive." sessionTimeout is how long the coordinator waits without a heartbeat before declaring a member dead and handing its partitions to someone else. So membership doesn't only change when you explicitly start or stop a consumer, it also changes if a consumer's eachMessage handler blocks too long to send a heartbeat in time. That's the same mechanism behind both a graceful join and a crash.

With pod-1 and pod-2 in the group, the printed assignment comes out as a clean 3-and-3 split across the topic's 6 partitions, one line per member.

Committed offsets and what lag measures

Every partition is an append-only log. Messages get monotonically increasing offsets as they're written, and the highest one is the log end offset. A consumer group tracks its own progress per partition as a committed offset, which is just the offset of the last message it has finished processing (or, with auto-commit, the last offset it had reached when the commit timer fired).

The gap between those two numbers is lag:

lag = logEndOffset - committedOffset   (per partition, summed across the topic)

Flat or zero lag means the group is keeping up. Steadily climbing lag means producers are outpacing consumers. Lag is the health signal I watch first, and the lag helper I wrote computes it from broker metadata rather than from anything the consumers self-report:

export async function lagWithAdmin(
  admin: Admin,
  groupId: string,
  topic: string,
): Promise<LagReport> {
  const [topicOffsets, groupOffsets] = await Promise.all([
    admin.fetchTopicOffsets(topic),
    admin.fetchOffsets({ groupId, topics: [topic] }),
  ]);

  const committed = new Map<number, number>();
  const entry = groupOffsets.find((g) => g.topic === topic);
  for (const p of entry?.partitions ?? []) committed.set(p.partition, Number(p.offset));

  const perPartition = topicOffsets.map((t) => {
    const high = Number(t.high);
    const low = Number(t.low);
    let c = committed.get(t.partition) ?? -1;
    if (c < 0) c = low; // group never committed here yet
    return { partition: t.partition, high, committed: c, lag: Math.max(0, high - c) };
  });

  return { total: perPartition.reduce((s, p) => s + p.lag, 0), perPartition };
}

The detail worth calling out is that c < 0 -> low fallback. If the group has never committed on a partition, treating its committed offset as zero would overstate lag back to the beginning of time on a topic that might have been retaining data for weeks. Instead it falls back to low, the log's earliest retained offset, which answers a more honest question: how much of what's currently available does this group still need to read. consumerLag() wraps this with its own admin client connect and disconnect so callers don't have to manage that lifecycle.

Watching lag drain

A live feed makes for a boring graph, because lag hovers near zero and never has anywhere to fall from. So I gave the run a deliberate burst-then-catch-up shape instead: 200,000 ticks produced across the 6 partitions in chunks of 10,000, all of it before any consumer connects.

const CHUNK = 10000;
for (let sent = 0; sent < TOTAL; sent += CHUNK) {
  const msgs = sim.batch(Math.min(CHUNK, TOTAL - sent)).map(tickToKafka);
  await producer.send({ topic: TOPIC, messages: msgs });
}

Then pod-1 and pod-2 join the group and start consuming, with a short auto-commit interval so the committed offset reflects live progress, and a small throttle so draining takes a few seconds instead of finishing before I can watch it:

await c.run({
  autoCommitInterval: 400, // commit often so lag tracks real progress
  eachMessage: async ({ message }) => {
    const tick = tickFromKafka(message.value);
    for (const rule of SAMPLE_ALERTS) evaluate(rule, tick);
    counter.n++;
    if (counter.n % 1000 === 0) await sleep(20); // gentle throttle
  },
});

While that runs, I poll consumerLag() every 400ms and print a row per sample:

for (let i = 0; i < 25; i++) {
  const { total } = await consumerLag(GROUP, TOPIC);
  console.log(`t+${(i * 0.4).toFixed(1)}s   lag=${total}   consumed=${counter.n}`);
  if (total === 0 && counter.n >= TOTAL) break;
  await sleep(400);
}

The shape is a ramp, not a cliff: lag starts in the tens of thousands (all 200,000 ticks are already sitting unconsumed) and falls steadily as both consumers chew through their 3 partitions each, hitting zero once counter.n catches up to TOTAL. That's the same curve you'd want on a real dashboard. A smooth drain tells you the consumers are healthy and proportionally scaled to the backlog; a flat or rising line while consumers are supposedly running tells you something's stuck.

Rebalancing: what happens when membership changes

Once lag has drained to zero, I add a third consumer, pod-3, to the same group and let it settle before reprinting the assignment:

const c3 = makeConsumer("pod-3", assignments);
await c3.connect();
await c3.subscribe({ topic: TOPIC, fromBeginning: true });
await c3.run({ eachMessage: async () => { counter.n++; } });
await sleep(3500);
printAssignments(assignments);

That single subscribe and run call is enough to trigger a rebalance: the coordinator has a new member to seat, so it recomputes the assignment across all three consumers and hands each of them their share. What was 3 and 3 becomes 2, 2, and 2. The same thing happens in reverse if a consumer disconnects or its heartbeat times out: the coordinator redistributes its partitions among the survivors.

sequenceDiagram
    participant Pod1 as pod-1
    participant Pod2 as pod-2
    participant Pod3 as pod-3
    participant GC as "Group Coordinator"
    Note over Pod1,Pod2: "Owns 3 partitions each"
    Pod3->>GC: "Join the group"
    GC->>Pod1: "Revoke partitions, pause fetching"
    GC->>Pod2: "Revoke partitions, pause fetching"
    GC->>GC: "Recompute assignment across 3 members"
    GC->>Pod1: "New assignment: 2 partitions"
    GC->>Pod2: "New assignment: 2 partitions"
    GC->>Pod3: "New assignment: 2 partitions"
    Note over Pod1,Pod3: "GROUP_JOIN fires per member, fetching resumes"

That diagram shows the classic, "eager" rebalance: every existing member briefly stops fetching, even the ones whose assignment doesn't end up changing, while the coordinator recomputes everything from scratch. Modern Kafka also supports cooperative/incremental rebalancing, which only moves the partitions that need to move and lets unaffected consumers keep fetching the whole time. Either way, rebalances are a normal part of operating a consumer group, not an error condition, but they aren't free. A group that rebalances constantly, because consumers are flaky or because eachMessage handlers are slow enough to miss heartbeats, spends more time renegotiating ownership than doing work.

The ordering guarantee survives the churn

The reassuring part is what a rebalance does not touch. It changes which consumer owns a partition. It never reshuffles the messages inside that partition. BBCA's ticks all still land in the same partition, because the key-to-partition hash never changes, so whichever consumer ends up owning that partition after a rebalance still sees BBCA's history in the exact order it was produced. Ownership moves; the log itself doesn't.

Redis Streams consumer groups look superficially similar from the outside. XREADGROUP also spreads work across a named group, but it has no partition concept underneath and no per-key ownership at all. Any consumer can pick up any entry, including two entries for the same symbol at the same time.

Which leaves me with the thing I want to chase next: with no partitions to hand out, what does Redis even revoke when a consumer dies? Its answer, the pending entries list, looks nothing like a rebalance.