A benchmark needs more than a throughput number. Hardware, payload size, batch size, warmup, and other activity on the machine affect the result. Without those conditions, a comparison is difficult to interpret.
After building the same stock ticker twice, I wanted to compare Kafka and Redis Streams on my machine. I wrote a test harness to measure throughput and latency.
This article describes the harness and its limits. It contains no measured throughput or latency results. Results from my laptop would need further tests before they could support a production decision.
Two ways into the same function
Two entry points call runBenchmark(). The CLI accepts a message count and payload size, with defaults of 15,000 messages and 64 bytes. The dashboard posts the selected values to an API route, which calls the same function on the server:
export async function POST(req: Request): Promise<Response> {
const body = await req.json();
const result = await runBenchmark(body.messages ?? 15000, body.payloadBytes ?? 64);
return Response.json(result);
}
Both paths clamp their inputs before doing anything:
const n = Math.max(1000, Math.min(50000, Math.floor(messages)));
const bytes = Math.max(8, Math.min(4096, Math.floor(payloadBytes)));
const m = Math.min(1200, n); // how many of those n messages get a latency sample
The message count ranges from 1,000 to 50,000, and the payload size ranges from 8 to 4,096 bytes. The harness measures latency for at most 1,200 messages to limit the duration of that phase.
The dashboard presets range from 5,000 to 50,000 messages and from 16 to 1,024 bytes. They remain within the same input limits.
What gets measured, and in what order
Two dimensions, each run against both systems, sequentially:
sequenceDiagram
participant H as Harness
participant K as Kafka
participant R as Redis
H->>K: throughput run "batched producer.send"
H->>K: latency run "producer + fresh consumer"
H->>R: throughput run "pipelined XADD"
H->>R: latency run "XADD + blocking XREAD"
Note over H: one phase at a time, never concurrent
The harness runs both Kafka phases first, then both Redis phases. Sequential runs avoid direct competition between the two tests for CPU, the Docker network bridge, and disk access. Other activity on my laptop can still affect either result.
Produce throughput
Both throughput functions do the same thing structurally: send n messages in batches of 1,000, time the whole loop, divide.
// Kafka: batched producer.send(), keyed across all 6 partitions
const producer = kafka.producer({ idempotent: true });
const BATCH = 1000;
const start = performance.now();
for (let i = 0; i < n; i += BATCH) {
const end = Math.min(n, i + BATCH);
const messages = [];
for (let j = i; j < end; j++) messages.push({ key: `k${j % 6}`, value });
await producer.send({ topic, messages });
}
const perSec = Math.round(n / ((performance.now() - start) / 1000));
// Redis: pipelined XADD, same batch size
const redis = createRedis();
const BATCH = 1000;
const start = performance.now();
for (let i = 0; i < n; i += BATCH) {
const end = Math.min(n, i + BATCH);
const pipe = redis.pipeline();
for (let j = i; j < end; j++) pipe.xadd(key, "*", "p", value);
await pipe.exec();
}
const perSec = Math.round(n / ((performance.now() - start) / 1000));
The Kafka run cycles keys k0 through k5 across a topic with 6 partitions. It includes the partition level parallelism described earlier. The Redis run writes to one stream key.
This difference limits the comparison. A further experiment could split Redis traffic across stream keys, with hash tags for related keys. The current harness does not do that.
The Kafka producer uses idempotent: true and acks=all. The default Compose setup has one node and creates benchmark topics with replicationFactor: 1. Thus, "all in-sync replicas" means one replica. These tests do not measure acknowledgment across multiple brokers.
End to end latency
Throughput measures messages per unit of time. Latency measures the time between the send operation and receipt. To measure latency, I run the producer and consumer concurrently. I record the send time in each message and calculate the elapsed time at receipt.
For Kafka, the harness creates a fresh consumer group (bench-lat-<timestamp>) and subscribes to the topic. It waits 900ms before the first message. This fixed delay reduced startup races in my tests, but it does not prove that the consumer is ready. Each message carries its send timestamp in a header:
await producer.send({
topic,
messages: [{ key: `k${i % 6}`, value, headers: { t: String(performance.now()) } }],
});
// in the consumer's eachMessage handler
const sent = Number(message.headers?.t?.toString() ?? "0");
latencies.push(performance.now() - sent);
The Redis writer sends entries with XADD. The reader uses a separate blocking connection. A blocking read would otherwise delay other commands on that connection. It issues raw XREAD ... BLOCK 1000 calls from $ (new entries only). The writer puts the send time in an entry field:
await writer.xadd(key, "*", "t", String(performance.now()), "p", value);
// read loop, dedicated blocking connection
const res = await reader.call("XREAD", "COUNT", "500", "BLOCK", "1000", "STREAMS", key, lastId);
// for each entry returned: latencies.push(performance.now() - Number(map.t))
The Redis measurement uses XREAD, rather than XREADGROUP, so it excludes pending entries list management. Both latency loops have a 20 second timeout. They send one message at a time, so they do not measure batch latency.
Percentiles, computed plainly
Once the harness has a list of latency samples, one shared function runs over both systems' results:
function percentiles(samples: number[]): Percentiles {
const s = [...samples].sort((a, b) => a - b);
const at = (q: number) => s[Math.min(s.length - 1, Math.floor(q * s.length))];
const sum = s.reduce((a, b) => a + b, 0);
return {
count: s.length,
avgMs: sum / s.length,
p50Ms: at(0.5),
p95Ms: at(0.95),
p99Ms: at(0.99),
minMs: s[0],
maxMs: s[s.length - 1],
};
}
The at(q) function sorts the samples and selects the index floor(q * length). This is an index based percentile estimate. It can differ from interpolation and from the nearest rank convention, which uses ceil(q * length) - 1 for positive quantiles. Use the same estimator when results are compared.
The harness discards no warmup samples. Startup costs, such as connection setup and consumer group coordination, remain in the mean and percentiles. With few samples, a small number of slow messages can substantially change the mean.
Why the shapes tend to differ
These architectural differences can explain test results. They do not replace measurements from the target deployment.
graph TD
subgraph Kafka
KP["Producer batches records"] --> KL["Leader log segment (page cache)"]
KL --> KF["fsync per broker flush policy"]
KF --> KA["acks=all: wait for in-sync replicas"]
end
subgraph Redis
RP["Client pipelines XADD"] --> RE["Single-threaded event loop"]
RE --> RM["In-memory stream (RAM)"]
RM --> RA["Async AOF append (everysec)"]
end
Kafka appends records to log segments and uses the operating system page cache. The broker does not normally synchronize each record to disk. Replication and acknowledgment settings are thus important to durability.
My harness sends batches of 1,000 records per producer.send() call. This batching reduces request overhead. Do not assume that Java producer settings such as linger.ms and batch.size apply to KafkaJS.
Redis stores stream data in memory and executes commands sequentially on its main command thread. Persistence depends on configuration. My Compose setup uses AOF with appendfsync everysec, which permits a window of data loss after a failure. Pipelining reduces network exchanges by sending multiple commands together.
The systems have different storage and replication costs. Measure both with the required payload, batching, and durability settings. The architecture alone does not establish which will be faster for that workload.
The caveats I keep attached to every run
A few things this benchmark is not, and should never be mistaken for:
- Local topology: the default Compose setup has one KRaft node with benchmark topics at
replicationFactor: 1. The separate three broker profile usesreplicationFactor: 3for the leader failure test. The benchmark does not use that profile or Redis Cluster. - Client overhead: KafkaJS and ioredis share the Node.js process with the harness. Garbage collection,
JSON.stringify, and promise scheduling affect the measurements. The results measure "Kafka through KafkaJS" and "Redis through ioredis," including client costs. - Local network: clients connect through Docker port mappings to
localhost. This setup does not represent network delay or failures between hosts, availability zones, or regions. - Single run: the harness has no warmup phase or repeated sustained load. Each run measures one machine under its current conditions.
A deployment test should vary producer and consumer concurrency, broker count, and Redis sharding. It should include the required disk synchronization behavior and payloads above 4,096 bytes if the application needs them. Run long enough to observe garbage collection and page cache effects on tail latency.
Reading the output without fooling myself
I inspect the distribution before the average. A p50 near the mean and a distant p99 indicate slow requests in the tail. I compare equal message counts and payload sizes, then repeat the run. A single 15,000 message run with a browser and IDE open only tests my local setup.
Use measurements from representative hardware, payloads, and load before a deployment decision. The local harness helps explore behavior, but its results need that additional context.
The final post uses the behavior from this series to compare where each system fits.
Loading comments...