The funny thing about queues is that they all look similar from far away. Something produces work, something stores it for a while, and something else consumes it later. Easy enough.
Then the real questions show up. Should the work disappear after one consumer handles it? Should five different services see the same event? Should a worker be able to replay yesterday's messages? Should the queue live in Postgres, Redis, a broker, or a managed cloud service? And what happens when a worker crashes right after charging a card but before acknowledging the message?
That is where the choice stops being a library preference and becomes a system design decision.
I used to lump all of these tools together as "queues." BullMQ, PgBoss, RabbitMQ, Kafka, SQS, SNS, ActiveMQ, IBM MQ, Redis Streams, NATS, Temporal. Same bucket, different logos. That mental model is too blurry. The better question is: what kind of waiting room does your system need?
The First Split
Before comparing products, I like to separate the problem into four shapes.
graph TD
A[Something needs to happen later] --> B{What kind of later?}
B --> C[Background job]
B --> D[Message between services]
B --> E[Replayable event history]
B --> F[Long running workflow]
C --> C1[PgBoss or BullMQ]
D --> D1[RabbitMQ, ActiveMQ, IBM MQ, SQS, Pub/Sub, Service Bus]
E --> E1[Kafka, Redis Streams, NATS JetStream]
F --> F1[Temporal]
A background job is usually owned by one application. Send an email. Resize an image. Retry a webhook. Generate a report.
A message broker is usually about communication between systems. Payment tells fulfillment that an order is paid. Inventory receives a command. A fraud service gets a copy of a transaction event.
An event stream is a history of facts. New consumers can replay it. Existing consumers can process at their own pace. Kafka lives here.
A workflow is a business process with memory. It waits, retries, sleeps for days, receives signals, and continues from where it left off. That is Temporal territory.
This split alone prevents a lot of bad decisions. Kafka is not a better BullMQ. RabbitMQ is not a worse Kafka. SNS is not SQS with a different name. Temporal is not a queue with extra steps. They are solving different shapes of waiting.
The Questions I Ask First
When someone asks "which queue should we use?" I usually want answers to these first:
- Is this work internal to one app, or communication between services?
- Does one consumer handle each message, or should many consumers see it?
- Do we need replay, or should messages disappear after processing?
- Do we need ordering? If yes, ordering by what?
- Can processing happen twice without breaking the business?
- Do we need delayed jobs, cron jobs, priorities, rate limits, or workflows?
- Do we want to operate infrastructure, or pay the cloud provider to do it?
- Is this a greenfield system, or are we integrating with older enterprise technology?
That last one matters. It is easy to talk as if the world starts at Kafka and SQS, but a lot of serious systems still run on ActiveMQ, IBM MQ, JMS, Beanstalkd, Gearman, and other older tools. Old does not automatically mean bad. Sometimes old means battle tested, boring, and already approved by every governance meeting you do not want to attend again.
A Quick Mental Model
Here is the simplest way I keep the categories straight.
graph LR
P[Producer] --> JQ[Job Queue]
JQ --> W1[Worker]
JQ --> W2[Worker]
P2[Producer] --> EX[Broker or Topic]
EX --> Q1[Consumer Queue A]
EX --> Q2[Consumer Queue B]
P3[Producer] --> LOG[Event Log]
LOG --> CG1[Consumer Group A]
LOG --> CG2[Consumer Group B]
In the first path, workers compete for jobs. One job usually gets handled by one worker.
In the second path, a broker routes messages. Different consumers may receive different copies depending on bindings or subscriptions.
In the third path, events stay in a log for some retention period. Consumers track their own position.
Once that difference clicks, the tools feel less random.
PgBoss
PgBoss is the tool I reach for mentally when the app already has PostgreSQL and the work is normal background processing.
The behavior is very concrete. A job is a row in Postgres. Workers claim rows using database locking, usually the same idea as FOR UPDATE SKIP LOCKED. One worker locks a job. Other workers skip it and take another job. The job moves through states like created, active, completed, failed, or expired.
The killer feature is transactional enqueue. You can create business data and enqueue the job in the same database transaction. If the transaction rolls back, the job never appears.
That sounds small until you have dealt with orphaned background jobs. Imagine creating an order, enqueueing process-order, then rolling back the order insert. If the queue is outside the database, the worker may receive a job for an order that does not exist. You can solve that with an outbox pattern, but PgBoss gives you the simpler version when Postgres is already the center of the system.
sequenceDiagram
participant App
participant DB as PostgreSQL
participant Worker
App->>DB: BEGIN
App->>DB: INSERT order
App->>DB: INSERT job into pgboss table
App->>DB: COMMIT
Worker->>DB: Claim job with row lock
Worker->>DB: Mark job completed
Where PgBoss feels good:
- You already run PostgreSQL.
- Job volume is moderate.
- You want fewer moving parts.
- You care about transactional enqueue.
- You like being able to inspect jobs with SQL.
Where PgBoss bites:
- Every job is database write load.
- Completed and failed jobs need retention discipline.
- Workers use database connections.
- Extreme throughput belongs somewhere else.
- It is not a general purpose service broker.
I wrote more about the PostgreSQL side in PgBoss vs BullMQ, and PgBoss also led me into PostgreSQL advisory locks.
BullMQ
BullMQ is the Redis flavored answer to background jobs, especially in Node.js systems. It feels fast because Redis is fast, and it has a very comfortable feature set: delayed jobs, retries, backoff, priorities, rate limits, repeatable jobs, flows, and dashboards.
Under the hood, BullMQ uses Redis data structures. Waiting jobs, active jobs, delayed jobs, completed jobs, failed jobs, locks, and events live in lists, sorted sets, hashes, and other Redis structures. Multi step changes use Lua scripts so Redis applies them atomically.
A worker takes a job, moves it to active, and keeps renewing a lock while it works. If the worker dies and the lock stops renewing, BullMQ can mark the job as stalled and put it back for another worker.
sequenceDiagram
participant Producer
participant Redis
participant Worker
participant Checker as Stall Checker
Producer->>Redis: Add job
Worker->>Redis: Move job to active and lock it
Worker->>Redis: Renew lock while processing
Note over Worker: Worker crashes
Checker->>Redis: Lock expired?
Checker->>Redis: Move job back to waiting
Where BullMQ feels good:
- Redis already exists in production.
- Low latency job pickup matters.
- You need priorities, delays, retries, rate limits, or job flows.
- You want a strong Node.js job queue experience.
- You are processing high volume app jobs.
Where BullMQ bites:
- Redis durability is a configuration choice, not magic.
- Redis memory pressure becomes queue pressure.
- Large payloads are a bad idea.
- Cross database atomicity needs an outbox pattern.
- It is not meant to be RabbitMQ or Kafka.
The big warning is simple: do not treat a Redis cache setup as a durable queue setup without checking persistence, failover, max memory policy, and backups. I talked about similar tradeoffs in caching strategies.
If the reason you are reaching for BullMQ is priority handling, the underlying idea is close to the data structure version I wrote about in priority queues.
RabbitMQ
RabbitMQ is where I stop thinking "background job library" and start thinking "broker topology."
Producers publish to exchanges. Exchanges route messages to queues. Consumers read from queues and acknowledge messages. The exchange type matters: direct, topic, fanout, headers. Bindings decide which queue gets what.
That routing model is RabbitMQ's best feature. You can have one payment event go to an email queue, a fulfillment queue, and a risk queue. You can route by exact key, topic pattern, or fanout. You can set prefetch so a consumer only receives a certain number of unacknowledged messages at a time.
graph LR
P[Producer] --> EX[Topic Exchange]
EX -->|payment.succeeded| Q1[Email Queue]
EX -->|payment.*| Q2[Risk Queue]
EX -->|payment.succeeded| Q3[Fulfillment Queue]
Q1 --> C1[Email Worker]
Q2 --> C2[Risk Worker]
Q3 --> C3[Fulfillment Worker]
Durability in RabbitMQ has layers. The exchange can be durable. The queue can be durable. The message can be persistent. If you miss one of those pieces, the setup may look safe but still surprise you during a restart.
Where RabbitMQ feels good:
- You need flexible routing.
- Several services consume different subsets of messages.
- You want mature acknowledgements and dead letter queues.
- You need classic request and reply, work queues, or topic routing.
- You want broker semantics without jumping to Kafka.
Where RabbitMQ bites:
- Topology can become messy.
- Long queues can hurt recovery and performance.
- It is not designed for long term event replay.
- Clustering and partitions need real operational understanding.
- Ordering gets tricky with multiple consumers and retries.
RabbitMQ is excellent when you actually need a broker. It is unnecessary weight if the only requirement is "send this email later."
ActiveMQ and Artemis
ActiveMQ is worth mentioning because many real systems still use it. Especially older Java enterprise systems. There is classic ActiveMQ, and there is Apache ActiveMQ Artemis, which is the newer broker architecture.
The mental model usually comes through JMS: queues, topics, sessions, acknowledgements, selectors, transactions, durable subscriptions. Queues are point to point. Topics are pub/sub. Durable subscriptions let subscribers receive topic messages that arrived while they were offline.
Classic ActiveMQ also shows up because it supports older integration needs and protocols. Artemis is generally the better option for new ActiveMQ style deployments, but in real life the decision is often less pure. The company may already have JMS contracts, old services, monitoring, runbooks, and people who know the failure modes.
Where ActiveMQ or Artemis feels good:
- The environment is Java or JMS heavy.
- Existing systems already depend on JMS semantics.
- Durable subscriptions matter.
- Protocol compatibility matters.
- Replacing the broker would create more risk than value.
Where it bites:
- Classic ActiveMQ can feel dated.
- Tuning and persistence configuration matter a lot.
- Newer web teams may not have the operational muscle for it.
- Documentation can feel split between classic ActiveMQ and Artemis.
- It is not the natural choice for analytics style event streams.
I would not choose classic ActiveMQ for a small greenfield web app. I also would not casually rip it out of a stable enterprise system just because the architecture diagram looks old.
IBM MQ
IBM MQ is not fashionable, but fashionable is not the point. It runs a lot of important enterprise workflows, especially in banks, insurance, mainframe integrations, and regulated environments.
IBM MQ thinks in queue managers, queues, channels, listeners, persistent messages, transactions, and governance. Queue managers can connect across machines through channels. Security and access control are central concerns. In many companies, IBM MQ is not a developer library. It is an enterprise integration platform with process around it.
Where IBM MQ feels good:
- You integrate with mainframes or long lived enterprise systems.
- The organization already standardizes on it.
- Governance matters more than developer convenience.
- You need proven durable messaging for critical workflows.
Where it bites:
- It is heavy for small teams.
- Local development is not fun compared with modern developer first tools.
- Licensing and process can slow delivery.
- It is not a cloud native event stream.
If your system already lives in a bank style environment, IBM MQ may be the boring correct answer. If you are building a new side project, please do not start there.
Kafka
Kafka is the one people over pick because it sounds scalable. It is also the one people under appreciate until they need replay.
Kafka is not a normal queue. It is a distributed append only log. Producers write records to topics. Topics are split into partitions. A partition is ordered. Consumers read from partitions and commit offsets. Consuming a message does not delete it. Kafka keeps records until retention removes them.
That one behavior changes the architecture. A fraud service, analytics service, notification service, and data warehouse pipeline can all read the same topic independently. A new service can be created later and replay old events. A bug can be fixed and a consumer can reprocess history.
graph LR
P[Payment Service] --> T[Kafka Topic: payments]
T --> CG1[Fraud Consumer Group]
T --> CG2[Analytics Consumer Group]
T --> CG3[Notification Consumer Group]
T --> CG4[Data Warehouse Consumer Group]
Ordering is per partition. If all events for accountId = 123 use the same key, they land in the same partition and keep account level order. If you want global ordering, you can use one partition, but then you give up parallelism. That tradeoff is not a footnote. It shapes the whole design.
Where Kafka feels good:
- Events are business facts worth keeping.
- Multiple consumers need independent reads.
- Replay matters.
- Throughput is high.
- Per key ordering is enough.
- You are ready for partitions, schemas, offsets, and consumer groups.
Where Kafka bites:
- Operational complexity is real.
- It is awkward for simple delayed jobs.
- Partition key design matters a lot.
- Schema evolution needs discipline.
- Small teams can overbuild with it very quickly.
Kafka is powerful when events are part of the product architecture. It is overkill when the requirement is just "run this task later."
SQS
SQS is boring in the best AWS way. You send a message. Workers poll. SQS stores it durably. You do not run the broker.
The important behavior is visibility timeout. When a worker receives a message, SQS does not delete it. SQS hides it for a period of time. If the worker finishes, it deletes the message. If the worker crashes, the timeout expires and the message becomes visible again.
sequenceDiagram
participant Worker
participant SQS
Worker->>SQS: Receive message
SQS-->>Worker: Message plus receipt handle
Note over SQS: Message is invisible
alt Worker succeeds
Worker->>SQS: Delete message
else Worker crashes
Note over SQS: Visibility timeout expires
SQS-->>Worker: Message can be delivered again
end
Standard SQS gives high throughput and at least once delivery. FIFO SQS gives ordering and deduplication constraints through message groups, but you need to design around the throughput and grouping model.
Where SQS feels good:
- You are already on AWS.
- You want a durable queue without broker operations.
- Workers can be idempotent.
- You want Lambda, ECS, or autoscaling integration.
- You need to absorb spikes safely.
Where SQS bites:
- Standard queues can deliver duplicates.
- Polling has latency and cost tradeoffs.
- Routing is intentionally simple.
- FIFO queues require careful message group design.
- Large payloads need S3 or another storage pattern.
The main discipline with SQS is idempotency. If processing a message twice breaks the business, the consumer design is not ready.
SNS
SNS is not a queue. It is fanout.
A producer publishes to a topic. SNS pushes that message to subscriptions. A subscription can be SQS, Lambda, HTTP, email, SMS, and other targets. The common reliable pattern is SNS topic to multiple SQS queues. Each consumer owns its own queue, backlog, retries, and dead letter queue.
graph LR
P[Publisher] --> SNS[SNS Topic]
SNS --> Q1[SQS: Email Consumer]
SNS --> Q2[SQS: Fraud Consumer]
SNS --> Q3[SQS: Analytics Consumer]
Q1 --> C1[Email Worker]
Q2 --> C2[Fraud Worker]
Q3 --> C3[Analytics Worker]
Subscription filters are useful. You can publish several event types to one topic and let each subscription receive only what it cares about.
Where SNS feels good:
- One event should notify many consumers.
- Publishers should not know subscribers.
- You are already in AWS.
- You pair it with SQS for durable per consumer queues.
Where SNS bites:
- It is not a queue by itself.
- Replay is not the normal model.
- Delivery behavior depends on subscriber type.
- Complex event choreography can become hard to trace.
- You are coupling to AWS.
If SQS is the waiting room, SNS is the announcement system.
Redis Streams
Redis Streams sits between a Redis queue and a lightweight event stream. It is not Kafka, but it is more structured than pushing jobs into a Redis list.
Producers append entries to a stream. Entries get IDs. Consumers can read directly, or join consumer groups. Redis tracks pending messages that were delivered but not acknowledged. If a consumer dies, another consumer can inspect and claim old pending messages.
Where Redis Streams feels good:
- Redis already exists.
- You want consumer groups without Kafka.
- Retention needs are modest.
- You want a light stream before committing to bigger infrastructure.
Where it bites:
- Retention uses Redis memory.
- Operational limits are Redis limits.
- The ecosystem is smaller than Kafka's.
- Huge historical replay is not its strength.
I see Redis Streams as a useful middle step. It is good when Kafka would be too much, but a plain queue is not enough.
Google Pub/Sub
Google Pub/Sub is the GCP managed version of event delivery for many teams. Publishers write to topics. Subscriptions receive from topics. Each subscription has its own delivery state, so multiple subscribers can consume the same topic independently.
Subscriptions can be pull based or push based. Messages need acknowledgement. If a message is not acknowledged before the deadline, it can be delivered again. Ordering is available with ordering keys, but you have to design for it.
Where Google Pub/Sub feels good:
- You are on GCP.
- You want managed event delivery.
- You need independent subscriptions.
- Push or pull delivery is useful.
- You do not want to operate Kafka.
Where it bites:
- It is cloud provider coupling.
- At least once delivery means idempotency still matters.
- Ordering requires explicit key design.
- It does not feel like RabbitMQ topology.
If AWS has SNS plus SQS, GCP teams often reach for Pub/Sub as the default event backbone.
Azure Service Bus
Azure Service Bus is the Azure managed broker. It has queues, topics, subscriptions, message locks, sessions, duplicate detection, and dead lettering.
The receive behavior is similar in spirit to other reliable queues. A consumer receives and locks a message. If it completes the message, the message is removed. If the lock expires before completion, the message can be delivered again.
Sessions are the interesting part. Messages with the same session ID can be processed together, which helps when you need ordered handling per account, per customer, or per workflow.
Where Azure Service Bus feels good:
- You are already on Azure.
- You need managed queues and topics.
- Sessions help your ordering model.
- Dead lettering and duplicate detection are important.
- You want more broker features than a basic queue.
Where it bites:
- Azure coupling is strong.
- Throughput and feature tiers need planning.
- It is not Kafka style replay.
- It has more concepts than a simple job queue.
For Azure heavy companies, this is often the practical answer, even if it gets less hype than Kafka.
NATS
NATS is fast, simple, and pleasant when your system wants lightweight service messaging.
Core NATS is subject based pub/sub. Publishers send messages to subjects like payments.created. Subscribers listen to exact subjects or wildcard patterns. Core NATS is not mainly a durable storage system.
JetStream adds persistence, streams, consumers, acknowledgements, replay, and retention. That moves NATS into a more durable messaging and streaming space while keeping a smaller feel than Kafka in many deployments.
Where NATS feels good:
- You need fast service to service messaging.
- You like simple subject based routing.
- You want a small operational footprint.
- JetStream gives enough durability for your case.
Where it bites:
- Core NATS is not durable by default.
- JetStream adds concepts you need to learn.
- Kafka has a larger stream processing ecosystem.
- It is less common in typical web app stacks.
NATS is one of those tools that feels elegant when it fits. It is not always the default, but it is worth knowing.
Beanstalkd and Gearman
Beanstalkd and Gearman are older job systems. I would not usually pick them for a new critical system today, but they are worth mentioning because they represent a simpler era of background processing.
Beanstalkd has tubes. Producers put jobs into tubes. Workers reserve jobs, process them, then delete, release, bury, or touch them. Reserved jobs have a time to run. If the worker does not finish, the job can return.
Gearman is more like function dispatch. A client submits work for a named function, and workers registered for that function execute it.
Where they feel good:
- You maintain an existing system that already uses them.
- The workload is simple and low risk.
- You need a tiny job server.
- Everyone understands the durability limits.
Where they bite:
- Smaller modern ecosystem.
- Less managed service support.
- Fewer observability and routing features.
- Easier to outgrow than SQS, BullMQ, RabbitMQ, or Kafka.
I would keep them if they are stable and boring. I would be cautious about introducing them fresh.
Temporal
Temporal is not a queue, but it enters queue conversations because teams often discover it after their queue based workers turn into a hidden workflow engine.
Temporal stores workflow history durably. The workflow code describes the process. Activities perform side effects like calling APIs or sending emails. Timers, retries, signals, and long running state are first class concepts.
The big difference is that Temporal remembers the process. A worker can crash, come back, and continue. A workflow can sleep for days. A human approval can arrive later as a signal. You do not have to hide all of that state in job payloads and custom tables.
Where Temporal feels good:
- The process lasts minutes, hours, or days.
- There are multiple business steps.
- You need durable timers.
- Retrying requires state.
- You are building a workflow, not just running a job.
Where it bites:
- It is not a simple queue replacement.
- Deterministic workflow rules take learning.
- It adds conceptual and operational weight.
- Tiny one step jobs do not need it.
My rule is simple: if the worker starts becoming a state machine, I look at Temporal.
The Mistakes I Watch For
The first mistake is choosing Kafka because it sounds scalable. Kafka is excellent when replay and event history matter. It is heavy when you only need to send emails after signup.
The second mistake is pretending at least once delivery means exactly once behavior. Most of these systems can deliver a message more than once. Consumers need idempotency keys, safe database writes, and careful external API calls.
The third mistake is forgetting dead letter queues. Retries are good until a poison message retries forever. Failed messages need a place to go and a way to be inspected.
The fourth mistake is saying "we need ordering" without saying the scope. Global ordering is expensive. Per account ordering is realistic. Per user ordering is common. No ordering is often fine.
The fifth mistake is using a queue to hide a slow system forever. A queue absorbs pressure, but it does not delete pressure. If producers create 10,000 messages per second and consumers process 1,000, you have not solved capacity. You have created a growing backlog.
The metrics I want on any serious queue are queue depth, oldest message age, processing rate, retry count, dead letter count, and worker error rate.
My Defaults
For a normal web app, I start with PgBoss if Postgres is already there and the workload is moderate. It keeps the architecture small.
If Redis is already reliable production infrastructure and the app needs fast background jobs, I reach for BullMQ.
For AWS service decoupling, SQS is my default. I add SNS when fanout appears.
For classic broker routing between services, RabbitMQ is still very reasonable.
For Java enterprise systems, I respect ActiveMQ or Artemis. For regulated enterprise integration, I respect IBM MQ. Not because they are exciting, but because boring reliability has value.
For replayable business events and high throughput pipelines, Kafka is the serious option, but only after accepting the operational cost.
For long running business workflows, I stop pretending a queue is enough and evaluate Temporal.
Summary Table
| System | What It Really Is | Best Fit | Pros | Cons |
|---|---|---|---|---|
| PgBoss | PostgreSQL backed job queue | Background jobs that should share database transaction safety | Simple stack, durable, transactional enqueue, inspectable with SQL | Adds database load, limited by Postgres throughput, not a broker |
| BullMQ | Redis backed job queue | Fast Node.js background jobs with delays, retries, priorities, and flows | Fast, great developer experience, rich job features | Redis durability and memory need real operations, not for replay |
| RabbitMQ | Classic message broker | Service messaging with routing, acknowledgements, and dead letter queues | Flexible exchanges, mature broker behavior, good work queue model | No long term replay, topology and clustering need discipline |
| ActiveMQ / Artemis | JMS style enterprise broker | Java enterprise systems and legacy protocol compatibility | Familiar JMS model, queues and topics, durable subscriptions | Classic ActiveMQ can feel dated, tuning and broker knowledge matter |
| IBM MQ | Enterprise integration messaging | Mainframe, banking, insurance, regulated cross system messaging | Extremely mature, reliable, strong governance and transaction story | Heavy, expensive, less friendly for small teams |
| Kafka | Distributed event log | Replayable business events, analytics pipelines, high throughput streams | Replay, consumer groups, high throughput, strong ecosystem | Operationally complex, awkward for simple jobs, partition design matters |
| SQS | Managed AWS queue | Durable AWS worker queues and service decoupling | Managed, durable, elastic, simple visibility timeout model | At least once delivery, limited routing, polling tradeoffs |
| SNS | Managed AWS fanout | One event delivered to many subscribers, usually through SQS queues | Simple fanout, filtering, publisher decoupling | Not a queue by itself, no normal replay, AWS coupling |
| Redis Streams | Lightweight Redis stream | Consumer groups and modest replay when Redis already exists | Fast, Redis native, better recovery than plain lists | Memory based retention, smaller ecosystem, not Kafka scale |
| Google Pub/Sub | Managed GCP pub/sub | GCP event delivery with independent subscriptions | Managed, push or pull delivery, easy fanout | GCP coupling, idempotency required, ordering needs design |
| Azure Service Bus | Managed Azure broker | Azure enterprise messaging with queues, topics, sessions, and dead letters | Managed broker features, sessions for ordering, duplicate detection | Azure coupling, more complex than a simple queue, not Kafka replay |
| NATS | Lightweight subject based messaging | Fast internal service messaging, with JetStream for durability | Fast, elegant subjects, small footprint | Core NATS is not durable, smaller ecosystem than Kafka |
| Beanstalkd / Gearman | Older simple job servers | Maintaining small existing job systems | Simple, lightweight, easy mental model | Limited modern durability, routing, observability, and managed support |
| Temporal | Durable workflow engine | Long running business processes with retries, timers, and state | Durable workflows, stateful retries, crash recovery | Not a simple queue, workflow determinism takes learning |