Choosing the Right Queue System

Queue systems store work until a consumer can process it. Their delivery, retention, and recovery rules determine which system fits an application.

Start with the required behavior. A job may belong to one worker, while an event may need five independent consumers. Some consumers need to replay earlier messages. Recovery also matters: a worker can crash after a card charge but before the message acknowledgment.

These requirements determine both the queue and the consumer design.

I used to put background job libraries, message brokers, event logs, and workflow engines in one category. BullMQ, PgBoss, RabbitMQ, Kafka, SQS, SNS, ActiveMQ, IBM MQ, Redis Streams, NATS, and Temporal all became "queues" in my notes. That label hid the differences in their delivery, history, and coordination models.

Jobs, brokers, streams, and workflows

I first separate the requirements into four categories.

graph TD
    A[Something needs to happen later] --> B{What kind of later?}
    B --> C[Background job]
    B --> D[Message between services]
    B --> E[Replayable event history]
    B --> F[Long running workflow]

    C --> C1[PgBoss or BullMQ]
    D --> D1[RabbitMQ, ActiveMQ, IBM MQ, SQS, Pub/Sub, Service Bus]
    E --> E1[Kafka, Redis Streams, NATS JetStream]
    F --> F1[Temporal]

A background job is usually owned by one application. Send an email. Resize an image. Retry a webhook. Generate a report.

A message broker is usually about communication between systems. Payment tells fulfillment that an order is paid. Inventory receives a command. A fraud service gets a copy of a transaction event.

An event stream is a history of facts. New consumers can replay it. Existing consumers can process at their own pace. Kafka lives here.

A workflow stores the state of a business process. It can wait for days, retry operations, receive signals, and continue from stored state. Temporal supports this type of work.

The categories help explain the product differences. BullMQ manages background jobs, RabbitMQ routes messages, Kafka retains events, SNS distributes notifications, and Temporal coordinates workflows.

The Questions I Ask First

When someone asks "which queue should we use?" I usually want answers to these first:

1. Is this work internal to one app, or communication between services?
2. Does one consumer handle each message, or should many consumers see it?
3. Do we need replay, or should messages disappear after processing?
4. Do we need ordering? If yes, ordering by what?
5. Can processing happen twice without breaking the business?
6. Do we need delayed jobs, cron jobs, priorities, rate limits, or workflows?
7. Do we want to operate infrastructure, or pay the cloud provider to do it?
8. Is this a greenfield system, or are we integrating with older enterprise technology?

Existing systems may already use ActiveMQ, IBM MQ, JMS, Beanstalkd, or Gearman. Their operational history, integrations, and organizational approvals affect the cost of replacement.

Delivery and retention

This model keeps the categories straight:

graph LR
    P[Producer] --> JQ[Job Queue]
    JQ --> W1[Worker]
    JQ --> W2[Worker]

    P2[Producer] --> EX[Broker or Topic]
    EX --> Q1[Consumer Queue A]
    EX --> Q2[Consumer Queue B]

    P3[Producer] --> LOG[Event Log]
    LOG --> CG1[Consumer Group A]
    LOG --> CG2[Consumer Group B]

In the first path, workers compete for jobs. One job usually gets handled by one worker.

In the second path, a broker routes messages. Different consumers may receive different copies depending on bindings or subscriptions.

In the third path, events stay in a log for some retention period. Consumers track their own position.

The delivery and retention rules give each category a distinct purpose.

PgBoss

PgBoss is the tool I reach for mentally when the app already has PostgreSQL and the work is normal background processing.

The behavior is very concrete. A job is a row in Postgres. Workers claim rows using database locking, usually the same idea as FOR UPDATE SKIP LOCKED. One worker locks a job. Other workers skip it and take another job. The job moves through states like created, active, completed, failed, or expired.

Transactional enqueue is a useful feature. You can create business data and enqueue the job in the same database transaction. If the transaction rolls back, the job never appears.

For example, an application might insert an order, enqueue process-order, and then roll back the order insert. If the queue is outside the database, the worker may receive a job for an order that does not exist. You can solve that with an outbox pattern, but PgBoss gives you the simpler version when Postgres is already the center of the system.

sequenceDiagram
    participant App
    participant DB as PostgreSQL
    participant Worker

    App->>DB: BEGIN
    App->>DB: INSERT order
    App->>DB: INSERT job into pgboss table
    App->>DB: COMMIT
    Worker->>DB: Claim job with row lock
    Worker->>DB: Mark job completed

PgBoss fits these requirements:

• You already run PostgreSQL.
• Job volume is moderate.
• You want fewer moving parts.
• You care about transactional enqueue.
• You like being able to inspect jobs with SQL.

PgBoss operating constraints:

• Every job is database write load.
• Completed and failed jobs need retention discipline.
• Workers use database connections.
• Extreme throughput belongs somewhere else.
• It is not a general purpose service broker.

I wrote more about the PostgreSQL side in PgBoss vs BullMQ, and PgBoss also led me into PostgreSQL advisory locks.

BullMQ

BullMQ stores background jobs in Redis for Node.js applications. Its features include delayed jobs, retries, backoff, priorities, rate limits, repeatable jobs, flows, and dashboards.

Internally, BullMQ uses Redis data structures. Waiting jobs, active jobs, delayed jobs, completed jobs, failed jobs, locks, and events live in lists, sorted sets, hashes, and other Redis structures. Multi step changes use Lua scripts so Redis applies them atomically.

A worker takes a job, moves it to active, and keeps renewing a lock while it works. If the worker dies and the lock stops renewing, BullMQ can mark the job as stalled and put it back for another worker.

sequenceDiagram
    participant Producer
    participant Redis
    participant Worker
    participant Checker as Stall Checker

    Producer->>Redis: Add job
    Worker->>Redis: Move job to active and lock it
    Worker->>Redis: Renew lock while processing
    Note over Worker: Worker crashes
    Checker->>Redis: Lock expired?
    Checker->>Redis: Move job back to waiting

BullMQ fits these requirements:

• Redis already exists in production.
• Low latency job pickup matters.
• You need priorities, delays, retries, rate limits, or job flows.
• You want a strong Node.js job queue experience.
• You are processing high volume app jobs.

BullMQ operating constraints:

• Redis durability depends on its persistence and replication settings.
• The queue shares Redis memory limits with other data.
• Large payloads are a bad idea.
• Cross database atomicity needs an outbox pattern.
• It is not meant to be RabbitMQ or Kafka.

Before you use Redis for a durable queue, check its persistence, failover, memory policy, and backups. I discussed similar tradeoffs in caching strategies.

BullMQ supports job priorities. I explained the underlying data structure in priority queues.

RabbitMQ

RabbitMQ routes messages through exchanges and queues. The relationship between them is the broker topology.

Producers publish to exchanges. Exchanges route messages to queues. Consumers read from queues and acknowledge messages. The exchange type matters: direct, topic, fanout, headers. Bindings decide which queue gets what.

That routing model is RabbitMQ's best feature. You can have one payment event go to an email queue, a fulfillment queue, and a risk queue. You can route by exact key, topic pattern, or fanout. You can set prefetch so a consumer only receives a certain number of unacknowledged messages at a time.

graph LR
    P[Producer] --> EX[Topic Exchange]
    EX -->|payment.succeeded| Q1[Email Queue]
    EX -->|payment.*| Q2[Risk Queue]
    EX -->|payment.succeeded| Q3[Fulfillment Queue]
    Q1 --> C1[Email Worker]
    Q2 --> C2[Risk Worker]
    Q3 --> C3[Fulfillment Worker]

Durability in RabbitMQ has layers. The exchange can be durable. The queue can be durable. The message can be persistent. If you miss one of those pieces, the setup may look safe but still surprise you during a restart.

RabbitMQ fits these requirements:

• You need flexible routing.
• Several services consume different subsets of messages.
• You want mature acknowledgements and dead letter queues.
• You need classic request and reply, work queues, or topic routing.
• You want broker semantics without jumping to Kafka.

RabbitMQ operating constraints:

• Topology can become messy.
• Long queues can hurt recovery and performance.
• It is not designed for long term event replay.
• Clustering and partitions need real operational understanding.
• Ordering gets tricky with multiple consumers and retries.

RabbitMQ fits when you need a broker. It is unnecessary weight if the only requirement is "send this email later."

ActiveMQ and Artemis

ActiveMQ still runs in many systems, especially older Java enterprise systems. The options are classic ActiveMQ and Apache ActiveMQ Artemis, the newer broker architecture.

The mental model usually comes through JMS: queues, topics, sessions, acknowledgements, selectors, transactions, durable subscriptions. Queues are point to point. Topics are pub/sub. Durable subscriptions let subscribers receive topic messages that arrived while they were offline.

Classic ActiveMQ also shows up because it supports older integration needs and protocols. Artemis is generally the better option for new ActiveMQ style deployments, but in real life the decision is often less pure. The company may already have JMS contracts, old services, monitoring, runbooks, and people who know the failure modes.

ActiveMQ or Artemis fits these requirements:

• The environment is Java or JMS heavy.
• Existing systems already depend on JMS semantics.
• Durable subscriptions matter.
• Protocol compatibility matters.
• Replacing the broker would create more risk than value.

Where it bites:

• Classic ActiveMQ can feel dated.
• Tuning and persistence configuration matter a lot.
• Newer web teams may not have the operational muscle for it.
• Documentation can feel split between classic ActiveMQ and Artemis.
• It is not the natural choice for analytics style event streams.

I would not choose classic ActiveMQ for a small greenfield web app. I also would not casually rip it out of a stable enterprise system just because the architecture diagram looks old.

IBM MQ

IBM MQ supports enterprise messaging, including banking, insurance, and mainframe integrations.

IBM MQ thinks in queue managers, queues, channels, listeners, persistent messages, transactions, and governance. Queue managers can connect across machines through channels. Security and access control are central concerns. In many companies, IBM MQ is not a developer library. It is an enterprise integration platform with process around it.

IBM MQ fits these requirements:

• You integrate with mainframes or long lived enterprise systems.
• The organization already standardizes on it.
• Governance matters more than developer convenience.
• You need proven durable messaging for critical workflows.

Where it bites:

• It is heavy for small teams.
• Local development is not fun compared with modern developer first tools.
• Licensing and process can slow delivery.
• It is not a cloud native event stream.

I would consider IBM MQ when existing enterprise integrations require it. Its administration and licensing costs would be difficult to justify for a small personal project.

Kafka

I consider Kafka when consumers need to retain and replay event history.

Kafka stores records in a distributed append only log. Producers write to topics, which contain ordered partitions. Consumers read partitions and commit offsets to record their progress. Reading a record does not delete it. Retention settings control how long records remain.

That one behavior changes the architecture. A fraud service, analytics service, notification service, and data warehouse pipeline can all read the same topic independently. A new service can be created later and replay old events. A bug can be fixed and a consumer can reprocess history.

graph LR
    P[Payment Service] --> T[Kafka Topic: payments]
    T --> CG1[Fraud Consumer Group]
    T --> CG2[Analytics Consumer Group]
    T --> CG3[Notification Consumer Group]
    T --> CG4[Data Warehouse Consumer Group]

Ordering is per partition. If all events for accountId = 123 use the same key, they land in the same partition and keep account level order. If you want global ordering, you can use one partition, but then you give up parallelism. The partition count sets this limit on parallel consumption.

Kafka fits these requirements:

• Events are business facts worth keeping.
• Multiple consumers need independent reads.
• Replay matters.
• Throughput is high.
• Per key ordering is enough.
• You are ready for partitions, schemas, offsets, and consumer groups.

Kafka operating constraints:

• Operational complexity is real.
• It is awkward for simple delayed jobs.
• Partition key design matters a lot.
• Schema evolution needs discipline.
• Small teams can overbuild with it very quickly.

Kafka is powerful when events are part of the product architecture. It is overkill when the requirement is just "run this task later."

SQS

AWS operates the SQS service. Producers send messages, SQS stores them, and workers poll for available messages.

The important behavior is visibility timeout. When a worker receives a message, SQS does not delete it. SQS hides it for a period of time. If the worker finishes, it deletes the message. If the worker crashes, the timeout expires and the message becomes visible again.

sequenceDiagram
    participant Worker
    participant SQS

    Worker->>SQS: Receive message
    SQS-->>Worker: Message plus receipt handle
    Note over SQS: Message is invisible
    alt Worker succeeds
        Worker->>SQS: Delete message
    else Worker crashes
        Note over SQS: Visibility timeout expires
        SQS-->>Worker: Message can be delivered again
    end

Standard SQS gives high throughput and at least once delivery. FIFO SQS gives ordering and deduplication constraints through message groups, but you need to design around the throughput and grouping model.

SQS fits these requirements:

• You are already on AWS.
• You want a durable queue without broker operations.
• Workers can be idempotent.
• You want Lambda, ECS, or autoscaling integration.
• You need to absorb spikes safely.

SQS operating constraints:

• Standard queues can deliver duplicates.
• Polling has latency and cost tradeoffs.
• Routing is intentionally simple.
• FIFO queues require careful message group design.
• Large payloads need S3 or another storage pattern.

The main discipline with SQS is idempotency. If processing a message twice breaks the business, the consumer design is not ready.

SNS

SNS is not a queue. It is fanout.

A producer publishes to a topic. SNS pushes that message to subscriptions. A subscription can be SQS, Lambda, HTTP, email, SMS, and other targets. The common reliable pattern is SNS topic to multiple SQS queues. Each consumer owns its own queue, backlog, retries, and dead letter queue.

graph LR
    P[Publisher] --> SNS[SNS Topic]
    SNS --> Q1[SQS: Email Consumer]
    SNS --> Q2[SQS: Fraud Consumer]
    SNS --> Q3[SQS: Analytics Consumer]
    Q1 --> C1[Email Worker]
    Q2 --> C2[Fraud Worker]
    Q3 --> C3[Analytics Worker]

Subscription filters are useful. You can publish several event types to one topic and let each subscription receive only what it cares about.

SNS fits these requirements:

• One event should notify many consumers.
• Publishers should not know subscribers.
• You are already in AWS.
• You pair it with SQS for durable per consumer queues.

SNS operating constraints:

• It is not a queue by itself.
• Replay is not the normal model.
• Delivery behavior depends on subscriber type.
• Complex event choreography can become hard to trace.
• You are coupling to AWS.

SNS distributes a published message to subscriptions. An SQS subscription can retain its copy until a worker processes it.

Redis Streams

Redis Streams sits between a Redis queue and a lightweight event stream. It is not Kafka, but it is more structured than pushing jobs into a Redis list.

Producers append entries to a stream. Entries get IDs. Consumers can read directly, or join consumer groups. Redis tracks pending messages that were delivered but not acknowledged. If a consumer dies, another consumer can inspect and claim old pending messages.

Redis Streams fits these requirements:

• Redis already exists.
• You want consumer groups without Kafka.
• Retention needs are modest.
• You want a light stream before committing to bigger infrastructure.

Where it bites:

• Retention uses Redis memory.
• Operational limits are Redis limits.
• The ecosystem is smaller than Kafka's.
• Huge historical replay is not its strength.

I see Redis Streams as a useful middle step. It is good when Kafka would be too much, but a plain queue is not enough.

Google Pub/Sub

Google Pub/Sub is the GCP managed version of event delivery for many teams. Publishers write to topics. Subscriptions receive from topics. Each subscription has its own delivery state, so multiple subscribers can consume the same topic independently.

Subscriptions can be pull based or push based. Messages need acknowledgement. If a message is not acknowledged before the deadline, it can be delivered again. Ordering is available with ordering keys, but you have to design for it.

Google Pub/Sub fits these requirements:

• You are on GCP.
• You want managed event delivery.
• You need independent subscriptions.
• Push or pull delivery is useful.
• You do not want to operate Kafka.

Where it bites:

• It is cloud provider coupling.
• At least once delivery means idempotency still matters.
• Ordering requires explicit key design.
• It does not feel like RabbitMQ topology.

If AWS has SNS plus SQS, GCP teams often reach for Pub/Sub as the default event backbone.

Azure Service Bus

Azure Service Bus is the Azure managed broker. It has queues, topics, subscriptions, message locks, sessions, duplicate detection, and dead lettering.

The receive behavior is similar in spirit to other reliable queues. A consumer receives and locks a message. If it completes the message, the message is removed. If the lock expires before completion, the message can be delivered again.

Sessions group messages with the same session ID, which helps when you need ordered handling per account, per customer, or per workflow.

Azure Service Bus fits these requirements:

• You are already on Azure.
• You need managed queues and topics.
• Sessions help your ordering model.
• Dead lettering and duplicate detection are important.
• You want more broker features than a basic queue.

Where it bites:

• Azure coupling is strong.
• Throughput and feature tiers need planning.
• It is not Kafka style replay.
• It has more concepts than a simple job queue.

For Azure heavy companies, this can be the practical answer, even if it gets less hype than Kafka.

NATS

NATS provides messaging between services with routing by subject.

Core NATS is subject based pub/sub. Publishers send messages to subjects like payments.created. Subscribers listen to exact subjects or wildcard patterns. Core NATS is not mainly a durable storage system.

JetStream adds persistence, streams, consumers, acknowledgements, replay, and retention. That moves NATS into a more durable messaging and streaming space while keeping a smaller feel than Kafka in many deployments.

NATS fits these requirements:

• You need fast service to service messaging.
• You like simple subject based routing.
• You want a small operational footprint.
• JetStream gives enough durability for your case.

Where it bites:

• Core NATS is not durable by default.
• JetStream adds concepts you need to learn.
• Kafka has a larger stream processing ecosystem.
• It is less common in typical web app stacks.

NATS fits systems that need lightweight subject based messaging. It is not a default for every workload.

Beanstalkd and Gearman

Beanstalkd and Gearman are older job systems. They represent a simpler era of background processing, so I would not usually pick them for a new critical system today.

Beanstalkd has tubes. Producers put jobs into tubes. Workers reserve jobs, process them, then delete, release, bury, or touch them. Reserved jobs have a time to run. If the worker does not finish, the job can return.

Gearman is more like function dispatch. A client submits work for a named function, and workers registered for that function execute it.

These tools fit these requirements:

• You maintain an existing system that already uses them.
• The workload is simple and low risk.
• You need a tiny job server.
• Everyone understands the durability limits.

Operating constraints:

• Smaller modern ecosystem.
• Less managed service support.
• Fewer observability and routing features.
• Easier to outgrow than SQS, BullMQ, RabbitMQ, or Kafka.

I would keep them if they are stable and boring. I would be cautious about introducing them fresh.

Temporal

Temporal manages workflows. Teams often consider it when their queue workers must coordinate several steps, retries, and waits.

Temporal stores workflow history durably. The workflow code describes the process. Activities perform side effects like calling APIs or sending emails. Timers, retries, signals, and long running state are first class concepts.

The big difference is that Temporal remembers the process. A worker can crash, come back, and continue. A workflow can sleep for days. A human approval can arrive later as a signal. You do not have to hide all of that state in job payloads and custom tables.

Temporal fits these requirements:

• The process lasts minutes, hours, or days.
• The process has multiple business steps.
• You need durable timers.
• Retrying requires state.
• You are building a workflow rather than running a single job.

Where it bites:

• It is not a simple queue replacement.
• Deterministic workflow rules take learning.
• It adds conceptual and operational weight.
• Tiny one step jobs do not need it.

My rule is simple: if the worker starts becoming a state machine, I look at Temporal.

The Mistakes I Watch For

The first mistake is choosing Kafka because it sounds scalable. Kafka is excellent when replay and event history matter. It is heavy when you only need to send emails after signup.

The second mistake is pretending at least once delivery means exactly once behavior. Most of these systems can deliver a message more than once. Consumers need idempotency keys, safe database writes, and careful external API calls.

The third mistake is forgetting dead letter queues. Retries are good until a poison message retries forever. Failed messages need a place to go and a way to be inspected.

The fourth mistake is saying "we need ordering" without saying the scope. Global ordering is expensive. Per account ordering is realistic. Per user ordering is common. No ordering is often fine.

A queue can absorb a temporary increase in traffic. It cannot compensate indefinitely for insufficient consumer capacity. If producers create 10,000 messages per second and consumers process 1,000, the backlog continues to grow.

The metrics I want on any serious queue are queue depth, oldest message age, processing rate, retry count, dead letter count, and worker error rate.

My Defaults

For a normal web app, I start with PgBoss if Postgres is already there and the workload is moderate. It keeps the architecture small.

If Redis is already reliable production infrastructure and the app needs fast background jobs, I reach for BullMQ.

For AWS service decoupling, SQS is my default. I add SNS when fanout appears.

For classic broker routing between services, RabbitMQ is still very reasonable.

For Java enterprise systems, I consider ActiveMQ or Artemis. For regulated enterprise integration, I consider IBM MQ. Existing operational experience can justify these choices.

For replayable business events and high throughput pipelines, Kafka is the serious option, but only after accepting the operational cost.

For business workflows that need persistent state across multiple steps, I evaluate Temporal.

Summary Table