PgBoss vs BullMQ: Job Queue Architectures Under the Hood

I use PgBoss at work and wanted to understand its implementation. I compared it with BullMQ. Both are Node.js job queues, but one stores queue state in PostgreSQL and the other in Redis. The PgBoss 12.27.0 repository and BullMQ documentation describe the same broad goal, with different storage and coordination models.

The problem both solve

You have work that needs to happen asynchronously. Send an email after a user signs up. Process a payment reconciliation file. Generate a report. Resize an uploaded image. Whatever it is, you do not want to do it in the request/response cycle.

You need:

• Reliability: jobs should not disappear if a worker crashes
• Concurrency: multiple workers should process jobs in parallel without duplicating work
• Scheduling: some jobs should run at a specific time or repeat on an interval
• Retry logic: failed jobs should be retried with backoff
• Ordering: some jobs need to be processed in order (per customer, per account, etc.)

Both libraries address these concerns, but the guarantees and operating costs depend on their storage model and configuration.

PgBoss: PostgreSQL as a job queue

PgBoss stores jobs in a PostgreSQL table, so an application that already runs PostgreSQL does not need a separate queue store. The job table documentation shows the queue state, payload, retry, and retention fields.

How it works

graph LR
    P[Producer] -->|INSERT| T[(pgboss.job table)]
    T -->|SELECT ... FOR UPDATE SKIP LOCKED| W1[Worker 1]
    T -->|SELECT ... FOR UPDATE SKIP LOCKED| W2[Worker 2]
    T -->|SELECT ... FOR UPDATE SKIP LOCKED| W3[Worker 3]

Job creation is just an INSERT:

import { PgBoss } from "pg-boss";

const boss = new PgBoss("postgres://localhost/mydb");
await boss.start();

// Create a job
await boss.send("send-email", {
  to: "[email protected]",
  subject: "Welcome!",
  body: "Thanks for signing up.",
});

Internally, PgBoss inserts a row into its job table with the queue name, payload, state, and scheduling metadata.

Job fetching uses SELECT ... FOR UPDATE SKIP LOCKED on the default PostgreSQL backend. The following is a simplified claim query, not the full PgBoss statement:

SELECT id, data
FROM pgboss.job
WHERE name = 'send-email'
  AND state < 'active'
  AND start_after <= now()
ORDER BY priority DESC, created_on, id
LIMIT 1
FOR UPDATE SKIP LOCKED;

FOR UPDATE locks the selected rows for the transaction. PostgreSQL's SKIP LOCKED option lets a worker skip rows another transaction already holds, which is useful for queue consumers. The claim and state update still run in a transaction, and retry or recovery behavior is handled separately by PgBoss. See the PostgreSQL locking clause documentation.

sequenceDiagram
    participant W1 as Worker 1
    participant DB as PostgreSQL
    participant W2 as Worker 2
    W1->>DB: SELECT ... FOR UPDATE SKIP LOCKED
    Note over DB: Row 1 locked by W1
    W2->>DB: SELECT ... FOR UPDATE SKIP LOCKED
    Note over DB: Row 1 skipped (locked), Row 2 returned
    W1->>DB: UPDATE state = 'completed'
    W1->>DB: COMMIT (lock released)
    W2->>DB: UPDATE state = 'completed'
    W2->>DB: COMMIT (lock released)

Job completion updates the row's state to completed or failed and commits the transaction, releasing the row locks.

PgBoss internals

The main pieces to keep in mind are:

Maintenance and retention: PgBoss periodically supervises queues, expires timed out or heartbeat stale jobs, and removes rows according to the queue's retention settings. Completed jobs are kept in the job table until deleteAfterSeconds, rather than being copied to a separate archive table. The job options and job table definition document those fields.

Schema coordination: Current PgBoss uses a transaction scoped pg_advisory_xact_lock while creating or migrating its schema. That is different from the supervisor that checks queues. The database backend documentation describes this lock's role.

// Simplified schema-migration coordination
await db.query("SELECT pg_advisory_xact_lock($1)", [SCHEMA_LOCK_KEY]);

Exponential backoff: Failed jobs can be retried with configurable delay and backoff. The retry_delay, retry_backoff, and retry_count fields in the job table hold the relevant state.

Job policies and scheduling: Queue policies cover throttling, singleton behavior, priority, and concurrency. Repeating jobs use cron expressions and are evaluated by the scheduler described in the scheduling API.

Where PostgreSQL adds work

Every enqueue and state transition is a PostgreSQL write. The queue can share the transaction and backup tools of the application. Queue traffic also competes with application traffic for database resources. Watch for:

• WAL volume and disk I/O from inserts and state changes. PostgreSQL records changes in its Write Ahead Log, while commit durability still follows the database's WAL and synchronous commit settings.
• Retained rows can contribute to table and index growth. Retention and partitioning limit the scope.
• Vacuum pressure from frequent updates to job state
• Connection pool pressure from worker polling and batch size

This repository has no PgBoss benchmark with a specified version, hardware, topology, payload, and repeatable workload. I cannot assign a throughput limit from this material.

Measure enqueue rate, claim rate, completion rate, queue depth, and worker pickup latency on the target deployment. Also measure database CPU, WAL volume, lock waits, and vacuum activity.

BullMQ: Redis as a job queue

BullMQ stores queue state in Redis and coordinates queue operations with Redis data structures and atomic scripts. The BullMQ architecture guide describes the wait, prioritized, delayed, active, completed, and failed states.

How it works

graph LR
    P[Producer] -->|Lua add script| Q[(Redis job keys and wait list)]
    Q -->|Worker wait and moveToActive script| A[Active state]
    A --> W1[Worker 1]
    A --> W2[Worker 2]
    W1 -->|Lua finish script| F[(Completed or failed sets)]

BullMQ uses multiple Redis data structures working together:

• Wait queue (Redis list): new jobs are pushed here.
• Active state: jobs currently being processed.
• Delayed set (Redis sorted set): jobs scheduled for the future, scored by their execution time.
• Completed and failed sets: finished jobs retained for tracking, subject to the queue's cleanup options.

Job creation:

import { Queue, Worker } from "bullmq";

const queue = new Queue("send-email", {
  connection: { host: "localhost", port: 6379 },
});

// Create a job
await queue.add("welcome", {
  to: "[email protected]",
  subject: "Welcome!",
  body: "Thanks for signing up.",
});

Internally, BullMQ runs a Lua script on Redis that atomically:

1. Creates a hash with the job data
2. Pushes the job ID onto the wait list
3. Publishes an event for waiting workers

Job fetching has two parts. A worker waits on Redis until work may be available. The moveToActive script then promotes eligible delayed jobs. It atomically moves the next waiting or prioritized job to the active state and creates its lock. For BullMQ 5.81.3, the moveToActive script shows this sequence. Other versions can change the command and script details.

Lua scripts for atomicity: BullMQ uses Redis scripts to coordinate several state changes in one operation. These operations include job creation, activation, completion, and recovery. The scripts do not provide a transaction with a separate PostgreSQL database.

BullMQ internals

Stalled job detection: A worker holds a lock while processing a job and renews it periodically. If the lock expires, BullMQ's stalled job checker can move the job back to waiting so another worker can process it. The stalled jobs guide calls out the at least once consequence: a job may run again after a lost lock.

sequenceDiagram
    participant W as Worker
    participant R as Redis
    participant SC as Stall Checker
    W->>R: Move job to active state
    W->>R: Set lock (configured duration)
    loop Renew lock before expiry
        W->>R: Renew lock
    end
    Note over W: Worker crashes!
    SC->>R: Check active state
    SC->>R: Lock expired?
    SC->>R: Move job back to waiting

Rate limiting: BullMQ supports a queue level rate limiter. Its rate limiting guide notes that the limit is global for a queue, so multiple workers share the same cap.

Job priorities: Jobs can have priorities from 1 through 2,097,151, with lower numbers taking precedence. The prioritized jobs guide documents the ordering and its sorted set cost.

Flows and dependencies: BullMQ supports parent child job relationships. A parent can remain in waiting-children until all children finish, as described in the flows guide.

Architecture comparison

This repository has no comparable benchmark for either queue. To compare them, record the library and database versions, hardware, topology, and payload size. Specify enqueue and worker concurrency, batch settings, retries, and persistence settings. Document a procedure that another person can repeat.

Measure enqueue and completion rates, queue depth, CPU, and memory use. Also record saturation, persistence behavior, and latency percentiles for job pickup and completion. The comparison above describes behavior, without numerical performance claims.

Choosing a queue for the workload

graph TD
    A[Choose Your Job Queue] --> B{Already have Postgres?}
    B -->|Yes| C{Need a measured performance target?}
    C -->|No| D[PgBoss<br/>Reuse the database]
    C -->|Yes| E{Can add Redis and benchmark it?}
    E -->|Yes| F[BullMQ<br/>Compare measured results]
    E -->|No| G[PgBoss<br/>Tune and measure PostgreSQL]
    B -->|No| H{Already have Redis?}
    H -->|Yes| F
    H -->|No| I[Pick based on what<br/>you're willing to operate]

Choose PgBoss when:

• You already have PostgreSQL and do not want to add Redis
• You need transactional job creation: enqueue a job only if the parent transaction commits
• You want queue data governed by PostgreSQL's transaction and backup policy

Choose BullMQ when:

• You already have Redis
• You need features like job priorities, flows, rate limiting, or Redis backed scheduling
• Your team is comfortable operating Redis
• You have a workload benchmark that justifies the additional queue store

Transactional enqueue

PgBoss can enqueue a job inside the same database transaction as the business write. The official transaction adapter documentation shows the db option used with ORM transaction handles:

import { fromKnex } from "pg-boss";

await knex.transaction(async (trx) => {
  // Create the order in the application transaction
  await trx("orders").insert({ id: orderId, status: "pending" });

  // Enqueue the processing job in the same transaction
  await boss.send(
    "process-order",
    { orderId },
    { db: fromKnex(trx) },
  );
});

// Both the order and the job commit or roll back together

If the transaction rolls back, the job insert rolls back with it. BullMQ's queue API writes to Redis and does not join a PostgreSQL transaction. Applications that must coordinate both writes usually need an outbox or another explicit coordination method.

With an outbox, the application writes an event to PostgreSQL in the business transaction and a separate publisher pushes that event to BullMQ. That gives the application a clear recovery point, but it is a different design from PgBoss's single transaction.

What I have learned from using PgBoss

Working with PgBoss at Xendit on the NEX product, a few practical lessons:

• Monitor your job table size: completed jobs remain until their retention setting removes them. Use deleteAfterSeconds and queue partitioning where appropriate, then watch table size, indexes, and vacuum behavior.
• Watch your connection pool: worker fetches, completion writes, batch size, and optional LISTEN/NOTIFY listeners all use database connections. Size the pool for the actual worker and application concurrency rather than assuming one connection per queue.
• Track the supervisor cycle: PgBoss expires jobs, checks heartbeats, and drives scheduling on an interval. If the instance or database is overloaded, those checks can run late. The worker documentation and job options describe the polling and expiration settings.
• Plan upgrades: the API surface changes between versions, and the internal table schema evolves. I spent a lot of time thinking about how to upgrade without downtime. The migration strategy matters.
• Identify the lock you are debugging: the default job claim uses PostgreSQL row locks. Advisory locks are used for schema coordination, not as a documented maintenance leader. I wrote a separate post on advisory locks, but the two paths solve different problems.

How I would choose

Neither PgBoss nor BullMQ is universally better. PgBoss keeps queue state in PostgreSQL and can join a business transaction. BullMQ keeps queue state in Redis and provides Redis backed priorities, flows, rate limiting, and scheduling. Those are architectural differences, not benchmark results.

If Redis is already part of the stack, BullMQ may be the simpler operational choice for those features. If PostgreSQL is the system of record and enqueue must commit with a business write, PgBoss avoids a cross store handoff. If throughput or pickup latency is a requirement, run the workload specific experiment described above before choosing.