I've been working with PgBoss for a while now at work, and at some point I started wondering: how does this thing actually work under the hood? And how does it compare to BullMQ, which seems to be the other popular choice in the Node.js ecosystem?
So I went and studied both. What I found is that they solve the same fundamental problem (reliable job processing) with completely different architectural philosophies. One leans on PostgreSQL, the other on Redis. The tradeoffs are fascinating.
The Problem Both Solve
You have work that needs to happen asynchronously. Send an email after a user signs up. Process a payment reconciliation file. Generate a report. Resize an uploaded image. Whatever it is, you don't want to do it in the request/response cycle.
You need:
- Reliability: jobs shouldn't disappear if a worker crashes
- Concurrency: multiple workers should process jobs in parallel without duplicating work
- Scheduling: some jobs should run at a specific time or repeat on an interval
- Retry logic: failed jobs should be retried with backoff
- Ordering: some jobs need to be processed in order (per customer, per account, etc.)
Both PgBoss and BullMQ provide all of this. The difference is how.
PgBoss: PostgreSQL as a Job Queue
PgBoss stores jobs in a PostgreSQL table. That's it. No additional infrastructure. If you already have a Postgres database (and you probably do), you already have the infrastructure for PgBoss.
How It Works
graph LR
P[Producer] -->|INSERT| T[(pgboss.job table)]
T -->|SELECT ... FOR UPDATE SKIP LOCKED| W1[Worker 1]
T -->|SELECT ... FOR UPDATE SKIP LOCKED| W2[Worker 2]
T -->|SELECT ... FOR UPDATE SKIP LOCKED| W3[Worker 3]
Job creation is just an INSERT:
import PgBoss from "pg-boss";
const boss = new PgBoss("postgres://localhost/mydb");
await boss.start();
// Create a job
await boss.send("send-email", {
to: "[email protected]",
subject: "Welcome!",
body: "Thanks for signing up.",
});
Under the hood, PgBoss inserts a row into its job table with the queue name, payload, state, and scheduling metadata.
Job fetching is where it gets interesting. PgBoss uses SELECT ... FOR UPDATE SKIP LOCKED:
SELECT id, data
FROM pgboss.job
WHERE name = 'send-email'
AND state = 'created'
AND start_after <= now()
ORDER BY created_on
LIMIT 1
FOR UPDATE SKIP LOCKED;
This is the magic. FOR UPDATE locks the row so no other worker can grab it. SKIP LOCKED means if a row is already locked by another worker, skip it and grab the next one. No blocking, no contention, no duplicate processing.
sequenceDiagram
participant W1 as Worker 1
participant DB as PostgreSQL
participant W2 as Worker 2
W1->>DB: SELECT ... FOR UPDATE SKIP LOCKED
Note over DB: Row 1 locked by W1
W2->>DB: SELECT ... FOR UPDATE SKIP LOCKED
Note over DB: Row 1 skipped (locked), Row 2 returned
W1->>DB: UPDATE state = 'completed'
W1->>DB: COMMIT (lock released)
W2->>DB: UPDATE state = 'completed'
W2->>DB: COMMIT (lock released)
Job completion updates the row's state to completed (or failed) and commits the transaction, releasing the lock.
PgBoss Internals
A few things happening behind the scenes that are worth understanding:
Maintenance cycle: PgBoss runs periodic maintenance to archive completed jobs, expire timed-out jobs, and handle retries. This is a background process within the PgBoss instance.
Advisory locks for coordination: PgBoss uses PostgreSQL advisory locks to ensure only one instance runs maintenance at a time, even across multiple application servers. When PgBoss starts, it acquires an advisory lock to claim responsibility for the maintenance cycle.
// Simplified version of what PgBoss does internally
await db.query("SELECT pg_try_advisory_lock($1)", [MAINTENANCE_LOCK_KEY]);
Exponential backoff: Failed jobs get retried with configurable backoff. The retry_delay and retry_count columns in the job table track this.
Job throttling and rate limiting: PgBoss supports throttling (max N jobs per queue per interval) using database queries to count recent jobs.
Cron scheduling: Repeating jobs use cron expressions. PgBoss checks on each maintenance cycle whether a scheduled job needs to create its next instance.
The Catch
PgBoss's throughput is limited by PostgreSQL's write throughput. For most applications, this is more than enough. But if you're pushing tens of thousands of jobs per second, you'll start feeling the pain of:
- Write amplification (every job insert gets written to the WAL, PostgreSQL's Write-Ahead Log, which is how it guarantees durability by recording every change to disk before confirming the write)
- Table bloat from completed jobs (mitigated by archiving, but still)
- Vacuum pressure from frequent updates to job state
- Connection pool pressure from long-polling workers
BullMQ: Redis as a Job Queue
BullMQ takes a completely different approach. It uses Redis as its data store, leveraging Redis's in-memory speed and atomic operations.
How It Works
graph LR
P[Producer] -->|LPUSH| WQ[Wait Queue<br/>Redis List]
WQ -->|BRPOPLPUSH| AL[Active List<br/>Redis List]
AL --> W1[Worker 1]
AL --> W2[Worker 2]
W1 -->|LREM + ZADD| CQ[Completed Set<br/>Redis Sorted Set]
BullMQ uses multiple Redis data structures working together:
Wait queue (Redis List): new jobs are pushed here Active list (Redis List): jobs currently being processed Delayed set (Redis Sorted Set): jobs scheduled for the future, scored by their execution time Completed/Failed sets (Redis Sorted Sets): finished jobs for tracking
Job creation:
import { Queue, Worker } from "bullmq";
const queue = new Queue("send-email", {
connection: { host: "localhost", port: 6379 },
});
// Create a job
await queue.add("welcome", {
to: "[email protected]",
subject: "Welcome!",
body: "Thanks for signing up.",
});
Under the hood, BullMQ runs a Lua script on Redis that atomically:
- Creates a hash with the job data
- Pushes the job ID onto the wait list
- Publishes an event for waiting workers
Job fetching uses BRPOPLPUSH (or its modern equivalent BLMOVE):
BRPOPLPUSH bull:send-email:wait bull:send-email:active 5
This atomically pops a job from the wait list and pushes it onto the active list. The B prefix means it blocks until a job is available (with a timeout). This is how BullMQ achieves near-zero latency between job creation and pickup.
Lua scripts for atomicity: BullMQ uses Redis Lua scripts extensively to ensure multi-step operations are atomic. Moving a job from active to completed, handling retries, managing rate limits, all happen in single atomic Lua script executions.
BullMQ Internals
Stalled job detection: If a worker crashes while processing a job, the job stays in the active list forever. BullMQ runs a "stalled job checker" that looks at the active list and checks if the worker holding each job is still alive (via lock renewal). If a worker hasn't renewed its lock, the job is moved back to the wait list.
sequenceDiagram
participant W as Worker
participant R as Redis
participant SC as Stall Checker
W->>R: Move job to active list
W->>R: Set lock (expire 30s)
loop Every 15s
W->>R: Renew lock
end
Note over W: Worker crashes!
SC->>R: Check active list
SC->>R: Lock expired?
SC->>R: Move job back to wait list
Rate limiting: BullMQ supports rate limiting natively using Redis's atomic increment + expire pattern. You can say "process at most 10 jobs per minute" and BullMQ handles the coordination across all workers.
Job priorities: Jobs can have priorities (1 through 2^21). BullMQ uses a combination of Redis lists and sorted sets to ensure higher-priority jobs are picked up first.
Flows and dependencies: BullMQ supports parent-child job relationships. A parent job can wait for all child jobs to complete before being processed itself. This is built using Redis sets to track dependencies.
Architecture Comparison
| Aspect | PgBoss | BullMQ |
|---|---|---|
| Data store | PostgreSQL | Redis |
| Job persistence | Disk (WAL + fsync) | In-memory (+ optional AOF/RDB) |
| Fetch mechanism | FOR UPDATE SKIP LOCKED | BRPOPLPUSH / Lua scripts |
| Latency | ~10-50ms (polling interval) | ~1-5ms (blocking pop) |
| Throughput ceiling | ~1,000-5,000 jobs/sec | ~10,000-50,000+ jobs/sec |
| Durability | Strong (ACID transactions) | Depends on Redis persistence config |
| Extra infrastructure | None (uses existing Postgres) | Requires Redis |
| Atomicity | SQL transactions | Lua scripts |
| Coordination | Advisory locks | Redis locks + Lua |
| Stalled job handling | Expiry-based (maintenance cycle) | Lock renewal + stall checker |
| FIFO guarantee | Per-queue ordering via ORDER BY | Per-queue via list ordering |
Note on throughput numbers: these are rough ballpark figures, not benchmarks. Actual throughput depends heavily on hardware, payload size, and configuration. Measure your own workload.
The Key Tradeoff: Simplicity vs Speed
graph TD
A[Choose Your Job Queue] --> B{Already have Postgres?}
B -->|Yes| C{Need >5K jobs/sec?}
C -->|No| D[PgBoss<br/>Simpler, no new infra]
C -->|Yes| E{Can add Redis?}
E -->|Yes| F[BullMQ<br/>Higher throughput]
E -->|No| G[PgBoss + optimize<br/>Batch inserts, partitioning]
B -->|No| H{Already have Redis?}
H -->|Yes| F
H -->|No| I[Pick based on what<br/>you're willing to operate]
Choose PgBoss when:
- You already have PostgreSQL and don't want to add Redis
- Your job volume is moderate (most apps fall here)
- You need transactional job creation: enqueue a job only if the parent transaction commits
- You value operational simplicity over raw throughput
- You want your jobs to survive by default (PostgreSQL durability)
Choose BullMQ when:
- You already have Redis
- You need high throughput or low latency
- You need advanced features like job priorities, flows, or rate limiting out of the box
- Your team is comfortable operating Redis
- Sub-10ms job pickup latency matters to your use case
The Hidden Superpower of PgBoss: Transactional Enqueue
This is the thing that doesn't get talked about enough. With PgBoss, you can enqueue a job inside the same database transaction as your business logic:
await db.query("BEGIN");
// Create the order
await db.query("INSERT INTO orders (id, status) VALUES ($1, 'pending')", [orderId]);
// Enqueue the processing job in the SAME transaction
await boss.send("process-order", { orderId }, { db: existingTransaction });
await db.query("COMMIT");
// Both the order AND the job are created atomically
If the transaction rolls back, the job is never created. No orphaned jobs, no phantom processing. This is impossible with BullMQ because the job store (Redis) is a completely separate system from your database.
With BullMQ, you'd have to handle this with patterns like the outbox pattern: write to an outbox table in your database transaction, then have a separate process poll the outbox and push to Redis. More moving parts, more things that can go wrong.
What I've Learned From Using PgBoss
Working with PgBoss at Xendit on the NEX product, a few practical lessons:
1. Monitor your job table size. PgBoss archives completed jobs, but if your archive table grows unbounded, your database will bloat. Set up retention policies.
2. Watch your connection pool. Each PgBoss worker holds a connection while polling. If you have many workers across many queue names, you can exhaust your connection pool fast.
3. The maintenance cycle matters. PgBoss's maintenance (expiring stalled jobs, scheduling cron jobs) runs on an interval. If your PgBoss instance is slow or overloaded, maintenance falls behind and stalled jobs take longer to recover.
4. Upgrading PgBoss major versions is non-trivial. The API surface changes between versions, and the internal table schema evolves. I spent a lot of time thinking about how to upgrade without downtime. The migration strategy matters.
5. Transaction-scoped locks are your friend. PgBoss's internal use of advisory locks for maintenance coordination is elegant. Understanding how this works (I wrote a separate post on advisory locks) helps you debug weird behavior.
Recap
A quick summary of everything covered:
PgBoss:
- Uses PostgreSQL as its job store (no extra infrastructure)
- Fetches jobs with
SELECT ... FOR UPDATE SKIP LOCKED(no blocking, no duplicate processing) - Uses advisory locks to coordinate maintenance across instances
- Supports transactional enqueue (enqueue a job inside the same DB transaction as your business logic)
- Throughput ceiling is lower (bound by PostgreSQL write speed)
- Great for moderate job volumes where operational simplicity matters
BullMQ:
- Uses Redis as its job store (in-memory, fast)
- Fetches jobs with
BRPOPLPUSH/BLMOVE(blocking pop, near-zero latency) - Uses Lua scripts for atomic multi-step operations
- Uses lock renewal + stall checker for crashed worker recovery
- Much higher throughput ceiling
- Advanced features out of the box: priorities, flows, rate limiting
When to pick which:
- Already have Postgres, no Redis, moderate volume? PgBoss
- Already have Redis, need speed or advanced features? BullMQ
- Need transactional enqueue (job only created if parent transaction commits)? PgBoss
- Need sub-10ms job pickup latency? BullMQ
- Don't want to add new infrastructure? PgBoss
Closing Thoughts
Neither PgBoss nor BullMQ is "better." They're different tools optimized for different constraints. PgBoss trades throughput for simplicity and transactional guarantees. BullMQ trades operational complexity for speed and features.
If someone asks me which to use, my first question is always: "Do you already have Redis in your stack?" If yes, BullMQ is a natural fit. If no, PgBoss lets you get reliable job processing without adding another piece of infrastructure to operate. And in my experience, the best infrastructure is the infrastructure you don't have to add.