What is a job queue, and why does my API need one?

Part 1 of 10: Reliable background jobs in Node.js, from zero.

By Sandro Dezerio, senior fullstack engineer.

The signup scenario in this post is made up. The timings and failure rates come from a simulation with stated assumptions, not from a real email provider.

Most APIs start the same way. A user signs up, you save them in the database, and you send a welcome email before returning a response. It works on your laptop. It keeps working for a while in production.

Then the email provider gets slow for an afternoon, and your signup page gets slow with it.

This post builds the smallest useful fix: a job queue in about 40 lines of TypeScript. It is the first step in a series that starts here and ends with reliable, multi-step workflows, including ones that call an LLM.

The problem

Here is the handler, in plain terms:

1. Save the user (fast)
2. Send the welcome email (slow, and it can fail)
3. Return “ok” to the user

Two things are wrong with this.

The user waits for work they don’t care about. They asked to sign up. They did not ask to wait for an email to leave your system.

A failure in a side task becomes a failure in the main task. If the email call times out, the handler throws, and the user sees an error. Their account exists, but they think it doesn’t. They try again and hit “email already in use”.

I ran into this myself building a PDF export feature: generating the file took long enough that the request routinely timed out, and the UI just sat there frozen waiting for an answer nobody needed right away. Moving the generation into a background job was the actual fix. The request returned immediately, and the UI picked up the finished file once it was ready, instead of holding the whole page hostage for a slow report.

The idea

Split the work into two parts:

• The request does the part the user needs: save the user, answer “ok”.
• Everything else goes into a queue, a line of jobs that a worker processes in the background.

The handler adds a job and returns straight away. The worker takes jobs from the line, one at a time, and does the slow work.

A queue in about 40 lines

export type Job<T> = { id: number; data: T };
export type Handler<T> = (job: Job<T>) => Promise<void>;

export class SimpleQueue<T> {
  private jobs: Job<T>[] = [];
  private nextId = 1;
  private running = false;
  private idleWaiters: Array<() => void> = [];
  readonly failed: Array<{ job: Job<T>; error: unknown }> = [];

  constructor(private handler: Handler<T>) {}

  /** Put a job in line and return right away. The work happens later. */
  add(data: T): number {
    const job = { id: this.nextId++, data };
    this.jobs.push(job);
    void this.drain(); // start the worker if it is not already running
    return job.id;
  }

  /** The worker: take jobs one at a time until the line is empty. */
  private async drain(): Promise<void> {
    if (this.running) return;
    this.running = true;
    while (this.jobs.length > 0) {
      const job = this.jobs.shift()!;
      try {
        await this.handler(job);
      } catch (error) {
        this.failed.push({ job, error }); // remember it, keep going
      }
    }
    this.running = false;
    this.idleWaiters.splice(0).forEach((resolve) => resolve());
  }

  /** Resolves when every job added so far has been handled. */
  onIdle(): Promise<void> {
    if (!this.running && this.jobs.length === 0) return Promise.resolve();
    return new Promise((resolve) => this.idleWaiters.push(resolve));
  }
}

export type Job<T> = { id: number; data: T };
export type Handler<T> = (job: Job<T>) => Promise<void>;

export class SimpleQueue<T> {
  private jobs: Job<T>[] = [];
  private nextId = 1;
  private running = false;
  private idleWaiters: Array<() => void> = [];
  readonly failed: Array<{ job: Job<T>; error: unknown }> = [];

  constructor(private handler: Handler<T>) {}

  /** Put a job in line and return right away. The work happens later. */
  add(data: T): number {
    const job = { id: this.nextId++, data };
    this.jobs.push(job);
    void this.drain(); // start the worker if it is not already running
    return job.id;
  }

  /** The worker: take jobs one at a time until the line is empty. */
  private async drain(): Promise<void> {
    if (this.running) return;
    this.running = true;
    while (this.jobs.length > 0) {
      const job = this.jobs.shift()!;
      try {
        await this.handler(job);
      } catch (error) {
        this.failed.push({ job, error }); // remember it, keep going
      }
    }
    this.running = false;
    this.idleWaiters.splice(0).forEach((resolve) => resolve());
  }

  /** Resolves when every job added so far has been handled. */
  onIdle(): Promise<void> {
    if (!this.running && this.jobs.length === 0) return Promise.resolve();
    return new Promise((resolve) => this.idleWaiters.push(resolve));
  }
}

There are three ideas in it:

• add puts the job at the end of the line and returns. It never waits for the work.
• drain is the worker. It runs until the line is empty, and only one drain runs at a time.
• If a job throws, the queue records it in failed and moves on. One bad job does not stop the rest.

Does it help? A simulation

I simulated a signup endpoint with these assumptions. All of them are illustrative:

• Saving the user takes 20 ms.
• The email provider takes 300 ms to answer.
• 10% of email calls fail with a timeout.
• 20 signups arrive one after another.

The simulation runs on a virtual clock and a seeded random generator, so it gives the same numbers every time. The code for the simulation is not included in this post.

The response time drops because the user no longer waits for the email. The errors disappear from the signup path because a failed email can no longer fail a signup.

The emails still fail, though. In the queued version, 2 emails failed in the background, and nothing retries them yet. The problem did not go away. It moved to a place where it can be handled without hurting the user.

What this queue does not do

This is a teaching tool, and its limits are the plan for the rest of the series:

• It runs one job at a time. After the last signup, the worker needed about 5.6 more seconds to send 20 emails. Part 2 covers concurrency.
• It never retries. A failed job is only recorded. Part 3 covers retries with backoff.
• It lives in memory. If the process restarts, every job still in line is lost. Part 6 moves the queue to Redis with BullMQ.
• It cannot run a job twice yet, but retries will change that. Once failed jobs are retried, the same job can run more than once. Part 5 covers idempotency.

In production you will usually reach for a library such as BullMQ instead of writing your own. Building the small version first makes it clear what the library is doing for you.

Takeaways

1. Do only what the user needs inside the request. Queue the rest.
2. A queue separates a slow or failing side task from the main task.
3. A queue does not remove failures. It gives you a place to handle them.

Try it

Save the queue from above as queue.ts. Next to it, save this as example.ts. It is a signup that saves the user, adds the welcome email to the queue, and answers right away:

import { SimpleQueue } from "./queue.ts";

const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

const emailQueue = new SimpleQueue<{ email: string }>(async (job) => {
  await sleep(300); // pretend to call the email provider
  console.log(`  [worker] welcome email sent to ${job.data.email}`);
});

async function signup(email: string) {
  await sleep(20); // pretend to save the user
  emailQueue.add({ email }); // returns immediately
  console.log(`signup ok: ${email}`);
}

async function main() {
  await signup("ana@example.com");
  await signup("luis@example.com");
  await signup("sofia@example.com");
  console.log("all signups answered, emails still on their way");
  await emailQueue.onIdle();
  console.log("queue is empty");
}

main();

import { SimpleQueue } from "./queue.ts";

const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

const emailQueue = new SimpleQueue<{ email: string }>(async (job) => {
  await sleep(300); // pretend to call the email provider
  console.log(`  [worker] welcome email sent to ${job.data.email}`);
});

async function signup(email: string) {
  await sleep(20); // pretend to save the user
  emailQueue.add({ email }); // returns immediately
  console.log(`signup ok: ${email}`);
}

async function main() {
  await signup("ana@example.com");
  await signup("luis@example.com");
  await signup("sofia@example.com");
  console.log("all signups answered, emails still on their way");
  await emailQueue.onIdle();
  console.log("queue is empty");
}

main();

Run it:

pnpm dlx tsx example.ts

pnpm dlx tsx example.ts

You should see all three signups answered first, and the emails go out afterwards, one at a time:

signup ok: ana@example.com
signup ok: luis@example.com
signup ok: sofia@example.com
all signups answered, emails still on their way
  [worker] welcome email sent to ana@example.com
  [worker] welcome email sent to luis@example.com
  [worker] welcome email sent to sofia@example.com
queue is empty

signup ok: ana@example.com
signup ok: luis@example.com
signup ok: sofia@example.com
all signups answered, emails still on their way
  [worker] welcome email sent to ana@example.com
  [worker] welcome email sent to luis@example.com
  [worker] welcome email sent to sofia@example.com
queue is empty

Try making the handler throw for one email. The other two still go out, and the failed job ends up in queue.failed.

The simulation behind the table is not part of this post. A more complete implementation, with the simulation, tests and the features from the next parts, will come later in the series.

Next: Part 2, “How many jobs should a Node.js queue run at once?”