Part 4 of 10: Reliable background jobs in Node.js, from zero.
By Sandro Dezerio, senior full-stack engineer.
The sign-up scenario is made up. It is the same one used in Part 1, Part 2 and Part 3. The failure rates come from a simulation with stated assumptions, not from real services.
A job with ten steps, where each step works 99% of the time, finishes about 90% of the time.
Nobody looks at a step that works 99% of the time and calls it unreliable. That is why the problem hides. Each step looks fine on its own, and the job as a whole fails about once in every ten runs.
In Part 3, every job was a single call: send one welcome email. Real jobs are rarely that small. This post looks at a job with several steps, shows what goes wrong when you retry it as one piece, and fixes it with the withRetry wrapper from Part 3, unchanged.
The job grows
Picture the sign-up job some time later. Sending the welcome email is no longer enough. The job now does ten things, in order:
- Save the user
- Create the billing customer
- Create the workspace
- Upload the default avatar
- Add the user to the CRM
- Subscribe the user to the newsletter
- Send the welcome email
- Notify the sales channel
- Record the analytics event
- Schedule the day-3 email
Each step calls something that can fail: a database, a payment provider, an email provider, an internal API. Assume each call fails 1% of the time, independently of the others. For a single call, 99% is a respectable number.
The arithmetic
Reliability multiplies. For the job to succeed, all ten steps must succeed, so the chance is 0.99 multiplied by itself ten times:
0.9910 = 0.904
About 96 jobs in every 1,000 fail somewhere along the way. With twenty steps, it would be about 18 in every 100.
To check this, and to see what each fix does, I wrote a simulation that runs 1,000,000 sign-up jobs through the queue from Part 2. Without retries, 90.398% of the jobs succeeded. The formula gives 90.438%, so the simulation and the math agree within sampling noise.
The failure rate is only half of the problem. Of the 96,024 jobs that failed, 85,902 failed after at least one step had already done its work. The user exists but has no workspace. The billing customer exists, but nobody got the welcome email. A failed job is often a half-finished job.
The obvious fix: retry the whole job
Part 3 already has a retry wrapper. The quickest fix is to wrap the whole job in withRetry, the same way Part 3 wrapped the email handler. With up to 3 attempts, the success rate jumps from 90.398% to 99.913%. That looks like a win, but the job now has a different problem.
When step 8 fails, the retry starts again from step 1. It saves the user again, creates the billing customer again, and sends the welcome email again. In the simulation, retrying the whole job did 462,891 steps a second time, and 29,564 users received the welcome email more than once. Nobody asked for either.
Some repeats are harmless, such as recording an analytics event twice. Others are not, such as a second billing customer for the same user.
The better fix: retry each step
The fix is to retry the step that failed, not the job. If step 8 fails, try step 8 again. Steps 1 to 7 already worked, so leave them alone.
The code adds one small file, workflow.ts. A job becomes a list of named steps, and the queue from Part 2 and the retry wrapper from Part 3 do not change:
import type { Handler } from "./queue.ts";
import { withRetry, type RetryOptions } from "./retry.ts";
/** One step of a job: a name and the work it does. */
export type Step<T> = { name: string; run: Handler<T> };
/** Run the steps in order. If one throws, the job stops there and the queue records the error. */
export function workflow<T>(steps: Step<T>[]): Handler<T> {
return async (job) => {
for (const step of steps) {
await step.run(job);
}
};
}
/** Give each step its own retries, so a failing step never repeats the steps before it. */
export function retryEachStep<T>(steps: Step<T>[], options: RetryOptions): Step<T>[] {
return steps.map((step) => ({ name: step.name, run: withRetry(step.run, options) }));
}workflow turns a list of steps into a normal handler, so the queue does not need to know that the job has steps. retryEachStep wraps each step with withRetry from Part 3. The two strategies differ only in where the wrapper goes:
- Whole job:
withRetry(workflow(steps), options) - Each step:
workflow(retryEachStep(steps, options))
Here is how a job with two steps looks:
import { SimpleQueue, type Job } from "./queue.ts";
import { retryEachStep, workflow, type Step } from "./workflow.ts";
const steps: Step<string>[] = [
{ name: "save the user", run: async (job: Job<string>) => console.log(`saving ${job.data}`) },
{ name: "send the welcome email", run: async (job: Job<string>) => console.log(`emailing ${job.data}`) },
];
const queue = new SimpleQueue<string>(
workflow(retryEachStep(steps, { maxAttempts: 3, baseDelayMs: 100, maxDelayMs: 2000, jitter: true })),
);
queue.add("ana@example.com");The three strategies side by side
These are the results for 1,000,000 simulated sign-ups, with up to 3 attempts in both retry strategies:
| Strategy | Succeeded | Failed jobs | Failed after some steps were done | Calls per job | Steps done again | Users with 2+ welcome emails |
|---|---|---|---|---|---|---|
| No retries | 90.398% | 96,024 | 85,902 | 9.56 | 0 | 0 |
| Retry the whole job | 99.913% | 867 | 865 | 10.57 | 462,891 | 29,564 |
| Retry each step | 99.999% | 9 | 8 | 10.1 | 0 | 0 |
Retrying each step beats retrying the whole job in every column. Only 9 jobs failed out of 1,000,000. No step ran twice. And it made fewer calls than retrying the whole job (10.1 per job compared with 10.57), because a retry repeats one call instead of several.
The success rates also match the formulas. Retrying the whole job needs one attempt in which all ten steps pass, so a job fails only when all three attempts fail: 1 – (1 – 0.9910)3 = 99.913%. Retrying each step needs each step to fail three times in a row, which happens once in a million calls: (1 – 0.013)10 = 99.999%.
Inngest, a tool for running multi-step workflows, uses the same idea: “Each step.run() has its own retry counter, so one failing step does not restart earlier completed steps.”
What retrying each step does not fix
A step that worked but did not say so. Sometimes the email provider sends the email, and the reply gets lost on the way back. The step sees a timeout and fails. A retry of that step sends the email again. I ran the simulation again with a lost reply on the email step 0.5% of the time:
| Strategy | Succeeded | Steps done again | Users with 2+ welcome emails |
|---|---|---|---|
| Retry each step, reply lost 0.5% of the time | 99.999% | 4,971 | 4,939 |
Retrying each step cannot tell the difference between “it did not happen” and “it happened, but I did not hear back.” That is the job of an idempotency key, which Part 5 covers.
The jobs that still fail. Of the 9 failed jobs, 8 stopped halfway through, with some steps done. Per-step retries make this rare, but not impossible, and queue.failed does not say which step failed or which steps already ran. Later parts cover where these jobs go (Part 7) and how to see what happened to each one (Part 8).
Failures that come together. The simulation assumes every call fails independently. Real failures are often correlated: if the email provider is down for five minutes, all three attempts fail. Treat these numbers as a way to build intuition, not as a prediction.
A restart. The queue still lives in memory. If the process dies after step 4, nothing remembers that steps 1 to 4 were done. Part 6 moves the queue to Redis.
Takeaways
- Reliability multiplies: ten steps at 99% give a job that succeeds about 90% of the time.
- Retry the step that failed, not the whole job, so the steps that worked do not run again.
- Per-step retries still repeat a step whose reply was lost, so side effects need idempotency keys.
Try it
This is a teaching version of multi-step jobs. A more complete implementation comes later in the series. Keep queue.ts from Part 2 and retry.ts from Part 3, and save the workflow.ts file from above next to them. Then save this as simulate.ts:
import { SimpleQueue, type Handler } from "./queue.ts";
import { withRetry, type RetryOptions } from "./retry.ts";
import { retryEachStep, workflow, type Step } from "./workflow.ts";
// Illustrative assumptions (not a real system):
const JOBS = 1_000_000; // signups to simulate
const P_FAIL = 0.01; // every call to a step fails 1% of the time, independently
const LOST_REPLY = 0.005; // table 2 only: the email goes out, but the reply is lost 0.5% of the time
const STEPS = [
"save the user", "create the billing customer", "create the workspace", "upload the default avatar",
"add to the CRM", "subscribe to the newsletter", "send the welcome email", "notify the sales channel",
"record the analytics event", "schedule the day-3 email",
];
const EMAIL = "send the welcome email";
// Up to 3 attempts. The simulation counts outcomes, not time, so it skips the waits.
const retry: RetryOptions = { maxAttempts: 3, baseDelayMs: 100, maxDelayMs: 2000, jitter: true, sleep: async () => {} };
// A small seeded random generator (mulberry32), so every run prints the same numbers.
function seededRandom(seed: number) {
let a = seed;
return () => {
a = (a + 0x6d2b79f5) | 0;
let t = Math.imul(a ^ (a >>> 15), 1 | a);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
// What each signup job carries: how many times each step really did its work.
type Signup = { done: Record<string, number> };
async function simulate(build: (steps: Step<Signup>[]) => Handler<Signup>, lostReply: number) {
const random = seededRandom(1);
let calls = 0;
const steps: Step<Signup>[] = STEPS.map((name) => ({
name,
run: async (job) => {
calls++;
if (random() < P_FAIL) throw new Error(`${name}: 503 service unavailable`);
job.data.done[name] = (job.data.done[name] ?? 0) + 1; // the side effect happened
if (name === EMAIL && random() < lostReply) throw new Error(`${name}: timeout, reply lost`);
},
}));
const queue = new SimpleQueue<Signup>(build(steps));
let repeatedSteps = 0;
let twoEmails = 0;
for (let start = 0; start < JOBS; start += 10_000) {
const batch: Signup[] = [];
for (let i = 0; i < 10_000; i++) {
const signup: Signup = { done: {} };
batch.push(signup);
queue.add(signup);
}
await queue.onIdle();
for (const signup of batch) {
for (const times of Object.values(signup.done)) repeatedSteps += times - 1;
if ((signup.done[EMAIL] ?? 0) > 1) twoEmails++;
}
}
const failed = queue.failed.length;
return {
succeeded: `${((100 * (JOBS - failed)) / JOBS).toFixed(3)}%`,
failed,
halfDone: queue.failed.filter(({ job }) => Object.keys(job.data.done).length > 0).length,
callsPerJob: Math.round((calls / JOBS) * 100) / 100,
repeatedSteps,
twoEmails,
};
}
const pct = (p: number) => `${(100 * p).toFixed(3)}%`;
const oneTry = (1 - P_FAIL) ** STEPS.length;
async function main() {
console.log(`1. ${JOBS.toLocaleString("en-US")} signups, ${STEPS.length} steps, each call fails ${P_FAIL * 100}% of the time`);
console.table([
{ strategy: "no retries", formula: pct(oneTry), ...(await simulate((s) => workflow(s), 0)) },
{ strategy: "retry the whole job", formula: pct(1 - (1 - oneTry) ** 3), ...(await simulate((s) => withRetry(workflow(s), retry), 0)) },
{ strategy: "retry each step", formula: pct((1 - P_FAIL ** 3) ** STEPS.length), ...(await simulate((s) => workflow(retryEachStep(s, retry)), 0)) },
]);
console.log(`2. Same, but the email provider loses the reply ${LOST_REPLY * 100}% of the time`);
console.table([
{ strategy: "retry each step", ...(await simulate((s) => workflow(retryEachStep(s, retry)), LOST_REPLY)) },
]);
}
main();Run it with Node.js 22, the version I tested:
pnpm dlx tsx simulate.tsIt runs 4,000,000 jobs, so it takes 15 to 20 seconds. The random generator is seeded, so you should see exactly these numbers:
1. 1,000,000 signups, 10 steps, each call fails 1% of the time
┌─────────┬───────────────────────┬───────────┬───────────┬────────┬──────────┬─────────────┬───────────────┬───────────┐
│ (index) │ strategy │ formula │ succeeded │ failed │ halfDone │ callsPerJob │ repeatedSteps │ twoEmails │
├─────────┼───────────────────────┼───────────┼───────────┼────────┼──────────┼─────────────┼───────────────┼───────────┤
│ 0 │ 'no retries' │ '90.438%' │ '90.398%' │ 96024 │ 85902 │ 9.56 │ 0 │ 0 │
│ 1 │ 'retry the whole job' │ '99.913%' │ '99.913%' │ 867 │ 865 │ 10.57 │ 462891 │ 29564 │
│ 2 │ 'retry each step' │ '99.999%' │ '99.999%' │ 9 │ 8 │ 10.1 │ 0 │ 0 │
└─────────┴───────────────────────┴───────────┴───────────┴────────┴──────────┴─────────────┴───────────────┴───────────┘
2. Same, but the email provider loses the reply 0.5% of the time
┌─────────┬───────────────────┬───────────┬────────┬──────────┬─────────────┬───────────────┬───────────┐
│ (index) │ strategy │ succeeded │ failed │ halfDone │ callsPerJob │ repeatedSteps │ twoEmails │
├─────────┼───────────────────┼───────────┼────────┼──────────┼─────────────┼───────────────┼───────────┤
│ 0 │ 'retry each step' │ '99.999%' │ 11 │ 10 │ 10.11 │ 4971 │ 4939 │
└─────────┴───────────────────┴───────────┴────────┴──────────┴─────────────┴───────────────┴───────────┘Change P_FAIL or add steps to STEPS and see how quickly the “no retries” row drops.
Previous: How to retry failed jobs in Node.js without making it worse (Part 3).
Next: Jobs that run twice: idempotency keys in Node.js (Part 5) looks at jobs that run twice and how idempotency keys stop the duplicates.
Leave a Reply