Capturing 10,000 URLs Without Melting Your Server

Capturing one screenshot is trivial. Capturing 10,000 across thousands of domains — without hitting rate limits, blowing your memory budget, or ending up with a folder full of half-rendered pages — is a different engineering problem entirely. Whether you're building a competitive monitoring tool, archiving marketing pages, or generating preview thumbnails for a directory site, you need a workflow built for volume.

This post walks through the architecture, code patterns, and gotchas of running a bulk screenshot API for multiple URLs at scale.

Why naive loops break at scale

The first instinct is usually a for loop over an array of URLs. It works for 50 URLs. At 5,000 you'll hit problems:

• Sequential execution is painfully slow. Even at 3 seconds per page, 5,000 URLs take over four hours.
• Unbounded concurrency crashes things. Spawning 5,000 parallel requests at once either melts your machine or gets you rate-limited.
• Transient failures compound. Without retries, a 1% failure rate means 50 missing screenshots and no easy way to find which ones.
• Memory leaks in headless browsers. If you're self-hosting Puppeteer or Playwright, a few hundred sequential captures will eat all your RAM.

The fix is a combination of bounded concurrency, retries with backoff, and offloading the actual rendering to a dedicated API.

Designing the bulk pipeline

A reliable bulk screenshot pipeline has four stages:

1. Input queue — a list of URLs with metadata (filename, format, viewport).
2. Worker pool — N parallel workers pulling jobs.
3. Screenshot service — the API doing the actual rendering.
4. Output sink — S3, local disk, or a CDN.

Pick your concurrency level

If you're using a hosted API like PxShot, your concurrency is bounded by your plan's rate limit, not your CPU. A typical sweet spot is 10–25 concurrent requests. If you're self-hosting headless Chrome, you'll be limited by RAM — assume roughly 300–500 MB per active page.

Use a queue, not an array

For one-off scripts an array with p-limit is fine. For production, push URLs into a real queue (BullMQ, SQS, or even a database table with a status column). This gives you:

• Crash-safe restarts
• Visibility into in-flight vs failed jobs
• The ability to re-run only failures

A working Node.js example

Here's a minimal pattern using p-limit against an HTTP screenshot API. Replace the endpoint with whichever service you use.

import pLimit from 'p-limit';
import fs from 'fs/promises';

const urls = JSON.parse(await fs.readFile('urls.json', 'utf8'));
const limit = pLimit(15);
const API_KEY = process.env.PXSHOT_KEY;

async function capture(url, attempt = 1) {
  const endpoint = `https://api.pxshot.dev/v1/screenshot?url=${encodeURIComponent(url)}&format=png&full_page=true`;
  try {
    const res = await fetch(endpoint, {
      headers: { Authorization: `Bearer ${API_KEY}` }
    });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    const buf = Buffer.from(await res.arrayBuffer());
    const name = url.replace(/[^a-z0-9]/gi, '_').slice(0, 80);
    await fs.writeFile(`out/${name}.png`, buf);
    return { url, ok: true };
  } catch (err) {
    if (attempt < 3) {
      await new Promise(r => setTimeout(r, 1000 * attempt));
      return capture(url, attempt + 1);
    }
    return { url, ok: false, error: err.message };
  }
}

const results = await Promise.all(urls.map(u => limit(() => capture(u))));
const failed = results.filter(r => !r.ok);
await fs.writeFile('failed.json', JSON.stringify(failed, null, 2));
console.log(`Done. ${results.length - failed.length} ok, ${failed.length} failed.`);

A few details worth noting:

• Concurrency of 15 — adjust based on your plan limits.
• Exponential backoff — 1s, 2s between retries.
• A failure log — so you can re-run only the broken URLs without re-rendering the rest.

Handling the messy real world

The code above assumes URLs that just work. In practice, a bulk job hits weird edge cases. Plan for them.

Slow-loading and SPA pages

Single-page apps often fire the load event before content renders. Most screenshot APIs let you pass a wait_until=networkidle or a custom delay. For unknown URLs, a 2–3 second buffer after networkidle catches most lazy-loaded content without ballooning your total runtime.

Cookie banners and consent walls

If your screenshots are full of GDPR popups, inject CSS to hide them. Many APIs (PxShot included) support a hide_selectors or custom_css parameter. A starter blocklist:

• [id*="cookie"], [class*="cookie"]
• [id*="consent"], [class*="consent"]
• .onetrust-pc-dark-filter, #onetrust-banner-sdk

Geo-blocked or login-walled pages

For pages requiring auth, look for APIs supporting custom headers, cookies, or session injection. For geo-restricted content, you'll want an API with regional endpoints.

Dedup and idempotency

If you re-run a bulk job, you don't want to re-capture pages that already succeeded. Use a content-addressable filename (hash of URL + viewport + format) and skip if it exists. This makes retries free.

Storage and naming conventions

At 10,000 files, a flat directory becomes a liability. Group output by:

• Date — screenshots/2024-11-14/ for time-series archives.
• Domain — screenshots/example.com/ for per-site monitoring.
• Job ID — screenshots/job_abc123/ for one-off runs.

Use a hash, not the URL, as the filename if URLs contain query strings or fragments. Store the URL-to-filename mapping in a small JSON or SQLite index alongside the files.

Cost math for bulk jobs

Before committing to a provider, run the numbers. A typical bulk screenshot API charges per successful capture. For a one-time job of 10,000 URLs at $0.002 per screenshot, you're looking at $20. For a monitoring system capturing 1,000 URLs daily, that's $60/month.

Compare that to self-hosting: a single VPS running Playwright at 10 concurrent pages handles maybe 50,000 captures per day, but you're also paying for engineering time, monitoring, browser updates, and failure handling. For most teams under 500k screenshots/month, hosted wins on total cost of ownership.

When to batch vs stream

Two patterns work for bulk jobs:

• Batch mode — submit all URLs, wait for all results, then process. Simpler code, but you can't start using results until everything finishes.
• Stream mode — write each result as it lands. Better for long jobs and dashboards showing live progress.

The Node example above is technically batch, but because we write each file inside capture(), it behaves like a stream. That's usually what you want.

Try it on your URL list

If you want to skip the infrastructure work, PxShot offers a free tier at pxshot.dev with enough monthly captures to test a real bulk workflow end-to-end. Drop your URL list into the script above, point it at your API key, and you'll have a folder of screenshots in minutes.