Fetching data: pages, timeouts and retries

A practical refresher. Read the contract, trace the examples, and attempt the exercise before opening its answer. Code blocks may share context; only ts check blocks are standalone checks.

HTTP failure is not necessarily rejection

fetch can resolve normally for 404, 429 or 500. Check response.ok or status, then parse the body and validate it as a separate step.

A timeout must cancel or otherwise bound the operation; Promise.race alone only stops waiting and leaves the losing work running.

Pass an AbortSignal to supported operations and clean up timers in finally. The examples are patterns, not a universal production retry policy.

async function fetchJson(url: string, timeoutMs: number): Promise<unknown> {
  if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) {
    throw new RangeError("timeoutMs must be positive and finite");
  }
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);
  try {
    const response = await fetch(url, { signal: controller.signal });
    if (!response.ok) {
      await response.body?.cancel();
      throw new Error(`HTTP ${response.status}`);
    }
    const value: unknown = await response.json();
    return value;
  } finally {
    clearTimeout(timer);
  }
}

Which failures to retry

Retry only failures that the operation contract treats as transient and safe. Typical candidates include selected network failures, 429 and selected 5xx responses; validation errors and most 4xx responses usually need a changed request.

Use bounded attempts, a total time budget, exponential backoff and jitter; consider Retry-After.

A retry can duplicate a write whose response was lost, so idempotency belongs in the design. Avoid multiplying retries across layers.

const sleep = (ms: number) =>
  new Promise<void>(resolve => setTimeout(resolve, ms));

async function retry<T>(
  operation: () => Promise<T>,
  attempts: number,
  shouldRetry: (error: unknown) => boolean,
): Promise<T> {
  if (!Number.isInteger(attempts) || attempts < 1) {
    throw new RangeError("attempts must be a positive integer");
  }
  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === attempts - 1 || !shouldRetry(error)) throw error;
      const ceiling = Math.min(5000, 200 * 2 ** attempt);
      await sleep(Math.random() * ceiling);
    }
  }
  throw new Error("Unreachable");
}

Cursor pagination

Read a page, process it, then advance to its next cursor. Detect repeated cursors instead of looping forever.

Cursor semantics and snapshot consistency come from the API; deduplication cannot recover records never delivered.

Decide whether individual page failures stop the import, retry that page or produce a resumable report. Do not advance the checkpoint before the page has been durably processed.

type Page<T> = { items: T[]; nextCursor: string | null };

async function* pages<T>(
  load: (cursor: string | undefined) => Promise<Page<T>>,
): AsyncGenerator<T[]> {
  let cursor: string | undefined;
  const seen = new Set<string>();
  while (true) {
    const page = await load(cursor);
    yield page.items;
    if (page.nextCursor === null) return;
    if (seen.has(page.nextCursor)) throw new Error("Cursor cycle");
    seen.add(page.nextCursor);
    cursor = page.nextCursor;
  }
}

for await (const batch of pages(loadPage)) {
  await mapLimit(batch, 8, processRecord);
}

Checkpoint and backpressure

Limit processing per page and persist the cursor after successful processing. This bounds memory without retaining the entire dataset.

In a stream, consumers must slow producers or buffer only within a declared bound.

A checkpoint should correspond to completed durable effects, not merely received data. Recovery must tolerate replay of already-completed work.

Practice

An API returns an endless repeated cursor and sometimes 429. What state should the importer keep, which failures are retried, and when is the checkpoint written?

Official references

• Node.js: stream