APIs, HTTP, JSON & Web Scraping

Branch: CS Foundations (Quant & Code region, Finance galaxy) · Node CS2.05 · ~16 focused hours · Mastery gate ≥ 85%

Every dataset you have touched so far in this program arrived as a tidy, already-solved problem: a CSV sitting in a folder, a table already loaded into a notebook, a price series someone else fetched for you. Out in the world, almost none of it starts that way. A real analyst's data begins its life behind a web API, a program on someone else's server that will hand you information only if you ask correctly, prove who you are, respect how often you're allowed to ask, and parse whatever shape of answer it decides to give you that day. Or it begins life worse: not exposed as an API at all, only visible as a number sitting inside a web page's HTML, meant for a human's eyes, that you must carefully and ethically extract. This module is about that entire boundary, the point where "data that exists somewhere on the internet" becomes "data sitting cleanly in your program", and about doing the crossing like an engineer: authenticated, rate-limited, resilient to failure, and honest about what you're allowed to touch.

There is one mechanical fact that shapes everything about how this module is taught, and it is worth stating up front rather than discovering by confusion three exercises in: the Python that runs live in your browser (Pyodide) cannot open a network socket. No live HTTP call you write here will ever leave your machine, not because of a permissions setting, but because the sandboxed runtime it executes in has no networking stack at all, by design (the same design that lets this program run entirely offline, with no server, no account, and no data ever leaving your device). So every exercise in this module that "makes a request" is really handing your code a bundled fixture response, a pre-captured, hand-authored stand-in for exactly what a real server would have sent back, and grading you on how correctly you parse it, retry it, paginate through it, and reason about whether you were even allowed to ask for it. This is not a workaround bolted on apologetically; it is, if anything, a better way to learn the discipline, because it means every hidden test in this module is perfectly deterministic, no flaky network, no rate limit you actually trip, no live site changing its HTML out from under your solution the day after you wrote it. You will still get the real skill: by the end of this module, if you point the exact same functions at a real, live endpoint from your own machine (outside the browser, where Python's sockets work normally), they will work unchanged. The module closes with a pointer to exactly that (a real, working, PC-side fetcher script already sitting in this program's own toolchain) so you can see the live wire the moment you want to.

One more framing worth carrying through the whole module: everything here is really one discipline wearing five hats. Parse the response correctly (JSON). Prove who you are, politely (auth). Don't ask faster than you're allowed to, and cope gracefully when you're told to slow down (rate limits). Ask only for what you're permitted to touch (robots.txt and terms of service). And when a failure is temporary, retry it smartly instead of either giving up too early or hammering a struggling server (resilience). A "resilient fetcher" (this module's payoff artifact) is simply a function that has internalized all five habits at once. Real engineers rarely think of these as five separate topics; they think of them as five things a production-grade client always does, and a toy script never does. This module is about becoming the kind of programmer who reaches for the production habit by default.

Learning objectives

By the end you can:

1. Explain the HTTP request/response cycle in plain language, method, URL, headers, body going out; status code, headers, body coming back, and correctly classify a status code into its 1xx–5xx family and state what that family means for your code's next move.
2. Explain what REST means as an architectural style (resources addressed by URL, verbs describing the action, statelessness) and why JSON became its dominant data format; parse and construct JSON confidently, including arrays nested inside objects nested inside arrays.
3. Explain why an API requires authentication (an API key or bearer token) and why a descriptive User-Agent is part of responsible, identifiable API use, and never hardcode a secret into source code.
4. Explain rate limiting from both sides of the wire (why a server imposes it, how a client detects it (HTTP 429, or a documented request budget)) and implement an exponential backoff retry policy that treats retryable and non-retryable failures differently.
5. State the robots.txt and Terms-of-Service ethics of automated fetching, and mechanically evaluate whether a given path is permitted under a given robots.txt using the real-world longest-match-wins rule.
6. Implement pagination (walking every page of a paged listing endpoint via dependency-injected fetch functions) with a hard guard against a runaway/misbehaving server.
7. Parse a small, well-defined HTML table using the standard-library html.parser, and explain why web scraping is treated as a lower-trust, more fragile data source than a documented JSON API.
8. Compose the above into a single resilient fetcher: authenticated, retryable, paginated, and ethically scoped, on both an Indian (₹) and a US ($) market-data scenario (R6).
9. (Copilot Discipline, R10.) Apply the three-law AI-coding discipline to a fetch-client task: understand the by-hand version first, use a coding assistant to accelerate the boilerplate, and review every line (including verifying that any API the assistant invents actually exists) before trusting it.

Prerequisites & connections

Builds on. CS1.01 (Python from zero) and CS1.02 (data structures, functions, idiomatic OOP) are assumed completely, this module writes ordinary functions over lists, dicts, and strings, and one small class (_TableParser, a subclass of the standard library's HTMLParser) without re-deriving what a class or a dictionary is. CS2.01 (command line, environments, reproducible projects) is assumed for the idea of an environment variable holding a secret, even though this module's exercises pass secrets as ordinary function arguments rather than reading os.environ directly (deliberately, a function that takes its key as a parameter is trivially testable, which is exactly the property you need in a sandboxed, no-network runtime; production code wires the real environment variable in at the call site, one layer up). Nothing from CS2.02 (Git), CS2.03 (SQL), or CS2.04 (systems) is required, though CS2.04's discussion of caching rhymes with this module's rate-limiting material, both are about not asking for something more often than you need to.

Feeds forward. CS2.06 (Software Engineering: Testing, Clean Code & AI-Assisted Dev) takes the dependency-injection pattern this module leans on constantly (fetch_with_retry(request, ...), paginate_all(fetch_page), passing the "thing that talks to the world" in as an argument instead of hardwiring it) and turns it into an explicit testing principle, and its dedicated AI-generated-code-review unit is the direct sequel to Objective 9 here. In the DA branch, DA1.02 (pandas I) and later modules routinely start from "you already have a clean list of dicts or a DataFrame", this module is the honest answer to the unasked question "clean from where, exactly?" for any dataset that started life behind an API rather than a bundled file. In the QD branch, QD1.01 (Market & Fundamental Data Pipelines + Vendor Landscape) is where this module's skills meet their natural home: real vendor APIs, real rate limits, real point-in-time correctness concerns, at production scale: QD1.01 will lean on the resilient-fetcher pattern you build here rather than re-teaching it. Nothing in this module depends on the QM branch (G1's statistics/ML concepts) or duplicates it; the boundary between "ML concepts" and "ML in code" (C-3) does not touch this module at all, since nothing here is statistical.

Your code pack. This module's graded exercises live in CS2.05.json (app/public/code-packs/CS2.05.json, referenced in-app by module id CS2.05), eight exercises running live in Pyodide with a structured fill-in-the-blank fallback for every one of them, so the module is 100% completable with zero runtime if your browser can't load the WASM engine. Every exercise that touches market-shaped data carries a hidden test on an Indian dataset and a hidden test on a US dataset (R6); none of them ever opens a socket.

Core teaching content

The request/response cycle, in one mental model

Picture a letter, not a phone call. Your program writes a request: a method (a verb: GET to read, POST to create, PUT/PATCH to update, DELETE to remove), a URL (which resource, and which server), a set of headers (metadata about the request, who you are, what format you'll accept, how large the body is), and, for POST/PUT/PATCH, a body (the actual data you're sending, usually JSON). The server reads the letter, does something, and writes back a response: a three-digit status code, its own headers, and usually a body. Nothing here is stateful in the way a phone call is, by design, each request/response pair is meant to stand alone (this is the "stateless" half of REST, discussed below), which is exactly why a single failed request can be retried in isolation without needing to "hang up and redial" some larger conversation.

The status code is the single most information-dense three characters in the whole exchange, and every professional fetcher branches on it before doing anything else:

• 1xx (informational): rare in application code; you will not handle these directly.
• 2xx (success): the request worked. 200 OK is the workhorse; 201 Created confirms a POST made a new resource; 204 No Content succeeds with an empty body (common for DELETE). Your code's happy path lives here.
• 3xx (redirection): the resource has moved; most HTTP client libraes follow these automatically, so you rarely branch on them by hand.
• 4xx (client error): you did something the server won't accept, and retrying identically will not help: 400 Bad Request (malformed request), 401 Unauthorized (missing/invalid credentials), 403 Forbidden (credentials fine, but you're not allowed), 404 Not Found (no such resource), 429 Too Many Requests (you can retry this one, see Rate limits, below, it is the deliberate exception to the "4xx means don't retry" rule).
• 5xx (server error): the server failed on its end, often transiently: 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable. These are usually worth retrying, because the failure may already be gone by your next attempt.

That last distinction (which failures are worth retrying and which are not) is the single most consequential piece of judgment this whole module builds toward, and it recurs everywhere: in the Rate limits section below, in Worked Example 3, and as the entire point of code-pack Exercise 6 (fetch_with_retry). Get it backwards (retry a 404 forever, or give up instantly on a 503) and you have written code that either wastes time hammering a server that will never say yes, or gives up on a problem that would have resolved itself a second later.

REST and JSON: the shape data comes in

REST (Representational State Transfer) is not a protocol or a library, it's an architectural style for designing APIs, built on a handful of ideas: resources are addressed by URL (/companies/RELIANCE, /companies/RELIANCE/quotes), verbs describe what you're doing to that resource (GET /companies/RELIANCE reads it, GET /companies/RELIANCE/quotes?page=2 reads a paged sub-collection), and the interaction is stateless, the server keeps no memory of your previous request, so every request must carry everything it needs (including, typically, your credentials) to be understood on its own. You do not need to memorize a checklist to recognize RESTful design; you need the instinct that a well-designed API's URLs read like a sentence about things, and its verbs read like a sentence about actions: GET /quotes/AAPL roughly means "give me the quote resource named AAPL," which is exactly why REST APIs feel navigable even before you've read their documentation.

JSON (JavaScript Object Notation) won as the near-universal data format for these APIs for reasons worth internalizing rather than just accepting: it maps directly onto data structures every language already has (objects → dicts, arrays → lists, plus strings, numbers, booleans, and null), it is human-readable text (you can eyeball a response and understand it, unlike a binary format), and it needs no schema to parse: json.loads on a string either succeeds and gives you a Python value, or raises. In Python, the mapping is exact and worth having cold: a JSON object {"a": 1} becomes a dict; a JSON array [1, 2, 3] becomes a list; JSON true/false/null become True/False/None; JSON numbers become int or float depending on whether they contain a decimal point or exponent. json.dumps goes the other way, turning a Python value back into JSON text, which is exactly how every hidden test's fixture in this module's code pack is built: a Python dict, turned into a JSON string via json.dumps, standing in for "what the wire actually carried."

The habit that separates a fragile JSON-parsing script from a production one is defensive extraction. A JSON response is a contract the far side controls, not you, a vendor can add a field, rename one, or simply not populate an optional one for a given period, and your code has to survive that without crashing the whole pipeline over one missing key. Naive code reaches straight in (doc["financials"]["revenue"]["value"]) and raises KeyError the first time any one of those three levels is absent. Code-pack Exercise 2 (safe_get) builds the standard fix: a small helper that walks a dotted path one segment at a time and returns a default the instant anything is missing, rather than letting the exception propagate. This single habit is worth more, in real fetching code, than almost anything else in this section, nested, optional JSON is not the exception in real APIs; it is the norm, especially in financial-data feeds where a company may simply not report a given line item for a given period.

{
  "skill": "Writing a defensive nested-JSON accessor without missing an edge case",
  "use": "Have a coding assistant draft the first version of a `safe_get`-style helper and a batch of edge-case tests (missing key, wrong type at an intermediate level, empty path) — then you read every line, run the tests yourself, and add any case the assistant's draft missed.",
  "tools": ["A coding-focused chat assistant or IDE copilot (Claude, GitHub Copilot, Cursor)", "The hidden-test harness in this module's own code pack, run locally, as your ground truth"],
  "prompt": "Write a Python function `safe_get(doc, path, default=None)` that walks a dict `doc` using a dotted string `path` like 'a.b.c', returning `default` the instant a key is missing OR the current value is not a dict. It must never raise. Then write 6 pytest-style test cases covering: a full valid path, a missing leaf, a missing branch, an intermediate value that is a list not a dict, an empty path, and an empty `doc`.",
  "verify": "Run every generated test yourself and read the implementation line by line before trusting it — a subtly wrong version (e.g. one that raises on a list intermediate instead of returning default, or one that treats an empty string path as a single empty-string segment instead of 'no path') will pass a *quick* glance and fail this module's hidden tests. Never accept a generated function's behavior on your say-so; run it against cases you designed yourself, not just the ones the assistant proposed.",
  "diy": "You must be able to write the walk-one-segment-at-a-time loop yourself, by hand, first — this Core section and code-pack Exercise 2 are exactly that gate. A copilot is for the boilerplate around a design you already own, never for owning the design.",
  "market": "US"
}

Authentication: proving who you are, without ever hardcoding it

Almost every non-trivial API requires you to identify yourself, for two reasons that are both entirely reasonable from the server's side: it wants to meter and bill usage (or at least know who to rate-limit), and it wants to be able to revoke access from one bad actor without punishing everyone else. Two patterns dominate: an API key sent in a custom header (X-API-Key: <key>) or as a query parameter, and a bearer token sent in the standard Authorization header (Authorization: Bearer <token>), the latter is the more modern convention and the one code-pack Exercise 3 (build_auth_headers) implements.

The single most important professional habit in this entire section has nothing to do with the wire format and everything to do with where the secret lives: never write a real API key as a string literal in your source code. It is tempting, especially in a quick script, to type API_KEY = "sk_live_abc123" at the top of a file and move on, and it is a genuine, repeated cause of real security incidents, because that literal now lives forever in your git history (even if you delete the line in a later commit), in any screen-share or screenshot of your editor, and in any log that happens to print your source. The professional pattern is to accept the key as a parameter, in a real program, read once from an environment variable or a secrets manager, and pass it down into whatever function needs it, and to fail loudly the moment it's missing, rather than silently sending an unauthenticated (and therefore probably-rejected, or worse, accidentally-permitted) request. "Fail loudly" here means exactly what it says: raise an exception with a clear message the instant a required credential is absent, rather than letting a None or an empty string quietly flow into a header and produce a confusing 401 three function calls later.

The other half of identifying yourself is less about secrecy and more about courtesy: a descriptive User-Agent header. Real fair-access policies (including the US SEC's EDGAR system, which this program's own toolchain talks to (see Reading & resources)) explicitly require a User-Agent that identifies the calling application and, ideally, a contact method, precisely so an operator who sees unusual traffic can reach out before blocking it outright. A client with no User-Agent, or a generic one that could belong to anyone, looks (to the server operator watching logs) indistinguishable from anonymous, badly-behaved scraping, even if your intent was entirely benign. Setting a real User-Agent costs you one line of code and buys you the benefit of the doubt.

Rate limits and exponential backoff: slow down before you're told to

A server that lets every client fetch as fast as their network allows would fall over under load from a single misbehaving script, so almost every production API imposes a rate limit, a cap on requests per unit time, per key. You discover you've hit it in one of two ways: the API's documentation states the limit up front (e.g., "5 requests per second"), letting you self-throttle proactively; or the server tells you directly, via 429 Too Many Requests, sometimes with a Retry-After header suggesting how long to wait. A resilient client treats both as the same underlying fact. "I am asking faster than I'm allowed to", and responds the same way: back off.

Exponential backoff is the standard shape of that response: instead of retrying immediately (which, under load, is exactly the wrong move, a fleet of clients all retrying instantly after a failure is precisely what causes a "thundering herd" that makes an already-struggling server worse), wait a delay that doubles with each attempt: 1 second, then 2, then 4, then 8, giving the server progressively more room to recover before you ask again. Code-pack Exercise 6 (fetch_with_retry) implements exactly this: delay = 2 ** (attempt - 1) for attempts 1, 2, 3, 4, giving the 1-2-4-8 sequence. (Production systems often add a small random "jitter" on top of the exponential delay, so that many clients backing off from the same failure don't all retry at the exact same instant either, worth knowing conceptually, though this module's graded exercises use the plain, deterministic doubling so that hidden tests can assert an exact schedule.)

The habit that makes backoff safe rather than merely present is distinguishing which failures deserve it. Re-read the status-code family list above: 429, 500, 502, 503 are retryable, they describe a transient condition that plausibly resolves on its own. 400, 401, 403, 404 are not, they describe a permanent fact about this specific request (it's malformed, you're not authenticated, you're not authorized, the resource doesn't exist), and retrying the exact same request will produce the exact same rejection every time, at the cost of wasted time and, worse, of looking like an attacker hammering an endpoint that will never say yes. A resilient fetcher raises immediately (no backoff consumed) the moment it sees a non-retryable status, and reserves its patience for the failures that might actually resolve.

Notice, too, how fetch_with_retry's signature is written: fetch_with_retry(request, max_attempts=4, sleep=time.sleep). Both request (the thing that actually talks to the world) and sleep (the thing that actually waits) are parameters, not hardwired calls, this is dependency injection again, and it is precisely what makes a timing-dependent function testable without a test suite that spends real wall-clock seconds waiting. A test passes a fake sleep that does nothing but record the delay it was asked for, so a hidden test can assert "this function asked to wait 1 second, then 2 seconds" in a few milliseconds of actual test time. This is not a workaround specific to this program's browser sandbox, it is how professional test suites handle any timing-dependent code, anywhere, and it is one of the most transferable engineering habits in this entire module.

robots.txt and Terms of Service: the ethics floor beneath every fetch

Before any automated program touches a site (whether through a documented API or by scraping a page meant for human eyes) two separate questions have to be answered, and conflating them is the single most common ethical misstep in this space. The first, mechanical question: does this site's robots.txt permit what I'm about to do? The second, human-judgment question: do this site's Terms of Service permit it, and would a reasonable person consider it fair use of the site's resources? robots.txt is a machine-checkable floor; ToS sits above it and requires actually reading and reasoning, not just parsing.

A robots.txt file (published, by convention, at the site's root: https://example.test/robots.txt) is organized into blocks: one or more User-agent: lines naming which crawler(s) a block applies to, followed by Disallow:/Allow: lines scoping what that crawler may touch. Code-pack Exercise 4 (is_allowed_by_robots) implements a simplified but realistic subset of the real specification (formalized as RFC 9309, see Reading & resources): no wildcards inside paths, only literal-prefix matching, but with the one rule that trips up every naive implementation and is exactly what real crawlers (Google's published behavior included) actually do: when both an Allow and a Disallow rule match a given path, the longer, more specific matched rule wins, not "whichever came first in the file," and not "any Disallow always wins regardless of specificity." A path can be broadly disallowed and specifically allowed at once (Disallow: /api/internal/ alongside Allow: /api/internal/public/), and the specific Allow correctly overrides the broad Disallow for anything under the public sub-path, precisely because it is the longer match. Get this backwards and your crawler either wrongly refuses to fetch something the site explicitly opened up, or, the more dangerous direction, wrongly fetches something the site tried to wall off.

Respecting robots.txt is necessary but not sufficient. A site's Terms of Service can prohibit automated access even where robots.txt is silent or permissive, and ToS violations are a matter of contract and law, not just etiquette, this is a real area where "I could technically reach the URL" and "I was allowed to" diverge, and where the honest answer is often "read the ToS, or ask, before you build the scraper," not "assume permission by default." A professional habit worth adopting for life, not just for this module: treat every scrape of a site you do not operate as requiring an affirmative answer to "am I permitted to do this," not merely a negative answer to "is anything technically stopping me." And even on the mechanical side, be a good citizen regardless of what's strictly required, set a real User-Agent (as above), space out requests rather than firing them as fast as the network allows, and cache anything you've already fetched rather than re-requesting it (a theme that also runs through CS2.04's discussion of caching).

Pagination: no listing endpoint gives you everything at once

Real listing endpoints ("give me every constituent of this index," "give me every filing this company has made") almost never return the whole answer in one response; returning ten thousand rows in a single payload is expensive for the server and unwieldy for the client, so the response comes back paged: a batch of items, plus a signal for whether more remain. A common, and this module's, contract: calling fetch_page(page) with page starting at 1 returns {"items": [...], "next_page": <int or None>}; a next_page of None means you have everything.

Code-pack Exercise 5 (paginate_all) implements the walk: call page 1, extend your running list with its items, and if next_page is truthy, call that page next; stop the moment next_page is None. The one piece of engineering discipline this simple loop needs that a first draft usually skips: a hard cap on how many pages you'll ever walk (this module's exercises use 1,000), raising an explicit error if you exceed it. A misbehaving or malicious API could in principle keep returning a next_page forever, and "never trust a remote service to terminate a loop for you" is a durable habit, the same instinct that makes you check an array index is in bounds before you use it, applied to a loop whose exit condition lives on someone else's server, not yours.

Notice, again, the shape of the function: paginate_all(fetch_page) takes the page-fetching function as a parameter rather than calling some hardwired requests.get(...) internally. This is the third time this section has used the phrase "dependency injection," and that repetition is deliberate, it is the single idea that makes every function in this module's code pack testable without a network, and it is worth you noticing, by the third occurrence, that it is one idea wearing three costumes (a fake page-fetcher, a fake request-maker, a fake sleep), not three separate tricks.

Web scraping: when there's no API at all, only a page meant for a person

Sometimes the only public copy of a number is embedded in a page's HTML, with no JSON API behind it at all, and this is where web scraping proper begins, and where the robots.txt/ToS ethics above matter the most, since a scraper is, by definition, using a page built for a human's eyes as if it were a documented data contract, which it was never designed to be. Everything downstream of "am I permitted to fetch this page" is a parsing problem: given HTML text, extract the structured data hiding inside its tags.

This program's runtime has no third-party scraping library available (no BeautifulSoup, no lxml), nor does it need one at this scale. Python's standard library ships html.parser.HTMLParser, an event-driven parser: you subclass it and override methods that fire as the parser encounters each piece of markup, handle_starttag(tag, attrs) when an opening td element begins, handle_data(data) for the text between tags, handle_endtag(tag) when that same element closes, and you accumulate state across those callbacks yourself. Code-pack Exercise 7 (extract_results_table) walks exactly this: a _TableParser tracks "am I currently inside a table cell" as a small state machine, appending completed cell text to the current row every time a td or th element closes, and completed rows to the table every time a tr element closes; the calling function then zips the first row (the header) against every subsequent row to build one dict per data row. This event-driven style feels unfamiliar the first time you meet it, you are not "walking down through" the document the way you'd walk a nested dict, you are reacting to a stream of tag-open/text/tag-close events fired at you, but it scales to documents far larger than would fit comfortably in memory as a full parsed tree, which is exactly why it is the standard-library choice.

The single durable lesson underneath the mechanics: web scraping is a lower-trust, more fragile data source than a documented JSON API, and an expert treats it that way. A JSON API is a published contract; a page's HTML is an implementation detail of someone's website design, subject to change the next time that team ships a redesign, with zero notice, and zero obligation to you. Code that scrapes a specific table element's structure will silently break, or worse, silently return wrong data that merely looks plausible, the day that page's markup changes even slightly. The professional response is not to avoid scraping (sometimes it's the only option), but to treat every scraped number with the same "verify before you trust it" discipline this program applies to AI output generally, a scraped figure is a claim to be checked against a more authoritative source wherever one exists, not a fact to be piped straight into a report.

Bringing it together: the resilient fetcher

A "resilient fetcher" (the phrase this module's own title-line description uses, and the shape of code-pack Exercise 8 (resilient_fetch_all)) is simply what happens when you compose the building blocks above instead of writing one more monolithic function that tries to do everything at once: retry logic (Exercise 6) wrapped around each individual page request, pagination logic (Exercise 5) walking the pages, with authentication (Exercise 3) and an ethics check (Exercise 4) applied before any of it runs. resilient_fetch_all does not re-implement retry logic from scratch; it calls the exact retry function from Exercise 6 once per page, and the pagination accumulation loop is the same shape as Exercise 5's, just with a retry-wrapped call standing in for the plain one. This is what real engineering mostly looks like day to day (gluing together small, independently-verified pieces) far more often than it looks like inventing something novel from a blank file, and it is exactly why this module builds each piece as its own graded, hidden-tested exercise before asking you to compose them.

One subtle bug belongs here explicitly, because it is common, easy to introduce, and easy to miss in review: when you build a small wrapper function inside a loop that needs to remember the loop variable's current value, lambda: fetch_page(page) inside a for loop over page. Python's closures capture the variable, not its value at creation time; by the time that lambda actually runs, page may have already changed to whatever it became on a later iteration. The fix is the lambda p=page: fetch_page(p) pattern used throughout this module's solutions: a default argument is evaluated at the moment the lambda is created, so p correctly freezes that iteration's value. This is called late binding, it bites working programmers in languages far more mainstream than any hypothetical here, and it is worth being able to name on sight, not just work around by accident.

{
  "skill": "Scaffolding a composed, resilient fetch client without losing track of what each piece is responsible for",
  "use": "Once you have your own working fetch_with_retry and paginate_all, use a coding assistant to draft the FIRST version of the composed resilient_fetch_all — boilerplate glue code is exactly the kind of task a copilot accelerates well — then read every line before you run it.",
  "tools": ["A coding-focused chat assistant or IDE copilot (Claude, GitHub Copilot, Cursor)", "This module's own hidden-test harness, run locally, as the ground truth for whether the draft actually works"],
  "prompt": "I have working functions `fetch_with_retry(request, max_attempts, sleep)` and the pagination loop shape used in `paginate_all(fetch_page)`. Draft `resilient_fetch_all(fetch_page, max_attempts, sleep)` that, for each page, wraps the page fetch in `fetch_with_retry` and accumulates the paginated results, capping at 1000 pages. Do not invent any library or method I have not shown you — use only what these two functions already provide.",
  "verify": "Read every line before running it, and specifically hunt for two things assistants get wrong on exactly this task: (1) a hallucinated convenience method that does not exist on the objects involved (assistants confidently invent plausible-sounding APIs — e.g. a nonexistent `requests.Session.paginate()` — far more often than they admit uncertainty), and (2) the late-binding closure bug described just above this callout, which a fast first draft reproduces constantly. Then run the actual hidden tests — a draft that LOOKS right and passes a single eyeball read can still fail a differently-shaped hidden fixture, which is exactly what those tests exist to catch.",
  "diy": "You must already have your own correct, by-hand `fetch_with_retry` and `paginate_all` passing their own hidden tests before this callout is relevant at all — composition only accelerates work built on a foundation you already own and understand, never a substitute for owning Exercises 5 and 6 yourself first.",
  "market": "IN"
}

That completes the module's mechanical core. Every one of the five hats (parsing, auth, rate limits, ethics, pagination) reduces to a small, independently testable function; a resilient fetcher is nothing more than those functions correctly wired together; and every one of them, in this program, is graded entirely offline, against a bundled fixture, with zero network call ever attempted. If you want to see the exact same ideas running against a real, live server, the Reading & resources section below points you at a script already sitting in this program's own repository that does precisely that, outside the browser, where Python's sockets work exactly as you'd expect.

Common mistakes & how experts think differently

1. "The JSON always has the fields I expect." It doesn't, and treating a nested lookup as guaranteed (doc["financials"]["revenue"]["value"], no guard) is the single most common way a data pipeline dies on one missing field from one vendor on one day. The expert writes defensive extraction (safe_get-style) as the default way to read any external JSON, not as a special case reserved for fields they already suspect are unreliable, because the fields that break are, almost by definition, the ones nobody suspected.
2. "A 4xx and a 5xx are basically the same kind of failure, just retry everything." They are opposite kinds of failure. A 5xx (or 429) describes a plausibly-transient server-side condition worth retrying with backoff; a 4xx (other than 429) describes a permanent fact about this specific request that retrying will not change. The expert's retry logic branches on this distinction explicitly and never retries a 404 or a 401 hoping it becomes true on the fifth attempt.
3. "Retry immediately, as fast as possible, to recover quickly." Immediate, unthrottled retries are precisely what turns one client's transient hiccup into a "thundering herd" that keeps a recovering server down. The expert backs off exponentially (and, in production, adds jitter) specifically because it gives the far side room to recover, at the modest cost of the individual request finishing a few seconds later.
4. "If the URL loads in my browser, I'm allowed to script against it." Loading is a technical fact; permission is a separate, human question governed by robots.txt and the site's Terms of Service, and the two can diverge sharply, a page can load perfectly in a browser while its ToS explicitly prohibits automated access. The expert checks both, in that order, before writing a single line of fetching code, and treats "I could reach it" as informationally worthless on the permission question.
5. "An empty Disallow: line means the crawler should disallow everything by default." It means the opposite, an empty Disallow: value is the standard idiom for "this rule disallows nothing," commonly used to explicitly signal "yes, come on in" for a User-agent block that would otherwise look restrictive. Misreading this specific idiom backwards causes a crawler to wrongly refuse access a site owner explicitly intended to grant.
6. "Whichever Allow/Disallow rule appears first in the file wins." Real robots.txt semantics use longest-match-wins, independent of file order, a broad Disallow: /api/ appearing before a narrow Allow: /api/public/ does not make the Disallow win just because it came first; the more specific rule governs regardless of position. An expert implementing or reasoning about a robots.txt parser checks specificity, never sequence.
7. "Hardcoding my API key in the script is fine, it's just for local testing." "Just for local testing" scripts are exactly the ones that get committed, shared, and copy-pasted into something more permanent; a real key in a git history is a real key exposed, full stop, no matter how quickly the literal line was later "fixed." The expert never types a real secret as a source-code literal, in a throwaway script or otherwise, it is a parameter, an environment variable, or a secrets-manager lookup, always.
8. "A 200 status means the data I got back is correct." A 200 means the transport succeeded (the server accepted the request and returned a response) and says nothing about whether the response body is well-formed or meaningful; a misconfigured server can return a 200 with an empty body, an error message disguised as JSON, or a stale cached page. The expert validates the shape and content of the body itself (exactly what parse_quote_response's defensive JSON handling does) rather than treating a 200 alone as proof the payload is trustworthy.
9. "Scraped HTML data is just as reliable as an API's JSON, once I've got my parser working." A JSON API is a maintained contract; a page's HTML is an implementation detail of someone's front-end, liable to change with zero notice and zero obligation to anyone scraping it, a scraper can silently start returning wrong-but-plausible data the moment the page's markup shifts slightly, with no error raised anywhere. The expert treats every scraped figure as lower-trust than a documented API response, and checks it against a more authoritative source before it enters anything that matters.
10. "An AI-generated fetch client is fine to ship once it runs without an error." Running without an error proves the happy path works; it proves nothing about whether the assistant invented a plausible-sounding method that doesn't actually exist on the library you're using, silently mishandled a non-retryable status, or reproduced the late-binding closure bug from the Core section. The expert reviews every line of AI-assisted fetch code against the specific failure modes this module just taught (hallucinated APIs, wrong retry/no-retry classification, closures over loop variables) and runs the hidden tests before trusting any of it, exactly the Copilot Discipline's third law.

The thread through all ten: an expert fetcher-writer assumes the far side is unreliable, adversarial-by-neglect, and entitled to say no, and builds code that survives a missing field, a transient failure, a permission boundary, and a fragile page structure as the default case, not as an afterthought bolted on once something breaks in front of a client. That posture, more than any individual function signature, is what this module is actually installing.

Worked examples

Worked example 1: Parsing an NSE-style quote response by hand (India, ₹)

An analyst's small research script needs the latest quote for an Indian large-cap. Pyodide cannot make the live call, so the fixture below stands in for exactly what a real quote endpoint would have returned, every figure here is illustrative and synthetic, not a claim about any company's actual traded price on any actual date.

raw = '{"symbol": "RELIANCE", "lastPrice": 2948.65, "currency": "INR", "asOf": "2026-07-28T15:30:00+05:30"}'

Solution. Step 1: parse the envelope. json.loads(raw) turns the string into a Python dict: {"symbol": "RELIANCE", "lastPrice": 2948.65, "currency": "INR", "asOf": "2026-07-28T15:30:00+05:30"}. If raw were not valid JSON at all (a truncated response, an HTML error page returned where JSON was expected, a real and common failure mode when a server is misconfigured or overloaded), json.loads raises json.JSONDecodeError, which parse_quote_response catches and re-raises as a single, clear ValueError("invalid quote response"), one exception type for the caller to handle, regardless of which of several ways the input could be broken.

Step 2, rename and re-type the fields you actually want. The source uses the vendor's own field names (lastPrice, asOf), names you do not control and should not propagate into your own codebase verbatim, since a second vendor will almost certainly name the same concept differently. parse_quote_response builds a new dict with your own stable keys (symbol, price, currency, as_of) decoupling the rest of your program from any one vendor's naming choices. Note also the explicit float(...) around the price: JSON numbers decode as either int or float depending on whether they contain a decimal point, and a price that happened to arrive as a whole number (lastPrice: 3000) would decode as a Python int without the explicit cast, float() makes the output type consistent regardless of what the wire happened to send.

Step 3, the missing-field case. Suppose the same endpoint, on a bad day, returned {"symbol": "RELIANCE"} with the rest of the fields simply absent. doc["lastPrice"] raises KeyError the moment the code reaches for it; the function's except (KeyError, TypeError, ValueError) catches this and raises the same clean ValueError("invalid quote response"), a caller one level up can handle "this quote is unusable" once, rather than needing to know about JSONDecodeError and KeyError and a bad type-conversion as three separate cases.

Read it. {"symbol": "RELIANCE", "price": 2948.65, "currency": "INR", "as_of": "2026-07-28T15:30:00+05:30"}, four clean, vendor-independent keys, or a single clear exception. This is code-pack Exercise 1.

Worked example 2: Defensive extraction from a deeply-nested filing-style JSON (US, $)

Real financial-statement APIs (SEC EDGAR's companyfacts endpoint is the real-world shape this fixture is modeled on) nest a single reported number four or five levels deep: company → concept → unit → list of period-tagged values. The figures below are illustrative and synthetic, not a claim about any company's actual filed results.

doc = {"cik": "0000320193",
       "facts": {"us-gaap": {"Revenues": {"units": {"USD":
           [{"val": 391035000000, "fy": 2025, "form": "10-K"}]}}}}}

Solution. Step 1: walk the happy path. safe_get(doc, "facts.us-gaap.Revenues.units.USD") splits the path on . into six segments and walks them one at a time: facts → the big nested dict; us-gaap → the taxonomy dict; Revenues → the concept dict; units → {"USD": [...]}; USD → the final value, which is a list of one dict, [{"val": 391035000000, "fy": 2025, "form": "10-K"}]. The walk reaches the end of the path and returns that list directly, this is the correct, intended result, and it is a list by design, because a concept can legitimately be reported in more than one unit or restated across filings.

Step 2, the scope boundary, made explicit. Suppose you then tried safe_get(doc, "facts.us-gaap.Revenues.units.USD.0.val"), reasoning "the 0 should grab the first list element." It doesn't, and this is deliberate, not a bug. safe_get only walks dict keys; at the point the walk reaches segment "0", current is the list from Step 1, and isinstance(current, dict) is False, so the function returns default (here, None) immediately, exactly as it would for a genuinely missing key. Indexing into a list by position is a different operation from looking up a dict key, and conflating the two inside one helper would make its failure behavior far harder to reason about, better to have one function that does one thing predictably, and a caller that explicitly indexes the list (safe_get(doc, "...USD")[0]["val"]) when it wants that.

Step 3, the field that was never reported. Ask for safe_get(doc, "facts.us-gaap.NetIncomeLoss.units.USD", []), and the walk fails at the NetIncomeLoss segment (it is simply not a key in doc["facts"]["us-gaap"] in this fixture, the filer's XBRL tagging didn't include it under this exact concept name for this period, a genuinely common real-world occurrence), returning the supplied default [] cleanly, with no exception anywhere in the caller's code.

Read it. The same three-line call shape (safe_get(doc, "<path>", <default>)) handles "field present," "field structurally out of scope," and "field genuinely absent" without the caller writing a single try/except of their own. This is code-pack Exercise 2.

Worked example 3: Tracing the exact retry/backoff schedule by hand (US, $)

A research script hits a (synthetic, illustrative) US market-data endpoint that happens to be having a bad few seconds: it returns 500 twice, then recovers and returns 200 with the real payload on the third attempt.

Solution. Step 1: attempt 1. request() returns {"status": 500, "body": None}. 500 is in the retryable set {429, 500, 502, 503}, so last_status is set to 500; since attempt (1) < max_attempts, the code calls sleep(2 ** (1 - 1)) = sleep(1), a one-second wait, and loops back for attempt 2.

Step 2: attempt 2. request() again returns {"status": 500, "body": None} (the fixture is built to fail exactly twice). Same branch as Step 1: last_status = 500, and this time sleep(2 ** (2 - 1)) = sleep(2), the delay has doubled, as exponential backoff requires.

Step 3: attempt 3. request() now returns {"status": 200, "body": "ok"}. The function returns response["body"], the string "ok", immediately, without consuming a fourth attempt or another sleep.

Step 4, the schedule, stated as a hidden test would check it. Across the whole call, request was invoked exactly 3 times, and sleep was invoked exactly twice, with arguments 1 then 2, the [1, 2] sequence that code-pack Exercise 6's hidden tests assert on the nose (using a fake sleep that just records its arguments, so the test finishes in milliseconds rather than actually waiting three seconds).

Read it. Contrast this with the same fixture under max_attempts=2: attempt 1 fails (500, sleep 1s), attempt 2 fails again (500), but now attempt (2) < max_attempts (2) is false, so no third sleep is scheduled, and the loop ends having exhausted its budget; the function raises RuntimeError("gave up after 2 attempts, last status 500"). The exact same flaky server, given a smaller retry budget, produces a hard failure instead of eventual success, which is exactly why choosing max_attempts sensibly (and reporting it honestly in an error message when it's exceeded) is itself part of the engineering judgment this module is teaching, not an incidental default value.

Worked example 4: robots.txt longest-match-wins, traced rule by rule (India, ₹)

A (synthetic, illustrative) Indian financial-data mirror publishes this robots.txt:

User-agent: *
Disallow: /private/
Disallow: /api/internal/
Allow: /api/internal/public/
Disallow:

User-agent: ResearchBot
Disallow: /
Allow: /public-data/

Your crawler identifies itself as "GenericBot/1.0" and wants /api/internal/public/report.json.

Solution. Step 1, which block applies? "genericbot/1.0" (lower-cased) is not a named User-agent in this file, so the crawler falls back to the User-agent: * block, the four rules Disallow: /private/, Disallow: /api/internal/, Allow: /api/internal/public/, and the trailing empty Disallow:.

Step 2: find every matching rule. Check each rule's value as a prefix of the requested path /api/internal/public/report.json: /private/ does not match (the path doesn't start with it); /api/internal/ does match, length 14, kind disallow; /api/internal/public/ does match, length 21, kind allow; the empty Disallow: is explicitly skipped (the "disallow nothing" idiom from Common Mistake #5, it never counts as a match at all).

Step 3: resolve by longest match. Two real matches: a 14-character Disallow and a 21-character Allow. The longer one, the 21-character Allow, wins, regardless of the fact that its shorter, broader Disallow sibling appeared first in the file. Result: allowed.

Step 4, the same file, a path that stays disallowed. Now request /api/internal/secret.json under the same * block. Matches: /api/internal/ (length 14, disallow); the more specific /api/internal/public/ does not match this path (it isn't a prefix of /api/internal/secret.json), so the only real match left is the 14-character Disallow, and the result is disallowed. The single character difference between public/report.json and secret.json after the same /api/internal/ prefix is exactly the kind of case a longest-match implementation must get right on purpose, not by accident.

Read it. Same file, same generic crawler, two paths one folder apart in name (one allowed, one not) and the only thing that decided it was which matching rule was longer, never which came first. This is code-pack Exercise 4.

Worked example 5: Scraping a small results table, traced as an event stream (US, $)

A (synthetic, illustrative) US small-cap's investor-relations page embeds its latest quarter's headline figures as a plain HTML table, with no JSON API behind it at all. Rendered in a browser it looks like the table below; the markup your script actually receives is one unbroken string built from a single table element holding two tr rows, a header row of three th cells, then a data row of three td cells, opened and closed in that exact order with no whitespace between them:

Solution. Step 1, the event stream HTMLParser actually sees. Feeding this string to _TableParser fires, in order: handle_starttag("table", []) (no handler reacts to table itself); handle_starttag("tr", []) → self._current_row = []; handle_starttag("th", []) → self._in_cell = True, self._cell_text = []; handle_data("Ticker") → appended to _cell_text (only because _in_cell is True); handle_endtag("th") → _in_cell = False, and "Ticker" (joined and stripped) is appended to _current_row.

Step 2, the rest of the header row. The same th/data/th-close cycle repeats for Last and ChangePct, leaving _current_row = ["Ticker", "Last", "ChangePct"]. Then handle_endtag("tr") fires: the completed row is appended to self.rows, and _current_row resets to None until the next tr element opens.

Step 3, the one data row. A fresh tr element starts a new _current_row = []; three td/data/close cycles build it up to ["AAPL", "213.40", "+1.1"]; that closing tr appends it to self.rows. After the table element closes (unhandled, no special action needed), parser.rows == [["Ticker", "Last", "ChangePct"], ["AAPL", "213.40", "+1.1"]].

Step 4, header/data split and the final zip. extract_results_table unpacks header, *data_rows = parser.rows, giving header = ["Ticker", "Last", "ChangePct"] and data_rows = [["AAPL", "213.40", "+1.1"]]; the return expression [dict(zip(header, row)) for row in data_rows] pairs each header label with the corresponding cell in each row, producing [{"Ticker": "AAPL", "Last": "213.40", "ChangePct": "+1.1"}].

Read it. Every value came back as a string ("213.40", not 213.40): HTML text is always text, and converting "213.40" to a real float where you actually need arithmetic is a separate, deliberate step this exercise leaves to the caller, not something extract_results_table should silently guess about (a cell might legitimately be non-numeric, a ticker, a date, a footnote marker, and a parser that tries to auto-detect and convert "numbers" invites exactly the kind of silent misclassification bug a careful engineer avoids). This is code-pack Exercise 7.

Worked example 6: Composing retry and pagination across two markets (India ₹ + US $)

A synthetic "resilient fetcher" scenario, deliberately combining both markets in one trace: page 1 of an Indian constituent listing fails twice with 500 before succeeding; page 2 (its next_page) succeeds cleanly and ends the listing. Separately, a US scenario: page 1 succeeds cleanly, page 2 recovers from a single 429, page 3 ends it.

Solution. Step 1. India, page 1. resilient_fetch_all calls _fetch_with_retry(lambda p=1: fetch_page(p), ...) for page 1. Internally, that retry call behaves exactly as Worked Example 3 traced: two 500s (sleeping 1s then 2s under a real sleep, or recording [1, 2] under a test's fake one), then a 200 whose body is {"items": ["RELIANCE", "TCS"], "next_page": 2}. resilient_fetch_all extends its running list with ["RELIANCE", "TCS"] and, seeing next_page = 2, moves on.

Step 2: India, page 2. A fresh call to _fetch_with_retry for page 2 succeeds on its first attempt: {"items": ["INFY"], "next_page": None}. The running list becomes ["RELIANCE", "TCS", "INFY"], and since next_page is falsy, the function returns immediately, three items, across two pages, one of which needed two retries and the other needed none, entirely invisibly to the caller.

Step 3: US, a different flaky pattern, same composed function. Page 1 succeeds immediately (["AAPL"], next_page: 2). Page 2's first attempt returns 429 (rate-limited, not a server error, but still in the retryable set), one backoff sleep, then a successful retry returns ["MSFT", "NVDA"], next_page: 3. Page 3 succeeds immediately (["AMZN"], next_page: None). Final result: ["AAPL", "MSFT", "NVDA", "AMZN"], four items across three pages, with exactly one retried page, and the same resilient_fetch_all code handling a 429 mid-pagination with no special-casing anywhere in the pagination logic itself, because the retry-vs-give-up decision was already fully handled one layer down, inside _fetch_with_retry.

Read it. Neither scenario required resilient_fetch_all to know or care which page would be flaky, or which retryable status it would see, that knowledge lives entirely inside the already-verified _fetch_with_retry, called fresh for each page. This is exactly the payoff of composition promised in the Core section: the calling function stays simple because the hard part was solved once, correctly, and reused. This is code-pack Exercise 8, this module's synthesis exercise.

Practice set

Work each problem fully before reading its solution (write before you peek). All fixtures below are illustrative/synthetic (R3) unless stated otherwise.

P1 (guided). Status-code triage. For each status below, say whether a resilient client should retry it (with backoff) or raise immediately: 503, 404, 429, 401, 502.

Solution. Retry with backoff: 503 (Service Unavailable (transient server condition), 429 (Too Many Requests) rate-limited, but retrying after backing off is exactly correct), 502 (Bad Gateway: transient upstream failure). Raise immediately, no retry: 404 (Not Found, the resource doesn't exist; retrying the identical request will never make it exist) and 401 (Unauthorized, a credentials problem that an identical retry cannot fix).

P2 (guided). JSON round-trip. What Python value does json.loads('{"a": [1, 2.5, null, true], "b": "x"}') produce? Give its exact type structure.

Solution. {"a": [1, 2.5, None, True], "b": "x"}, a dict with two keys; "a" maps to a list containing an int (1), a float (2.5), None (from JSON null), and True (from JSON true); "b" maps to a plain str.

P3 (guided). safe_get trace. Given d = {"x": {"y": 5}}, what does safe_get(d, "x.y.z", "fallback") return, and why?

Solution. "fallback". The walk reaches x → {"y": 5}, then y → 5. The next segment, z, is checked against current = 5: isinstance(5, dict) is False, so the function returns the default immediately rather than raising TypeError for trying to subscript an int.

P4. The missing-field exception path. parse_quote_response('{"symbol": "X", "lastPrice": 10}'), currency and asOf are both missing. What happens, and what single exception type does the caller see?

Solution. The JSON parses fine (json.loads succeeds), but building the return dict reaches for doc["currency"], which raises KeyError. The function's own except (KeyError, TypeError, ValueError) catches this and re-raises ValueError("invalid quote response"), the caller sees one exception type regardless of which field was missing or which of the three underlying error types actually occurred.

P5. Auth header construction. What does build_auth_headers("tok_789", "myscript/2.0") return, exactly?

Solution. {"Authorization": "Bearer tok_789", "User-Agent": "myscript/2.0", "Accept": "application/json"}.

P6. The falsy-key edge case. build_auth_headers("", "myscript/2.0"), what happens, and why does an empty string count as "missing" here rather than being treated as a (weird but technically present) key?

Solution. It raises ValueError("missing api_key"). The check is if not api_key:, and not "" is True in Python, an empty string is falsy, and treating it as "no key" is the correct behavior, since a real API would never issue a genuinely valid key that happens to be the empty string; an empty string in that slot is always a bug upstream (a config that failed to load, a default that was never overridden), and failing loudly on it is exactly right.

P7. Backoff schedule, longer budget. Using delay = 2 ** (attempt - 1), list the sleep durations for attempts 1 through 5 (assuming every attempt through the fifth fails and needs a retry).

Solution. Attempt 1 → sleep 1; attempt 2 → sleep 2; attempt 3 → sleep 4; attempt 4 → sleep 8; (attempt 5, if it is the final attempt and max_attempts=5, sleeps nothing, no retry follows the last attempt). So across 5 attempts the recorded sleep sequence is [1, 2, 4, 8], one fewer sleep than attempts, because you never back off after the attempt that exhausts your budget.

P8. Non-retryable inside a retry budget. fetch_with_retry is called with max_attempts=5 against a request that returns {"status": 403, "body": None} on its very first call. How many total calls does request receive, and what does the function do?

Solution. Exactly one call. 403 is not in the retryable set {429, 500, 502, 503}, so the function raises RuntimeError("non-retryable status 403") immediately on the first attempt, it never consumes a second attempt, and no sleep is ever called.

P9. Exhausted retries, exact message. fetch_with_retry is called with max_attempts=3 against a request that always returns {"status": 502, "body": None}. What exception, with what exact message, is eventually raised?

Solution. RuntimeError("gave up after 3 attempts, last status 502"). All three attempts return the retryable 502; after the third (with no further attempts remaining), the loop ends and the function raises using max_attempts and the last-seen status.

P10. robots.txt, the empty-Disallow idiom. A site's User-agent: * block contains only one rule: Disallow: (nothing after the colon). Is a crawler identifying as "AnyBot" allowed to fetch /anything/at/all? Why?

Solution. Yes, allowed. An empty Disallow: value is the standard idiom meaning "this rule disallows nothing", it is explicitly skipped during matching (never counted as a match), so with no other rules in the block, nothing is disallowed at all, and the function's default (best_allow = True) stands.

P11. robots.txt: longest match, a harder tie. A block has Disallow: /reports/2026/ (length 15) and Allow: /reports/2026 (length 12, note: no trailing slash). For the path /reports/2026/q3.pdf, which rule wins, and what's the result?

Solution. Both are checked as prefixes: /reports/2026/ (15 chars) matches /reports/2026/q3.pdf; /reports/2026 (12 chars, no trailing slash) also matches, being a literal prefix of the same string. The longer rule (the 15-character Disallow) wins over the shorter 12-character Allow, so the path is disallowed. (Notice how the missing trailing slash on the Allow rule cost it four characters of specificity and, with it, the tie-break, a reminder that in robots.txt, exact rule text matters down to the character.)

P12. robots.txt: grouped User-agent lines. A file reads: User-agent: BotA / User-agent: BotB / Disallow: /x/ / (blank line) / User-agent: * / Allow: /x/. Does BotA get the * block's Allow: /x/ rule too, or only its own group's Disallow: /x/?

Solution. Only its own group's rule. Consecutive User-agent: lines with no rule between them share one block, BotA and BotB both belong to the group carrying Disallow: /x/, and that grouping ends the moment a rule line is seen; the following User-agent: * starts an entirely separate block. _parse_robots looks up the rules for BotA specifically (finding the Disallow: /x/ group) and, having found a match, never falls through to the * block at all, the * block is only a fallback for agents with no named block of their own.

P13. Pagination trace. fetch_page returns, for pages 1/2/3: {"items": [1], "next_page": 2}, {"items": [2, 3], "next_page": 3}, {"items": [4], "next_page": None}. What does paginate_all(fetch_page) return, and how many calls does fetch_page receive?

Solution. [1, 2, 3, 4], via exactly 3 calls to fetch_page (pages 1, 2, then 3, after which next_page is None and the loop returns).

P14. The runaway-pagination guard. fetch_page(page) always returns {"items": [page], "next_page": page + 1}, it never stops. What does paginate_all do, and after how many calls?

Solution. After exactly 1,000 calls (the hard cap), paginate_all raises RuntimeError("too many pages (possible infinite pagination loop") rather than looping forever) the guard exists precisely for a misbehaving server like this synthetic one.

P15. HTML table, header-only edge case. extract_results_table(html) is called on a table element that opens, holds one tr row of two th cells (Symbol, Price), and closes, no data rows at all. What is returned?

Solution. []. parser.rows contains exactly one row (the header); header, *data_rows = parser.rows correctly unpacks data_rows as an empty list, and the list comprehension over an empty data_rows produces [], no special-casing needed beyond the if not parser.rows: return [] guard for the zero-rows-at-all case (a table element that closes immediately, with not even a header row inside it).

P16. The late-binding closure bug, made concrete. A learner writes fetchers = [lambda: fetch_page(p) for p in [1, 2, 3]] (no default argument) instead of using the p=page default-argument pattern. If all three lambdas in fetchers are called after the list is fully built, what does each one actually fetch, and why?

Solution. All three call fetch_page(3), every one of them, regardless of which position in the list. A lambda (or any closure) that references a loop variable directly captures the variable itself, not a frozen snapshot of its value at creation time; by the time any of the lambdas is actually called, the loop has finished and p holds its final value, 3, which every lambda now sees. The fix is exactly the pattern this module's solutions use: lambda p=p: fetch_page(p), a default-argument value is evaluated once, at the moment the lambda object is created, correctly freezing that iteration's value inside the lambda's own scope.

Applied mini-project

Build a resilient client for a bundled "mystery API," end to end.

This project asks you to exercise every idea in this module against one coherent, offline fixture rather than eight separate toy inputs. Open this node's Code tab and load the bundled mystery-api fixture set (a small collection of Python functions simulating a flaky, paginated, authenticated, robots.txt-governed listing endpoint, every call is local, deterministic, and network-free, exactly like the rest of this module).

Do this, in order:

1. Read the fixture's own robots.txt string first, before writing any fetch code at all, and determine (using is_allowed_by_robots) whether your assigned User-Agent string may fetch the assigned path. If the answer is no, stop: write one sentence explaining which rule blocked you and why, and treat that as your deliverable for this step (real engineering sometimes ends here, and knowing when to stop is the actual skill).
2. Build your headers with build_auth_headers, using the fixture's provided API key and a User-Agent you write yourself that would genuinely identify your (hypothetical) script and a contact method, in the spirit of a real fair-access policy.
3. Fetch every page, composing fetch_with_retry and pagination exactly as resilient_fetch_all does, against the fixture's flaky, paginated mock endpoint (you do not need to reinvent resilient_fetch_all's shape from scratch, reuse it, adapted to this fixture's exact contract).
4. Parse every record defensively with a safe_get-style walk, extracting a specific nested field the fixture sometimes omits, confirm your code neither crashes nor silently returns wrong data on the records missing it.
5. Write a two-paragraph incident note, as if for a teammate: paragraph one states what your fetcher did (how many pages, how many retries, any records skipped and why); paragraph two states, honestly, what would need to change for this to run against the real, live version of this API outside Pyodide (at minimum: swap the fixture functions for a real HTTP client call, and re-check the robots.txt/ToS check against the site's actual current policy, since either can change over time and your earlier check does not stay valid forever).

Deliverable and self-check. One notebook or script containing: the robots.txt determination and its one-sentence justification; the constructed headers; the full fetch-and-parse run with its final record count; and the two-paragraph incident note. Score yourself out of 10: 2 points for correctly gating step 1 on the robots.txt check before writing fetch code; 2 points for headers that would fail loudly on a missing key, not just happen to work with the one key you were given; 3 points for a fetch that correctly composes retry and pagination (spot-check: does it survive a page that fails twice before succeeding, without either giving up early or double-counting that page's items?); 1 point for the defensive parse not crashing on a record missing the target field; 2 points for an honest, specific incident note (a note that says only "it worked" scores 0 here, the point is naming what would actually have to change for a live run). Pass at 8/10, with no zero on any item.

Case Lab: none mapped yet for this node (this module has no live Case Lab hook, see the module's cross-links above for where this pattern gets used against real market-data pipelines).

Reading & resources

Everything below is a place to learn the mechanism; none of it substitutes for reading a real API's own documentation before you build against it.

HTTP, REST, and JSON fundamentals:

• MDN Web Docs, HTTP overview and HTTP response status codes, the clearest free reference for the request/response cycle and every status-code family used in this module. [Free] [Beginner]
• restfulapi.net, What is REST: a compact, practical primer on REST's resource/verb/statelessness ideas, without over-formalizing them. [Free] [Beginner]
• Python's own documentation for the json module, the authoritative reference for exactly how Python types map to and from JSON. [Free] [Beginner]
• Roy Fielding's original dissertation, Architectural Styles and the Design of Network-based Software Architectures (2000), Chapter 5, the primary source for REST, for when you want the real thing behind the popularized version. [Free] [Advanced]

Auth, rate limits, and resilience:

• Any major cloud provider's public API rate-limiting documentation (e.g., a "best practices for retries" guide), the vocabulary and backoff conventions used in production are near-universal across vendors; read whichever one you can find free access to. [Free] [Intermediate]
• Marc Brooker (Amazon), Exponential Backoff and Jitter, the widely-cited explanation of why jitter matters on top of plain exponential backoff, referenced conceptually in this module's Core section. [Free] [Intermediate]

robots.txt and scraping ethics:

• RFC 9309, Robots Exclusion Protocol: the actual, formal specification behind the simplified subset this module teaches; read this once you're comfortable with the simplified version and want the full rule set (wildcards included). [Free] [Advanced]
• Google Search Central, How Google interprets the robots.txt specification, a practitioner-level walkthrough of the same longest-match-wins semantics this module's Exercise 4 implements, from the crawler operator most people have heard of. [Free] [Intermediate]
• Python's own documentation for html.parser.HTMLParser, the authoritative reference for the exact callback methods this module's _TableParser overrides. [Free] [Beginner]

See the real thing (PC-side, optional, ungraded):

• This program's own tools/fetch_edgar.py is a real, working Python script (run outside the browser, on your own machine, where sockets work normally) that fetches real US company data from SEC EDGAR's public API: it sets a real, descriptive User-Agent (reading a contact address from an environment variable, exactly as this module's Common Mistakes section recommends), respects EDGAR's fair-access request pacing, and parses real nested JSON with the same defensive spirit as safe_get. Reading it after finishing this module's exercises is the fastest way to see "the exact same ideas, against a real socket", nothing about it will look unfamiliar. [Free] [Intermediate], external to the graded app, run entirely at your own option; this program never requires it and never runs it for you.
• SEC EDGAR's own developer documentation for its data.sec.gov JSON APIs, including its fair-access policy (the real-world source of this module's User-Agent discussion), the primary source behind Worked Example 2's fixture shape. [Free] [Intermediate]

Teach it back & journal

Feynman prompt. Explain to a friend who has only ever used a phone app (never written a line of fetching code) why a "resilient fetcher" needs five separate habits rather than just "downloading the data": parsing defensively (fields go missing), proving who you are without ever writing your secret key into a file, backing off instead of hammering a struggling server, checking whether you're even allowed to ask, and walking through every page rather than assuming the first response is everything. If your friend can, unprompted, explain why retrying a 404 forever is pointless while retrying a 503 is often correct, using the idea that one describes a permanent fact about the request and the other a transient fact about the server's current state, you have installed the module's central piece of judgment.

Journal prompt. Think of the last time you personally hit a "please wait" or "too many requests" message on any website or app, as a human, not as a programmer. Write down what you did in response, did you retry immediately, over and over? Wait a while? Give up? Now write one paragraph connecting your own very human backoff instinct (or lack of one) to the exponential-backoff logic this module just had you implement in code, the same "give the other side room to recover" reasoning applies whether the "other side" is a rate-limited server or a person you've just annoyed. Then write a second paragraph naming one specific habit from this module: defensive JSON parsing, checking robots.txt before writing a single fetch call, never hardcoding a secret, or reviewing AI-generated fetch code for hallucinated APIs, that you are most likely to skip under time pressure in your own future work, and one concrete thing you will do this week to make skipping it harder (a lint rule, a checklist, a habit of writing the robots.txt check first, before any other line of the script).

This module's flashcards and mastery quiz are wired into the app: see the node's Quiz and Reviews.