APIs, HTTP, JSON & Web Scraping

Learning objectives

By the end you can:

1. Explain the HTTP request/response cycle in plain language, method, URL, headers, body going out; status code, headers, body coming back, and correctly classify a status code into its 1xx–5xx family and state what that family means for your code's next move.
2. Explain what REST means as an architectural style (resources addressed by URL, verbs describing the action, statelessness) and why JSON became its dominant data format; parse and construct JSON confidently, including arrays nested inside objects nested inside arrays.
3. Explain why an API requires authentication (an API key or bearer token) and why a descriptive User-Agent is part of responsible, identifiable API use, and never hardcode a secret into source code.
4. Explain rate limiting from both sides of the wire (why a server imposes it, how a client detects it (HTTP 429, or a documented request budget)) and implement an exponential backoff retry policy that treats retryable and non-retryable failures differently.
5. State the robots.txt and Terms-of-Service ethics of automated fetching, and mechanically evaluate whether a given path is permitted under a given robots.txt using the real-world longest-match-wins rule.
6. Implement pagination (walking every page of a paged listing endpoint via dependency-injected fetch functions) with a hard guard against a runaway/misbehaving server.
7. Parse a small, well-defined HTML table using the standard-library html.parser, and explain why web scraping is treated as a lower-trust, more fragile data source than a documented JSON API.
8. Compose the above into a single resilient fetcher: authenticated, retryable, paginated, and ethically scoped, on both an Indian (₹) and a US ($) market-data scenario (R6).
9. (Copilot Discipline, R10.) Apply the three-law AI-coding discipline to a fetch-client task: understand the by-hand version first, use a coding assistant to accelerate the boilerplate, and review every line (including verifying that any API the assistant invents actually exists) before trusting it.

Prerequisites & connections

Builds on. CS1.01 (Python from zero) and CS1.02 (data structures, functions, idiomatic OOP) are assumed completely, this module writes ordinary functions over lists, dicts, and strings, and one small class (_TableParser, a subclass of the standard library's HTMLParser) without re-deriving what a class or a dictionary is. CS2.01 (command line, environments, reproducible projects) is assumed for the idea of an environment variable holding a secret, even though this module's exercises pass secrets as ordinary function arguments rather than reading os.environ directly (deliberately, a function that takes its key as a parameter is trivially testable, which is exactly the property you need in a sandboxed, no-network runtime; production code wires the real environment variable in at the call site, one layer up). Nothing from CS2.02 (Git), CS2.03 (SQL), or CS2.04 (systems) is required, though CS2.04's discussion of caching rhymes with this module's rate-limiting material, both are about not asking for something more often than you need to.

Feeds forward. CS2.06 (Software Engineering: Testing, Clean Code & AI-Assisted Dev) takes the dependency-injection pattern this module leans on constantly (fetch_with_retry(request, ...), paginate_all(fetch_page), passing the "thing that talks to the world" in as an argument instead of hardwiring it) and turns it into an explicit testing principle, and its dedicated AI-generated-code-review unit is the direct sequel to Objective 9 here. In the DA branch, DA1.02 (pandas I) and later modules routinely start from "you already have a clean list of dicts or a DataFrame", this module is the honest answer to the unasked question "clean from where, exactly?" for any dataset that started life behind an API rather than a bundled file. In the QD branch, QD1.01 (Market & Fundamental Data Pipelines + Vendor Landscape) is where this module's skills meet their natural home: real vendor APIs, real rate limits, real point-in-time correctness concerns, at production scale: QD1.01 will lean on the resilient-fetcher pattern you build here rather than re-teaching it. Nothing in this module depends on the QM branch (G1's statistics/ML concepts) or duplicates it; the boundary between "ML concepts" and "ML in code" (C-3) does not touch this module at all, since nothing here is statistical.