DA1.03 named the destination without building it: "a pandas script runs unattended on a cron job… the natural path to everything QD2.04 will formalize." Here is that formalization. A schedule states when a job should run without a human present to click a button, the classic form, cron, specifies a set of allowed minutes, hours, days-of-month, months, and days-of-week, and the scheduler's job, mechanically, is to walk forward through time and find the next moment that satisfies every one of those fields simultaneously. Code pack exercise QD2.04-e04 (next_run_minute) builds a deliberately simplified two-field version of exactly that search, a minute-set and an hour-set, either of which may be "any", because the search itself, not the extra fields, is the part worth internalizing: walk forward one minute at a time from right after the current moment, and stop at the first candidate whose minute (if constrained) and whose hour (if constrained) both match. Get the schedule's fields backwards, or forget that "the next run" must be strictly after the current moment, a job asked for its next run at the exact instant it is scheduled to fire must still find tomorrow's occurrence, not report that it is due right now, and you have built the single most common real production bug in this entire space: a job that fires at the wrong hour, once a day, forever, until someone notices the timestamps in the log look off by a fixed offset.
Advertisement
Why do scheduled jobs need to be idempotent?
The property that makes a schedule survivable in production is idempotency, and it exists because of an unglamorous fact about real schedulers: almost none of them guarantee exactly-once delivery. A scheduler restarts mid-tick, a network call to trigger your job times out after the job actually started, a retry policy fires because a health check was slow, all of these produce at-least-once delivery, meaning your pipeline must be prepared to be invoked twice, with identical inputs, for what is logically the same run. A pipeline that is not built for this will, on that (inevitable, eventually) day, silently double-book a trade in a paper ledger, double-count a trial in an experiment log, or double-charge an API rate limit, and because the duplicate invocation looks, from the outside, exactly like a legitimate second run, nothing about the failure looks unusual until someone notices the numbers don't add up. Code pack exercise QD2.04-e07 (append_if_new) is the general fix: every entry you ever log (a scheduled run, a research trial, a promotion decision) carries its own stable identifier, and appending to the ledger first checks whether that identifier already exists; if it does, the append is silently a no-op, and the existing entry is left exactly as it was. The same one function serves two purposes that look unrelated until you actually build them: it keeps your scheduler honest about duplicate firings, and (as the research-integrity half of this module will make explicit) it keeps your experiment ledger honest about duplicate or silently-repeated trials. Both are the identical problem: don't let a system pretend something happened once when it actually happened zero or two times.
What is exponential backoff with full jitter?
The last piece of a survivable schedule is what happens when a run fails and must be retried. Retrying immediately, every time, for every failed job across a whole fleet, produces a thundering herd, every failed job hammering the same downstream service at the same instant, which is its own outage. The standard fix is exponential backoff with full jitter: for retry attempt i (counting from zero), the maximum possible delay grows exponentially (base × 2^i) capped at some ceiling so it never grows unboundedly, and the actual delay used is a uniformly random value drawn between zero and that capped maximum, so that jobs which failed at the same instant do not all retry at the same instant a second time either. Code pack exercise QD2.04-e05 (backoff_delays) builds this from your own seeded generator instance: random.Random(seed), never the module-level shared random state, for precisely the reason CS2.01's seeded-walk exercise first taught you: a shared global generator leaks state between unrelated calls, and this module's whole thesis is that a retry policy's timing is itself part of a pipeline's behavior, and therefore must be just as reproducible, on demand, as any number the pipeline computes.