Branch: CS Foundations (quant-cs) · Node CS2.01 · ~12 focused hours · Understanding gate ≥ 85%

Every function you have written so far in this branch, the return calculators, the classifiers, the loops over price series in CS1.01 through the algorithms of CS1.05, ran inside a box this program built for you. You opened a module, a Python interpreter was already waiting, the right packages were already installed, and your code either worked or it didn't, for reasons that lived entirely inside the code itself. That box is a kindness, and it is also a lie of omission. The instant you copy one of those same functions onto your own laptop, paste it into a file, and try to run it the way a real developer does (from a terminal, against your own operating system) a whole invisible layer that the sandbox was quietly managing suddenly becomes your problem: which Python runs when you type python3, whether the packages your code imports are actually installed for that Python, what directory you're standing in when a relative file path is resolved, which values of which environment variables are currently set, and whether the exact versions of everything involved are the same today as they will be in six months. None of that is "the code." All of it decides whether the code runs.

This node is the twelve hours where you stop being a guest in someone else's sandbox and start being responsible for the room the code lives in. Here is the whole module compressed into one sentence you will spend those hours earning the right to believe: the code is never the whole program, the shell, the interpreter, the installed packages, and the working directory around it are just as load-bearing, and none of that surrounding state travels with the file when you email it, push it, or hand it to a colleague. A program that is logically correct and a program that is reproducible are two different, independent properties, and the gap between them is where an enormous fraction of real-world "but it worked yesterday" and "it works on my machine" incidents live. Closing that gap is not glamorous. It is also the single most load-bearing, most transferable skill in this entire twelve-module CS branch, because every module after this one. Git, databases, systems, APIs, testing, and every notebook you will ever open in the Data Science or Quant Developer branches, silently assumes you can already do what this node teaches explicitly.

Concretely, by the end of these twelve hours you will drive a command-line shell as fluently as you drive a spreadsheet: navigating a filesystem, composing small tools into pipelines, and reading exactly what an error message is telling you instead of pattern-matching it to "restart and hope." You will understand environment variables and the PATH mechanism precisely enough to diagnose, in seconds, why the wrong Python or the wrong package version just ran. You will build an isolated virtual environment for every project you touch, so that no two projects on your machine can silently poison each other's dependencies, the single most common cause of "it works on my machine" in the entire industry. You will lay out a project the way research repositories at real quant shops and research desks are laid out, with a place for code, a place for data, a place for tests, and a README that tells a stranger, or you, eight months from now, exactly how to rebuild the whole thing from nothing. And you will learn to make a run reproducible: pinned dependencies, a documented data snapshot, and a seeded source of randomness, so that "run it again" reliably means "get the same answer," not "roll the dice."

One more thing before the mechanism: this node is deliberately not about writing more Python logic. Every exercise in its code pack is graded, but the actual subject (a real shell, a real filesystem, a real PATH) lives outside any browser sandbox this app can inspect. So the graded exercises here model the reasoning precisely in Python (PATH resolution, precedence merges, permission math, exit-code chains, reproducibility fingerprints) while the applied mini-project asks you to do the physical thing, once, on your own machine, and grade yourself against a rubric. Both halves matter. Neither substitutes for the other.

Learning objectives

By the end you can:

Navigate and manipulate a filesystem entirely from a command-line shell, the working directory, absolute versus relative paths, and the core verb set (list, change directory, make a directory, copy, move, remove), without ever opening a file browser.
Compose small, single-purpose tools into a pipeline using pipes and redirection (|, >, >>, <), and explain the three-stream model (stdin, stdout, stderr) that makes that composition possible.
Explain precisely what an environment variable is and how a shell resolves a bare command name via the PATH variable, including exactly what happens, and in which order, when two versions of the same tool are both "on the path."
Build, activate, and reason about an isolated Python virtual environment (venv); explain why environment isolation exists by naming the specific failure mode ("it works on my machine") it defeats.
Read and write dependency version constraints in a requirements.txt (==, >=, ~=, unpinned), and explain (with a concrete example) exactly how an unpinned dependency silently breaks reproducibility weeks or months after it was written.
Lay out a small project directory the way a real research repository is organized (a place for source, data, notebooks, tests, and a documented entrypoint), and state precisely what belongs in version control versus what must be excluded and regenerated.
Run a Jupyter notebook the way it is meant to be run for anything you intend to trust; name the "hidden state" failure mode caused by out-of-order or repeated cell execution, and apply the "Restart & Run All" discipline that defeats it.
Define reproducibility precisely, the same code, the same pinned environment, the same data snapshot, and the same seed must produce the same result, and build a small run that survives being deleted and rebuilt from scratch on a different machine.
Read and set file permissions (chmod, symbolic and octal notation) and a shebang line well enough to diagnose a "permission denied" error on sight, and predict which commands in an && / || / ; chain actually execute given a set of exit codes.
(Productivity objective: R10 duality.) Use an AI copilot to accelerate the unglamorous parts of environment work, drafting a shell one-liner, a requirements.txt skeleton, or a diagnosis of a cryptic error message, while applying the three-law Copilot Discipline: understand the by-hand version first, then accelerate, then always review before anything touches a real filesystem.

The duality, stated once (R10). This module carries two objectives at once, as every module in this program does. The understanding objective (items 1–9) is what the mastery gate rewards; you earn the pass by explaining the mechanism and by getting the graded code-pack exercises' hidden tests to go green, not by having watched someone else do it. The productivity objective (item 10) is the payoff you keep afterward: knowing when a copilot can safely draft a shell command or a dependency file for you, and knowing exactly what "review it first" has to mean when the thing you're about to run can delete real files. A tool can never buy you a pass on the gate, and it must never run unread against anything you can't afford to lose.

Prerequisites & connections

Builds on. CS1.01 through CS1.05, you can already write correct, idiomatic Python and reason about data structures and algorithms. Every one of those modules ran your code inside this program's own sandboxed runner, which quietly handled the interpreter, the packages, and the working directory for you. This node is the branch's sequential next step after the CS1.05 crown (Data Structures & Algorithms): it does not ask you to write cleverer logic, it asks you to take the logic you already know how to write and make it run, correctly, the same way twice, on a real machine that isn't managing any of this for you. If a term below (function, loop, dictionary) is unfamiliar rather than just "phrased differently," that is a CS1.01–CS1.05 gap, not a CS2.01 one.

Feeds forward. CS2.02 (Version Control with Git) is the very next node and leans on this one directly, every Git operation you will learn happens inside the project layout and the shell fluency this node builds; CS2.02 does not re-teach cd or a terminal prompt. CS2.03 (Databases & SQL) has you running queries and small scripts from the same command line, against the same kind of project layout. CS2.04 (Systems & Data-Intensive Thinking) assumes you already think of "the environment" as a real, inspectable thing, not a black box. CS2.05 (APIs, HTTP, JSON & Web Scraping) reuses this node's .env-for-secrets pattern verbatim for API keys, never hardcoded, never committed. CS2.06 (the branch's second crown: Software Engineering, Testing & AI-Assisted Dev) assumes a comfortable venv and a pinned requirements.txt as table stakes before it even starts on pytest and packaging. Beyond this branch, every notebook and script you will touch in the Data Science (quant-ds) and Quant Developer (quant-dev) branches assumes these habits, as a soft, recommended companion, never a hard gate (R11): you may pursue those branches in parallel with or ahead of this one, but you will feel this node's absence the first time an unpinned dependency quietly changes your backtest's answer.

R1: reference, don't duplicate. This node teaches the personal-workstation foundation of reproducibility: one project, one venv, one pinned requirements.txt, one seed, one README. QD2.04 (Reproducible Research, Deployment & the Quant Stack) is the deep, production-scale version of the same idea, scheduled and logged pipelines, secrets management at scale, and López de Prado's anti-overfitting research protocols, and it builds directly on the four-legged definition of reproducibility this node establishes; nothing there is re-taught here, and nothing here is re-derived there. Likewise, full version-control mechanics (commits, branches, .gitignore in depth) belong to CS2.02, not here, this node touches what belongs in version control only long enough to motivate why the shell and the environment matter, and points there for the how.

Core teaching content

The box this whole branch has been running inside

Before opening anything new, name what you have been standing on. Every code exercise in CS1.01–CS1.05 executed inside this app's own sandboxed Python runner: a single, fixed interpreter version, a fixed set of pre-installed packages, no real filesystem to get lost in, and no possibility of a stray environment variable changing the outcome. That sandbox is not a simplification of "real" programming (it is real Python, executing real logic) but it is real programming with one enormous variable held constant: the environment. The moment you leave the sandbox for your own laptop, that variable stops being constant, and a huge fraction of the debugging you will do for the rest of your career is debugging in that surrounding layer, not in your logic. Experienced developers do not experience this as a separate skill from "programming", they experience not-knowing-this as a wall that stops otherwise-correct code from ever running at all. This module builds you a door through that wall.

The shell as a place, not just a program

A command-line shell is a program that reads text you type and translates it into instructions for the operating system, but the far more useful mental model is that it puts you somewhere: at any moment, your shell session has a current working directory, and every relative reference you type is resolved against that one location. pwd (print working directory) tells you where you are; ls lists what's there; cd <path> moves you. The critical distinction that trips up nearly everyone at first is absolute versus relative paths. An absolute path starts from the very top of the filesystem (on Linux/macOS, with a leading /; conventionally /home/kavya/projects/nse-pipeline) and means the same location no matter where you currently are. A relative path (data/raw.csv, ../shared/config.yaml) is resolved against your current working directory, the same relative path typed from two different directories points at two entirely different files, and this is the single most common source of "file not found" errors among people who understand cd but haven't yet internalized that every relative path silently carries an invisible "from here" attached to it. Two special relative segments matter: . means "this directory" (usually a no-op, occasionally meaningful to a tool that wants an explicit path) and .. means "one directory up." A chain like ../../data means "go up two levels, then into data", and you will trace exactly this by hand in Worked Example 3 and Practice P1.

The rest of the core verb set is short and worth memorizing outright rather than looking up every time: mkdir <name> makes a new directory; cp <src> <dst> copies a file (add -r to copy a directory and everything inside it); mv <src> <dst> moves or renames a file, there is no separate "rename" command, moving a file to a new name in the same directory is renaming it; rm <file> deletes a file permanently, with no undo, no recycle bin, and rm -rf <dir> recursively force-deletes an entire directory tree without asking again, the two most dangerous keystrokes in this entire module, and the reason the Copilot Discipline callout below insists you read every character of any AI-suggested command containing them before you press Enter. Wildcards (*.csv matches every file ending in .csv in the current directory) let you address many files with one command, and are themselves a small trap: rm *.csv in the wrong directory deletes every CSV in that directory, no confirmation, no exceptions.

Composing small tools: pipes and redirection

Every running program on a Unix-like system is handed three connected streams by default: stdin (standard input, where it reads from, by default your keyboard), stdout (standard output, where its normal results go, by default your screen), and stderr (standard error, a separate channel for error/diagnostic messages, also by default your screen, which is why errors and normal output can appear interleaved even though they are, underneath, two different streams). The entire Unix philosophy of "small tools, each doing one thing well, combined" rests on the fact that you can redirect these streams instead of accepting the defaults. The pipe operator | takes the stdout of the command on its left and feeds it in as the stdin of the command on its right, letting you chain tools: cat trades.txt | grep NSE | wc -l reads a file, keeps only lines containing NSE, and counts how many are left, three tiny programs, none of which knows the other two exist, composed into one answer. You will trace exactly this pipeline by hand in Practice P7.

Redirection sends a stream to or from a file instead of the terminal. > redirects stdout into a file, completely replacing that file's previous contents, this is the single gotcha that causes the most real-world data loss in this whole topic, because command > out.txt run twice does not append two results, it silently throws away the first result and keeps only the second. >> appends to a file instead of truncating it, the safe choice whenever you mean "add to," not "replace." < redirects a file's contents in as a command's stdin, as if you had typed the file's contents by hand. You will predict the exact contents of a file after a >/>> sequence in Practice P6, and it is worth internalizing right now, before you need it under pressure: if you are not certain whether a command truncates or appends, assume it truncates, and never point > at a file whose current contents you have not already backed up or don't need.

Environment variables and PATH: how the shell finds anything

An environment variable is a named piece of text that lives in your current shell session (and, if you say so, in every program that session launches), export API_KEY=abc123 sets one for the rest of that session; it vanishes when the terminal closes unless you also add it to a shell profile file (.bashrc, .zshrc) that runs automatically every time a new session starts. This is exactly the mechanism you will use in CS2.05 to hand an API key to a script without ever writing that key into the script itself, a pattern worth adopting as a reflex starting now, not just when a module tells you to.

The single most important environment variable for everything else in this node is PATH, a colon-separated (Linux/macOS; semicolon on Windows) list of directories, searched strictly in order, left to right, every time you type a bare command name like python3 or git. The shell walks the list, checks each directory for an executable file with that exact name, and runs the very first match it finds, it does not check whether a later directory has a "better" or "more recent" version, and it does not merge the two. This single fact explains an entire category of confusing bugs: if you have two Pythons installed, say, one your operating system shipped with, and one that a virtual environment (below) puts on the path ahead of it, then whichever one's directory comes first in PATH is the one that runs, every time, with zero regard for which one has the packages you actually need installed. The command which <name> (or where on Windows) asks the shell to tell you, right now, exactly which directory it would use, the single most useful two-second diagnostic in this entire module, and the first thing to run the instant a script complains about a missing package you are sure you installed. You will compute a PATH resolution by hand in Worked Example 3 and in Practice P2, and the code pack's exercise CS2.01-e01 (which) asks you to implement the exact search rule in Python.

{
  "skill": "Writing a shell one-liner, or diagnosing a cryptic pip/venv error message",
  "use": "Turn a plain-English task ('one command that lists every .csv file under data/ modified in the last day') into a shell one-liner, or paste an error message verbatim and ask what it means and which of a few likely fixes actually applies to your situation.",
  "tools": ["Chat / coding assistants — Claude, ChatGPT, Gemini", "Shell-aware IDE copilots — GitHub Copilot, Cursor, a terminal-integrated assistant"],
  "prompt": "I'm on <macOS/Linux/WSL>, using bash. I need a single command that <goal, in plain English, naming the exact paths/patterns involved>. Show the command, then explain each flag and each operator in it in one line.",
  "verify": "Read every flag out loud before you run it — especially anything containing `rm`, `sudo`, a bare `*`, or a pipe into `sh`/`bash`. Test it first against a throwaway copy of the directory, or with a harmless stand-in command, never against your only copy of real data. A generated command that looks entirely plausible can still delete or overwrite the wrong thing — plausible is exactly what a language model optimizes for, not 'safe on your filesystem.'",
  "diy": "You must already know what redirection, pipes, and wildcards do (the two sections above) before you can safely judge whether a generated command does what it claims — without that, you cannot verify it, you can only hope.",
  "market": "IN"
}

Isolating environments: why "it works on my machine" happens

Install enough Python packages onto one system-wide Python and you will eventually hit the defining failure mode of this whole topic: project A needs pandas 1.5 and project B needs pandas 2.1, and a single system Python can only have one version installed at a time. Upgrade for B and you silently break A; pin for A and B can't run at all. This is not a rare edge case, it is the default outcome of using one Python installation for every project you will ever write, and it is the single most common reason a script that "works on my machine" fails, in some subtly different way, on a colleague's.

The fix is environment isolation: a virtual environment (venv, part of the Python standard library) is a self-contained folder holding its own copy of the Python interpreter (or a lightweight pointer to one) and its own, completely separate set of installed packages. You create one with python3 -m venv venv_name and turn it "on" for your current shell session with source venv_name/bin/activate (macOS/Linux), which does exactly one load-bearing thing: it silently prepends that venv's own bin/ directory to the front of your PATH. That is the entire mechanism. There is no magic, activating a venv is just a PATH edit, which is precisely why Worked Example 3's by-hand PATH trace and this section belong together. Your prompt typically grows a (venv_name) prefix as a visual reminder, but the reliable check (the one that catches a stale prompt or a half-broken activation) is always the same two-second command from the previous section: which python3. deactivate reverses it, restoring your previous PATH. Conda environments (common in data-science and quant workflows) solve the identical isolation problem with a more powerful tool that also manages non-Python dependencies and multiple Python versions at once, this node teaches venv hands-on because it ships with every Python install and needs nothing extra, and asks you to carry the concept, one interpreter, one package set, per project, always, forward to conda or any other environment manager you meet later, rather than re-learning isolation from zero each time.

Once you're inside an isolated environment, pip install <package> installs a package into that environment only, and pip install package==2.1.0 pins it to an exact version rather than "whatever is newest today." pip freeze prints every currently installed package with its exact installed version, in the standard name==version format, redirect that output into a file (pip freeze > requirements.txt, using the redirection operator from two sections ago) and you have captured, in one plain-text file, the precise environment your code was written and tested against. pip install -r requirements.txt reads that file back and installs every pin exactly. A requirements.txt line can also use looser constraints: >=1.24 means "1.24 or any later version, with no upper bound at all" (the most permissive and the riskiest for reproducibility, since a major version released two years from now still satisfies it); ~=1.11.0 (the "compatible release" operator, from the official version-specification standard, PEP 440, see Reading & Resources) means "1.11.x patch releases only, never 1.12"; and no constraint at all means "whatever is newest at install time," which is not really a pin at all, it is a promise to drift. You will rank these by reproducibility risk in Practice P5, and the code pack's CS2.01-e03 (satisfies) has you implement the actual constraint-checking logic, including the classic gotcha that version numbers must be compared as integers, component by component, never as plain text: "1.10.0" is a later release than "1.9.0", even though, read as a string, "1.10.0" sorts before "1.9.0" alphabetically. A tool that compares versions as strings will silently misorder your dependencies the very first time a package crosses from single- to double-digit minor versions.

Reproducible project layout

A project that anyone (including future you) can pick up, understand, and rebuild follows a small number of conventions that matter far more than any specific folder name. A src/ (or a top-level package folder named after the project) holds the actual code. A data/ folder, often split into data/raw/ (never edited, exactly as received) and data/processed/ (regenerable output of your code run against the raw data), holds inputs and derived intermediate outputs, kept visibly separate so nobody accidentally treats a processed file as if it were the source of truth. A notebooks/ folder holds exploratory Jupyter work (see the next section for the discipline that keeps it honest). A tests/ folder holds the automated checks CS2.06 will teach you to write properly. A requirements.txt (or the more modern pyproject.toml) at the top level pins every dependency. And a README.md states, in plain language a stranger could follow with no other context, exactly how to rebuild the environment and reproduce a result, this is not bureaucratic decoration, it is the single artifact that turns "I think I remember how I ran this" into an actual, checkable fact.

Two disciplines cut across the whole layout. First, config over hardcoding: a file path, an API key, a database URL, or a "which market am I processing today" flag should live in a configuration source, an environment variable, a .env file, a small config file, never baked as a literal string inside your logic, because the moment it needs to change for a different machine, a different analyst, or a different market, hardcoded values force you to edit and re-test code that was otherwise perfectly correct. Second, and non-negotiably: secrets never go in committed code. An API key or database password belongs in a .env file (a plain-text file of KEY=value lines, loaded into environment variables at runtime) that is explicitly excluded from version control, the project instead commits a .env.example template showing every expected key name with a placeholder value, so a new teammate knows exactly what to fill in without ever seeing a real secret. CS2.02 teaches the Git mechanics (.gitignore, what to commit versus exclude) in full; this node's job is narrower and comes first: know which files are source-of-truth-you-must-keep (code, the pinned dependency file, the README, raw data you cannot regenerate) and which are regenerable-therefore-disposable (a venv folder, cached bytecode, most processed data), you will sort exactly this list in Practice P10.

Running Jupyter notebooks the way they're meant to be run

A Jupyter notebook's defining feature, you can run cells in any order, any number of times, and each one remembers whatever state earlier executions left behind: is also its defining trap. A notebook's cells run top to bottom on the screen, but nothing forces you to execute them top to bottom, exactly once; click back up to an earlier cell, tweak a value, and re-run it, and the notebook's actual execution history no longer matches its visual order at all. This is hidden state: a variable can hold a value that no single visible pass through the notebook, read top to bottom, would ever produce, and the notebook gives you no visual warning that this has happened. You will predict exactly this outcome, and why it is dangerous, not just quirky, in Practice P11.

Two practical consequences follow. First, a Jupyter kernel (the running Python process actually executing your cells) is tied to a specific Python interpreter, exactly the same way a terminal session is; that interpreter should be the one inside your project's venv, selected explicitly when you start the kernel, or your notebook will silently run against whatever Python Jupyter happened to default to, with whatever packages that Python has, unrelated to your project's pinned requirements.txt. Second, and this is the discipline that actually defeats hidden state: before you trust a notebook's output (before you write a number from it into a report, or claim a result reproduces) Restart the kernel and Run All cells, top to bottom, from a clean slate, exactly once. If the output changes when you do this, whatever you were looking at a moment ago was hidden state, not a real result. Production tools like papermill and nbconvert exist precisely to execute a notebook end-to-end, non-interactively, as if it were a script (worth knowing by name, not needed hands-on here) and the moment a piece of exploratory notebook work is meant to run unattended or be trusted repeatedly, the right move is usually to graduate it into a plain .py script, which cannot execute out of order because it has no "click an earlier cell" affordance to misuse in the first place.

Permissions and executable scripts

Every file on a Unix-like system carries a permission string like -rwxr-xr-x, read as three groups of three: what the file's owner can do, what its group can do, and what everyone else can do, each group independently granting read, write, and execute. The same permissions are also written as three octal digits, one per group, where each digit from 0–7 is the sum of read (4), write (2), and execute (1) that group has: 7 = rwx (4+2+1, everything), 5 = r-x (4+1, read and execute but not write), 6 = rw- (4+2), 0 = --- (nothing). chmod 755 script.sh sets owner to rwx, group and other to r-x, the classic "runnable by anyone, editable only by the owner" permission for a script. You will convert between octal and symbolic form by hand in Worked Example 4 and Practice P3, and the code pack's CS2.01-e05 has you implement the conversion in Python, including the realistic bug it's designed to catch: get the digit order backwards and a symmetric permission like 644 still looks right by accident, while an asymmetric one like 750 comes out silently wrong.

A file needs its execute bit set before the operating system will run it directly as ./script.sh at all, a text file with perfectly correct contents but permissions -rw-r--r-- (644, no execute bit anywhere) fails with "Permission denied" before your code is even considered, which is the single most common cause of learners believing a script is "corrupted" when it is, in fact, working exactly as designed (see Common Mistakes). But the execute bit alone is not sufficient for a script written in a language like Python: the operating system also needs to know which interpreter to hand the file to, and that is the job of the shebang line, the very first line of the file, starting with #!, conventionally #!/usr/bin/env python3 for a Python script (the env indirection finds whichever python3 is first on the current PATH, tying this section back to the PATH section above rather than hardcoding one machine's specific interpreter path). Get both right (the shebang line present, and the execute bit set via chmod +x) and ./script.py runs directly. Get either wrong, or simply skip both, and python3 script.py (explicitly naming the interpreter on the command line) always works regardless, because it never needs the execute bit or a shebang to know what to do. You will diagnose exactly this "permission denied vs. missing shebang" pair, and give both fixes, in Practice P15.

Chain reaction: command sequencing and exit codes

Every command that finishes running produces an exit code, a small integer, 0 meaning success and any non-zero value meaning some kind of failure (the specific non-zero value is tool-specific and rarely needs memorizing; only "zero versus non-zero" matters for chaining). Three operators let you sequence commands based on those codes, and their differences are exactly the kind of thing that looks interchangeable until a CI pipeline silently deploys a broken build. ; runs the next command unconditionally, regardless of what the previous one returned, it's a plain separator, nothing more. && runs the next command only if the previous one succeeded (exit code 0), "run this, and if that worked, then run this." || runs the next command only if the previous one failed (non-zero), "run this, or if that failed, run this instead" (a common pattern for fallback or alerting logic). When a command is skipped by && or ||, it does not "count" as having run for the purposes of evaluating whatever operator comes after it, the exit code that the next operator checks is always the exit code of the last command that actually executed, not a fixed position in the chain. You will trace a four-command chain mixing && and || in Worked Example, wait, in Practice P8, and the code pack's CS2.01-e06 (run_chain) has you implement precisely this rule; the surprising result you'll hit there (a failing middle step does not necessarily stop a later step from running, once a || has intervened) is exactly the kind of thing worth predicting before you trust a real CI script.

Making a run reproducible end-to-end

Pull every thread in this node together and reproducibility resolves into four concrete, checkable legs, none of which is optional and none of which substitutes for another: (1) the exact code and exact dependency versions (a pinned requirements.txt, not a floating one); (2) a fixed seed for every source of randomness the run touches, so "run it again" means "get the identical numbers," not "get a plausible-looking new set of numbers"; (3) the exact data snapshot used, identified well enough (an id, a seed, a snapshot date, see the dataset manifest fields the code pack references) that "the same data" is a checkable fact, not an assumption; and (4) a documented, exact sequence of commands, the README's "how to reproduce" section, specific enough that someone with none of your memory of the project could rebuild the environment and rerun it and land on the same answer. Miss any one of the four and you have something that merely looks reproducible: correct code with an unpinned dependency, or pinned dependencies with an unseeded random draw, or a perfectly deterministic script pointed at a data file that quietly got overwritten last Tuesday. You will work through exactly this failure, end to end, in Worked Example 6, and the code pack's CS2.01-e07 (a seeded random walk) and CS2.01-e08 (a reproducibility fingerprint combining the pinned dependencies with the dataset snapshot's own id/seed/date into one short, checkable string) are the two exercises that operationalize legs 2–3 directly in code, including a genuinely useful, order-independence property: two teammates who happened to install their packages in a different sequence, or list them in a different order, must still get the identical fingerprint if their environments are truly the same, which means the fingerprint has to be built by sorting before hashing, never by hashing whatever order a dictionary happens to iterate in.

It is worth naming the tool this whole discipline is heading toward, even though it sits outside what this sandbox can run: a container (Docker being the dominant example) packages the code, the interpreter, the dependencies, and often the operating system itself into one image that runs identically anywhere a container engine is installed, the closest thing to "leg 1 through 4, automated and enforced by the tooling rather than by your own discipline." This node does not teach containers hands-on, because building and running one needs a real container engine this sandbox cannot provide; QD2.04 picks up reproducibility at exactly this production scale. What matters here is the sequencing: a container automates a discipline you must already understand by hand, the same way a calculator automates arithmetic you must already understand, reach for it later without ever having done the four legs by hand yourself, and the first time the container doesn't behave the way you expect, you will have nothing to reason from.

{
  "skill": "Scaffolding a requirements.txt or a project README's 'how to reproduce' section",
  "use": "Generate a first-draft requirements.txt from a description of what your script imports, and a first-draft README section describing the exact venv-creation, activation, install, and run commands needed to rebuild the environment from nothing.",
  "tools": ["Chat / coding assistants — Claude, ChatGPT", "IDE copilots that can scan your project's imports — GitHub Copilot, Cursor"],
  "prompt": "Here are the imports my script uses: <list>. Draft a requirements.txt pinning each to a specific, currently real, released version, and a short README 'How to reproduce' section with the exact venv-creation, activation, install, and run commands for <macOS/Linux/Windows>.",
  "verify": "Delete the environment and rebuild it FROM the generated requirements.txt, then rerun the script, before trusting any of it. A model can hallucinate a package name that doesn't exist or a version number that was never published — the only real check is the rebuild actually working. If it fails or drifts from what you expected, the draft was wrong, not your setup.",
  "diy": "You must already be able to write the pins and the project layout by hand (the two sections above) — a copilot only saves typing on a first draft you were always going to verify by literally rebuilding the environment from it.",
  "market": "US"
}

Common failure modes you will now recognize instantly

Put together, this module gives you a short, reusable checklist for the single most common class of bug a working analyst hits that has nothing to do with logic: "the code is right, but it didn't run/gave a different answer." which python3 and which pip catch a PATH/venv mismatch in seconds. Reading a requirements.txt for floating versus pinned constraints catches silent dependency drift before it costs you an afternoon. Restarting the kernel and running all cells catches hidden notebook state before you trust a number from it. And asking "what exactly changed, code, environment, data, or seed?" catches the rest. The next section turns eight specific, real misconceptions in this space into the instinct that replaces them.

Common mistakes & how experts think differently

"The error said to install the package, so I'll just pip install it and move on." Installing a package with no virtual environment activated installs it into whatever Python is currently first on your PATH, often the system Python every other tool on your machine also uses. Do this across a few projects with conflicting version needs and you have quietly built the exact "it works on my machine" trap this node exists to prevent. The expert's reflex, before any install, is which python3, confirm you're inside the project's own venv first, every time, not just the first time.

"My requirements.txt lists every package I use, so it's reproducible." Listing packages with no version pins (or loose ones like numpy with nothing after it) is not a pin, it's a promise to drift, whoever installs from that file next month gets whatever is newest then, which may behave subtly differently from what you tested against. The expert treats an unpinned dependency exactly like an unseeded random number generator: syntactically fine, silently non-reproducible.

"It printed the right number in my notebook, so the logic is correct and the script version will match." A notebook's on-screen cell order and its actual execution history can diverge the moment you re-run an earlier cell, the number on your screen may depend on a stale, hand-edited variable no single top-to-bottom pass would ever produce. The expert never trusts a notebook result without first doing "Restart & Run All" from a clean kernel, exactly once, and treats any discrepancy as the real answer, not the on-screen one.

"Permission denied, the file must be corrupted." A -rw-r--r-- (644) permission string is a perfectly intact, perfectly readable file that simply has no execute bit set for anyone, the operating system refuses to run it directly, on principle, before your code is ever considered. The expert's first move on "Permission denied" is ls -l to read the permission string, not a re-download or a rewrite.

"The PATH variable is basically magic; I don't need to understand exactly how it works." PATH is a plain, ordered list, searched left to right, first match wins, no cleverness, no "best" match, no merging of two installations. Every "wrong version ran" bug in this whole topic is this one mechanical fact, misunderstood. The expert can trace a PATH resolution by hand, the way Worked Example 3 does, in under a minute, because it is genuinely that mechanical once you've done it once.

"This shell command from Stack Overflow (or an AI) looks right, so it's safe to run." A command can be perfectly grammatical, entirely plausible-looking, and still delete the wrong directory, overwrite the wrong file, or install something you didn't intend, especially anything containing rm -rf, sudo, an unquoted wildcard, or a pipe into sh/bash. The expert reads every flag of a command they didn't write themselves before running it against anything they can't afford to lose, and tests it against a throwaway copy first whenever there's any doubt, exactly the discipline both ai-augment-json callouts in this node insist on.

"> and >> are basically the same thing, I'll just use whichever." > silently replaces a file's entire previous contents; >> adds to the end. Running the same > command twice does not double your output, it destroys the first run's output entirely, with no warning and no recovery. The expert defaults to >> whenever the intent is "add to," and reserves > deliberately for "I mean to replace this, and I've already confirmed I don't need what's currently there."

"Reproducibility is a DevOps/Docker problem, that's not something I need to think about." Containers automate the four-legged discipline this node teaches (pinned code and deps, a seed, a data snapshot, a documented run), they do not replace the need to understand it. A learner who reaches for a container without first having pinned a requirements.txt by hand, seeded a random draw by hand, and written a "how to reproduce" README by hand has automated a discipline they don't actually have, and the first time the container misbehaves, they have nothing to reason from.

The thread through all eight: the expert has fully internalized that correct logic and reproducible execution are two separate properties, checked two separate ways, and that the checking tools (which, ls -l, "Restart & Run All", reading a requirements.txt line by line) take seconds each and turn "it should work" into "I confirmed it works," every single time, before trusting a result enough to act on it.

Worked examples

Worked example 1: "ModuleNotFoundError: No module named 'pandas'" (India, ₹)

Kavya, an analyst in Bengaluru, is handed a colleague's script, build_nse_report.py, which reads a bundled end-of-day NSE-style price file and prints a small report, the colleague's exact words are "it just works, run it." The file's project folder already contains a venv/ directory and a requirements.txt pinning pandas==2.1.0. Kavya opens a terminal, cds into the project folder, and runs python3 build_nse_report.py. It fails immediately: ModuleNotFoundError: No module named 'pandas'. She is confused, the colleague clearly has pandas installed, since the script runs fine for them.

Solution. Step 1, name the mechanism before guessing a fix. A ModuleNotFoundError means the Python that just ran genuinely does not have that package installed for it specifically, it says nothing about whether pandas is installed somewhere on the machine. The question is never "is pandas installed?"; it is "is pandas installed for the exact Python that python3 currently resolves to?"

Step 2: run the two-second diagnostic. which python3 prints /usr/bin/python3, the operating system's own, system-wide Python, not anything inside the project. python3 -m pip show pandas against that same Python confirms: not installed there. The colleague's venv/ folder exists on disk, but Kavya never activated it, her shell session's PATH still has the system Python's directory ahead of anything project-specific, exactly the mechanism from the PATH section above.

Step 3, fix it precisely, not by reinstalling at random. source venv/bin/activate, the prompt grows a (venv) prefix, and, more reliably, which python3 now prints something like /home/kavya/projects/nse-pipeline/venv/bin/python3, confirming the venv's own interpreter is now first on PATH. pip install -r requirements.txt installs the pinned pandas==2.1.0 into that environment (if it isn't already there). Re-running python3 build_nse_report.py now succeeds, printing a line for an illustrative NSE-listed stock, say RELIANCE: start=2820.00 end=2847.60 return=0.98% (illustrative, synthetic figures, not a live market quote), using exactly the format_report-style output pattern from CS1.01.

Read it. Nothing about Kavya's logic was ever in question, the script's code was correct the entire time. The bug lived one layer below the code, in which interpreter a bare python3 happened to resolve to, and it dissolved the instant she checked that layer directly instead of guessing at package reinstalls. This is the exact diagnostic reflex (which python3 first, always) that the rest of this node exists to install.

Worked example 2: The backtest that gives two different answers (US, $)

Sam, an analyst in Chicago, runs the same backtest script against an illustrative US equity sleeve on two machines (Sam's own laptop, and a colleague's) and gets two different Sharpe ratios: 1.42 on one machine, 1.58 on the other (illustrative figures). Same script, same file, copied byte-for-byte. Sam's first instinct is that one machine's copy of the file must be stale or corrupted.

Solution. Step 1: rule out the file itself. A checksum of the script on both machines matches exactly, it is genuinely the identical code. Whatever is different is not in the file; it is in the environment around it, exactly this node's opening claim.

Step 2, check the two things a "same code, different answer" bug is almost always hiding. First, requirements.txt: it lists pandas and numpy with no version pins at all. pip show pandas on each machine reveals two different installed versions, resolved independently, months apart, by two separate pip install -r requirements.txt runs against an unpinned file, exactly the drift Common Mistake #2 names. Second, the backtest's resampling step calls Python's global random.random() directly, with no seed ever set, meaning every single run, on every machine, draws a genuinely different sequence of random numbers, whether or not the dependency versions also differed.

Step 3, fix both legs, not just one. pip freeze on the machine whose result is trusted, saved as the new, fully pinned requirements.txt (pandas==2.1.0, numpy==1.26.0, illustrative exact versions). The resampling step is rewritten to build its own generator explicitly: rng = random.Random(seed), with seed an explicit parameter threaded in from the top of the script rather than left to the shared global generator, exactly the pattern the code pack's CS2.01-e07 has you implement. Both machines rebuild their environment from the newly pinned file and rerun with the same explicit seed. Both now print the identical Sharpe ratio, say 1.50 (illustrative), byte-for-byte, every time, on either machine.

Read it. Two completely independent failures were stacked on top of each other, and either one alone would have been enough to make "same code, different answer" true. This is precisely why reproducibility is defined as four legs, not one: fixing only the dependency pins, or only the seed, would have left the bug half-alive, waiting to resurface the next time only the other leg happened to differ.

Worked example 3: Tracing a PATH resolution by hand (mechanism, illustrative)

Kavya's PATH, after activating her project's venv, is (illustratively) this three-directory list, in order: /home/kavya/projects/nse-pipeline/venv/bin, /usr/local/bin, /usr/bin. Each directory's relevant contents (illustrative): the venv's own bin/ has python3 (with pandas installed inside that venv) and pip; /usr/local/bin has a separately-installed python3 (no pandas) and git; /usr/bin has the operating system's own python3 (no pandas) and awk. What runs when Kavya types python3, and what runs when she types git?

Solution. Step 1, apply the rule exactly, one directory at a time, in order. For python3: check /home/kavya/projects/nse-pipeline/venv/bin first, it contains a python3, stop immediately; this is the match, regardless of what either later directory also contains. The venv's own Python runs, and it has pandas installed, so any script importing pandas succeeds.

Step 2, trace a case where the first directory does NOT have a match. For git: check /home/kavya/projects/nse-pipeline/venv/bin first, no git there (venvs isolate Python packages, not general system tools) (move to /usr/local/bin) it has git: stop; this is the match. The third directory, /usr/bin, is never even inspected, because the search already stopped one directory earlier.

Step 3, now trace the failure case from Worked Example 1. Before activation, Kavya's PATH was just /usr/local/bin:/usr/bin (no venv directory present at all, since it was never prepended). Typing python3 there checks /usr/local/bin first, it has a python3, no pandas, stop; this is the match, and it is the wrong one for her purposes, which is exactly the ModuleNotFoundError she hit.

Read it. Three different commands, one identical rule applied three times: search the list in order, stop at the first match, never look further and never compare quality. which <name> performs exactly this trace instantly instead of you doing it by hand, but doing it by hand once, as you just did, is what makes the output of which legible instead of magical. The code pack's CS2.01-e01 asks you to implement this exact search rule as a function.

Worked example 4: "Permission denied" is not corruption (chmod, by hand)

A learner writes a small shell script, run_pipeline.sh, saves it, and runs ./run_pipeline.sh. The shell replies: bash: ./run_pipeline.sh: Permission denied. cat run_pipeline.sh shows the file's contents are completely intact and correctly written. The learner's first instinct is to assume the file somehow got corrupted during saving.

Solution. Step 1, read the permission string before assuming anything about the contents. ls -l run_pipeline.sh prints a line starting -rw-r--r--. Read it in its three groups of three: owner rw- (read, write, no execute), group r-- (read only), other r-- (read only). No execute bit is set, for anyone, anywhere in that string.

Step 2, convert to octal and confirm what's missing. rw- = read(4) + write(2) = 6; r-- = read(4) alone = 4; r-- again = 4. The mode is 644, an extremely common default permission for a newly created text file, and the reason nearly everyone hits this exact error the first time they write a script rather than just editing existing ones. The operating system will happily let you read and edit this file; it refuses, on principle, to execute it directly, because no execute bit is present for the user attempting to run it.

Step 3, fix it, and verify the fix the same way you diagnosed the problem. chmod +x run_pipeline.sh sets the execute bit for all three classes (owner, group, other), chmod +x with no u/g/o prefix is shorthand for "everyone." ls -l run_pipeline.sh now shows -rwxr-xr-x, owner rwx (7), group r-x (5), other r-x (5), mode 755, the classic "anyone can run it, only the owner can edit it" script permission. ./run_pipeline.sh now runs (assuming the file also has a correct shebang line, see the section above; a missing shebang is a second, independent failure mode traced fully in Practice P15).

Read it. "Permission denied" is the operating system refusing to even attempt execution, it is evidence about one specific permission bit, and nothing whatsoever about whether the file's contents are correct or intact. The fix took three commands and about ten seconds once the permission string was actually read instead of guessed at. The code pack's CS2.01-e05 has you implement the octal-to-symbolic conversion in code, including a version of the exact bug (digit order reversed) that would make this diagnosis silently wrong for an asymmetric mode.

Worked example 5: Who wins? Tracing environment-variable precedence by hand

A small pipeline reads its configuration from four layers, from lowest to highest precedence: built-in defaults in the code itself, a project .env file, the real shell environment (shell_env, whatever is currently export-ed), and explicit command-line --flag overrides (cli_overrides). For one run, the four layers are (illustrative): defaults = {"LOG_LEVEL": "INFO", "MAX_ROWS": "1000", "REGION": "US"}; dotenv = {"MAX_ROWS": "5000", "VENDOR_KEY": "sandbox-key-001"}; shell_env = {"REGION": "IN", "VENDOR_KEY": "prod-key-778"}; cli_overrides = {"LOG_LEVEL": "DEBUG"}. What is the final, fully resolved configuration?

Solution. Step 1, start empty and apply layers in precedence order, lowest first, letting each later layer overwrite any key it also sets. Start: {}. Apply defaults: {"LOG_LEVEL": "INFO", "MAX_ROWS": "1000", "REGION": "US"}.

Step 2: apply the .env file. It sets MAX_ROWS (overwriting 1000 with 5000) and introduces a new key, VENDOR_KEY. Running total: {"LOG_LEVEL": "INFO", "MAX_ROWS": "5000", "REGION": "US", "VENDOR_KEY": "sandbox-key-001"}.

Step 3: apply the shell environment. It overwrites REGION (US → IN) and overwrites VENDOR_KEY (the sandbox key → a production key). Running total: {"LOG_LEVEL": "INFO", "MAX_ROWS": "5000", "REGION": "IN", "VENDOR_KEY": "prod-key-778"}.

Step 4, apply the CLI overrides, the highest layer. It overwrites only LOG_LEVEL (INFO → DEBUG). Final configuration: {"LOG_LEVEL": "DEBUG", "MAX_ROWS": "5000", "REGION": "IN", "VENDOR_KEY": "prod-key-778"}.

Read it. Every key's final value is decided by exactly one rule (whichever layer touched it last, in the fixed precedence order) applied independently, key by key; a layer that never mentions a given key simply leaves whatever the previous layers already decided untouched. Reverse any two layers in this precedence order and at least one key's final value changes, which is exactly why "which layer wins" has to be a fixed, documented convention for a whole team, not an assumption each person makes independently. The code pack's CS2.01-e02 (resolve_env) has you implement this exact four-layer merge, and Practice P9 gives you a second scenario to trace on your own.

Worked example 6: Reproducibility, end to end, across two markets (India ₹ and US $)

Two analysts share one research repository: one processes an illustrative NSE-style equity file (₹) from Mumbai, the other an illustrative US EOD-style equity file ($) from Chicago, using the same shared pipeline code. Both pipelines include a Monte-Carlo-style resampling step. The requirements.txt currently has no version pins, and the resampling step uses the shared global random generator with no seed. Both analysts, independently, get results they cannot reproduce even on their own machine a week later: let alone match each other's.

Solution. Step 1, separate the four legs and audit each one independently, rather than guessing at a single fix. Code and dependencies: the code itself is shared and identical (checked via version control, CS2.02's territory), but the unpinned requirements.txt means each analyst's own pip install resolved a slightly different pandas/numpy version, on different days. Seed: the resampling step draws from the shared global generator, seeded from OS entropy by default, genuinely different random numbers on every single invocation, on every machine, including reruns on the exact same machine. Data snapshot: each analyst's raw data file has an id and a date, but neither is recorded anywhere a rerun could check against, "the file currently on my disk" is not a verifiable claim six months from now. Documentation: there is no README section stating the exact commands used to set up and run the pipeline.

Step 2, fix each leg with the tool this node built for it. pip freeze from a known-good environment becomes the new, fully pinned requirements.txt, committed and shared. The resampling step is rewritten to accept an explicit seed parameter and build its own random.Random(seed) generator, never the shared global one, with the seed itself recorded in the run's configuration. Each dataset gets a manifest entry recording, at minimum, its id, a seed (for any dataset that is itself synthetically generated), and a snapshotDate. The README gains a "How to reproduce" section naming the exact venv-creation, activation, install, and run commands.

Step 3, verify the fix with a fingerprint, not just a feeling. A small fingerprint(dependencies, dataset_manifest) function, exactly the code pack's CS2.01-e08, combines the sorted, pinned dependency list with the dataset manifest's id/seed/snapshotDate into one short hash string. Both analysts compute it: the Mumbai analyst's NSE-side run and the Chicago analyst's US-side run naturally produce different fingerprints from each other (different dataset_manifest.id (nse-eod versus us-eod) and different underlying data), which is correct and expected; but each analyst, rerunning their own side twice, now gets the identical fingerprint both times, and a teammate who rebuilds the identical environment from the pinned requirements.txt and processes the identical dataset id/seed/snapshot gets that same fingerprint too, regardless of what order they happened to pip install things in.

Read it. Nothing here required cleverer code, the actual pipeline logic never changed. What changed was making every one of the four legs an explicit, checkable fact instead of an implicit assumption, and building one small tool (the fingerprint) whose entire job is to make "did anything actually change?" a yes/no question instead of a guess. That is the whole discipline this node teaches, applied once, completely, end to end.

Practice set

Work each problem fully before reading its solution. Every path, permission string, and PATH list below is illustrative/synthetic (R3) unless stated otherwise.

P1 (guided). Absolute vs. relative, traced by hand. Starting at /home/kavya/projects, run, in order: cd nse-pipeline, then cd ../../, then cd data/raw. What does pwd print after all three?

Solution. /home/kavya/projects → cd nse-pipeline → /home/kavya/projects/nse-pipeline. cd ../../ goes up two levels from there: one level up is /home/kavya/projects, two levels up is /home/kavya. Then cd data/raw (relative, from /home/kavya) → /home/kavya/data/raw. Each cd is resolved against wherever the previous one left you, there is no "reset to start" between commands.

P2 (guided). PATH resolution, a second trace. PATH lists, in order: /opt/miniconda3/envs/quant/bin, /usr/local/bin, /usr/bin, /bin. Contents: the first directory has python3 and jupyter; the second has python3 and git; the third has python3 and awk; the fourth has sh. Which directory answers which jupyter? Which answers which git?

Solution. jupyter exists only in the first directory (/opt/miniconda3/envs/quant/bin), that is the match, found immediately. git does not exist in the first directory, so the search continues; it is found in the second directory (/usr/local/bin), the third and fourth directories are never even inspected, because the search already stopped.

P3 (guided). Octal to symbolic, a second value. Convert octal mode 640 to its 9-character symbolic permission string, and say, in one sentence, what kind of file commonly uses this permission.

Solution. Digits 6, 4, 0 → owner rw- (4+2), group r-- (4), other --- (nothing) → rw-r-----. This is a common permission for a file containing secrets shared within one team but readable by nobody else on the machine: e.g., a config file with database credentials, readable by the owner and their group, invisible to everyone else.

P4. Confirming a venv activation, reliably. Before activating a project's venv, which python3 prints /usr/bin/python3. After running source venv/bin/activate, what is the one command you should run to reliably confirm the venv is now active, and what output would confirm it (in general terms, not an exact path)?

Solution. Run which python3 again. A successful activation is confirmed by the output now pointing inside the venv's own folder: something ending in .../venv/bin/python3 rather than /usr/bin/python3. The prompt growing a (venv) prefix is a helpful visual cue but is not the reliable check (a stale or customized prompt can lie); which python3 reports the actual PATH resolution, which is the mechanism itself, not a cosmetic hint about it.

P5. Ranking pinning strategies by reproducibility risk. Rank these four requirements.txt lines from safest to riskiest for reproducibility, and justify the ranking in one sentence each: pandas==2.1.0, numpy>=1.24, scipy~=1.11.0, requests (no constraint at all).

Solution. Safest to riskiest: (1) pandas==2.1.0: an exact pin, never resolves to anything else, ever. (2) scipy~=1.11.0, allows only patch-level updates within 1.11.x, a small and usually-safe drift window. (3) numpy>=1.24, allows any future version with no upper bound, including a major version that could change behavior entirely. (4) requests, no constraint at all, meaning "whatever is newest right now," which isn't really a pin, it's a promise that the exact version used is never even recorded.

P6. Redirection, traced by hand. An empty file out.txt is acted on by, in order: echo "line1" > out.txt, echo "line2" >> out.txt, echo "line3" > out.txt, echo "line4" >> out.txt. What are the final contents of out.txt?

Solution. Step 1: > writes "line1\n", replacing the (empty) prior contents. Step 2: >> appends "line2\n" → file now "line1\nline2\n". Step 3: > truncates again (this is the gotcha) destroying both prior lines and writing only "line3\n". Step 4: >> appends "line4\n". Final contents: "line3\nline4\n", line1 and line2 are gone entirely, silently, the moment the second > ran.

P7. Pipe composition, traced by hand. trades.txt contains five lines: NSE,TCS,BUY / NSE,INFY,SELL / US,AAPL,BUY / NSE,TCS,SELL / US,MSFT,BUY. What does cat trades.txt | grep NSE | wc -l print?

Solution. cat trades.txt streams all five lines to grep NSE, which keeps only lines containing the text NSE, three lines survive (NSE,TCS,BUY; NSE,INFY,SELL; NSE,TCS,SELL). wc -l counts the lines it receives on its stdin: 3.

P8. Exit-code chaining, the surprising case. Commands lint (exit 0), test (exit 1), build (exit 0), deploy (exit 0) are chained as lint && test || build && deploy. Which commands actually run?

Solution. lint always runs first (exit 0). && after lint: previous exit was 0, so test runs (exit 1, fails). || after test: the last command that ran (test) failed, so build runs (exit 0, succeeds). && after that: the last command that ran (build) succeeded, so deploy runs too. All four commands run, a failing middle step (test) does not stop the chain, because the intervening || "rescues" it the moment build succeeds. This is exactly why reading an &&/|| chain requires tracking the exit code of the last command that actually ran, not assuming any fixed pattern.

P9. Environment-variable precedence, a second trace. defaults = {"THREADS": "4"}, dotenv = {} (empty), shell_env = {"THREADS": "8", "DEBUG": "true"}, cli_overrides = {"DEBUG": "false"}. Same precedence as Worked Example 5 (defaults < dotenv < shell_env < cli_overrides). What is the final configuration?

Solution. Start {} → +defaults → {"THREADS": "4"} → +dotenv (empty, no change) → {"THREADS": "4"} → +shell_env → {"THREADS": "8", "DEBUG": "true"} (overwrites THREADS, adds DEBUG) → +cli_overrides → {"THREADS": "8", "DEBUG": "false"} (overwrites DEBUG only). THREADS ends at 8 because nothing above shell_env mentions it again.

P10. What belongs in version control? A project directory contains: src/, data/raw/, data/processed/, venv/, __pycache__/, notebooks/, requirements.txt, README.md, .env. Sort these into "commit to version control" versus "exclude and regenerate/never commit," and give the one-sentence reason for the one item that is not like the others.

Solution. Commit: src/, notebooks/, requirements.txt, README.md (and usually data/raw/, since raw data often cannot be regenerated from anything else). Exclude, regenerate instead: venv/ (rebuild from requirements.txt), __pycache__/ (regenerated automatically by Python itself), data/processed/ (regenerable by rerunning the code against the raw data, document that assumption in the README). The one item that is not like the others: .env. Every other excluded item is excluded because it's regenerable and bulky; .env is excluded because it holds secrets, committing it would leak credentials into the project's history permanently, which is a security failure, not merely a tidiness one. (Full .gitignore mechanics: CS2.02.)

P11. The out-of-order notebook. Cell 1: x = 10. Cell 2: x = x * 2. Cell 3: print(x). A learner runs cell 1, then cell 2, then (while still deciding what to write in cell 3) clicks back and re-runs cell 2 two more times, then finally runs cell 3. What does cell 3 print, and why is the on-screen notebook order misleading about it?

Solution. Cell 2 actually executed three times total (once, then twice more), each time doubling whatever x currently held: 10 → 20 → 40 → 80. Cell 3 prints 80. Anyone reading the notebook top to bottom (cell 1, cell 2, cell 3, each apparently run once) would predict 20, and would be wrong, because the notebook's visual order says nothing about how many times, or in what actual sequence, each cell was executed. Only "Restart the kernel & Run All," which forces exactly one top-to-bottom pass, would reveal the true, reproducible answer (20, in that clean case) and expose that 80 was hidden state, not a real result.

P12. Path normalization, a third case. Resolve the relative path ../processed/../clean/2026.csv against base /data/nse/raw.

Solution. Combined: /data/nse/raw/../processed/../clean/2026.csv. Process segment by segment with a stack: data → push; nse → push; raw → push (stack: [data, nse, raw]); .. → pop ([data, nse]); processed → push ([data, nse, processed]); .. → pop ([data, nse]); clean → push ([data, nse, clean]); 2026.csv → push. Final stack: [data, nse, clean, 2026.csv] → /data/nse/clean/2026.csv.

P13. Determinism, reasoned through. You call a seeded random-walk function with step_pct=0.0 twice, same seed: seeded_walk(2026, 100.0, 4, 0.0). What do you get both times, and why? Separately: a colleague calls seeded_walk(2026, 100.0, 4, 0.01) (a non-zero step size) on their own laptop, a month later. Will they get the exact same list you would, calling it with those same four arguments? Why or why not?

Solution. With step_pct=0.0, every random draw is uniform(-0.0, 0.0), which is always exactly 0.0, so both calls return the identical trivial list, [100.0, 100.0, 100.0, 100.0, 100.0] (five elements: the start plus four steps), regardless of the seed at all. For the colleague's non-trivial call: yes, they get the exact same list, because a seeded generator's entire output sequence is a deterministic function of the seed value and the sequence of calls made to it, not of the wall clock, the operating system, or which physical machine is running it. "Same seed, same inputs, same code" is sufficient on its own; "same day" or "same machine" is not required and does not matter.

P14. Why order-independence matters for a fingerprint. Two teammates build the identical logical dependency dictionary, but insert the key-value pairs in a different order. Explain why a fingerprint built as "|".join(f"{k}={v}" for k, v in deps.items()) (then hashed) would differ between them even though their environments are truly identical, and name the one-line fix.

Solution. A Python dict iterates its items in insertion order, so two dicts holding the exact same key-value pairs, built in a different sequence, produce a different string from that join, and therefore a different hash, even though the content is identical. Fix: sort the items (by key) before joining or serializing, so the resulting string, and hash, depends only on the dictionary's content, never on the order it happened to be built in.

P15. Permission denied, with no shebang. A file analyze.py has permissions -rw-r--r-- (644) and does not start with a #! shebang line. Running ./analyze.py fails. You run chmod +x analyze.py and try again, it still fails, or behaves strangely. Explain why the shebang matters even after the permission is fixed, and give two complete, independent ways to actually run the script.

Solution. chmod +x fixes only the permission half of the problem (adds the execute bit, 644 → 755). But ./analyze.py also asks the operating system to figure out, on its own, which program should interpret the file's contents, that is exactly the shebang line's job (#!/usr/bin/env python3 as literally the first line), and without it the OS has no reliable way to know this text file should be handed to Python. Two complete fixes, either one sufficient on its own: (1) add the shebang line as the file's first line, keep the execute bit set, then run ./analyze.py directly; or (2) skip both the shebang and the execute bit entirely and run python3 analyze.py, explicitly naming the interpreter on the command line, this always works and never needs either fix, because you are telling the OS which program to use instead of asking it to guess.

P16. Reproducibility across two markets, computed by hand. A seeded, deterministic simulation is called twice with step_pct=0.0 (so every path is trivially flat by construction): once starting at an illustrative NSE-style level of ₹24,000, once starting at an illustrative US-style level of $5,500, both with steps=6 and the identical seed. Two teammates (one in Mumbai, one in Chicago) each independently make both calls on their own laptops next week. Of the four resulting lists (2 teammates × 2 calls), how many pairs are guaranteed identical to each other, and which pairs are guaranteed to differ?

Solution. Each list has steps + 1 = 7 elements, and since step_pct=0.0, every element in a given call equals that call's own starting value repeated seven times: the ₹ call is [24000.0] * 7, the $ call is [5500.0] * 7. Guaranteed identical: Mumbai's ₹-call list and Chicago's ₹-call list (identical seed, start, steps, step size, determinism depends on the four inputs alone, never on who runs it or when); likewise Mumbai's $-call list and Chicago's $-call list. That is two guaranteed-identical pairs. Guaranteed to differ: any ₹-call list compared to any $-call list, because they were built with a different start argument (₹24,000 vs. $5,500), the shared seed does not make two calls with different inputs converge on the same output; it only guarantees that the same call, repeated, converges on itself.

Applied mini-project

Build a reproducible mini research project.

This app's sandbox cannot see your real operating system, so (like every module in this branch) the graded portion and the hands-on portion are two separate halves. The graded half is the CS2.01 code pack (code-packs/CS2.01.json, module id CS2.01): eight exercises that model every mechanism in this node precisely in Python, each with a fill-in, reorder, or choose-the-correct-implementation fallback so the whole pack is completable with zero live Python runtime. This project is the other half: the physical thing, done once, in your own terminal, on your own machine, self-checked against the rubric below.

Do this, in order, on your own laptop:

Lay out the project. Create a fresh directory. Inside it, create src/, data/raw/, notebooks/, tests/, an empty README.md, and an empty requirements.txt.
Isolate the environment. Create a virtual environment (python3 -m venv venv), activate it, and confirm activation the reliable way (which python3, not just the prompt). pip install a small, real data-handling package (e.g. pandas) and numpy. Run pip freeze > requirements.txt to pin exact versions.
Write a tiny, seeded script. In src/, write a script that: reads (or synthesizes inline) two small illustrative price series (one framed in ₹ (an NSE-style name), one framed in $ (a US-style name)) computes a simple total return for each, and runs one small step involving randomness (even something as simple as a seeded shuffle or a seeded random sample) using an explicit random.Random(seed) you set yourself, never the shared global generator. Print a small report line for each market.
Prove reproducibility, don't just claim it. Run the script twice in a row and confirm the output is byte-for-byte identical. Then delete the entire venv/ folder, recreate it from scratch, reinstall strictly from your pinned requirements.txt, and rerun the script a third time, confirm the output still matches exactly.
Document it. Write a "How to reproduce" section in README.md specific enough that a stranger with none of your memory of this project could follow it exactly: the venv-creation command, the activation command, the install command, and the run command, in order.
Self-check. Score yourself, 2 points each: (a) the layout matches the conventions taught above (a place for source, data, tests, a pinned dependency file, a README); (b) the environment is genuinely isolated and the pins are exact (==), not floating; (c) the script runs via one command with no manual edits required; (d) you actually deleted and rebuilt the environment and confirmed identical output, not merely reasoned that it "should" work; (e) the README's reproduce section is complete and correct enough that you could hand it to someone else. Pass at 8/10, with no zero on any item. If step 4's rebuild ever produces a different result, you have found a real gap in one of the four legs, go find which one before moving on; that debugging is the whole point of the exercise.

Lab: open the CS2.01 code pack in this node's Code tab: eight exercises (CS2.01.json) covering PATH resolution (CS2.01-e01), environment-variable precedence (CS2.01-e02), dependency-version constraints (CS2.01-e03), path normalization (CS2.01-e04), permission bits (CS2.01-e05), command-chaining exit codes (CS2.01-e06), seeded determinism (CS2.01-e07), and reproducibility fingerprints (CS2.01-e08). Every exercise ships a fill-in-the-blank, reorder-the-lines, or choose-the-correct-implementation fallback as its definition of record, so the entire pack is completable offline even with no Python runtime available in your browser.

Reading & resources

Everything below is an anchor to learn the mechanism from and verify your own understanding against, read it, then go confirm the behavior yourself in a real terminal; do not treat a tutorial's description as a substitute for trying the command.

The core spine:

MIT, The Missing Semester of Your CS Education, the shell, environment variables, and version-control lectures are exactly this node's syllabus, taught by the people who wrote the tools. [Free] [Beginner–Intermediate]
The Art of Command Line (a widely used, community-maintained reference on GitHub), a dense, well-organized cheat sheet of shell idioms once you have the fundamentals. [Free] [Beginner–Advanced]
explainshell.com, paste any real command you don't fully understand and see every flag and operator explained piece by piece. [Free] [Beginner]

Official references (primary-source grade):

The Python Packaging User Guide (packaging.python.org), the authoritative, current guidance on venv, pip, and requirements.txt. [Free] [Beginner–Intermediate]
PEP 440, Version Identification and Dependency Specification, the actual specification behind ==, >=, ~=, and every other version constraint operator this node teaches. [Free] [Advanced]
The official Jupyter documentation, especially its notes on kernels and reproducible execution. [Free] [Beginner]
Docker's own "Get Started" documentation, a conceptual pointer only; this node does not teach containers hands-on, but this is the right place to start once you do. [Free] [Intermediate]

Reproducibility as its own discipline:

Sandve, Nekrutenko, Taylor & Hovig, Ten Simple Rules for Reproducible Computational Research (PLOS Computational Biology, 2013), a short, widely cited paper that turns "reproducibility" from a vague virtue into the same kind of concrete checklist this node builds. [Free] [Intermediate]
QD2.04 (Reproducible Research, Deployment & the Quant Stack), the production-scale continuation of everything in this node's last two sections: scheduled pipelines, secrets at scale, and anti-overfitting research protocols. Read this node first; QD2.04 assumes it.

Do this, not just read: open a real terminal right now and run which python3, then echo $PATH and read the list it prints, one directory at a time. Create one throwaway virtual environment, activate it, run which python3 again, and watch the answer change. Ten minutes of watching PATH actually change in front of you will teach you more than an hour of reading about it.

Teach it back & journal

Feynman prompt. Explain to a friend who has only ever written code that either "worked" or "had a bug" why a THIRD category exists: code that is completely correct and still fails, or gives a different answer, purely because of what's around it. Use the sentence "the code is never the whole program, the shell, the interpreter, the packages, and the working directory around it are just as load-bearing." Then walk them through one real diagnostic (which python3) and have them predict, before you tell them, what it would print on their own laptop right now, and why that answer might not be the Python their editor is secretly using. If they can, unprompted, ask "wait, which Python does my script actually import against?" after hearing that, you have installed the reflex this node exists to build.

Journal prompt. Pick one real project you're already working on (for this course or otherwise) and, this week, actually apply all four legs of reproducibility to it: pin every dependency exactly, add an explicit seed anywhere randomness appears, record the exact identity of whatever data it depends on, and write the "how to reproduce" section of its README. Then do the real test: delete your virtual environment and rebuild it from nothing, and rerun everything. Log what broke. Was it a dependency that had quietly drifted? A path that was secretly absolute when you thought it was relative? A random call you didn't know was unseeded? Write one paragraph on this: which single habit from this node, checking which python3 before assuming a package problem, treating an unpinned dependency as untrustworthy by default, or refusing to trust a notebook result without "Restart & Run All", would most change how reliably your work reproduces, and what will you do this week to make it automatic.

This module's flashcards and mastery quiz are wired into the app: see the node's Quiz and Reviews.

Command Line, Environments & Reproducible Projects