pandas I — Data Wrangling

Learning objectives

By the end you can:

1. Explain what a Series and a DataFrame actually are (a labeled one-dimensional array, and a dict-like collection of aligned Series sharing one index) and state why the index, not row position, is what pandas aligns on whenever two pandas objects meet.
2. Read a real CSV into a DataFrame with pd.read_csv, and treat .shape, .dtypes, .head()/.tail(), .info(), and .describe() as a mandatory ritual, never trusting a frame's contents until you have looked at all five.
3. Select data correctly and safely: build a boolean mask from one or more conditions, choose .loc (label-based) versus .iloc (position-based) deliberately, and combine a mask with a column list inside a single .loc[mask, cols] call rather than chaining two separate [] lookups.
4. Add and derive columns the vectorized way (df["x"] = ..., .assign()) as the default, and explain (with a concrete case) why row-by-row .apply(axis=1) is a last resort, not a habit.
5. Handle missing data on purpose: distinguish isna()/notna(), choose deliberately between dropna() and fillna()/ffill()/bfill(), and perform every fill or drop per group (per ticker, per account) whenever the data holds more than one entity.
6. Find and remove duplicate rows correctly with duplicated()/drop_duplicates(), using subset= and keep= precisely enough to never over-delete a genuine second trade or under-delete an accidental resubmission.
7. Perform group-wise computation with the split-apply-combine pattern: .groupby(...).agg(...) (including named aggregation) to collapse to one row per group, versus .groupby(...).transform(...) to broadcast a per-group result back onto every original row, and state which shape each produces before you run either.
8. Combine tables correctly: choose the right pd.merge how= (inner/left/right/outer) for the question being asked, explain in one sentence why validate= exists, and reach for pd.concat (not merge) when you are stacking rows, not joining on a key.
9. Reshape between wide and long form with .melt() and .pivot_table(), and say, for a given next step (a groupby, a merge, a plot), which shape it actually needs.
10. Assemble the operations above into one short, ordered pipeline (read → inspect → clean → merge/reshape → group → sort) against real India (₹, NSE-style) and US (\$, NYSE-style) data, matching this module's code pack (DA1.02, exercises DA1.02-e01 through DA1.02-e08).
11. (Productivity objective: R10 duality.) Use an AI coding assistant to scaffold repetitive pandas boilerplate and to explain an unfamiliar error message, while reviewing every line it produces for the silent, non-crashing bugs pandas code is especially prone to, a wrong axis, a merge that quietly drops or multiplies rows, a fill that should have been grouped and wasn't, and never trusting an AI-authored transformation until you have checked its output against a small case you can verify by hand.

The duality, stated once (R10). Objectives 1–10 are the understanding objective: the gate (quiz ≥ 85%, code-pack ≥ 85% with hidden tests green) rewards being able to write and trace this pandas yourself, on data you have never seen before. Objective 11 is the productivity objective: knowing how a real data professional uses an AI copilot on top of that understanding, never instead of it. A copilot can type a groupby call faster than you. It cannot pass this module's hidden tests for you, and it cannot tell you (without your own verification) whether the row count it just produced is the row count you actually wanted.

Prerequisites & connections

Builds on. DA1.01 (NumPy & Vectorized Computation), this branch's first node, establishes the array-and-vectorization mindset that pandas is built directly on top of, a DataFrame's numeric columns are NumPy arrays underneath, and the habit of reaching for a whole-array operation instead of a Python-level loop carries straight over into every df["x"] = ... line in this module. If you have not taken DA1.01, this module still holds together as long as you already have ordinary Python fluency, variables, functions, for loops, list and dict literals, from any source; this program's own recommended, non-blocking route to that fluency is CS1.01/CS1.02, but per R11 no branch hard-gates on another. Within the quant-ds branch itself, DA1.01 → DA1.02 is a real, sequential, enforced chain; DA1.01 has no prerequisite of its own and opens on day one, exactly like CS1.01 and QD1.01 do for their own branches.