NumPy & Vectorized Computation

Branch: Data Science & ML (quant-ds) · Finance galaxy, Quant-Code region · Node DA1.01 · ~16 focused hours · Gate: quiz ≥ 85% + code-pack ≥ 85% (hidden tests green on at least 85% of exercises)

Every one of the eight functions you wrote in CS1.01 worked one price at a time: a loop walked the list, one iteration touched one number, and the count of iterations was the count of days. That is a completely correct way to compute a return series, and it is also, the moment your data stops being seven days long and starts being seven hundred thousand rows of tick data, the wrong way to write it, not because it gives the wrong answer, but because it makes the machine do far more work than the computation actually requires, one Python-level decision at a time, for every single element. This module is where you stop writing loops over numeric data and start writing vectorized code: whole-array expressions that describe the computation once and let a compiled, contiguous-memory engine (NumPy) apply it to every element, every row, every column, all at once. The shift is not cosmetic. It is the single most consequential habit change between "someone who can write Python" and "someone who can do numerical and financial computation in Python," and every module after this one (pandas, statistics, machine learning, backtesting, portfolio optimization, risk systems) is built directly on top of the array object and the vectorized habits you install here.

NumPy (short for Numerical Python) is the foundation underneath essentially the entire Python data and quant stack. pandas' DataFrame is, underneath its row/column labels, a collection of NumPy arrays. scikit-learn's entire input and output contract is NumPy arrays. SciPy, statsmodels, and even the from-scratch neural network you will build in this branch's crown module (DA2.03) are NumPy arrays and NumPy operations, arranged differently. If you learn NumPy's array mental model precisely here, what a shape is, when an operation copies data and when it shares it, exactly how broadcasting decides whether two differently-shaped arrays can combine, you will find every later module in this branch easier, because you will already own the substrate they are all built on. If you skip past it, you will spend the rest of this branch fighting shape errors you cannot diagnose.

Everything in this module points at the same two running datasets you will meet across the whole quant-ds branch: a bundled end-of-day snapshot of daily closing prices from India's NSE (nse-eod, 15 fictional tickers, 500 trading days) and from the US's NYSE-style market (us-eod, 15 fictional tickers, 500 trading days), the same market-data convention CS1.01 used, now represented as NumPy arrays and, once you reach the matrix sections, as 2D arrays of many tickers at once. Every specific number used for teaching in this module is explicitly synthetic and illustrative (R3), small, hand-traceable series built to teach a mechanism, not real historical quotes for any named company, while this module's code pack's hidden tests draw on both an India-flavored and a US-flavored case for every exercise that touches market data, so that a solution that only works on the one example you happened to test by hand is caught immediately. Every idea below becomes a graded, auto-checked exercise in this module's code pack (pack id DA1.01, exercises DA1.01-e01 through DA1.01-e10), real NumPy, executed live in your browser, checked against hidden tests you cannot see and cannot hardcode your way past. If you can explain why an operation broadcasts the way it does, you can code it; if you can code it, the pack will prove it.

Learning objectives

By the end you can:

1. Explain why a NumPy array outperforms a native Python list for numeric work, contiguous, single-dtype memory and vectorized operations that run as a tight, compiled loop instead of the Python interpreter stepping through one element, one bytecode dispatch, at a time, and predict, before running anything, that a vectorized operation on a real-sized array will be an order of magnitude (or more) faster than the equivalent explicit loop.
2. Create and inspect ndarrays: np.array, np.zeros/np.ones/np.arange/np.linspace, and read off an array's shape, dtype, ndim, and size to predict what any expression involving it will produce, including when np.asarray copies data and when it does not.
3. Correctly predict whether a given indexing operation (basic slicing versus fancy (integer-list) or boolean indexing) returns a view (shares memory with the original; mutating it mutates the original) or a copy (independent; mutating it does nothing to the original), and use this to explain why a chained indexing assignment can silently do nothing at all.
4. Write elementwise vectorized arithmetic and comparison expressions (ufuncs) with no explicit loop, and use a boolean mask to filter and then aggregate in one line (returns[returns > 0].mean()).
5. State NumPy's broadcasting rule precisely, align shapes from the trailing dimension; two dimensions are compatible when they are equal, or when one of them is 1, and use it to predict, before running code, whether two given shapes will broadcast together, what the result shape will be, or that NumPy will raise a ValueError.
6. Perform axis-aware reductions (sum/mean/std/min/max/cumsum/cumprod, axis=0 vs. axis=1) and correctly diagnose the classic np.cov rowvar trap on (days, tickers)-shaped return data, where the default silently computes the wrong-shaped result.
7. Use matrix multiplication (@) to turn a returns matrix and a weight vector into a vector of per-day portfolio returns, and a quadratic form (weights @ cov_matrix @ weights) to compute portfolio variance, strictly as NumPy mechanics, not portfolio theory.
8. Detect and avoid three concrete dtype/NaN pitfalls: silent integer overflow in fixed-width dtypes, an in-place true-division cast error on integer arrays, and NaN silently propagating through an ordinary .mean()/.sum() with no error at all.
9. Generate reproducible pseudo-random arrays with NumPy's modern Generator API (np.random.default_rng(seed)) and explain why a fixed seed is exactly what makes a hidden test, or a report: deterministic and re-runnable.
10. Apply everything above to the module's running problem: given price and return data from an Indian (₹, NSE-style) and a US ($, NYSE-style) snapshot, compute vectorized returns, cumulative growth, a standardized return matrix, a covariance matrix, and portfolio variance, first by hand on a small case, then in real, running, tested NumPy code (this module's code pack, DA1.01).
11. (Productivity objective: R10 duality.) Use an AI coding assistant to scaffold NumPy boilerplate for parts of a task you already understand by hand, while never trusting an AI-generated axis, rowvar, or broadcasting choice without tracing the shapes yourself and running this module's tests.

The duality, stated once (R10). Objectives 1–10 are the understanding objective: the gate (quiz ≥ 85%, code-pack ≥ 85% with hidden tests green) rewards being able to trace shapes, predict broadcasting outcomes, and write correct vectorized code yourself. Objective 11 is the productivity objective: knowing how a real developer uses a copilot on top of that understanding, never instead of it. An assistant can generate a plausible-looking np.cov(matrix) in half a second, it cannot tell you, and will not warn you, that it just silently computed the wrong shape. Only tracing the shape yourself, and running this module's hidden tests, can.

Prerequisites & connections

Builds on. Nothing is a hard prerequisite gate, like every branch's first node, DA1.01 unlocks from day one. But this module assumes real fluency with core Python at roughly CS1.01/CS1.02's level: variables, functions with return, lists, indexing, and if/for/while control flow. If writing a small function or tracing a loop by hand still feels shaky, spend an hour in CS1.01 first, this module does not re-teach those fundamentals (R1), it immediately builds a new layer on top of them: the array object, and the habit of describing a computation once, over a whole array, instead of one element at a time. No calculus and no linear-algebra background is assumed; the two matrix operations this module uses (matrix multiplication and a quadratic form) are taught here from the ground up, mechanically, with no theory beyond "here is what shape goes in and what shape comes out."

Feeds forward. Directly into DA1.02 (pandas I), where you will discover that a pandas DataFrame is, underneath its row and column labels, built out of exactly the NumPy arrays and dtype rules taught here, every vectorization habit from this module carries over unchanged, now with labels attached. DA1.04 (EDA & Visualization) and DA1.05 (Statistics in Code) compute directly on arrays this module's mechanics produce. DA2.01 (ML I. Supervised Learning) hands scikit-learn NumPy arrays as its entire input/output contract; an axis or shape mistake there is the same mistake this module teaches you to catch here. DA2.03 (this branch's crown, Deep Learning, NLP & Time-Series) grades a from-scratch NumPy multilayer perceptron, forward and backward passes that are, mechanically, nothing but more layers of the exact broadcasting and @ operations this module introduces; master the mechanics here and DA2.03's code is a scale-up, not a new language. Further out, the entire QD (Quant Developer) branch (signal construction, a backtesting engine, portfolio optimization, risk systems, performance attribution) is vectorized NumPy code end to end; a soft, non-blocking recommendation (R11) to have this module solid first, never a gate.

Cross-branch companions (soft, not gates). CS1.01/CS1.02 (Python foundations) are the natural on-ramp if coding itself is still new. C-3 binding boundary, stated once and clearly: this module and its code pack teach NumPy as a computational tool, how to represent numeric and financial data efficiently and compute on it correctly (vectorization, broadcasting, reductions, linear algebra mechanics, reproducible randomness). The statistical meaning of a quantity like covariance or correlation, what it tells you about co-movement, diversification, or risk, and the theory behind it: belongs to DA1.05 (Statistics in Code, which cross-links Finance-Plus's QM1.02) and to QD1.04 (Portfolio Optimization in Code, where portfolio_variance's mechanics here grow into an actual mean-variance optimization problem with constraints). This module computes the numbers, precisely and fast; it does not re-derive what they mean or how to optimize them. Reference those modules when you want the theory; nothing here duplicates it.

Core teaching content

1. Why vectorization exists: the case against the loop

Start with the plainest possible version of the problem this whole module solves. You have n daily closing prices and you want n - 1 daily returns, prices[i+1] / prices[i] - 1 for each i. CS1.01 wrote this as a loop: initialize an accumulator or a result list, walk an index from 1 to n, do one division and one subtraction per iteration, append the result, repeat. That code is entirely correct. It is also, mechanically, doing something wasteful: for every single element, the Python interpreter has to re-discover that prices[i] is a float, re-look-up what / means for two floats, re-check bounds on the list access, re-dispatch the subtraction, the same handful of bookkeeping steps, over and over, once per element, in the slow, general-purpose, interpreted way Python does everything.

A NumPy array sidesteps essentially all of that bookkeeping, for two structural reasons. First, a NumPy ndarray stores its elements in one contiguous block of memory, all of the same fixed dtype (int64, float64, and so on), not, like a Python list, as a list of pointers to separately-allocated Python objects scattered through memory. Second, an operation like prices[1:] / prices[:-1] is dispatched once, for the whole array, to a tight, compiled C loop that already knows the exact dtype and the exact memory layout it is working with, no per-element type-checking, no per-element bytecode dispatch, just raw arithmetic over a block of numbers it already understands. (For very large matrix operations, some of that compiled code may additionally be backed by a multi-threaded linear-algebra library depending on your NumPy build, but the primary source of the speedup, and the one to hold as your mental model, is the elimination of per-element interpreter overhead, not "it runs on multiple cores.")

This is not a hypothetical difference. Measured directly, in this module's own authoring environment, on a 200,000-element synthetic price array: computing daily returns with an explicit Python for loop took about 32 milliseconds; the exact same computation as prices[1:] / prices[:-1] - 1 took about 0.9 milliseconds, roughly a 35× speedup, for identical results. (A Python list-comprehension version, still a loop, just written more compactly, measured about 18 milliseconds: faster than the explicit loop, but still an order of magnitude behind vectorized NumPy.) Treat the exact multiplier as illustrative and hardware/version-dependent, your machine, your NumPy build, and your array size will all move the number, but treat the order of magnitude and the direction as durable: for numeric work at any real scale, an explicit Python loop over array elements is close to the slowest correct way to write the computation, and a vectorized NumPy expression is close to the fastest, for the identical answer. Once your data is 500 trading days across 15 tickers instead of 7 days across one, this stops being an academic point and starts being the difference between a report that runs instantly and one that visibly stalls.

2. The ndarray: shape, dtype, and how arrays are built

The central object in NumPy is the ndarray (n-dimensional array). Four properties tell you almost everything you need before touching a single element:

• shape: a tuple of sizes, one per dimension. A 1D array of 500 daily prices has shape == (500,). A 2D matrix of 500 days across 15 tickers has shape == (500, 15), by this module's convention, always rows are observations (days), columns are variables (tickers), a convention you will need to hold precisely once broadcasting and np.cov enter the picture (§5–6).
• dtype: the single, fixed data type every element shares (int64, float64, bool, and others). Every element in an array has the same dtype; there is no mixing a float and a str in the same array the way a Python list allows.
• ndim: the number of dimensions (1 for a plain price series, 2 for a days-by-tickers matrix).
• size: the total element count (the product of every entry in shape).

You build arrays several ways. np.array([100.0, 102.0, 99.5]) wraps an existing Python list or nested list of lists (for a 2D array) into a real ndarray, inferring a dtype from the values. np.zeros(500) and np.ones((500, 15)) build arrays of a given shape pre-filled with 0.0/1.0. np.arange(0, 10, 2) builds [0, 2, 4, 6, 8] (like Python's range, but a real array you can immediately do arithmetic on). np.linspace(0, 1, 5) builds 5 evenly spaced points from 0 to 1 inclusive: [0.0, 0.25, 0.5, 0.75, 1.0].

Two conversion functions look like they do the same thing and, critically, do not. np.array(x) always makes a new, independent copy of x's data, whether or not x was already an array. np.asarray(x) is smarter: if x is already an ndarray of the requested dtype, it hands back the exact same array object, no copy at all; it only copies when a copy is genuinely required (converting a plain list into an array, or changing the dtype). This module's code pack solutions consistently open every function with prices = np.asarray(prices, dtype=float) for exactly this reason: it is the cheap, safe way to say "treat this as a float array, whatever it currently is", accepting a plain Python list or an existing array, either coercing it once if genuinely needed or handing it straight through, untouched, if it already qualifies. Reach for np.array only when you specifically want a fresh, independent copy, for instance, when you must guarantee you are never accidentally about to mutate a caller's array (§3, and this module's clip_returns/fill_missing_with_mean exercises).

{
  "skill": "Building correct intuition for a NumPy mechanism you have not fully internalized yet (e.g. broadcasting, axis semantics, view-vs-copy)",
  "use": "Have a chat or coding assistant draft a plain-language explanation plus ONE tiny, concrete, fully-traceable example of a NumPy rule you are shaky on, then have it generate a few more shape-prediction questions so you can quiz yourself — using AI to accelerate building the right mental model, not to replace actually verifying it.",
  "tools": ["AI coding assistants — Claude, ChatGPT, GitHub Copilot", "A live Python/NumPy REPL, as your own independent verification"],
  "prompt": "Explain NumPy's broadcasting rule to me using exactly one worked example with two SMALL arrays whose shapes and values I can trace completely by hand. Then give me three more pairs of shapes and ask me to predict, before I run anything, whether they broadcast together, what the result shape would be, or that NumPy raises a ValueError.",
  "verify": "An explanation of a mechanism is not the mechanism. Run every example yourself in a real Python/NumPy session and confirm the actual shapes and values match what was claimed — exactly the discipline this module's own Worked Examples model. Never accept a claimed shape, a claimed broadcasting outcome, or a claimed speedup number without checking `.shape` or timing it yourself.",
  "diy": "You must already be able to state the broadcasting rule in your own words and trace at least one example by hand (this section and §5) before using AI to generate more practice — otherwise you cannot tell a correct explanation from a confidently wrong one.",
  "market": "IN"
}

3. Indexing and slicing: the view-versus-copy fault line

This is the single most consequential mechanical fact in this entire module, because getting it wrong produces bugs that do not crash, they silently give a wrong, plausible-looking answer, or silently do nothing at all.

Basic slicing: a[1:3], a[:-1], a[::2], returns a view: a new array object, but one that shares the same underlying memory as the original. Mutating a value through the view mutates the original array too, because there is, underneath, only one block of memory. Verified directly: a = np.array([10, 20, 30, 40, 50]); b = a[1:3]; b[0] = 999 leaves a as [10, 999, 30, 40, 50], the assignment through b reached straight into a's own memory.

Fancy indexing (an explicit list or array of integer positions, a[[0, 2]]) and boolean indexing (a mask, a[a > 15]) both return a genuine, independent copy. Mutating the result changes nothing about the original. Verified directly: a2 = np.array([10, 20, 30, 40, 50]); c = a2[[0, 2]]; c[0] = -1 leaves a2 completely unchanged at [10, 20, 30, 40, 50], because c is a fresh array holding copies of the selected values, with no memory link back to a2 at all. Boolean masking behaves identically: d = a2[a2 > 15]; d[0] = -1 again leaves a2 untouched.

Why this matters goes beyond "know the rule": it produces a specific, nasty, silent failure mode called chained indexing. Writing a[a > 0][0] = 999 looks, at a glance, like "set the first positive element to 999." It is not. Python evaluates this in two separate steps: first a[a > 0] builds a brand-new copy (boolean indexing, per the rule above); then [0] = 999 mutates position 0 of that throwaway copy, which is immediately discarded, having never been assigned to anything. The net effect on a is exactly nothing, with no error, no warning, and code that reads as if it obviously should have worked. Verified directly: a = np.array([1, -2, 3, -4]); a[a > 0][0] = 999 leaves a as [1, -2, 3, -4], completely unchanged. The fix is to never chain a second index onto the result of a boolean/fancy selection when your intent is to write: mutate through a single indexing operation on the original array instead (a[a > 0] = 999) which is not chained at all (it is one __setitem__ call directly against a's own memory) and genuinely does update every positive element in place. Hold the general principle: reading through a chain of indexing operations is completely safe and idiomatic; writing through more than one indexing step in the same statement is close to always a bug.

4. Vectorized arithmetic, comparisons, and ufuncs

Ordinary arithmetic operators (+ - * / **) and comparison operators, > < >= <= == !=, all work elementwise on arrays automatically, with no loop, no comprehension, nothing beyond the plain expression. prices[1:] / prices[:-1] - 1 computes every daily return in one call. returns > 0 does not produce a single True/False, it produces a whole new boolean array, the same shape as returns, one True/False per element, recording whether that return was positive. These elementwise operations are technically called ufuncs (universal functions) in NumPy's own vocabulary, you do not need the term to use them, but you will see it in NumPy's own documentation and error messages, so it is worth recognizing.

The genuinely powerful move is combining a boolean array with indexing to filter and then aggregate in a single vectorized expression. returns[returns > 0] reads as "every return where the corresponding mask entry is True", the up-days only, with the down-days and flat-days dropped entirely, not zeroed out. Chain .mean() onto that and returns[returns > 0].mean() computes "the average gain, counting only the days that actually gained", a genuinely different, and often more useful, number than the plain average of every day (which counts every down-day too, diluting the figure). This exact pattern is this module's average_gain_on_up_days exercise (DA1.01-e08). np.where(condition, if_true, if_false) is the elementwise ternary version of the same idea when you need to keep every element but choose between two computed values per position, rather than dropping the elements that fail the test.

Keep one edge case in view for later: if returns happens to contain zero up-days (every return is ≤ 0), returns[returns > 0] is a genuinely empty array, and .mean() on an empty array is not an error, it silently returns nan, along with a RuntimeWarning ("Mean of empty slice"). It does not crash your program; it quietly hands you a nan that will then poison anything downstream that adds or averages it in with other numbers (§9 covers exactly this propagation behavior in depth).

5. Broadcasting: one rule, three shapes of consequence

Broadcasting is the rule that lets NumPy combine two arrays of different shapes in a single elementwise operation, without you writing a loop or manually repeating data to make the shapes match. State the rule precisely, because "it just works" is not good enough once a real bug depends on knowing exactly when it does and does not: compare the two shapes starting from the trailing (rightmost) dimension and working backward. Two dimensions are compatible when they are equal, or when (at least) one of them is 1. If one shape has fewer dimensions than the other, treat the missing leading dimensions as 1. Wherever a dimension is 1, that array is conceptually "stretched" (with no actual memory copy) to match the other array's size along that axis.

Case 1, a scalar against an array. prices / prices[0] * 100.0 (this module's rebase_to_100, DA1.01-e03) broadcasts a bare scalar (prices[0], a single number with, conceptually, shape ()) against every element of prices. This is the simplest and most familiar case: the scalar applies everywhere.

Case 2, a 1D vector against a 2D matrix, aligned on the trailing axis. Given a (days, tickers) matrix and a (tickers,) vector of per-column means, matrix - col_means broadcasts correctly with no reshaping needed at all, because the trailing dimension of matrix (tickers) matches the only dimension of col_means exactly. Trace a genuinely tiny, fully hand-computable case: small_matrix = [[10.0, 100.0], [12.0, 108.0], [14.0, 92.0]], a (3, 2) matrix. Its column means are [12.0, 100.0] (the two columns' means, computed with axis=0, see §6). Subtracting: small_matrix - [12.0, 100.0] broadcasts the (2,) vector against every one of the 3 rows, giving [[-2.0, 0.0], [0.0, 8.0], [2.0, -8.0]]: row 1's first column (10.0 - 12.0 = -2.0) and second column (100.0 - 100.0 = 0.0), and so on down every row, with the same [12.0, 100.0] pair reused each time. This module's standardize_columns exercise (DA1.01-e04) is exactly this pattern, one step further: subtract the column means and divide by the column standard deviations, both computed with axis=0, both (tickers,)-shaped, both broadcasting cleanly against the (days, tickers) matrix with zero reshaping.

Case 3, the failure that teaches the rule. Now flip the orientation: suppose, instead of per-column (per-ticker) means, you had computed per-row (per-day) means, a (days,)-shaped vector, and tried the same subtraction against a (days, tickers) matrix. Verified directly: for a (4, 3) matrix and a (4,) vector, matrix - row_vector raises ValueError: operands could not be broadcast together with shapes (4,3) (4,). Read the rule against this failure and it becomes obvious why: comparing shapes from the trailing dimension, 3 (the matrix's tickers axis) is compared against 4 (the vector's only axis), neither equal nor 1, so NumPy refuses rather than guess. The fix is to make your intent about which axis you mean to broadcast against explicit, by reshaping the (4,) vector into a genuine (4, 1) column vector: row_vector.reshape(-1, 1) or the equivalent row_vector[:, None] (None inserts a new length-1 axis), after which (4, 1) against (4, 3) compares 1 against 3 (compatible: the 1 stretches) on the trailing axis, and 4 against 4 (equal) on the next, broadcasting cleanly, this time genuinely applying one value per row across every column in that row. The general lesson: a broadcasting failure is not a bug in NumPy, it is NumPy correctly refusing to guess which axis you meant, and the fix is never "try random reshapes until it stops complaining" but "state explicitly, with reshape or [:, None], which axis your vector is meant to align against."

6. Reductions and axes: sum, mean, std, and the np.cov trap

A reduction collapses one or more axes of an array down to a single summary value per remaining position: sum, mean, std, min, max, argmin/argmax (the position of the min/max, not its value), and the running (non-collapsing) variants cumsum/cumprod. Called with no axis argument, a reduction collapses the entire array to one number. Called with axis=0 on a (days, tickers) matrix, it collapses axis 0 (the days axis) and leaves one value per ticker, a (tickers,)-shaped result, exactly the column means and column standard deviations §5's broadcasting examples used. Called with axis=1 instead, it collapses the tickers axis and leaves one value per day, a (days,)-shaped result. Getting axis=0 and axis=1 backward does not raise an error; it silently hands you the other dimension's aggregate, labeled with the same variable name you gave the one you meant.

This exact axis confusion has a famous, specific, real trap built directly into NumPy's own np.cov function, and this module's covariance_matrix exercise (DA1.01-e06) is built to make you meet it once, deliberately, rather than discover it by accident in a real report. np.cov's rowvar argument defaults to True, which tells np.cov to assume each row is a variable and each column is one observation of it, the opposite of this module's (days, tickers) convention, where rows are days (observations) and columns are tickers (variables). Verified directly: calling the correct np.cov(matrix, rowvar=False) on a (4, 3) matrix (4 days, 3 tickers) returns the intended (3, 3) tickers-by-tickers covariance matrix; calling the default np.cov(matrix) (silently rowvar=True) on the exact same input returns a (4, 4) matrix instead, the wrong shape entirely, with no error, no warning, just a confidently-returned matrix answering an entirely different, unintended question ("how do these 4 days co-vary, treating each day as if it were a variable measured 3 times"). The fix is one keyword argument, rowvar=False, and the discipline worth installing is: before calling any NumPy statistics function on tabular data, explicitly check which axis it assumes is the "variable" axis, never trust a default without reading the one line of documentation that states it.

A brief, fully hand-traceable worked case makes the actual covariance computation (not just the shape trap) concrete. Two tickers' daily returns over 4 days: ticker A [0.02, -0.01, 0.03, 0.00], ticker B [0.01, 0.01, -0.02, 0.04]. Both have mean 0.01. Deviations from the mean: A is [0.01, -0.02, 0.02, -0.01]; B is [0.00, 0.00, -0.03, 0.03]. Multiply the deviations pairwise and sum: (0.01)(0.00) + (-0.02)(0.00) + (0.02)(-0.03) + (-0.01)(0.03) = 0 + 0 - 0.0006 - 0.0003 = -0.0009. Divide by n - 1 = 3 (the sample covariance convention np.cov uses by default): -0.0009 / 3 = -0.0003. That number (a negative sample covariance) is exactly what np.cov returns as the off-diagonal entry when you feed it these same two columns with rowvar=False; the two diagonal entries are each ticker's own variance by the identical formula (each deviation squared instead of cross-multiplied), 0.000333... for A and 0.0006 for B. Hold the mechanics, not the interpretation: why a negative covariance means "these two series tend to move in opposite directions" is DA1.05/QM1.02 territory; this module only guarantees you can compute the number correctly and know which axis it was computed along.

7. Linear algebra mechanics: dot products and matrix multiplication

Two NumPy operations carry essentially all of this branch's linear-algebra needs, and neither requires any theory beyond "what shapes go in, what shape comes out."

Matrix multiplication, spelled @ in modern Python/NumPy (equivalently np.matmul or, for the specific vector-vector and matrix-vector cases here, np.dot), combines a (days, tickers) matrix with a (tickers,) weight vector to produce a (days,) result: returns_matrix @ weights. Shape-wise, @ requires the trailing dimension of the left operand to match the leading dimension of the right operand (here, tickers matches tickers), and the result keeps whichever dimensions are left over (here, just days). Mechanically, each entry of the result is a weighted sum: day i's output is sum(returns_matrix[i, j] * weights[j] for j in range(tickers)), computed, for the whole matrix, in one call. This is precisely how you turn a matrix of per-ticker daily returns and a vector of portfolio weights into a vector of per-day portfolio returns (this module's portfolio_returns, DA1.01-e05): each day, blend that day's per-ticker moves according to how much of the portfolio sits in each ticker.

The second operation is a quadratic form: weights @ cov_matrix @ weights, chaining two matrix multiplications (a vector times a matrix, then the result (itself a vector) times the original vector again) down to a single scalar. This exact expression is the standard formula for portfolio variance given a covariance matrix of the underlying tickers' returns and a weight vector for how the portfolio is split across them (this module's portfolio_variance, DA1.01-e07). Trace a small, fully-checkable case: two assets, equal weights [0.5, 0.5], covariance matrix [[0.0004, 0.0001], [0.0001, 0.0009]] (asset 1's variance 0.0004, asset 2's 0.0009, their covariance 0.0001). By hand, the same quantity expands to w1² · σ1² + w2² · σ2² + 2·w1·w2·σ12 = 0.25 · 0.0004 + 0.25 · 0.0009 + 2 · 0.25 · 0.0001 = 0.0001 + 0.000225 + 0.00005 = 0.000375, and weights @ cov_matrix @ weights on exactly those inputs returns 0.000375, matching the hand expansion precisely. Name what you have and have not just done: you have computed a number correctly and fast, via matrix mechanics. You have not touched portfolio optimization, choosing which weights minimize this variance, subject to constraints, is the subject of an entire later module (QD1.04), and this module deliberately stops at "here is how to compute the formula," never wandering into "here is how to optimize it."

8. Reproducible randomness: seeding with the modern Generator API

Every module in this branch that needs randomly-generated data, a synthetic price path, a bootstrap resample, a Monte Carlo simulation (fully developed later, in Finance-Plus's QM1.03 and this branch's own later modules), needs that randomness to be reproducible: the same seed must always produce the exact same sequence of "random" numbers, so that a hidden test, a report, or a colleague re-running your code gets an identical result, not a different roll of the dice every time. NumPy's modern, recommended way to do this is the Generator API: rng = np.random.default_rng(seed) creates a seeded random-number generator object, and every draw from rng (rng.normal(...), rng.uniform(...), and others) is fully determined by that seed, call default_rng(42) twice, draw the same sequence of operations from each, and you get bit-for-bit identical arrays back, every time, on every machine. (You may also encounter the older, legacy np.random.seed(...) / bare np.random.normal(...) global-state style in existing code, it still works, but the Generator API is the modern, safer convention, because it does not rely on hidden global mutable state that one part of a program can silently disturb for another.)

This matters for more than tidiness. This module's own code pack hidden tests are deterministic specifically because every seeded value they check against was generated once, with a fixed seed, and recorded, exactly the discipline R3 requires of this entire program ("every number real-cited or explicitly synthetic (seeded)"). When you eventually simulate a price path, for instance, compounding a sequence of rng.normal(0, 0.01, size=n) daily returns via the cumulative_growth mechanics from §1/§7, recording the seed alongside the result is what turns "a random-looking chart" into "an exact, re-derivable, checkable computation." Holding the mechanics here (how to seed, how to draw, why it reproduces) is this module's job; what a particular random model implies about markets (geometric Brownian motion, volatility clustering, and so on) is intentionally left to the modules that teach that theory directly.

9. NaN: silent propagation, and the nan-aware fix

Real financial data has gaps, a missing quote, a holiday your calendar didn't account for, a vendor feed that dropped a row. NumPy's standard way to represent "this value is missing" in a numeric array is np.nan ("not a number"), and its behavior is deliberately, dangerously contagious: almost any arithmetic operation involving a nan silently produces another nan, with no error at all. Verified directly: np.array([10.0, np.nan, 30.0]).mean() returns nan, not an error, not a warning about the missing value, just a completely valid-looking nan that will happily propagate into every downstream calculation that touches it, until something finally notices the whole result has quietly become unusable.

The fix is a family of NaN-aware functions that explicitly skip missing values instead of letting them poison the result: np.nanmean, np.nansum, np.nanstd, and others, each the direct nan-safe counterpart of its ordinary reduction. np.nanmean(np.array([10.0, np.nan, 30.0])) returns 20.0, the mean of the two values that are actually present, correctly ignoring the gap. Combine this with np.isnan(values), which returns a boolean mask marking exactly which positions are missing, to build the standard fill-with-mean cleanup pattern this module's fill_missing_with_mean exercise (DA1.01-e10) drills: make a safe copy of the array (never mutate a caller's array in place, §3's lesson, again), compute a boolean mask of where the nans are, compute the nan-aware mean of the whole array (correctly ignoring its own gaps), then assign that single mean value into every masked position. Verified on a tiny case: [10.0, nan, 30.0] becomes [10.0, 20.0, 30.0], the single gap filled with the mean of the two real values, and both real values left completely untouched. Hold the general discipline: the moment you compute a plain .mean()/.sum()/.std() on data you have not personally confirmed is nan-free, you have introduced a silent failure mode, either confirm there are no gaps, or use the nan-aware function deliberately, every time, as a reflex rather than an afterthought.

10. Two dtype pitfalls worth meeting on purpose

NumPy's single-dtype-per-array rule (§2) buys real speed, and it costs two specific, verified gotchas worth meeting here, deliberately, rather than in a live report.

In-place true division on an integer array raises an error, it does not truncate. Ordinary division, a / 2, always promotes to a float64 result in NumPy 3-style true division, even when a is an integer array: np.array([1, 2, 3]) / 2 correctly returns [0.5, 1.0, 1.5], with no silent truncation trap (unlike some older languages' integer-division default). But the in-place version, a /= 2, is different: it tries to write the (necessarily float64) result back into a's own, still-int64 memory, and NumPy refuses, verified directly: a = np.array([1, 2, 3]); a /= 2 raises numpy.exceptions.UFuncTypeError: Cannot cast ufunc 'divide' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'. This is an honest, loud crash, not a silent bug, but it surprises people expecting a /= 2 and a = a / 2 to be interchangeable shorthand for the same operation, when for an integer array they are not: the second always works (it simply creates a brand-new float64 array and rebinds the name a to it, same as CS1.01's re-assignment model); the first tries, and fails, to keep the original integer memory.

Fixed-width integer dtypes silently overflow. int8 can represent only -128 to 127; ask it to hold something outside that range and it does not raise an error, it silently wraps around, two's-complement style, with, in this NumPy version, not even a warning printed. Verified directly: np.array([90], dtype=np.int8) + np.array([50], dtype=np.int8), mathematically 140, returns [-116] (140 - 256 = -116), silently, confidently, and with no diagnostic at all. This almost never bites you if you simply default to float64 (or NumPy's own default integer width, int64, which has enormously more headroom) for financial calculations, which is exactly why every solution in this module's code pack opens with np.asarray(x, dtype=float). It bites hard, and silently, the moment code somewhere upstream, a file loader, another library's default, a deliberately memory-squeezed dtype choice, has narrowed your data into a small fixed-width integer type without your noticing, and a later addition or accumulation quietly wraps.

11. Putting it together: the module's pipeline, end to end

Every mechanism above chains into one small, coherent pipeline, and this module's Applied mini-project asks you to build exactly this, on both markets. Start from raw prices (a (days,) array). Vectorize into daily returns with slicing (§1, to_returns). Compound them into a growth curve with np.cumprod (§7's cousin, cumulative_growth), and notice, as a genuine sanity check rather than a coincidence, that the last element of a returns-array's cumulative growth always equals prices[-1] / prices[0] exactly (both are the same telescoping product, just computed two different ways; if they disagree, one of your two functions has a bug). Rebase the raw price series to start at 100 with scalar broadcasting (§5, rebase_to_100) for chart-ready, market-agnostic comparison. Stack several tickers' returns into a (days, tickers) matrix and standardize every column with axis-aware reductions and vector-against-matrix broadcasting (§5–6, standardize_columns). Blend tickers into one portfolio return series with matrix multiplication (§7, portfolio_returns), and compute that portfolio's variance with a covariance matrix (mind the rowvar trap, §6) and a quadratic form (§7, covariance_matrix and portfolio_variance). Filter to the up-days only with boolean masking (§4, average_gain_on_up_days), winsorize extreme values without mutating the source array (§3's copy discipline, clip_returns), and patch any missing observations with a nan-aware mean (§9, fill_missing_with_mean). Ten mechanisms, ten functions, one pipeline, and, from here forward, every module in this branch is this same pipeline, applied to bigger data, with more structure (pandas' labels), more theory (statistics, ML), and, eventually, real money on the line (the QD branch). Get the array mechanics genuinely solid here, once, and you will spend the rest of the branch thinking about the finance and the models, not fighting a shape error you cannot explain.

{
  "skill": "Reviewing AI-generated NumPy code for the specific silent bugs this module teaches (rowvar, chained indexing, broadcasting axis, dtype)",
  "use": "Ask an AI assistant to draft a small NumPy function for a task you already understand by hand, then deliberately hunt it for this module's named silent-failure patterns before trusting a single line — using AI to produce a first draft fast, never to certify that draft as correct.",
  "tools": ["AI coding assistants — Claude, ChatGPT, GitHub Copilot", "This module's code pack's own hidden tests, as the actual verdict"],
  "prompt": "Write a NumPy function that computes the covariance matrix of a (days, tickers) return matrix (or: standardizes each column of one). After you draft it, state explicitly which axis or rowvar choice you made and why, so I can check it myself.",
  "verify": "Check the axis/rowvar choice against this module's own rule yourself — do not take the assistant's stated justification on faith; an assistant that gets rowvar backwards will still explain its answer fluently and confidently. Then run the draft against a small case you can verify by hand (as in this module's Worked Examples) AND against this module's code pack (`DA1.01`) hidden tests, which are built specifically to catch a rowvar default left unfixed, a chained-indexing no-op, or a broadcast against the wrong axis.",
  "diy": "You must already be able to state the broadcasting rule and the rowvar convention yourself (§5–6 above) before you can tell whether an AI's confident one-line fix is actually correct — this module's own hidden tests, not the assistant's confidence, are what actually decide.",
  "market": "US"
}

Common mistakes & how experts think differently

1. Confusing a view with a copy. A basic slice (a[1:3]) shares memory with the original array, mutate the slice and you mutate the source. Fancy indexing (a[[0,2]]) and boolean indexing (a[a>0]) both return independent copies, mutate the result and the source is untouched. The expert habit: before writing through any indexed result, ask "did I get here by a plain slice, or by a list/boolean selection?", because the answer changes whether your write reaches the original data or vanishes into a throwaway copy.
2. Chained indexing that silently does nothing. a[a > 0][0] = 999 builds a boolean-indexed copy, mutates that copy, and discards it, a is completely unchanged, with no error and no warning. This is not a hypothetical: it is precisely what this module's average_gain_on_up_days structured-fallback exercise asks you to spot. The expert habit: to write through a condition, use a single indexing step directly against the original array (a[a > 0] = 999) never a second index chained onto the first.
3. Trusting np.cov's default rowvar=True on (days, tickers) data. The default assumes each row is a variable; this module's convention is the opposite (rows are days, columns are tickers). Left at its default, np.cov(matrix) silently returns a matrix of the wrong shape (days-by-days instead of tickers-by-tickers) with no error at all. The expert habit: before calling any NumPy statistics function on tabular data, explicitly state which axis holds your variables, here, rowvar=False, always, deliberately, never left to the default.
4. Broadcasting a vector against the wrong axis. Subtracting a (days,) per-row vector from a (days, tickers) matrix without reshaping raises ValueError: operands could not be broadcast together with shapes (4,3) (4,) (exact message verified in this module's own authoring), a loud, honest failure, but only if the shapes genuinely conflict; a coincidentally matching size can let a wrong-axis broadcast through silently instead. The expert habit: state which axis a vector is meant to align against explicitly, with .reshape(-1, 1) or [:, None], rather than reshaping by trial and error until the error message stops appearing.
5. Letting NaN propagate through a plain .mean(). A single missing value anywhere in an array silently turns an ordinary .mean()/.sum()/.std() into nan, with no error, and that nan then poisons everything computed from it. The expert habit: the moment you have not personally confirmed an array is gap-free, default to the nan-aware function (np.nanmean and its siblings) as a reflex, not an afterthought triggered only after a report already came out wrong.
6. Assuming fixed-width integer dtypes have Python's unlimited-size integers. np.int8([90]) + np.int8([50]) silently wraps to -116 (140 - 256), with no error and, in current NumPy, no warning either. The expert habit: default to float64 (or NumPy's ordinary default integer width) for financial calculations, and treat a deliberately narrow fixed-width dtype as something that needs its value range checked, not assumed safe.
7. Expecting a /= 2 and a = a / 2 to behave identically on an integer array. The second always works (it creates a new float64 array and rebinds the name). The first tries to write a float64 result back into a's existing int64 memory and raises UFuncTypeError: Cannot cast ufunc 'divide' output from dtype('float64') to dtype('int64')..., an honest crash, but a surprising one if you assumed the two spellings were interchangeable shorthand. The expert habit: treat in-place operators on integer arrays as a deliberate, checked choice, not a harmless shortcut.
8. Treating a hardcoded array literal as if it were the vectorized solution. def to_returns(prices): return [0.1, -0.1] passes a single visible test built on [100.0, 110.0, 99.0] perfectly, and fails immediately the moment a hidden test calls it with any other array, because it is not computing anything, it is reciting one memorized answer, exactly the trap CS1.01 named for loop-based code and this module's hidden tests are built to catch for vectorized code too. The expert habit, unchanged from CS1.01: write the formula, not the answer, a genuinely vectorized function is indifferent to which specific numbers you hand it, India or US, visible or hidden, because it never memorized any of them in the first place.

Worked examples

A note on the data: every array below is explicitly synthetic (seeded, illustrative (R3)) small enough to trace completely by hand. The seven-day NSE series (₹) and seven-day US series ($) are the same two illustrative series CS1.01 used, reused deliberately so this module's vectorized code visibly reproduces the answers you already trust from that module's loop-based code. Nothing here is a real historical quote for any named company.

Worked example 1: The loop-versus-vectorized speedup, measured (mechanism, illustrative)

A learner has a working, loop-based to_returns and wonders whether rewriting it as prices[1:] / prices[:-1] - 1 is worth the trouble on a 200,000-row intraday price file.

Solution. Step 1, name what changes and what does not. Both versions compute the identical formula, element by element; neither is more "correct" than the other, this is purely a question of how much work the machine does to get the same answer. Step 2, name why the loop is slower, mechanically. The explicit Python loop asks the interpreter to re-discover, on every single one of 200,000 iterations, that it is dividing two floats, to dispatch that operation through Python's general-purpose bytecode machinery, and to bounds-check a list access, the same handful of bookkeeping steps, paid again and again, once per element. The vectorized expression pays that bookkeeping cost once, for the whole array, then runs a tight compiled loop that already knows the dtype and memory layout. Step 3, read the measured numbers. Timed directly, in this module's own authoring environment, on exactly this task at n = 200,000: the explicit loop took about 32 ms; the list-comprehension version (still a loop, just terser) took about 18 ms; the vectorized expression took about 0.9 ms, roughly a 35× speedup over the explicit loop, for byte-identical results (np.allclose confirms this directly). Step 4, generalize correctly, and not too far. Treat the exact multiplier as illustrative, it depends on your hardware, your NumPy build, and how large n is (the gap widens further as n grows, and narrows toward negligible for a handful of elements), but treat the order of magnitude as a durable, repeatable fact about vectorization versus explicit Python loops on numeric arrays, not a one-off benchmark trick. Read it. The lesson to actually keep is not "NumPy is fast" as folklore, it is the mechanical reason why: eliminating per-element interpreter overhead over contiguous, single-dtype memory. Once you can state why, you can predict, without benchmarking every time, that any explicit loop over a large numeric array is a strong candidate for vectorization, and any vectorized expression that reproduces a hand-traced small example correctly is very likely correct and fast at scale, simultaneously.

Worked example 2: A silent bug from confusing a view with a copy (India, ₹)

A learner is cleaning an NSE-style closing-price array and wants to zero out the first two entries (perhaps flagging them as "before the listing was liquid enough to trust") while keeping a separate, untouched copy of the original series for a later comparison chart. They write: original = np.array([3500.00, 3552.00, 3518.00, 3560.00]); flagged = original[:2]; flagged[:] = 0.0, expecting original to remain untouched because they "only modified flagged."

Solution. Step 1, classify the indexing operation before predicting the outcome. original[:2] is basic slicing, a start:stop slice with no list or boolean mask involved, and basic slicing returns a view, not a copy (§3). Step 2, predict, then check, what a view implies. Because flagged shares its underlying memory with original, writing through flagged[:] = 0.0 writes directly into the same memory original also reads from. There is, mechanically, only one array here wearing two names. Step 3, the actual, verified result. original becomes [0.0, 0.0, 3518.00, 3560.00], not the untouched [3500.00, 3552.00, 3518.00, 3560.00] the learner expected, because flagged's mutation reached straight through to it. Step 4, the fix, and why it works. To genuinely keep an independent copy, the learner needed flagged = original[:2].copy() (or, more simply, flagged = np.array(original[:2]), since np.array always copies), either forces a brand-new, memory-independent array, so mutating flagged afterward cannot possibly touch original. Read it. The bug did not crash, did not warn, and produced a completely plausible-looking (if wrong) result, exactly the profile of a view/copy confusion. The single habit that catches this before it ships: whenever you are about to mutate the result of an indexing operation, first ask "was that a basic slice, or a fancy/boolean selection?", and if you are not sure, or if you specifically need independence, call .copy() explicitly rather than guessing.

Worked example 3: Broadcasting a column-standardization, traced completely (US, $)

Two US tickers' daily returns over three days sit in a small matrix, returns = [[1.0, 10.0], [2.0, 12.0], [3.0, 14.0]] (illustrative units, chosen for clean arithmetic rather than realistic return magnitudes). Standardize both columns to mean 0, standard deviation 1, by hand, then reconcile with standardize_columns.

Solution. Step 1, reduce with axis=0 to get per-column statistics. Column 1 ([1.0, 2.0, 3.0]) has mean 2.0 and population standard deviation ≈0.8165; column 2 ([10.0, 12.0, 14.0]) has mean 12.0 and the identical standard deviation ≈1.633 (twice column 1's spread, since column 2 is exactly column 1 scaled by 2 and shifted). Both are (2,)-shaped vectors, one entry per column. Step 2: broadcast the subtraction. returns - [2.0, 12.0] aligns the trailing dimension (2 columns against the (2,) vector's own 2 entries) with no reshaping required (§5, Case 2) and gives [[-1.0, -2.0], [0.0, 0.0], [1.0, 2.0]], every row's column 1 entry minus 2.0, every row's column 2 entry minus 12.0. Step 3: broadcast the division. Dividing that result by [0.8165, 1.633] (again a clean (2,)-against-(3,2) broadcast) gives, for row 1: -1.0 / 0.8165 ≈ -1.2247 and -2.0 / 1.633 ≈ -1.2247, the same standardized value in both columns, because column 2 was built as an exact linear rescaling of column 1, so standardizing removes that rescaling entirely and leaves identical shapes. Verified directly through standardize_columns: the full result is [[-1.2247, -1.2247], [0.0, 0.0], [1.2247, 1.2247]], matching the hand trace to the same precision. Step 4: sanity-check the defining property. Standardized columns must have mean 0 and standard deviation 1, by construction, and re-computing the mean of either output column ((-1.2247 + 0 + 1.2247) / 3) confirms 0, exactly as the formula guarantees. Read it. Broadcasting did the "apply this per-column adjustment to every row" work with zero explicit looping and zero manual reshaping, because the shapes were already aligned on the trailing axis by construction (§5, Case 2), the discipline worth keeping is checking, before you trust any broadcast, exactly which axis your vector's single dimension is meant to line up against.

Worked example 4: The covariance sign, and the rowvar trap made concrete (India, ₹)

Two NSE-style tickers' daily returns over four days: ticker A [0.02, -0.01, 0.03, 0.00], ticker B [0.01, 0.01, -0.02, 0.04]. Compute their sample covariance by hand, then reconcile with covariance_matrix, and separately demonstrate what the rowvar default would have done wrong.

Solution. Step 1: means. Mean of A: (0.02 - 0.01 + 0.03 + 0.00) / 4 = 0.01. Mean of B: (0.01 + 0.01 - 0.02 + 0.04) / 4 = 0.01. Step 2: deviations from each mean. A's deviations: [0.01, -0.02, 0.02, -0.01]. B's deviations: [0.00, 0.00, -0.03, 0.03]. Step 3, multiply the deviations pairwise, sum, and divide by n - 1. Products: (0.01)(0.00) = 0, (-0.02)(0.00) = 0, (0.02)(-0.03) = -0.0006, (-0.01)(0.03) = -0.0003; sum = -0.0009; divide by n - 1 = 3: -0.0003. Step 4: reconcile with covariance_matrix(np.column_stack([A, B])). The off-diagonal entry of the returned 2 x 2 matrix is exactly -0.0003, matching the hand computation; the diagonal entries are each ticker's own variance by the same deviations-squared formula (≈0.000333 for A, 0.0006 for B). *Step 5, now compute what np.cov's default would have returned, on purpose, as the cautionary half of this example.* Calling np.cov on the same (4, 2) matrix without rowvar=False treats each of the 4 rows as a variable and each of the 2 columns as an observation of it, the reverse of what was intended, and returns a (4, 4) matrix instead of the intended (2, 2): the wrong shape, computing a completely different (and here, meaningless) quantity, with no error raised at any point. Read it. Two lessons in one example: first, the covariance computation itself is nothing more than "average product of deviations from the mean," entirely traceable by hand on a small case; second, and the reason DA1.01-e06 exists at all, the exact same function call with one keyword argument omitted does not fail loudly, it silently answers a different question, in the wrong shape, and only checking the shape of what came back (2x2 versus 4x4) catches it before it reaches a report.

Worked example 5: Portfolio return and variance via matrix mechanics (US, $)

A two-asset US portfolio holds weights [0.5, 0.5]. Two days of per-asset returns sit in a (2, 2) matrix, [[0.02, 0.04], [0.01, -0.02]], and the two assets' covariance matrix is [[0.0004, 0.0001], [0.0001, 0.0009]]. Compute the portfolio's daily returns and its variance, entirely by hand, then reconcile with this module's functions.

Solution. Step 1, portfolio returns via a weighted sum, one day at a time, by hand. Day 1: 0.5 x 0.02 + 0.5 x 0.04 = 0.01 + 0.02 = 0.03. Day 2: 0.5 x 0.01 + 0.5 x (-0.02) = 0.005 - 0.01 = -0.005. Step 2: reconcile with portfolio_returns([0.5, 0.5], [[0.02, 0.04], [0.01, -0.02]]), matrix multiplication (returns_matrix @ weights) performs exactly this weighted sum for every day at once, and returns [0.03, -0.005], matching the hand computation exactly. Step 3: portfolio variance, expanded by hand. For two assets, weights @ cov_matrix @ weights expands to w1² σ1² + w2² σ2² + 2 w1 w2 σ12. Substituting: (0.5)²(0.0004) + (0.5)²(0.0009) + 2(0.5)(0.5)(0.0001) = 0.25(0.0004) + 0.25(0.0009) + 0.5(0.0001) = 0.0001 + 0.000225 + 0.00005 = 0.000375. Step 4: reconcile with portfolio_variance([0.5, 0.5], [[0.0004, 0.0001], [0.0001, 0.0009]]), which returns 0.000375, matching the hand expansion to the last digit, because the two chained matrix multiplications are, mechanically, nothing but this same weighted-sum-of-variances-and-covariances formula, computed without you having to write out the expansion by hand every time. Read it. Notice precisely what has, and has not, been claimed: this example proves the formula computes correctly and shows you exactly what shapes go in and what number comes out. It says nothing about whether [0.5, 0.5] is a good choice of weights, or how you would find better ones, that question, and the constrained optimization needed to answer it, belongs entirely to QD1.04, later in this program; this module's job stops at "the arithmetic is correct and fast."

Worked example 6: NaN, filled correctly, and the plain-mean trap it avoids (India, ₹)

An NSE-style price series has one missing day (perhaps a data-vendor outage) recorded as np.nan: [3500.00, np.nan, 3518.00, 3560.00]. A learner first tries prices.mean() "just to get a quick average," then fills the gap properly.

Solution. Step 1, predict, then verify, the plain-mean trap. prices.mean() on an array containing even one nan returns nan, silently, with no error, because ordinary arithmetic treats any operation touching a nan as itself undefined, and that undefined-ness propagates through the whole reduction. Verified: np.array([3500.00, np.nan, 3518.00, 3560.00]).mean() is exactly nan, not a partial or approximate average of the three real values. Step 2, the nan-aware fix. np.nanmean computes the mean of only the non-missing values, correctly ignoring the gap: np.nanmean([3500.00, np.nan, 3518.00, 3560.00]) returns 3526.0, the average of 3500.00, 3518.00, and 3560.00 alone (10578.00 / 3 = 3526.0), with the missing day correctly excluded from both the sum and the count. Step 3, fill the gap, without disturbing the real values. fill_missing_with_mean builds a boolean mask of where the nans are (np.isnan(prices)), computes that same nan-aware mean once, and assigns it only into the masked position(s): the result is [3500.00, 3526.0, 3518.00, 3560.00], the one gap replaced with 3526.0, and the three genuine observations completely unchanged (verified: abs(result[0] - 3500.00) < 1e-6 and likewise for the other two real entries, both trivially true). Step 4, confirm the function copies rather than mutates. Because fill_missing_with_mean opens with .copy() on the array before writing into it, the original prices array, if it happened to be a real ndarray the caller still holds a reference to, not just a fresh list, is left completely untouched by the fill, exactly the discipline §3 and this module's clip_returns exercise both drill. Read it. Two separate lessons chain together here: NaN propagation is silent and total (one gap poisons an entire plain .mean()), and the fix is not "remember to double-check for NaN after the fact", it is to make the nan-aware function your default reflex the moment real-world data (which always, eventually, has gaps) is in play.

Practice set

Work every problem fully (predicting the answer before checking it) using nothing but this module's Core teaching content. Every problem below uses fresh numbers, distinct from the module's running NSE/US series and its Worked Examples, so you cannot simply recall an answer instead of computing one.

P1 (guided). dtype prediction. State the dtype NumPy assigns to np.array([1, 2, 3]), to np.array([1.0, 2, 3]), and give the values (not just the dtype) of np.array([1, 2, 3]) / 2.

Solution. np.array([1, 2, 3]) → int64 (every element is a whole-number literal, no decimal point). np.array([1.0, 2, 3]) → float64 (the presence of even one decimal-point literal makes NumPy infer a single shared float64 dtype for the whole array, recall every element must share one dtype). np.array([1, 2, 3]) / 2 → [0.5, 1.0, 1.5]: ordinary (non-in-place) true division always produces a float64 result even from an integer array, so there is no silent truncation here, the classic truncation trap only bites the in-place /= operator (§10, P15 below), never plain /.

P2 (guided). Broadcasting shape prediction. For a (3, 4) matrix, predict whether each of the following broadcasts successfully, and if so, the result shape: (a) matrix + a (4,) vector; (b) matrix + a (3,) vector; (c) matrix + a (3, 1) vector.

Solution. (a) Succeeds, result shape (3, 4), the trailing dimensions 4 and 4 match exactly. (b) Fails, raising ValueError: operands could not be broadcast together with shapes (3,4) (3,), the trailing dimensions 4 and 3 are neither equal nor is either 1, so NumPy refuses rather than guess which axis was intended. (c) Succeeds, result shape (3, 4), comparing trailing-first, 4 against 1 is compatible (the 1 stretches to 4), then 3 against 3 matches; this is exactly the .reshape(-1, 1) / [:, None] fix for case (b)'s failure, applied deliberately up front.

P3 (guided). View or copy? Given a = np.array([10, 20, 30, 40]), state what a equals after each of the following, run independently from the original a: (i) b = a[:2]; b[:] = 0; (ii) c = a[[0, 1]]; c[:] = 0.

Solution. (i) a becomes [0, 0, 30, 40], a[:2] is a basic slice, returning a view that shares memory with a, so writing through b writes directly into a. (ii) a remains [10, 20, 30, 40], completely unchanged, a[[0, 1]] is fancy indexing, returning an independent copy, so writing through c has no effect at all on a. The only syntactic difference between the two indexing expressions (a slice versus an explicit list of positions) is exactly what determines whether the later write reaches the original array or vanishes into a discarded copy.

P4. Vectorized returns, fresh prices. Compute to_returns([2000.0, 2080.0, 2038.4]) by hand.

Solution. [2080.0/2000.0 - 1, 2038.4/2080.0 - 1] = [0.04, -0.02], a clean +4% followed by -2%, computed with the same slicing formula (prices[1:] / prices[:-1] - 1) as every other array this module touches, regardless of length.

P5. Cumulative growth, fresh returns. Compute cumulative_growth([0.05, -0.02, 0.03]) by hand, and confirm your last value against the telescoping shortcut from §11.

Solution. [1.05, 1.05 x 0.98, 1.05 x 0.98 x 1.03] = [1.05, 1.029, 1.05987]. The telescoping check: if these three returns had instead been computed from an underlying price series via to_returns, the last value, 1.05987, would have to equal prices[-1] / prices[0] exactly, a genuine cross-check you can run on any returns-to-growth pipeline, not a coincidence specific to this example.

P6. Rebase to 100, fresh prices. Compute rebase_to_100([80.0, 88.0, 76.0]) by hand.

Solution. [80.0/80.0 x 100, 88.0/80.0 x 100, 76.0/80.0 x 100] = [100.0, 110.0, 95.0], the first element is always exactly 100.0 by construction, and every other element preserves its original proportional relationship to the first.

P7. Standardize columns, fresh 2-column matrix. For matrix = [[4.0, 200.0], [6.0, 220.0], [8.0, 180.0]], compute the column means and column standard deviations, and state (without computing every entry) what the standardized matrix's column means and standard deviations must equal, by construction.

Solution. Column means (axis=0): [6.0, 200.0] ((4+6+8)/3 and (200+220+180)/3). Column standard deviations: ≈1.633 and ≈16.33 (the second column's spread is exactly 10x the first's, since column 2 is column 1 scaled by 10 plus an offset). By construction, standardize_columns guarantees every output column has mean exactly 0 and standard deviation exactly 1, that is the entire definition of standardizing, regardless of the specific input values, which is exactly why this module's hidden tests check sum(column) / n ≈ 0 directly rather than only checking individual entries.

P8. Portfolio returns, fresh weights and matrix. Compute portfolio_returns([0.7, 0.3], [[0.03, -0.02], [-0.01, 0.04]]) by hand.

Solution. Day 1: 0.7(0.03) + 0.3(-0.02) = 0.021 - 0.006 = 0.015. Day 2: 0.7(-0.01) + 0.3(0.04) = -0.007 + 0.012 = 0.005. Result: [0.015, 0.005], each day, the weighted blend of that day's two per-ticker returns, exactly the returns_matrix @ weights computation performed once for the whole matrix.

P9. Covariance sign, opposite-moving series. Two series move mostly in opposite directions: x = [0.02, -0.01, 0.03, -0.02], y = [-0.01, 0.02, -0.015, 0.01]. Without computing the exact number, predict the sign of their covariance, then verify.

Solution. Negative. When one series tends to be above its own mean exactly when the other tends to be below its own mean, the deviations you multiply together are consistently opposite in sign, so their products are consistently negative, and summing consistently-negative products yields a negative total. Verified: covariance_matrix on column_stack([x, y]) returns an off-diagonal entry of ≈-0.000358: negative, as predicted. (Two series that tend to move together would show consistently same-signed products and a positive covariance instead, the general rule this example is built to make concrete, not a coincidence of this particular pair.)

P10. Portfolio variance, fresh weights and covariance. For weights = [0.6, 0.4] and cov_matrix = [[0.0009, -0.0002], [-0.0002, 0.0004]], compute the portfolio variance by hand.

Solution. w1² σ1² + w2² σ2² + 2w1w2 σ12 = (0.36)(0.0009) + (0.16)(0.0004) + 2(0.6)(0.4)(-0.0002) = 0.000324 + 0.000064 + (-0.000096) = 0.000292. Notice the negative covariance term actually reduces total portfolio variance below what either asset held alone would suggest, the arithmetic reason diversification across negatively- or weakly-correlated assets lowers portfolio risk, though (per this module's C-3 boundary) why that matters for portfolio construction is QD1.04's territory, not this one's.

P11. Average gain on up-days, fresh returns. For returns = [-0.03, 0.02, 0.05, -0.01, 0.04, -0.02], compute average_gain_on_up_days by hand.

Solution. Up-days (strictly positive): [0.02, 0.05, 0.04]. Their mean: (0.02 + 0.05 + 0.04) / 3 = 0.11 / 3 ≈ 0.03667. The three negative entries are excluded entirely from both the sum and the count, not averaged in as zeros, a meaningfully different (and larger) number than the plain mean of all six returns would give.

P12. Clip returns, fresh array and bounds. Compute clip_returns([0.08, -0.12, 0.02, 0.20, -0.03], -0.10, 0.10) by hand.

Solution. Each value is capped to [-0.10, 0.10] independently: 0.08 (inside the band, unchanged), -0.12 → -0.10 (below the floor, clipped up), 0.02 (unchanged), 0.20 → 0.10 (above the ceiling, clipped down), -0.03 (unchanged). Result: [0.08, -0.10, 0.02, 0.10, -0.03].

P13. Fill missing with mean, two gaps. For values = [100.0, nan, 120.0, nan, 140.0], compute the nan-aware mean used to fill, and the fully filled array.

Solution. np.nanmean averages only the three real values: (100.0 + 120.0 + 140.0) / 3 = 360.0 / 3 = 120.0. Both gaps are filled with that same single value (the mean is computed once, from the whole array, then broadcast into every missing position): [100.0, 120.0, 120.0, 120.0, 140.0]. Notice the middle, already-present value (120.0) happens to equal the fill value here purely by the coincidence of this example's numbers, the function does not know or care that they match; it only ever assigns the computed mean into positions where np.isnan was True.

P14. Integer overflow, fresh dtype. Predict the result of np.array([90], dtype=np.int8) + np.array([50], dtype=np.int8), and show the arithmetic that explains it.

Solution. Mathematically 90 + 50 = 140, but int8's range is -128 to 127, so 140 overflows and wraps: 140 - 256 = -116, returned silently with no error and no warning. The wraparound arithmetic is exactly "subtract the dtype's full range (256 for an 8-bit type) until the value falls back inside [-128, 127]", the same two's-complement behavior underlying every fixed-width integer overflow, in any language that uses it.

P15. In-place division on an integer array. Predict what happens when you run a = np.array([10, 20, 30]); a /= 3, and explain why it differs from a = np.array([10, 20, 30]); a = a / 3.

Solution. a /= 3 raises UFuncTypeError: Cannot cast ufunc 'divide' output from dtype('float64') to dtype('int64') with casting rule 'same_kind', true division always produces a float64 result, and NumPy refuses to silently narrow that back down into a's existing int64 memory in place. a = a / 3, by contrast, succeeds, returning [3.333..., 6.666..., 10.0], because it does not try to reuse a's old memory at all, it computes a brand-new float64 array and simply rebinds the name a to point at it, exactly CS1.01's re-assignment model (a name is rebound to a new value, nothing is mutated in place).

P16. AI-generated code, reviewed. An AI coding assistant, asked to write a function returning the covariance matrix of a (days, tickers) return matrix, produces: def covariance_matrix(returns_matrix):\n return np.cov(returns_matrix). Is this correct? If not, exactly what goes wrong, and how would this module's hidden tests catch it?

Solution. It is wrong. np.cov's rowvar argument was left at its default, True, which assumes each row is a variable, the opposite of this module's (days, tickers) convention, where rows are days and columns are tickers. On, say, a (4, 3) input (4 days, 3 tickers), this draft silently returns a (4, 4) matrix, the wrong shape entirely, instead of the intended (3, 3) tickers-by-tickers covariance matrix, with no error or warning at any point. This module's hidden tests catch it immediately and unambiguously: they assert on the shape of the result (len(result) == 3 and len(result[0]) == 3) before ever checking individual values, so a rowvar-default submission fails on the very first assertion, every time, regardless of which specific numbers were used. This is precisely why this module's ai-augment-json callouts insist on tracing the axis/rowvar choice yourself and running the tests before trusting an assistant's confident one-liner: the bug here reads as entirely plausible, idiomatic code, and only checking the actual returned shape against what you expected catches it.

Applied mini-project

Build a one-page vectorized portfolio-analytics mini-report, for both markets.

Using two small synthetic 4-day, 3-ticker return matrices, an India-flavored one, nse_matrix = [[0.02, -0.01, 0.01], [0.01, 0.02, -0.02], [-0.01, 0.03, 0.02], [0.03, 0.00, 0.01]], and a US-flavored one, us_matrix = [[0.01, 0.02, -0.01], [-0.02, 0.00, 0.03], [0.03, -0.02, 0.01], [0.02, 0.04, -0.01]], plus a portfolio-weight vector of your own choosing for each (three weights, summing to 1.0), write one small NumPy program (in this module's Practice/code area, or in any Python environment you have) that, for each market, computes and prints:

1. The standardized return matrix (standardize_columns), and a one-line confirmation that every column's mean is (numerically) 0 and standard deviation is 1.
2. The covariance matrix (covariance_matrix), printed with its shape stated explicitly (e.g. "3 x 3"), as a deliberate self-check against the rowvar trap.
3. The portfolio return series for your chosen weights (portfolio_returns), one value per day.
4. The portfolio variance for the same weights and the covariance matrix from step 2 (portfolio_variance), a single number.
5. The average gain on up-days only of your portfolio return series (average_gain_on_up_days), and one sentence stating how it differs from the plain average of every day (including down-days).

This is deliberately the integration task: every function above is a graded, individually-tested exercise in this module's code pack (DA1.01); the mini-project is where you wire five of them together into one coherent pipeline and confirm it holds together end to end, for both an India and a US matrix, without re-deriving any formula from scratch. Before you trust a single number, hand-verify at least the covariance matrix's shape and one entry, and the portfolio variance, for one of the two markets (exactly as the Worked Examples did) and confirm your program agrees.

Self-check rubric, score yourself out of 10, pass at 8/10 with no zero on any item:

• (2 pts) Both markets present and correctly labeled, the report clearly distinguishes the India-flavored figures from the US-flavored figures, with no mixed-up numbers.
• (2 pts) Every one of the five computed quantities is present for both markets, standardized matrix, covariance matrix, portfolio returns, portfolio variance, and average gain on up-days; nothing skipped.
• (2 pts) The covariance matrix's shape was explicitly checked and stated (tickers × tickers, not days × days), the concrete, printed defense against the rowvar trap, not just a value that happens to look plausible.
• (2 pts) At least the portfolio variance was independently hand-verified (shown as a written expansion, w1²σ1² + w2²σ2² + 2w1w2σ12, not just "I ran it and it looked fine") for at least one market, before being trusted.
• (2 pts) The up-days-only sentence is numerically correct and specific, it states why the two averages differ (down-days excluded entirely versus counted as real values), not a vague gesture at "it's different because of rounding."

Keep this program, the "compute by hand on a small case, write the general vectorized function, verify by re-computing" loop you just ran end to end is the exact discipline every later module in this branch, and every one of the DA1.01 code-pack's own hidden tests, will hold you to.

Reading & resources

Everything below supports what this module teaches; none of it substitutes for actually writing and running the code yourself. Free unless marked; tags show the ramp from first-contact to depth.

• NumPy's own official documentation, especially NumPy: the absolute basics for beginners and the NumPy fundamentals user guide (numpy.org), the primary, authoritative reference for every function and rule in this module, written and maintained by the library's own developers. [Free] [Beginner]
• NumPy's own Broadcasting user guide page: the canonical statement of the exact rule taught in §5, with more worked shape examples than this module has room for. [Free] [Beginner–Intermediate]
• Jake VanderPlas, Python Data Science Handbook, the NumPy chapters, a friendly, example-driven deep dive into arrays, broadcasting, and aggregations, freely readable online (also available as a paid print/ebook). [Free online / Paid print] [Beginner–Intermediate]
• Nicolas P. Rougier, From Python to NumPy, a free online book that goes considerably deeper into vectorized thinking and performance than this module has scope for; the natural next stop once these fundamentals are solid. [Free] [Intermediate–Advanced]
• "100 NumPy exercises" (a well-known open GitHub problem set, rougier/numpy-100), short, sharp, individually-checkable drills covering exactly this module's mechanisms (indexing, broadcasting, reductions) and considerably more; excellent spaced-repetition practice once you have this module's core down. [Free] [Beginner–Advanced]
• NumPy's own Linear algebra documentation page (numpy.linalg, and the @ operator), the authoritative reference for matrix multiplication and the handful of related operations this module touches. [Free] [Intermediate]

Do this, not just read. Open any Python environment (this module's own code editor is enough) and, for ten minutes, deliberately trigger every failure this module named, on purpose: divide an integer array in place and read the UFuncTypeError; add two int8 arrays that overflow and watch the silent wraparound; broadcast two genuinely incompatible shapes and read the ValueError carefully; compute .mean() on an array with one nan planted in it and watch a single missing value quietly poison the whole result. Deliberately breaking things you now know the name for, and watching NumPy's own error messages confirm exactly what this module predicted, will cement the mechanism far faster than reading about it a second time, and every one of these five small experiments is something you will meet for real, unannounced, before this module's code pack is done.

Teach it back & journal

Feynman prompt. Explain to someone who already knows basic Python, but has never used NumPy, why prices[1:] / prices[:-1] - 1 is a genuinely complete, correct way to compute every daily return in one line, not a clever shortcut that happens to work on one example, but the general vectorized form of the exact loop they would otherwise write by hand. Then explain, separately and just as carefully, why a[a > 0][0] = 999 can look completely reasonable and still silently do nothing at all, using the words "view" and "copy" correctly, and stating precisely which of the two indexing steps in that expression is the one that throws the mutation away. If your listener can, unprompted, predict that np.cov(matrix) on a (days, tickers) matrix is likely returning the wrong shape rather than just "maybe slightly off," you have installed this module's central discipline: shapes are not a formality to check after something goes wrong, they are the first and fastest thing to verify before you trust any array result at all.

Journal prompt. Over the next few days, keep a short log of every NumPy shape error, nan you did not expect, or silently-wrong-shaped result you hit while working through this module's code pack: the exact operation you ran, what shape or value you expected, what you actually got, and which of this module's named mechanisms (view vs. copy, broadcasting, rowvar, NaN propagation, dtype overflow) turned out to explain it. How many of your bugs were a loud, honest crash (a ValueError or UFuncTypeError) versus a silent wrong answer with no error at all, and which kind took longer to actually find? Then write a short paragraph: which single habit from this module: checking .shape before trusting any array result, tracing whether an indexing operation was a view or a copy before writing through it, or stating the rowvar/axis convention explicitly rather than trusting a default, would most change how confidently you'd hand a piece of your own vectorized code to someone else to rely on, and what will you do differently the next time you reach for np.cov, a broadcast, or an in-place operator because of it?