Branch: Data Science & ML (quant-ds, Quant-Code region) · Node DA1.04 · ~16 focused hours · Gate ≥ 85% (quiz + code pack)

Every chart makes a claim, whether its author meant it to or not. A bar chart sorted alphabetically claims that alphabetical order is what matters; a line chart with two y-axes claims that two differently-scaled series are fair to eyeball against each other; a histogram with ten arbitrary bins claims that ten is the right amount of detail for this exact dataset. Most of the time nobody chose these claims on purpose (they are matplotlib's defaults, or Excel's defaults, or muscle memory from the last chart somebody made) and a reader has no way to tell an intentional claim from an accidental one. This module exists to make the claim intentional every time. Here is the whole module compressed into the one sentence you will spend sixteen hours earning the right to believe: a chart's form must follow the comparison it exists to make (not aesthetic preference, not a library default, not "that's what Excel gives you") and exploratory data analysis is the disciplined, numeric habit of knowing exactly what that comparison is before you ever call a plotting function.

This is the code-side half of a split this program draws on purpose. G1's QM branch (QM1.01, QM1.02) already gave you the theory, what a distribution is, what skewness and the Central Limit Theorem mean, what makes a correlation statistically real rather than noise. This module does not re-derive any of that; it hands you the implementation: how to compute a five-number summary, a principled histogram bin count, an outlier flag, and a correlation matrix in actual, gradable Python, and how to turn each of those numbers into a matplotlib exhibit that says exactly one true thing clearly. Distributions, outliers, and correlation are the three systematic checks every real dataset gets before anyone is allowed to build a model or write a memo about it, not because it is traditional, but because a model fit on a distribution you never looked at, or a correlation you never sanity-checked, is a model built on an assumption nobody tested. Exploratory data analysis (EDA) is that testing, done first, done numerically, done before a single picture gets drawn.

One promise for the hours ahead: by the end you will be able to take any bundled dataset in this program (an India or a US price history, a cross-sectional table of company fundamentals, a macro time series) and, without being told what to look for, run the same three-step drill: describe it (five-number summary, moments, a properly-binned histogram), interrogate it (outliers via Tukey's fences, a correlation matrix), and exhibit it (the one chart type that matches the comparison, built and labeled so a stranger reads the right message in five seconds). That drill, run on both an India (₹) and a US ($) dataset in this module's worked examples and its nine-exercise code pack, is the entire discipline this node installs, everything downstream in quant-ds, from DA1.05's hypothesis tests to DA2.01's model diagnostics to QD1.02's factor-evaluation heatmaps, assumes you already have it.

Learning objectives

By the end you can:

Explain what exploratory data analysis is and why it must happen (numerically, not just visually) before any model is fit or any chart is drawn, and name the three systematic checks (distribution, outliers, correlation) that make up a disciplined first look at a dataset.
Compute a full five-number summary plus mean, sample standard deviation, and a moment-based skewness in code (NumPy), stating precisely which convention each statistic uses (sample vs. population denominator, linear-interpolation quantiles) and reading what a nonzero skew implies about a distribution's shape.
Choose a histogram's bin count by a principled rule (Freedman–Diaconis, with a Sturges fallback) rather than an arbitrary default, and explain (with a concrete before/after) how the choice of bins changes the story a histogram appears to tell.
Detect outliers with Tukey's IQR fences, implement the mask in code, and reason about when a flagged point is a genuine extreme worth keeping versus a data error worth investigating, never an automatic delete.
Compute a Pearson correlation matrix in code as a descriptive summary of co-movement, build the heatmap it feeds, and state precisely why "the correlation is 0.42" is not the same claim as "the correlation is statistically significant" (that test is QM1.02's, not this module's).
State and apply the dataviz grammar this module needs in code: map a comparison type (ranking, trend over time, distribution, relationship between two variables) to its correct chart type, and implement that mapping with matplotlib's object-oriented API (fig, ax = plt.subplots()), not the bare pyplot state machine.
Identify and eliminate the specific chartjunk failures this module grades against: an unsorted ranking bar chart, a dual (twinx) y-axis comparing unlike scales, a missing or unit-free axis label, and a generic, non-committal title, and implement the fix (sort, rebase to a common index, label, title the comparison) in code.
Build a rolling mean and volatility band to smooth a noisy time series, and explain why plotting raw daily noise over a multi-year window is usually the wrong exhibit.
Build the numeric machinery behind a box-plot comparison (matplotlib's own five-number-plus-whiskers convention) so that comparing several groups' distributions in one exhibit is a computed fact, not a matplotlib default you trusted blindly.
(Productivity objective: R10 duality.) Use an AI coding copilot to accelerate writing pandas/matplotlib EDA boilerplate once you have already decided, from this module's grammar, which chart the comparison calls for, and verify its output by running this module's own hidden tests and confirming every matplotlib call it used actually exists.

The duality, stated once (R10). Objectives 1–9 are the understanding objective the mastery gate rewards: you pass by demonstrating you can compute the numbers and choose the chart yourself, closed-book. Objective 10 is the productivity payoff you keep afterward, once you already know which chart a comparison calls for, a copilot legitimately speeds up the ax.set_* boilerplate. A tool can draft the code; it cannot decide what the chart should say, and that decision is the entire content of this module.

Prerequisites & connections

Builds on. This node assumes real fluency with the three modules immediately before it in the quant-ds branch. DA1.01 (NumPy & Vectorized Computation) is where the array operations this module leans on constantly: np.mean, np.std with an explicit ddof, np.quantile, vectorized arithmetic over a whole column at once, stopped being new; this module does not re-teach vectorization, it applies it to real EDA questions. DA1.02 (pandas I. Data Wrangling) is where loading a CSV, filtering rows, and grouping by a key became routine; every exercise and worked example below loads one of the program's bundled datasets and immediately filters it (df[df.ticker == '...'], df[df.fiscal_year == ...]) exactly the way DA1.02 taught. DA1.03 (pandas II. Cleaning, Time Series & the Excel↔Python Bridge) is the closest and most load-bearing neighbor: this module assumes the data arriving in each exercise is already clean (no stray nulls, correctly typed dates): DA1.03 is where that cleaning discipline was built, and this module's rolling-window work (DA1.04-e09) is the direct continuation of DA1.03's time-series indexing. If any of the three feels shaky, the shakiness will surface here as a wrong sample-vs-population ddof, a groupby that returns the wrong shape, or a rolling window that silently includes a row it shouldn't: go back, not around.

Feeds forward. DA1.05 (Statistics in Code) takes this module's descriptive correlation and distribution work and adds the inferential layer on top (a hypothesis test for whether a correlation is real, a formal normality check, a regression's diagnostics) and explicitly cross-links back to this module for the descriptive numbers it starts from. DA2.01 (ML I. Supervised Learning) and DA2.02 (ML II. Unsupervised, Feature Engineering & Model Discipline) both assume the "look at your data before you touch it" reflex this module installs: a train/test split fit on a feature nobody histogrammed, or a leak silently hiding in a correlation nobody checked, is exactly the kind of mistake EDA exists to catch first. QD1.02 (Signal & Factor Construction and Evaluation) reuses this module's correlation-matrix and distribution machinery directly, cross-sectionally, on factor panels, the heatmap you build here on four synthetic factors in Worked Example 7 is the same computation QD1.02 runs at production scale to check for double-counted signals.

Cross-links (own the code, not the theory). Per the binding C-3 boundary: G1's QM branch owns the statistics: QM1.01 (Probability, Distributions, Sampling & Estimation) derives what a distribution, a skew, and the Central Limit Theorem are; QM1.02 (Hypothesis Testing, Correlation & Regression) derives when a sample correlation is statistically distinguishable from zero. This module never re-derives either, it computes a skew and a correlation coefficient in code and hands you the number; if you want to know why the formula looks the way it does, or whether your r = 0.42 clears a significance bar, that theory lives in QM1.01/QM1.02, referenced, not repeated. Symmetrically, this module's dataviz grammar is a deliberately narrow, code-implementation slice of a fuller subject that the Presentation galaxy's G4/PR2 branch (Chart-Choice Grammar, Decluttering & Focus, One Message Per Chart) owns end to end, including the shared dataviz heuristic module, Gestalt principles, and dashboard layout, this module teaches exactly enough of that grammar, in matplotlib, to make your pandas output correct by construction; PR2 is where the deeper design narrative, the pre-attentive-attribute theory, and the eye for "one series too busy" actually live. Galaxies never gate each other (R11): you can do this module with zero PR2 exposure, and PR2 with zero code, but the two are the same discipline seen from two sides.

Core teaching content

What EDA actually is, and why it comes before the chart

Treat "exploratory data analysis" as a specific, three-part habit, not a vague synonym for "looking at data." Before you fit a model, write a memo, or draw a single exhibit, you run three systematic checks on the dataset in front of you: describe its distribution (what does a typical value look like, and how spread out and lopsided is it), flag its outliers (which points sit far enough outside the pack that they need a human decision, not a silent default), and check its correlations (which columns move together, and by how much). Skipping this and going straight to a chart or a model is how a right-skewed expense-ratio column gets summarized by a mean that no individual fund actually has, how a single data-entry typo (a revenue of 269,000 instead of 269) silently dominates every regression fit afterward, and how two features that are 98% correlated get fed into a model as if they were independent information. None of those failures announce themselves, they look like ordinary numbers until someone runs the three checks. That is the entire justification for a node that, on its face, looks like "just compute some statistics": the statistics are the tripwire that catches a bad dataset before it wastes the far more expensive hours of modeling or writing on top of it.

The three checks are also, not coincidentally, exactly the inputs a well-chosen chart needs. A histogram needs a distribution and a bin count; a box plot needs the same five numbers matplotlib's own whisker convention is built from; a heatmap needs a correlation matrix. This module's structure follows that fact: every section below builds one numeric check in code, then shows the one exhibit type that check exists to feed.

The five-number summary and moments, computed, not assumed

A describe() call is not a black box, it is five specific numbers plus a mean and a spread, each with a definition you should be able to state and recompute by hand on a small example. Given a list of values, the minimum and maximum need no comment; the median (Q2, the 50th percentile) is the middle value once sorted; Q1 and Q3 (the 25th and 75th percentiles) split the lower and upper halves the same way. This module uses linear interpolation between the two nearest ranks for a percentile that doesn't land exactly on a data point, the default method NumPy's np.quantile and pandas' .quantile() both use, and the one every hidden test in this module's code pack checks against. (Other interpolation conventions exist and are perishable across library versions and settings, if you ever get a quantile that disagrees with a colleague's by a hair, "which interpolation method?" is the first question to ask, not a bug report.)

Two numbers deserve extra care because their denominator convention is easy to get backwards. The sample standard deviation (what you almost always want when you have a sample, not the whole population) divides the sum of squared deviations from the mean by n − 1, not n (ddof=1 in NumPy's argument name; ddof stands for "delta degrees of freedom," and forgetting to set it is the single most common mistake this module's Common Mistakes section names). The population standard deviation (ddof=0, dividing by n) is what you use when the data in front of you genuinely is the entire population you care about, and, not obviously, it is also the convention this module's skewness formula needs internally: skew = mean((x − mean)^3) / population_std^3. Notice the inconsistency and hold it on purpose rather than "fixing" it: the skewness formula's units only cancel correctly against the population-convention denominator, even when you separately report a sample-convention standard deviation elsewhere in the same summary. A skew of 0 describes a symmetric distribution; positive skew means a long right tail (a few very large values pulling the mean above the median, common in return series after a rally, in revenue across companies of wildly different sizes, in almost any variable that is bounded below by zero); negative skew is the mirror image. Reading a skew's sign before you look at a single chart is often enough to predict, correctly, whether the histogram you are about to draw will look lopsided.

DA1.04-e01 in the code pack is exactly this computation: a function summary_stats(values) returning a dict with all nine numbers above, graded against a hand-checkable five-value example and (hidden, so a solution that merely hardcodes the visible answer fails immediately) real daily-return series pulled from this program's bundled US and India end-of-day price history.

Choosing histogram bins: the Freedman–Diaconis rule

"Just use ten bins" is the single most common piece of unexamined chartjunk in this entire module, because a histogram's shape is not a fixed property of the data, it is a joint property of the data and the bin width somebody chose, and two analysts looking at the identical column can walk away with opposite impressions of its shape if one used 5 bins and the other used 50. Too few bins and real bimodality or a fat tail gets smoothed into invisibility; too many and ordinary sampling noise gets dressed up as fine structure nobody should trust. The fix is a principled rule, not a fixed number.

The Freedman–Diaconis rule (Freedman & Diaconis, 1981) sets the bin width from the data's own spread and size: bin width = 2 × IQR / n^(1/3), where IQR = Q3 − Q1 (the same linear-interpolation quartiles from the section above) and n is the sample size. Divide the data's full range by that width and round up to get the bin count: n_bins = ceil((max − min) / bin_width). The n^(1/3) term is deliberate, as a sample grows, FD adds bins slowly (proportional to the cube root of n, not linearly), because a bigger sample earns you finer resolution but not without limit; the IQR term (rather than the full range or the standard deviation) makes the rule robust to a handful of extreme outliers, which would otherwise blow the bin width out under a naive range-based rule. When the data is degenerate, every value identical, or the whole sample crammed into one quartile so IQR = 0. FD's formula divides by zero and must fall back to something else; this module's convention (and its code pack's, DA1.04-e02) is the older Sturges' rule (Sturges, 1926): n_bins = ceil(log2(n) + 1), a simpler rule that assumes roughly normal data and undercounts bins badly on skewed or heavy-tailed data, which is exactly why FD is the default and Sturges is only the emergency fallback.

Worked Example 2 below runs this on a real US return series and shows the gap between FD's answer and a naive bins=10 directly, the difference is not cosmetic.

Outliers: Tukey's fences, and the z-score alternative from QM1.01

An outlier, for this module's purposes, is not "a suspicious-looking number", it is a value flagged by a stated, computable rule, because "suspicious-looking" is exactly the judgment a systematic check exists to replace. The rule this module uses is Tukey's fences (Tukey, 1977): using the same Q1, Q3, and IQR from the sections above, a value is flagged if it falls below Q1 − k × IQR or above Q3 + k × IQR, with k = 1.5 the conventional default (a stricter k = 3.0 is sometimes used for "extreme" outliers only). Tukey's fences are attractive for exactly the reason FD's bin width is: they are built from quartiles, which barely move even if the data already contains a few genuine extremes, so the fence itself does not get dragged outward by the very points it is trying to flag.

QM1.01 already gave you a second, related tool: the z-score, z = (x − μ)/σ, which standardizes a value by how many standard deviations it sits from the mean, usually flagging |z| > 2 or |z| > 3 as extreme. The two methods are not interchangeable, and knowing when they disagree is the actual skill. A z-score's denominator is the standard deviation, which the outlier itself inflates, sometimes enough to make its own z-score look unremarkable (a single enormous value can drag σ up so far that z for that same value stops clearing the threshold). Tukey's fences, built from quartiles instead, are far less sensitive to this self-masking effect, which is exactly why this module defaults to IQR-based flagging for cross-sectional and return-series work and treats the z-score as QM1.01's complementary tool for genuinely close-to-normal data.

Flagging is not deleting. A value outside Tukey's fences is a candidate for a human decision, not an automatic removal, it might be a data-entry error (a revenue reported in the wrong currency scale, a decimal point in the wrong place) worth fixing or excluding, or it might be the single most informative row in the dataset (the one company whose earnings actually did triple that year). DA1.04-e03's hidden tests flag exactly one company's net income as an outlier in each of a real US and a real India fundamentals slice. Worked Example 3 walks through why that flagged company is worth a second look, not a deletion.

Correlation as description, not proof

A Pearson correlation matrix answers one narrow, descriptive question: for every pair of numeric columns, how strongly and in which direction do they move together, on a scale from −1 (perfectly opposite) through 0 (no linear relationship) to +1 (perfectly together)? Building it in code is direct, for each pair of equal-length series, standardize and take the covariance, or simply call np.corrcoef on the stacked array, and the result is always symmetric (corr[a][b] == corr[b][a]) with a diagonal of exactly 1.0 (every series is perfectly correlated with itself). DA1.04-e07 builds exactly this matrix as a dict-of-dicts, checked against np.corrcoef on a real four-factor synthetic panel plus a forward return column.

State the boundary explicitly, because it is the exact seam this module shares with QM1.02: a Pearson r you compute here is a descriptive summary of the sample you have: full stop. Whether that r is large enough to be statistically distinguishable from zero in the wider population it was sampled from is a completely separate question, with its own test, its own sample-size dependence, and its own assumptions, and that test is QM1.02's H0: ρ = 0 machinery, not this module's. A heatmap built here can show you that two factors barely move together in this sample; whether that near-zero reading would hold up in a formal test, or whether an r = 0.42 clears significance at a stated α, is a question this module deliberately declines to answer, so QM1.02 does not get re-derived twice in two different galaxies of this program.

{
  "skill": "Using an AI coding copilot to draft pandas/matplotlib EDA boilerplate — without letting it choose the chart or hide a hallucinated API call",
  "use": "Once you already know, from this module's grammar, which chart type a comparison calls for and which specific numbers must feed it, ask a copilot to draft the mechanical matplotlib calls (axis labels, tick rotation, figure sizing, color) so you move faster on the part that has only one right answer once the decision is made.",
  "tools": ["GitHub Copilot / Cursor — inline completion inside an editor", "Claude / ChatGPT — chat-based code drafting for a described chart"],
  "prompt": "Using matplotlib's object-oriented API (fig, ax = plt.subplots()), draft a <CHART TYPE> of <DATA>, sorted <ORDER> by <VALUE>, with a y-axis labeled '<UNIT>' and a title stating <THE SPECIFIC COMPARISON>. Do not add a legend, gridlines, dual axis, or a different chart type unless I asked for it.",
  "verify": "Read every matplotlib call before running it and confirm each one — ax.set_xticklabels, ax.spines, ax.twinx, whatever it used — genuinely exists on an Axes object in the matplotlib version you're running (check the official docs, not the assistant's confident say-so; a plausible-looking but hallucinated keyword argument is one of the most common AI-generated-code failures). Then run this module's own hidden tests: DA1.04-e04 checks sort order and labels; DA1.04-e06 checks that no second Axes was silently created via twinx().",
  "diy": "You must already be able to state, unprompted, which chart type this module's grammar calls for and why, before you hand the boilerplate to a copilot — that judgment is this module's entire content, and a copilot that drafts a fluent, wrong chart choice has not saved you any real work.",
  "market": "US"
}

The dataviz grammar, in code: form follows the comparison

Every chart exists to make exactly one comparison legible, and the comparison's type (not taste, not habit) determines the chart's correct form. Hold four pairings as the load-bearing subset this module grades against (the fuller grammar, including composition/part-to-whole charts, diagrams, and dashboard layout, is G4/PR2's territory):

Ranking a set of categories ("which of these fifteen companies earns the most revenue") → a bar chart, sorted by value, never alphabetically or by whatever order the data happened to arrive in. Alphabetical order answers a question nobody asked.
A trend over time ("has this price grown") → a line chart. When the comparison is between two or more series on different original scales, the fix is never a second y-axis (more on this below), it is rebasing every series to a common starting index.
The shape of a single distribution ("how spread out and skewed are these returns") → a histogram with a principled bin count (Freedman–Diaconis, above), or, for comparing the same statistic across several groups at once, a box plot.
The relationship between two numeric variables ("do these two factors move together") → a scatter plot for a single pair, or a correlation heatmap for many pairs at once (the matrix from the previous section, rendered as color).

The discipline is mechanical, not aesthetic: name the comparison in one sentence before you write a single plt call, and the chart type falls out of that sentence. "I want to show which company earns the most" names a ranking, which names a sorted bar chart, there was never a decision to make about pie charts, 3-D bars, or a dual axis, because none of those forms answer a ranking question in the first place.

Killing chartjunk in code: sorting, rebasing, and honest labels

Chartjunk (a term from Edward Tufte's work on the data-ink ratio) is any part of a chart that does not serve the comparison, or, worse, actively misleads about it. This module grades three specific, code-implementable chartjunk failures, because each one is common, each one is a single concrete line of code to fix, and each one is exactly what this module's hidden tests check.

Unsorted ranking bars. A bar chart meant to show "who is biggest" that plots bars in input order (or alphabetical order) makes the reader do the ranking in their head, work the chart exists to do for them. The fix, in code, is one line: sorted(zip(labels, values), key=lambda p: p[1], reverse=True) before you ever call ax.bar. DA1.04-e04 grades exactly this: the bar heights, in the order they appear on the Axes, must equal the values sorted descending, not the caller's original order.

The dual-axis trap. Comparing a ₹1,200 stock against a $60 stock (or a revenue column against a margin percentage) on one chart tempts a second y-axis (ax.twinx()), and a dual axis lets its author choose each axis's zero point and scale independently, which means two people can read the same chart and walk away with opposite, both-defensible stories about which series "grew more." The honest fix is rebasing: divide every series by its own first observation and multiply by 100, so every series starts at the identical value and only genuine relative growth separates them afterward, all on one shared axis. DA1.04-e05 computes the rebasing; DA1.04-e06 builds the actual two-line chart and is graded, concretely, on len(ax.figure.axes) == 1, a real, checkable proxy for "no second axis was created."

Missing or unit-free labels, and a title that doesn't commit to a message. A y-axis labeled nothing, or labeled "value", forces a reader to guess whether they are looking at dollars, rupees, a percentage, or an index level. A title that reads "Chart 1" or "Revenue Data" describes the chart's category, not its message, compare that to a title that states the actual comparison, "Fiscal-year revenue by company, largest to smallest", which tells a reader what to look for before their eyes even reach the bars. DA1.04-e04's hidden tests check that the y-axis label states a unit and that the title names the comparison, not merely that a title string exists.

Smoothing the noise: rolling means and volatility bands

Plotting two years of raw daily closing prices produces a jagged, spiky line whose day-to-day zigzags usually carry no information a reader can act on, the exhibit that actually reads is a rolling mean (the trend a reasonable window of recent history implies) optionally paired with a rolling standard deviation (how much the series has been bouncing around that trend lately, often shown as a shaded band above and below the mean). A trailing rolling window of size w computed at position i uses observations i − w + 1 through i; the first w − 1 positions have no full window behind them and are conventionally dropped rather than padded with a placeholder. DA1.04-e09 builds rolling_mean_std(values, window) this way, checked against pandas.Series.rolling(window).mean()/.std() on a real 500-day US and India price history with a 20-day window, long enough to smooth genuine daily noise, short enough to still track a real regime change within a few weeks rather than a few quarters.

Two failure modes sit on either side of this choice, and naming them protects you from both. Too short a window barely smooths anything and mostly reproduces the raw noise you were trying to escape. Too long a window smooths past genuine structural changes (a company's business model shifting, a macro regime change) turning a real inflection point into an invisible blur. There is no universal correct window; there is only "does this window's length match how fast the thing you're tracking actually changes," a judgment call this module expects you to state explicitly rather than default silently.

Comparing distributions across groups: what a box plot is actually built from

When the comparison is "how does this distribution differ across several groups at once" (several tickers' return distributions, several months' worth of a metric) a box plot compresses each group's five-number summary (plus outliers) into one small, stackable glyph, which is exactly what makes it the right tool for a small-multiples-style comparison (the fuller "small multiples, one message per chart" discipline is G4/PR2.03's; this module owns computing the numbers correctly). The part worth building by hand once, because it is not quite what "1.5 times the IQR" sounds like: a box plot's whiskers do not extend to the raw fence value Q1 − 1.5×IQR, they extend only to the most extreme actual data point still inside that fence. Points genuinely beyond the fence are drawn as individual fliers (dots), not folded into the whisker.

DA1.04-e08 builds this exact structure: boxplot_stats(values, k=1.5) returning min, q1, median, q3, max, whislo, whishi, and fliers, and its hidden tests check the result against matplotlib.cbook.boxplot_stats, the very function ax.boxplot() calls internally. Building it yourself once is the difference between trusting a box plot's whiskers as a fact about your data and trusting them as an unexamined default; Worked Example 6 runs this on a real US and India macro time series side by side.

Figure, Axes, and why the object-oriented API is the default here

matplotlib exposes two overlapping APIs, and this module standardizes on one deliberately. The pyplot state-machine API (plt.plot(...), plt.title(...)) implicitly tracks a single "current" figure and axes behind the scenes, convenient for a one-off, throwaway plot in a notebook cell, and a source of quiet bugs the moment you build more than one chart in the same script, because "current" can silently point somewhere you didn't expect. The object-oriented API: fig, ax = plt.subplots(), then every subsequent call as a method on that specific ax (ax.bar(...), ax.set_title(...), ax.legend()), makes the target of every instruction explicit, and is what every exercise in this module's code pack is built around, for a very concrete reason: it is what lets the hidden tests inspect your chart's actual properties (ax.patches, ax.lines, ax.get_title(), len(ax.figure.axes)) directly and deterministically, the same way a human reviewer would look at the chart and judge it against the grammar above. A Figure is the whole canvas (and can hold several Axes, the individual plot areas within it); grading this module's exhibits by inspecting the Axes object your function returns (rather than comparing rendered pixels) is exactly what makes "did you sort the bars," "did you label the axis," and "did you avoid a second Axes" into machine-checkable facts instead of a human's subjective read of an image.

Common mistakes & how experts think differently

"describe() is a black box; I don't need to know what's inside it." Every number a summary statistic reports rests on a convention someone chose: sample vs. population denominator, which quantile interpolation, which skewness formula, and two tools that disagree on a number are usually just using two different conventions, not disagreeing about the data. The expert can state, for any summary number they report, exactly which convention produced it, and recomputes by hand on a tiny example whenever a result looks surprising.
Forgetting ddof=1 on a sample standard deviation. NumPy's np.std defaults to ddof=0 (population), which understates a sample's true spread, a small, silent bias that compounds badly across many downstream calculations (a Sharpe ratio, a confidence interval) built on top of an understated standard deviation. The expert treats "which ddof?" as a question to answer explicitly every time, never a default to trust blindly.
"I'll just use bins=10 (or whatever the default is)." A histogram's apparent shape is a joint property of the data and the bin width, and an arbitrary bin count can hide real structure or invent fake structure with equal ease. The expert derives the bin count from the data itself (Freedman–Diaconis by default) and treats a bare, unexamined default bin count as exactly as suspicious as an unexamined default anywhere else in a pipeline.
Treating every flagged outlier as an automatic deletion. Tukey's fences (or a z-score) name a candidate for a human decision, possibly a data error worth fixing, possibly the single most informative row in the dataset. The expert investigates before acting, and never silently drops a flagged point without a stated, defensible reason recorded somewhere a reviewer can see.
Using a z-score to hunt for outliers in a distribution you haven't checked for skew. A z-score's own denominator (the standard deviation) is inflated by the very extreme values it is trying to flag, and a heavily skewed distribution can hide a genuine outlier's z-score under the conventional |z| > 2 or |z| > 3 threshold entirely. The expert checks the skew first (Core content, above) and defaults to the quartile-based IQR fence (far less sensitive to this self-masking effect) whenever the data is not close to symmetric.
"A correlation of 0.42 means these two things are really connected." A Pearson r computed here is a descriptive fact about the one sample in front of you; whether it is statistically distinguishable from zero in the wider population, and whether it is economically large enough to matter, are two entirely separate, further questions (QM1.02's territory for the first; plain economic judgment for the second). The expert states the descriptive number and refuses to let it silently become a claim about significance or about causation, which a bare r never establishes either way.
Reaching for a second y-axis whenever two series are on different scales. A dual axis lets its author pick each axis's own zero point and scale independently, which means the SAME chart can be read to support opposite conclusions about "which grew more" depending only on where each axis happened to start. The expert's reflex is rebasing to a common index on one shared axis, not a second axis, and treats ax.twinx() as a request that should almost always be answered with "rebase instead," not granted.
Believing "the whiskers reach 1.5 times the IQR." They reach only the most extreme actual data point still inside that fence, a subtly different, and stricter, claim that matters the moment your data doesn't happen to have an observation exactly at the fence. The expert who has built boxplot_stats by hand once, as this module does, never makes this mistake again, because the distinction is baked into the code rather than remembered as trivia.

The thread through all eight: the expert never accepts a statistic, a bin count, an outlier flag, or a chart's form as a tool's silent default, every one of them is a choice with a stated convention behind it, and EDA is the discipline of making that choice deliberately, in code, before a single picture or a single model gets built on top of it.

Worked examples

Worked example 1: Summary statistics and a right-skewed return series (India, ₹)

A synthetic India-listed company, ticker NILAM (illustrative/synthetic, from this program's bundled NSE-style end-of-day file), closed at ₹1,129.08 on the first session of a 500-day window and ₹802.05 on the last. Before drawing anything, describe the daily-return series numerically.

Solution. Step 1, compute the daily simple returns. For each pair of consecutive closes, r_t = close_t / close_{t-1} − 1, giving 499 daily returns.

Step 2, the five-number summary and moments (computed directly, in percent for readability). Mean daily return −0.053%, sample standard deviation 1.756% (ddof=1), minimum −4.72%, Q1 −1.27%, median −0.166%, Q3 1.10%, maximum 5.14%.

Step 3, read the skew. The moment-based skew computes to +0.192: modestly positive. Notice what that sign, on its own, already predicts before any chart is drawn: the right tail (the biggest up-days) stretches further from the center than the left tail (the biggest down-days) does, which is visible directly in the raw numbers too, the maximum (+5.14%) sits further from the median than the minimum (−4.72%) does. A histogram of this series, whenever you build one, should show a longer tail reaching right than left; if it didn't, that would be the signal to recheck the arithmetic, not the chart.

Read it. Every one of these seven numbers is checkable by hand on a small slice of the same data, and none of them required a single line of plotting code, that is the entire point of this module's opening section: describe first, in numbers you can defend, before a picture ever enters the room. This is Exercise DA1.04-e01; its hidden tests recompute every one of these numbers independently (via NumPy) on this exact NILAM series and on a separate US series, so a solution that merely hardcodes this worked example's five values fails immediately on both.

Worked example 2: Freedman–Diaconis versus a naive ten bins (US, $)

A synthetic US-listed company, ticker CRSTNE (illustrative/synthetic), has the same 499-return series structure as Worked Example 1. An analyst reaches for plt.hist(returns, bins=10) out of habit. Check that habit before trusting the picture it produces.

Solution. Step 1, the quartiles and IQR. Q1 = −1.215%, Q3 = +1.202%, so IQR ≈ 2.417 percentage points. The full range of the series is 9.39 percentage points (from its minimum to its maximum).

Step 2: Freedman–Diaconis bin width and count. bin width = 2 × 2.417 / 499^(1/3) ≈ 0.610 percentage points. Dividing the full range by that width and rounding up: 16 bins.

Step 3: compare against the naive default. Ten bins, applied to the same 9.39-point range, forces a bin width of 0.939 points, roughly 54% wider than FD's own answer. A wider bin blurs together return days that FD's narrower bins would keep visually distinct, quietly smoothing out real structure in the tails (exactly the days Worked Example 1's skew computation says are stretching furthest from the center) in favor of a rounder, more "default-looking" histogram shape.

Read it. Nothing about the data changed between the two histograms, only the bin width did, and the bin width was never a property of the data until this module's rule derived it from the data's own spread and size. This is Exercise DA1.04-e02; its hidden tests recompute the Freedman–Diaconis bin count independently (via numpy.histogram_bin_edges(..., bins='fd')) on this CRSTNE series and on a separate, differently-shaped India series, so a hardcoded 16 fails the moment the hidden data's spread differs from this worked example's.

Worked example 3: Flagging a genuine outlier before you rank anything (India, ₹)

Fifteen synthetic India-listed companies (illustrative/synthetic fundamentals) report FY2024 net income, ranging from ₹41.77 crore up to a startling ₹503.82 crore for one company, ticker MYNAH, more than double the next-highest figure. Before building any ranking chart, decide whether MYNAH belongs in it as reported.

Solution. Step 1, compute the fences. Across all fifteen companies, Q1 = ₹121.18 crore, Q3 = ₹183.20 crore, IQR ≈ 62.02 crore. The upper fence: Q3 + 1.5 × IQR = 183.20 + 93.03 = ₹276.22 crore.

Step 2: apply the rule. MYNAH's ₹503.82 crore sits well above the ₹276.22 crore fence: flagged. Every other company (the next-highest, ₹243.46 crore, sits comfortably inside it) is not.

Step 3: decide, don't delete. A flag is a question, not a verdict. Investigate before acting: is ₹503.82 crore a currency-scale or decimal-placement error (perhaps a figure meant to be reported in a different unit), or is it a company that genuinely had an exceptional year, a one-off asset sale, a legal settlement, an unusually strong operating year? This module cannot answer that from the number alone; what it can do is make certain the question gets asked rather than skipped, by turning "does this look weird" into a stated, computable rule that flags MYNAH the same way every time, on every re-run, regardless of who is looking.

Read it. Notice how directly this feeds the very next section's exhibit: if MYNAH's figure survives investigation as genuine, DA1.04-e04's sorted bar chart will show it as an honestly enormous outlier, sitting far above its next-nearest peer, which is exactly the shape a correct, honest ranking chart of this company should have. This is Exercise DA1.04-e03; its hidden tests independently reconstruct this same fence calculation on the real bundled India fundamentals table and on a parallel real US table (where a different single company, not MYNAH, is the one that clears the fence), so a solution that hardcodes "MYNAH" or any single company name fails the moment the hidden market differs.

Worked example 4: A correctly sorted, correctly labeled ranking exhibit (US, $)

Fifteen synthetic US-listed companies report FY2025 revenue. The largest, ticker NRTHBY, reports $1,067.04 million, nearly double the second-largest, CRSTNE at $527.87 million, down to the smallest, IRNVLE at $80.48 million. Build the ranking chart this comparison calls for.

Solution. Step 1, name the comparison, in one sentence, before touching matplotlib. "Which of these fifteen companies earns the most revenue" is a ranking question, per the grammar above, that names a bar chart, sorted by value, full stop; there was never a decision to make about a pie chart or a 3-D chart, because neither answers a ranking question.

Step 2: sort before you plot. sorted(zip(tickers, revenues), key=lambda p: p[1], reverse=True): NRTHBY first, IRNVLE last, every company in between in strictly descending order. Plotting in the table's original (arbitrary) row order instead would force a reader to scan every bar and do the ranking by eye, exactly the work this chart exists to do for them.

Step 3, label with units, and title the actual message. Y-axis: "Revenue ($ million)", states both the quantity and its scale, so nobody has to guess whether they are reading millions or thousands. Title: "Fiscal-year revenue by company, largest to smallest", states the comparison being made, not a placeholder like "Chart 1" or "Revenue Data".

Read it. NRTHBY's bar, sorted correctly to the far left and nearly twice its nearest competitor's height, is the single most important visual fact in this exhibit, and it would have been just as true, and much harder to see at a glance, in an unsorted chart. This is Exercise DA1.04-e04; its hidden tests check the actual Axes object your function returns, bar heights in strictly sorted order, matching the real bundled US and India fundamentals tables exactly, plus a title and y-axis label that both name "revenue", not merely that some chart got drawn.

Worked example 5: The dual-axis trap, and the rebasing fix (India, ₹)

Two synthetic India-listed tickers, ARUNIMA (closing at ₹503.10 at the start of a 500-day window, ₹587.30 at the end) and SAGARIKA (₹616.04 to ₹726.15 over the same window), sit at very different price levels. An analyst wants one chart answering "which one actually grew more."

Solution. Step 1: reject the tempting dual axis. Plotting both raw price series on one chart with a second y-axis for SAGARIKA (ax.twinx()) looks like it solves the "different scales" problem, until you notice that whoever draws that chart gets to choose each axis's own starting point and range independently, and a slightly different choice on either axis can visually favor either stock, regardless of which one actually grew more in percentage terms. That is not a hypothetical risk; it is the entire mechanism of the trap.

Step 2: rebase both series to 100. rebased = [p / prices[0] × 100 for p in prices]. ARUNIMA: 587.30 / 503.10 × 100 ≈ 116.74. SAGARIKA: 726.15 / 616.04 × 100 ≈ 117.87. Both series now start at the identical value (100), on the identical single y-axis, and whatever gap remains between their endpoints is a genuine, undistorted statement about relative growth.

Step 3, read the honest answer. SAGARIKA's rebased index (117.87) edges out ARUNIMA's (116.74), the two stocks grew almost the same amount in percentage terms over this window, with SAGARIKA very slightly ahead, a conclusion that would have been easy to distort with a badly chosen dual axis and impossible to distort with a shared, rebased one.

Read it. Rebasing did not add information the raw prices didn't already contain, it removed the authorial discretion a dual axis would have handed the chart's builder, which is exactly why "rebase to a common index" is this module's answer to "these two series are on different scales," never a second axis. This pairs Exercise DA1.04-e05 (the rebasing computation) with DA1.04-e06 (the actual two-line chart, graded on len(ax.figure.axes) == 1, a concrete, checkable proxy for "no second axis got created", plus each line's rebased start value); both are checked against real bundled India and US ticker pairs different from the ones narrated here.

Worked example 6: Comparing two macro distributions with a box plot (India ₹ & US $, together)

Fifteen years of monthly year-on-year CPI inflation are available for both the US (via a bundled synthetic FRED-style series) and India (via a bundled synthetic RBI-style series), 180 observations each. Compare their distributions in one exhibit.

Solution. Step 1: name the comparison. "How does the spread and level of inflation compare between these two economies" is a distribution comparison across two groups, the grammar above names a box plot (or, for a single distribution alone, a histogram; here, with exactly two groups to compare side by side, a box plot is the more compact, more directly comparable form).

Step 2, compute each economy's five numbers plus whiskers. US CPI: Q1 = 0.45%, median = 1.48%, Q3 = 2.04%, whislo = −1.00% (the coldest month in the window, itself the series minimum, sits inside the fence, no fliers below), whishi = 2.94% (also the series maximum, also inside the fence). India CPI: Q1 = 2.83%, median = 3.41%, Q3 = 4.44%, whislo = 1.63%, whishi = 6.64%. Neither series produces any flagged fliers at k = 1.5, both distributions, whatever else differs between them, stay within their own fences across the whole window.

Step 3, read the comparison a box plot makes legible in one glance. India's entire box (Q1 to Q3, 2.83% to 4.44%) sits well above the US's (0.45% to 2.04%), a level difference visible immediately from where the two boxes sit relative to each other, without needing to read a single number off either axis. That is precisely what a box plot is for: five (or seven, counting the whiskers) numbers per group, compared side by side, in far less space than two full histograms would need.

Read it. Notice what did NOT require re-deriving anything from QM1.01: this module computed the level and spread of two real distributions and rendered the honest comparison; whether either distribution is "normal enough" for some further test, or whether the level difference is statistically significant across a longer history, is QM1.01/QM1.02 territory, not asked here. This is Exercise DA1.04-e08; its hidden tests check boxplot_stats against matplotlib.cbook.boxplot_stats, the function ax.boxplot() calls internally, on this exact US and India CPI series, so a solution that approximates the whisker convention instead of implementing it exactly will disagree with the real bundled data on at least one of the two markets.

Worked example 7: A correlation heatmap on a four-factor panel (neutral, synthetic)

A synthetic cross-sectional panel carries four style factors (value, momentum, quality, and low-volatility) plus a forward return column, across sixty fictional tickers and forty-eight months (illustrative/synthetic; not a claim about any real factor's real-world efficacy). Build the correlation matrix a heatmap of this panel would need.

Solution. Step 1, compute the full 5×5 matrix. Every factor correlates with itself at exactly 1.0 (the diagonal), and the matrix is symmetric by construction. Off-diagonal, this specific panel shows: value–momentum ≈ 0.006, value–quality ≈ 0.015, value–low-vol ≈ 0.026, momentum–quality ≈ 0.014, momentum–low-vol ≈ −0.001, quality–low-vol ≈ 0.025, every pair of factors here sits close to zero, meaning (in this synthetic panel) the four style factors were constructed to be close to mutually independent, not double-counting the same underlying signal.

Step 2, check each factor against the forward return. value–forward-return ≈ 0.099, momentum–forward-return ≈ 0.122, quality–forward-return ≈ 0.085, low-vol–forward-return ≈ −0.009. Three of the four factors show a small positive co-movement with next-period return in this seeded sample; the fourth is close enough to zero that this sample alone says nothing meaningful about it either way.

Step 3, say precisely what this does and does not establish. This heatmap is a descriptive fact about this one seeded panel: full stop. It does not, on its own, establish that any of these four numbers would survive a formal significance test, that they would replicate on a different sample, or that a trading strategy built on them would earn back its own costs; QD1.02 (Signal & Factor Construction and Evaluation) is where that harder evaluation (information coefficient, decay, turnover, cross-sectional neutralization) actually happens, using this exact same correlation-matrix machinery as its starting point.

Read it. A correlation heatmap built honestly answers "how do these things move together in the sample I have," and nothing more; the discipline is stating that boundary out loud every time, not treating a heatmap's color as a verdict. This is Exercise DA1.04-e07; its hidden tests check the full matrix against pandas.DataFrame.corr() on the real bundled factor panel, entry by entry, at a tight numerical tolerance.

Worked example 8: A rolling trend and volatility band on real noise (US, $)

A synthetic US-listed company, ticker NRTHBY (the same company from Worked Example 4's revenue ranking), closed at $310.45 on day one of a 500-day window and climbed, with substantial day-to-day noise, to $923.75 at its peak. Plotting all 500 raw daily closes produces a jagged, spiky line. Smooth it.

Solution. Step 1, choose a window and compute the rolling statistics. A 20-day trailing window: the first computable point covers days 1–20 (mean $296.30, standard deviation $9.22 across that first window); the window then slides forward one day at a time, dropping the oldest observation and adding the newest, ending, after 481 total rolling points from 500 raw closes, at a much higher level (mean $854.41, standard deviation $40.17 across the final 20-day window) with visibly wider swings than the series started with.

Step 2, read what the smoothing reveals that the raw noise obscured. The rolling mean traces NRTHBY's actual climb from roughly $300 to roughly $850–900 as a comparatively smooth line; the rolling standard deviation, plotted as a shaded band around that mean, shows the stock's day-to-day volatility itself rising over the window (from about $9 in the first computable window to about $40 near the end), a second, genuinely useful fact (the stock did not just rise, it got choppier while rising) that a bare raw-price line would have buried in noise rather than revealed as a trend in its own right.

Step 3, name the trade-off this window length makes. A 20-day window is short enough to still track a multi-week shift in this company's trajectory without lagging it by a full quarter, and long enough to smooth out ordinary single-day noise; a much shorter window (5 days) would barely smooth anything, and a much longer one (100+ days) would blur a genuine multi-month inflection point into invisibility.

Read it. Nothing about NRTHBY's underlying prices changed between the raw chart and the smoothed one, only which fact the exhibit makes legible did. This is Exercise DA1.04-e09; its hidden tests check both the rolling mean and the rolling standard deviation against pandas.Series.rolling(20) on this exact NRTHBY series and on a separate real India series, at a tight numerical tolerance across all 481 (or the India series' equivalent count of) rolling points, not merely the first or last one.

Practice set

Work each problem fully (compute by hand or trace the logic on paper) before reading its solution. Percentages and currency figures are illustrative/synthetic unless stated otherwise.

P1 (guided). Sample vs. population standard deviation. For the tiny dataset [2.0, 4.0, 4.0, 4.0, 5.0, 5.0, 7.0, 9.0], compute both the sample standard deviation (ddof=1) and the population standard deviation (ddof=0). Which is larger, and why is that true in general?

Solution. Mean = 5.0. Sum of squared deviations = 9+1+1+1+0+0+4+16 = 32. Sample variance = 32/7 ≈ 4.571, sample std ≈ 2.138. Population variance = 32/8 = 4.0, population std = 2.0. The sample version is always larger (or equal, only when n is infinite) because it divides by the smaller number (n − 1 < n), which is exactly the correction that makes the sample statistic an unbiased estimate of the population's true spread.

P2 (guided). Linear-interpolation quantiles by hand. For [10, 20, 30, 40] (already sorted, n = 4), compute Q1 using linear interpolation between ranks (position = 0.25 × (n − 1)).

Solution. Position = 0.25 × 3 = 0.75. That sits 75% of the way from index 0 (10) to index 1 (20): 10 + 0.75 × (20 − 10) = 17.5. Q1 = 17.5.

P3 (guided). Reading a skew's sign. A distribution has mean = 8.2, median = 6.5. Without computing the exact skew value, state whether the distribution is left- or right-skewed, and why.

Solution. Right-skewed. When the mean sits above the median, a long right tail (a handful of unusually large values) is pulling the mean upward while the median (which only cares about the middle-ranked value, not the extreme ones) stays comparatively anchored. Mean below median is the mirror-image signal for left skew.

P4. Freedman–Diaconis by hand. For a sample of n = 64 values with Q1 = 10, Q3 = 18, minimum = 2, maximum = 50, compute the Freedman–Diaconis bin width and the resulting bin count.

Solution. IQR = 8. bin width = 2 × 8 / 64^(1/3) = 16 / 4 = 4.0. range = 50 − 2 = 48. n_bins = ceil(48 / 4.0) = 12.

P5. When Freedman–Diaconis breaks. A dataset has 30 identical values (IQR = 0). What does the FD formula compute for bin width, why is that a problem, and what does this module's fallback rule give instead?

Solution. bin width = 2 × 0 / 30^(1/3) = 0, division by (or multiplication resulting in) a zero bin width, which would demand infinitely many bins, a nonsensical result. The Sturges fallback instead gives n_bins = ceil(log2(30) + 1) = ceil(4.907 + 1) = ceil(5.907) = 6.

P6. Tukey's fences by hand. For Q1 = 40, Q3 = 60 (so IQR = 20), compute the lower and upper fences at k = 1.5, and state whether a value of 95 is flagged.

Solution. Lower fence = 40 − 1.5×20 = 10. Upper fence = 60 + 1.5×20 = 90. 95 > 90, so yes, it is flagged as an outlier.

P7. Why IQR beats a bare z-score here. A dataset of 20 ordinary values plus one enormous outlier has a mean and standard deviation both strongly pulled toward the outlier. Explain, in one or two sentences, why the outlier's own z-score can end up unremarkable even though it is visibly extreme, and why its Tukey-fence status is unaffected by this.

Solution. The standard deviation used in the z-score's denominator is itself inflated by the very outlier being tested, so z = (x − mean)/std can shrink toward an unremarkable value even for a genuinely extreme point, the outlier partially masks itself. Q1 and Q3 barely move under the influence of one extreme point (they are determined by the bulk of ordinary values, not the tails), so the IQR-based fence keeps flagging the same point regardless of how extreme it is.

P8. Correlation matrix properties. A correlation matrix computed from real data has an entry corr['a']['b'] = 0.73 but the code has a bug and returns corr['b']['a'] = 0.68. Name the property this violates and what it tells you about the bug.

Solution. It violates symmetry: a Pearson correlation matrix must always satisfy corr[a][b] == corr[b][a], since the correlation between a and b is definitionally the same quantity as the correlation between b and a. Two different values for the same pair means the implementation is computing something order-dependent (a bug), not a genuine property of correlation.

P9. Descriptive versus inferential, in one sentence. A colleague says "the correlation between these two stocks is 0.42, so the relationship is real." Name precisely what has been conflated.

Solution. A descriptive Pearson r = 0.42, computed on one sample, has been conflated with a claim of statistical significance: whether that 0.42 is distinguishable from zero in the wider population the sample was drawn from is a separate question with its own test (QM1.02's H0: ρ = 0), not something the bare correlation coefficient establishes on its own.

P10 (India, ₹). Naming the chart. You want to show which of twelve India-listed companies had the highest year-on-year revenue growth. Name the chart type this comparison calls for, and the one line of code (in words) that must happen before plotting.

Solution. A bar chart, because the comparison is a ranking. Before plotting: sort the twelve companies by growth value, descending, plotting in the data's original (arbitrary) order would force a reader to do the ranking themselves.

P11 (US, $). Spotting the dual-axis trap. A colleague proposes plotting a $40 stock and a $400 stock on one chart with a second y-axis so "both lines are visible." What is wrong with this plan, and what should replace it?

Solution. A dual axis lets its author choose each axis's own scale and starting point independently, so the same chart can be made to visually favor either stock regardless of which one actually grew more, an unfalsifiable, authorial choice masquerading as an objective picture. The fix: rebase both series to 100 at their own first observation and plot both on one shared axis.

P12. Rolling window arithmetic. Given 100 daily observations and a rolling window of 25, how many rolling (mean, std) pairs are produced, using this module's trailing-window convention (no leading padding)?

Solution. 100 − 25 + 1 = 76 rolling pairs, the first full window ends at index 24 (the 25th observation), and one more pair is produced for every subsequent day through the 100th.

P13. The whisker convention, precisely. For a dataset with Q1 = 20, Q3 = 30 (IQR = 10, upper fence = 45), the actual data contains the values ..., 38, 41, 47, 52. State whishi and the fliers, applying this module's exact convention.

Solution. 41 is the largest value still ≤ 45 (the fence), so whishi = 41, not 45 itself, and not 47, which is beyond the fence. Fliers: [47, 52], every value strictly greater than the fence, drawn individually rather than folded into the whisker.

P14. Reading two box plots side by side. Group A's box spans Q1 = 5 to Q3 = 9; Group B's box spans Q1 = 14 to Q3 = 22, with no overlap between the two boxes at all. What can you say about the two groups' distributions from this alone, and what can you NOT yet say?

Solution. You can say Group B's typical (middle-50%) values sit substantially higher than Group A's, with no overlap in that middle range, a clear level difference visible at a glance. You cannot yet say this difference is statistically significant, or that it would hold up on a different sample, that inferential question belongs to a formal two-sample test (QM1.02's territory), not to reading two box plots.

P15. Skew and the mean-vs-median gap, applied. A dataset of company revenues has mean = ₹340 crore and median = ₹185 crore. Predict the sign of the skew, and explain what kind of companies are likely driving the gap.

Solution. Positive (right) skew: the mean sitting far above the median signals a small number of very large companies pulling the average upward while most companies cluster at a lower, more typical revenue level the median reflects. This is the ordinary shape of a cross-sectional revenue column across companies of very different sizes, not an anomaly requiring investigation on its own.

P16. Choosing between a histogram and a box plot. You need to compare the distribution of monthly returns across eight different funds at once. Would a single histogram or a set of box plots better serve this comparison, and why?

Solution. Box plots: eight overlapping histograms on one axis (or eight separate histograms) are hard to compare directly at a glance, while eight box plots side by side compress each fund's five-number summary into a compact, directly comparable glyph, which is exactly the "distribution across several groups at once" case this module's grammar assigns to a box plot rather than a histogram.

P17. FD bin count sensitivity to sample size. Two samples share the same Q1, Q3, minimum, and maximum, but one has n = 100 and the other n = 800 (eight times as many observations). Compute the ratio of their Freedman–Diaconis bin widths, and state which sample gets more bins.

Solution. Bin width is proportional to 1/n^(1/3). 800^(1/3) / 100^(1/3) = (800/100)^(1/3) = 8^(1/3) = 2. So the larger sample's bin width is half the smaller sample's, meaning it gets roughly twice as many bins, a larger sample earns finer resolution, but only at the comparatively slow cube-root rate FD deliberately builds in, not linearly with n.

P18. The three EDA checks, in order. Name the three systematic checks this module opens with, and explain in one sentence why running them in a fixed order (rather than jumping straight to a chart) matters.

Solution. Distribution (five-number summary, moments, a well-binned histogram), outliers (Tukey's fences), and correlation (the descriptive matrix). Running them before any chart or model matters because each check can reveal a problem, a data error masquerading as an extreme value, a lopsided distribution a naive summary would misrepresent, a redundant pair of near-duplicate columns, that would otherwise silently distort every picture or model built on top of the unexamined data.

Applied mini-project

A full EDA pass on one bundled dataset, in both markets.

This project is the module's synthesis: run the complete three-step drill (describe, interrogate, exhibit) on a real bundled US dataset and its India counterpart, producing three publication-quality matplotlib exhibits that pass this module's own dataviz-grammar checklist.

The problem. Pick one paired US/India dataset from this program's bundle (e.g., us-eod/nse-eod, or sec-fundamentals/india-fundamentals). For both markets:

Describe. Compute the five-number summary, mean, sample standard deviation, and skew for one numeric column of your choice (e.g., daily returns, or a fundamentals metric). State the skew's sign and what it predicts about the shape of a histogram of the same data.
Choose bins. Compute the Freedman–Diaconis bin count for that same column, and compare it in one sentence against what a naive bins=10 would have given.
Flag outliers. Compute the Tukey-fence mask (k = 1.5) across an appropriate cross-section (e.g., across all tickers/companies for one period), name which entity (if any) is flagged, and state (with reasoning, not a coin flip) whether you would investigate it further or accept it as genuine.
Correlate. Build a correlation matrix across at least three numeric columns available in your chosen dataset, and state, in one sentence, what it does and does not establish (the QM1.02 boundary from Core content).
Exhibit, three times, each against a stated checklist. Build: (a) a sorted, labeled ranking bar chart (sort descending, unit-bearing y-axis label, a title stating the comparison); (b) a rebased, single-axis line-chart comparison of the US and India versions of your chosen series (no twinx, both series starting at 100); (c) a box-plot comparison of the US and India distributions of your chosen column, side by side.

Deliverable and self-check. One short write-up (or notebook) containing: both markets' five-number summaries and skews with your one-sentence shape prediction; both bin counts with the naive-bins=10 comparison; the outlier flag and your investigate-or-accept reasoning; the correlation matrix and its one-sentence descriptive-not-inferential statement; and all three exhibits, each with its title and axis labels visible in the output. Score yourself 2 points each for: (a) both markets' summary statistics are computed correctly and the skew prediction is stated before any histogram is drawn, not fitted to it afterward; (b) the FD bin count is computed correctly and honestly compared against a naive default; (c) the outlier decision is reasoned, not asserted; (d) the correlation matrix is symmetric with a unit diagonal and its descriptive boundary is stated explicitly; (e) all three exhibits pass this module's own checklist (sorted bars, no dual axis, unit-bearing labels, a title stating the comparison). Pass at 8/10, with no zero on any item.

Code Lab: open the DA1.04 code pack in the Practice tab: nine exercises, DA1.04-e01 through DA1.04-e09, covering the five-number summary and skew, Freedman–Diaconis bin choice, Tukey-fence outlier flagging, a sorted and labeled ranking bar chart, index rebasing, a dual-axis-free comparison chart, a correlation matrix, box-plot statistics, and a rolling mean/volatility band, and clear every one of its hidden tests, not merely the single visible example shown for each.

Reading & resources

Everything below is an anchor to build intuition from and check your own reasoning against, never a substitute for computing a number yourself. Free unless marked; level tags run from intuition to full rigor.

Building the intuition (read these first):

Wes McKinney, Python for Data Analysis (the pandas creator's own book), the standard reference for exactly the load-, filter-, and describe-style EDA workflow this module assumes. [$ Paid] [Beginner–Intermediate]
Claus O. Wilke, Fundamentals of Data Visualization, a free, full-text-online treatment of "match the chart to the data type," the closest single-source anchor for this module's chart-choice grammar (shared territory with G4/PR2, read there for the fuller design narrative). [Free online / $ Paid print] [Beginner–Advanced]
The Matplotlib documentation's own "Quick start guide" and "Anatomy of a Figure", the authoritative, current reference for the object-oriented Figure/Axes API this module standardizes on. [Free] [Beginner]

Going deeper (for when you want the full rigor or the original sources):

Freedman, D., & Diaconis, P. (1981), "On the histogram as a density estimator: L2 theory", the original derivation of the bin-width rule this module uses by default. [Free/academic access varies] [Advanced]
John W. Tukey, Exploratory Data Analysis (1977), the origin of the box-and-whisker plot and the 1.5×IQR fence convention this module implements exactly. [$ Paid, widely available secondhand] [Intermediate–Advanced]
Edward Tufte, The Visual Display of Quantitative Information, the source of "chartjunk" and the data-ink ratio this module's decluttering rules apply in code (shared canon with G4/PR2, do not re-read as new material if you've already covered it there). [$ Paid] [Advanced]

Official, perishable references:

The NumPy documentation for numpy.quantile / numpy.percentile, confirm the default interpolation method (linear, in every version this module was written against) for your installed version before trusting a quantile that disagrees with a colleague's. [Free] [Beginner] (as of July 2026)
The Matplotlib documentation for matplotlib.cbook.boxplot_stats, the exact function ax.boxplot() calls internally, and the authoritative reference for the whisker convention DA1.04-e08 implements by hand. [Free] [Intermediate]

Do this, not just read: load one bundled dataset yourself, call .describe() on a numeric column, and then compute the same five numbers by hand from np.quantile calls until they match exactly. Then plot the same column twice (once with bins=10, once with your own Freedman–Diaconis bin count) side by side, and look at the two pictures for as long as it takes to genuinely see the difference. Ten minutes of watching a bin-count decision change what a histogram appears to claim will install this module's central habit faster than any amount of additional reading.

Teach it back & journal

Feynman prompt. Explain to a colleague who already knows basic pandas but has never heard the phrase "dataviz grammar" why the bar chart they just built, sorted alphabetically, no axis label, a dual y-axis comparing a stock priced in the hundreds against one priced in the thousands, is not "basically fine, it shows the data." Use this module's central idea: a chart's form must follow the comparison it exists to make, and walk through, out loud, what comparison their chart is actually claiming to make (alphabetical order? which axis's scale is "correct"?) versus what comparison they actually wanted to show. Then explain, using Worked Example 3's flagged company, why "this number looks weird, I'll just delete it" is not the same discipline as "this number clears a stated, computable fence, so I'm going to go find out why before I decide." If your colleague can, unprompted, look at a new chart and name the exact chartjunk it contains and the one-line fix for each (sort the bars, rebase instead of a dual axis, label the axis with a unit) you have installed the mechanism this module exists to teach.

Journal prompt. For one real dataset you work with this week (at work, in another course, anywhere), run this module's three-step drill before you build a single chart: compute the five-number summary and skew for one numeric column, flag outliers with Tukey's fences, and build a correlation matrix if you have two or more numeric columns. Write down, in one sentence each: what the skew's sign predicted about the shape you'd see, whether anything got flagged and what you decided to do about it (investigate, keep, or exclude, and why), and one pair of columns whose correlation surprised you either by being higher or lower than you expected. Then build exactly one chart from that data and check it against this module's grammar: does its form match the comparison you actually wanted to make, or did you reach for a familiar default? Log which one habit, computing the bin count instead of defaulting to ten, sorting the ranking, rebasing instead of a dual axis, or stating the descriptive-vs-inferential boundary out loud, would most change how a reviewer reads your next chart, and commit to making it automatic this week.

This module's flashcards and mastery quiz are wired into the app: see the node's Quiz and Reviews.

EDA & Visualization