K-means needs you to choose k before it runs. Hierarchical clustering does not: it produces the whole nested family of groupings at once and leaves the choice of k to a cut.
How does hierarchical clustering build a dendrogram?
The agglomerative version starts with every observation as its own cluster and repeatedly merges the two closest, recording the distance at which each merge happened. The divisive version starts with one cluster and splits recursively, and it is rarer because the split search is expensive. Plot the merge history with distance on the vertical axis and you have a dendrogram; a horizontal cut at any height reads off a clustering, and the number of vertical lines it crosses is k.
Between two single points, distance is unambiguous. Between two clusters it is a choice, and the choice is called the linkage:
| Linkage | Distance between clusters | Behaviour |
|---|---|---|
| Single | smallest distance between any pair across the two | chains together strung-out groups; merges early |
| Complete | largest distance between any pair across the two | forces compact groups; merges late |
| Average (UPGMA) | mean of all cross-pairs | between the other two; the common default |
| Ward | increase in within-cluster sum of squares | favours equal-sized spherical clusters |
Advertisement
How do you use hierarchical clustering on stock returns?
For return series the natural distance is built from correlation, and d = 1 − ρ is the standard simple choice, running from 0 for perfectly correlated pairs to 2 for perfectly opposed ones. Worked Example 5 clusters ten stocks that way, and it makes two points a dendrogram picture usually hides. Cutting the same average-linkage tree at 0.50 gives six clusters and at 0.66 gives five, because one stock joins the Indian block in between. And cutting at 0.72, one single height, gives five clusters under average linkage and two under single linkage. Same data, same distance matrix, same cut, and the count of clusters differs by a factor of two and a half.
So a clustering result is not a fact about the world until you have named three things: the distance, the linkage and the cut height. State them, and state whether the cut height was chosen before or after you saw the memberships. Two further habits earn their keep. Compare the tree from at least two linkages and report the merges that disagree, since those are your unstable observations. And check stability by refitting on subsamples, because a correlation matrix estimated from 72 monthly observations on 10 assets has 45 free parameters and is noisier than it looks.