Statistics

Statistics is the craft of learning from data while keeping track of how sure you can be. Data is a record of things you looked at. Each thing you looked at, a person, a shop order, a day of weather, is an observation (also called a case or a unit). Each property you wrote down about it, an age, a price, a temperature, is a variable: it varies from one observation to the next.

This little survey of a class has 5 observations (the rows) and 6 variables (the columns). Statisticians write the number of observations as $n$, so here $n=5$. You will meet this class again throughout the module.

Observation

One thing you recorded: a person, an order, a day. A row of the table.

Variable

A property recorded for every observation: age, price. A column.

Value

What one variable is for one observation: Ben’s age is 22.

Data set

All the values together, usually as a table.

The first question to ask of any variable is what kind it is, because the kind decides which charts and which numbers make sense. A categorical variable puts each observation in a group: how someone travels (bus, bike, walk, car), a blood type, a colour. A numerical (or quantitative) variable is a measured or counted amount: a height, a number of siblings. A useful test: does it make sense to add two values, or take their average? The average height of the class means something; the average of “bus” and “bike” does not.

Numerical variables come in two kinds. A discrete variable takes separate values with gaps between them, usually counts: 0, 1, 2 or 3 siblings, never 1.5. A continuous variable can take any value in a range: a height could be 170.5 cm, 170.52 cm or 170.523 cm, limited only by how finely you measure. Rounding a continuous variable to whole centimetres does not make it discrete; the thing itself is still continuous.

An ordinal variable is categorical with a natural order: poor < fair < good < excellent, or school year 1, 2, 3. You can say which is higher, but not by how much: is “good” to “excellent” the same step as “poor” to “fair”? Nobody measured it. A nominal variable has no order at all: bus, bike, walk, car.

Watch out for numbers that are only names. A postcode, a phone number, a football shirt number or a product code is written in digits but is nominal: the average shirt number of a team is a number with no meaning. Data sets often store categories as codes (1 = bus, 2 = bike), and a computer will happily average them if you ask.

The population is every observation you want to know about: all the students in a country, every bottle a factory fills this year. You can rarely look at all of it, so you look at a sample, a part of it, and reason from the sample to the population. A number that describes a population, such as the true average height of all students, is a parameter; it is fixed but usually unknown. A number worked out from a sample is a statistic; it changes from sample to sample.

This course computes each statistic in small C++ programs, as Algebra does, so nothing is a black box. Here is all the C++ it needs, in one program. Run it, then change the waits.

1#include <cmath>2#include <iostream>3#include <vector>4
5double largest(const std::vector<double>& values) {6  double best = values[0];7  for (double v : values) {  // each value in turn8    if (v > best) best = v;9  }10  return best;11}12
13int main() {14  std::vector<double> waits = {3, 4, 5, 5, 8};15  int n = waits.size();16  bool longWait = largest(waits) > 6;17  std::cout << "n = " << n << ", largest = " << largest(waits) << "\n";18  if (longWait) std::cout << "someone waited long\n";19  std::cout << "7 / 2 = " << 7 / 2 << ", 7.0 / 2 = " << 7.0 / 2 << "\n";20  std::cout << "sqrt(2) = " << std::sqrt(2.0) << "\n";21  std::cout << "2 to the 10th = " << std::pow(2, 10) << "\n";22  for (int i = 0; i < n; i++) std::cout << waits[i] - 5 << " ";23  std::cout << "\n";24  return 0;25}

• Every value has a type: int (whole numbers), double (decimals), bool (true or false). std::vector<double> is a list of doubles: v.size() is its length and v[i] the value at index i, counting from 0.
• A function says what it returns, then what it is given: double largest(const std::vector<double>& values). The const …& reads the caller’s list without copying or changing it.
• for (double v : values) visits each value; for (int i = 0; i < n; i++) counts with an index. if (…) decides.
• #include <cmath> brings std::sqrt, std::pow, std::exp, std::log and std::abs. A struct groups named values: a row of a table, or a function’s two answers.
• A function can be handed to another as a std::function<double(double)>, and a lambda, [](double x) { return x * x; }, writes one in place.

Data is easiest to work with in a tidy shape: each row is one observation, each column is one variable, and each cell holds one value. Spreadsheets drift away from this: a column per year, two values in one cell, totals mixed in with the rows. Tidy data is what every tool, and every program in this course, expects.

In code, a tidy table is a list of records: a std::vector of structs, one per observation, with the same fields. That is how the class survey looks in C++.

1struct Student {2  std::string name;3  int age;4  double heightCm;5  std::string travel;  // "bus", "bike", "walk" or "car"6  int siblings;7};8
9std::vector<Student> students = {10    {"Ana", 19, 164.2, "bus", 2},11    {"Ben", 22, 181.0, "bike", 0},12    {"Chloe", 20, 170.5, "walk", 1}};13
14std::vector<double> heightsOf(const std::vector<Student>& rows) {15  std::vector<double> heights;  // one column16  for (const Student& s : rows) heights.push_back(s.heightCm);17  return heights;18}

No measurement is perfect. Random error scatters readings around the true value, some too high and some too low: a shaky hand, a rounding, a noisy sensor. Averaging several readings shrinks it. Systematic error, or bias, pushes every reading the same way: a scale that reads 2 kg heavy, shoes left on for a height. Averaging does nothing for bias. In the words of measurement, low random error is precision and low bias is accuracy.

Before trusting a number, ask how it was collected. In an observational study you record what happens without changing anything. A survey is one kind: you ask people questions. In an experiment you deliberately change something, such as giving half the plants a fertiliser, and compare. Logs are records a system writes as it runs: website visits, card payments, sensor readings. They are huge and cheap, but nobody designed them to answer your question.

Faced with a new data set, look before you calculate. How many observations? Which variables, of which kinds? Are any values missing (an empty cell: in C++, a std::optional<double> holding std::nullopt, nothing, instead of a number) or impossible (a height of 1700 cm, a typing slip)? For a categorical variable, count how many observations fall in each group. Those counts are its frequencies, and they are the first picture of it.

1#include <iostream>2#include <map>3#include <optional>4#include <string>5#include <vector>6
7int main() {8  std::vector<std::string> travel = {"bus", "bike", "walk", "bus",9                                     "car", "bus", "walk"};10  std::vector<std::optional<double>> heights = {11      164.2, 181.0, 170.5, std::nullopt, 158.9, 1755, 172.1};12
13  std::map<std::string, int> counts;14  for (const std::string& mode : travel) counts[mode]++;15  std::cout << "observations: " << travel.size() << "\n";16  for (const auto& [mode, count] : counts)17    std::cout << "  " << mode << ": " << count << "\n";18
19  int missing = 0;20  for (const std::optional<double>& h : heights) {21    if (!h) missing++;22    else if (*h < 50 || *h > 250)23      std::cout << "impossible height: " << *h << "\n";24  }25  std::cout << "missing heights: " << missing << "\n";26}

Two small programs that every analysis starts with: counting the groups of a categorical variable, and finding the missing values of a numerical one.

1#include <map>2#include <string>3#include <vector>4
5std::map<std::string, int> frequencies(6    const std::vector<std::string>& values) {7  std::map<std::string, int> counts;8  // TODO: count each value9  return counts;10}

1#include <optional>2#include <vector>3
4std::vector<double> cleanHeights(5    const std::vector<std::optional<double>>& heights) {6  return {};7}

• Data is observations (rows) of variables (columns); $n$ is the number of observations.
• Categorical variables are nominal (no order) or ordinal (ordered); numerical ones are discrete (counts) or continuous (measurements).
• Digits can be labels: postcodes and shirt numbers are nominal.
• A sample is studied to learn about a population; statistics estimate parameters.
• Tidy data: one observation per row, one variable per column, one value per cell.
• Random error averages away; bias does not.
• How data was collected, survey, observation, experiment or logs, limits what it can show.
• First look: count, find missing and impossible values, before any calculation.

The distribution of a variable says which values it takes and how often. The simplest way to write it down is a frequency table: each value or group, and its frequency, the number of observations with it. Dividing by the total $n$ gives the relative frequency, the share of the data, which lets you compare groups of different sizes. For an ordered variable, the cumulative frequency adds up the counts so far.

A bar chart draws one bar per category, its height the frequency (or relative frequency). The bars have gaps between them, because the categories are separate things. For a nominal variable, sort the bars by height so the eye can rank them; for an ordinal one, keep the natural order.

A continuous variable has too many different values for a bar each. A histogram splits the number line into equal intervals called bins, counts the observations in each, and draws touching bars, because the bins cover the line with no gaps. A value on a boundary must go in exactly one bin; the usual rule is that a bin includes its left edge and not its right, written $[10,20)$.

The same 30 commutes, with bins of 5 and of 20 minutes. Bins that are too narrow give a jagged picture where every bump is noise; bins that are too wide hide the shape, here the long tail of slow commutes. There is no single right width: try a few and pick one that shows the shape without the noise. Rules of thumb exist, such as about $\sqrt{n}$ bins (about 5 or 6 for 30 values), as a starting point.

For a small data set you can draw every value. A dot plot puts one dot per observation above a number line, stacking equal values. A stem-and-leaf plot splits each number into a stem (the leading digits) and a leaf (the last digit), and writes the leaves beside their stem. Turned on its side, it is a histogram that still shows the exact numbers.

1Siblings of 18 students (dot plot)2
3  •4  •  •5  •  •  •6  •  •  •7  •  •  •  •8  •  •  •  •  •9 ─┼──┼──┼──┼──┼─10  0  1  2  3  411
12Exam marks of 18 students (stem | leaf)13
14  5 | 2 815  6 | 1 4 7 716  7 | 0 2 3 5 8 917  8 | 1 3 4 818  9 | 0 5          key: 7 | 3 means 73

Look at a histogram from far away and describe its shape. Symmetric: the two sides are near mirror images, as with adult heights. Skewed right (positively skewed): a long tail to the right, as with incomes, house prices and the commute times, where most values are modest and a few are very large. Skewed left: the long tail is on the left, as with marks on an easy test. A peak is a mode; a bimodal shape has two, which often means two groups are mixed together.

A box plot (box-and-whisker plot) summarises a numerical variable with five numbers. Sort the data. The median is the middle value. The lower quartile $Q_1$ is the middle of the lower half, and the upper quartile $Q_3$ is the middle of the upper half. The box runs from $Q_1$ to $Q_3$, so it holds the middle half of the data, with a line at the median. The whiskers reach out to the smallest and largest values that are not unusually far away; values beyond them are drawn as single points. The next modules define these numbers exactly.

A chart is an argument, and it can be a dishonest one. The most common trick is a truncated axis: two bars of 50 and 52, drawn on an axis that starts at 49, look as if one is three times the other. Bar lengths are only honest when the axis starts at 0. Other tricks: pictures scaled in two directions (doubling a coin’s width and height quadruples its area), uneven bins or time steps, a 3D tilt that makes near slices look bigger, and two y-axes chosen so any two lines cross where you like.

• A distribution is the values a variable takes and how often; a frequency table writes it down.
• Relative frequency is frequency divided by $n$; cumulative frequency adds up counts in order.
• Bar charts are for categories (gaps); histograms are for numbers in bins (touching bars).
• The bin width changes the picture: try a few.
• Dot plots and stem-and-leaf plots keep every value of a small data set.
• Describe shape: symmetric, skewed right or left (named after the tail), unimodal or bimodal.
• A box plot shows minimum, $Q_1$, median, $Q_3$ and maximum; the box holds the middle half.
• Check every chart’s axes: bars must start at 0.

Asked “how long do people wait at this café?”, you want one number that stands for all the waits: a measure of centre, or average. There are three in everyday use, and they answer slightly different questions. The mean shares the total out equally. The median is the middle value. The mode is the most common value. On tidy, symmetric data they agree; on skewed data they can be far apart, and choosing the wrong one tells a misleading story.

Measure of centre

One number that stands for the typical value of a variable.

Mean

The sum of the values divided by how many there are.

Median

The middle value once the data is sorted.

Mode

The value that appears most often.

Call the values $x_1,x_2,\dots ,x_n$. The sample mean, written $\bar{x}$ and read “x bar”, adds them all up and divides by $n$. The symbol $\sum$ (capital sigma) means “add up”: ${\sum }_{i=1}^n{x}_i$ is $x_1+x_2+\cdots +x_n$. The mean of a whole population is written $\mu$ (the Greek letter mu).

1#include <vector>2
3double mean(const std::vector<double>& values) {4  double sum = 0;5  // TODO: add up the values6  return sum;7}

Put equal weights on a ruler at each value. The mean is where the ruler balances. In numbers: the deviations $x_i-\bar{x}$, how far each value is from the mean, always add up to exactly zero. The values above the mean pull as hard as the values below. Drag the pivot and watch the sum of deviations: it is zero only at the mean.

To find the median, sort the values from smallest to largest. With an odd number of values, the median is the one in the middle, at position $\frac{n+1}{2}$. With an even number, there are two middle values, and the median is their mean. Half the data is at or below the median and half at or above it.

The mode is the value that appears most often. It is the only measure of centre for a nominal variable: there is no mean or median of “bus, bike, walk”, but the most common way to travel is a fine summary. A data set can have two modes (bimodal) or more, or none worth naming if every value appears once. For continuous data, where values rarely repeat exactly, the mode means the highest bar of the histogram.

A small firm pays its seven staff 25, 27, 28, 30, 32 and 35 thousand a year, and its owner 200 thousand. The mean salary is $377/7\approx 53.9$ thousand: more than six of the seven people earn. The median is 30 thousand, which describes a typical member of staff well. The mean uses every value’s size, so one huge value pulls it; the median uses only the order, so it barely notices. A measure that an extreme value cannot move much is called resistant (or robust).

In a symmetric distribution the mean and the median are about the same. In a skewed one, the values in the long tail pull the mean towards the tail, while the median stays with the bulk. So for data skewed right, as incomes, house prices and waiting times usually are, the mean is above the median; skewed left, it is below. Comparing the two is a quick test of skew.

Sometimes values should not count equally. A course grade where homework is worth 20%, a midterm 30% and a final 50% gives each mark a weight $w_i$. The weighted mean multiplies each value by its weight, adds up, and divides by the total weight. The plain mean is the special case where every weight is 1.

1#include <vector>2
3double weightedMean(const std::vector<double>& values,4                    const std::vector<double>& weights) {5  // TODO: two running totals6  return 0;7}

A trimmed mean sorts the data, removes the same number of values from each end, and takes the mean of the rest. A 20% trimmed mean of 10 values drops the 2 smallest and the 2 largest. It sits between the mean (trim nothing) and the median (trim everything but the middle): it uses most of the data but resists outliers. Judges’ scores in diving and gymnastics drop the highest and lowest for exactly this reason.

1double trimmedMean(std::vector<double> values, int k) {2  std::sort(values., values.end());3  std::vector<double> kept(values.begin() + ,4                           values.end() - );5  double sum = 0;6  for (double v : kept) sum += v;7  return sum / kept.;8}

• The mean $\bar{x}=\frac{1}{n}\sum x_i$ shares the total equally; deviations from it add up to 0.
• The median is the middle of the sorted data, or the mean of the two middle values.
• The mode is the most common value, and the only centre for nominal data.
• Outliers and skew drag the mean towards the tail; the median resists them.
• Right skew: mean > median. Left skew: mean < median.
• A weighted mean divides $\sum w_i{x}_i$ by $\sum w_i$; never average averages of groups of different sizes.
• A trimmed mean drops the same number of values at each end.
• In code, sort a copy, and halve two whole numbers with / 2.0: (8 + 9) / 2 is 8.

Two bakeries both sell loaves that weigh 500 g on average. At the first, five loaves weigh 498, 499, 500, 501 and 502 g. At the second, 460, 480, 500, 520 and 540 g. Same mean, very different bakeries: at the second you might get a loaf 40 g light. Spread (or variability, or dispersion) measures how far the values stray from each other and from their centre. A centre without a spread is half a summary.

The range is the largest value minus the smallest. It is easy to compute and to explain, but it depends on only two values, the two most extreme ones, so a single outlier or typo can change it completely. It also tends to grow as you collect more data, because a bigger sample is more likely to contain an extreme value.

The quartiles split sorted data into four parts with about the same number of values. The median is the middle quartile, $Q_2$. Here is the simple method: the lower quartile $Q_1$ is the median of the values below the median, and the upper quartile $Q_3$ is the median of the values above it (with an odd $n$, the median itself is left out of both halves). The interquartile range, $\text{IQR}=Q_3-Q_1$, is the spread of the middle half of the data. It ignores the top and bottom quarters, so outliers cannot move it.

An outlier is a value far from the rest. A common rule, due to John Tukey, draws two fences 1.5 IQRs beyond the quartiles, and flags any value outside them. In a box plot, the whiskers stop at the most extreme values inside the fences, and the outliers are drawn as points. Flagged does not mean wrong: an outlier may be a typo, or the most interesting value in the data. Look at it before deciding.

1std::pair<double, double> fences(double q1, double q3) {2  double iqr = ;3  return {q1 -  * iqr,  + 1.5 * iqr};4}

The IQR uses only the quartiles. To use every value, start from the deviations $x_i-\mu$, how far each value is from the mean. Their sum is always 0 (the last module showed why), so their mean is useless. Squaring each deviation makes it positive, and big deviations count much more. The mean of the squared deviations is the variance. For a whole population of $N$ values with mean $\mu$, it is written ${\sigma }^2$ (sigma squared).

Eight values with mean $\mu =40/8=5$. The squared deviations add up to 32, so ${\sigma }^2=32/8=4$. The variance is in squared units (square grams, square minutes), which are hard to picture, so its square root, the standard deviation $\sigma =2$, is what people report: roughly, the typical distance of a value from the mean, in the data’s own units.

Usually your data is a sample, and you want to estimate the population’s spread. You do not know $\mu$, so you measure deviations from $\bar{x}$ instead. But $\bar{x}$ is, by its definition, the point closest to this sample (in squared distance), closer than $\mu$ is. So the squared deviations from $\bar{x}$ are a little too small on average. Dividing by $n-1$ instead of $n$ corrects this exactly. The result is the sample variance $s^2$, and $s$ is the sample standard deviation.

Another way to see it: the $n$ deviations from $\bar{x}$ must add up to 0, so once you know $n-1$ of them, the last is fixed. Only $n-1$ are free to vary: the sample has $n-1$ degrees of freedom left for estimating spread.

For a bell-shaped distribution, the standard deviation is the distance from the centre to where the curve bends from curving down to curving up. Drag σ: the centre stays put, the curve spreads out, and because the total area always stands for 100% of the data, a wider curve must be lower. You will meet this curve, the normal distribution, properly in a later module.

Is 78 in a maths test better than 85 in English? Not if the English test was easy. A z-score (standard score) says how many standard deviations a value is above or below the mean. Positive is above, negative below, 0 is exactly average. Because it has no units, it compares values from different scales: test marks, heights and temperatures alike.

1#include <vector>2
3std::vector<double> zScores(const std::vector<double>& values,4                            double mean, double sd) {5  std::vector<double> z;6  for (double x : values) z.push_back(x);  // TODO7  return z;8}

For data with a bell shape (a normal distribution, coming later), the standard deviation tells you a lot: about 68% of the values are within 1 SD of the mean, about 95% within 2 SDs, and about 99.7% within 3. This is the empirical rule. A z-score beyond ±3 is then very rare, which makes it a natural flag for an outlier in bell-shaped data.

The empirical rule needs a bell shape. Chebyshev’s inequality needs nothing: for any distribution, at least $1-\frac{1}{k^2}$ of the values lie within $k$ standard deviations of the mean, for any $k>1$. So at least 75% are within 2 SDs and at least 88.9% within 3, whatever the data. The guarantee is weak because it must hold even for the strangest shapes, but it is always true.

• Spread is how far values stray from each other and from the centre.
• Range = max − min: simple, but set by two extreme values.
• Quartiles split the sorted data into quarters; IQR = $Q_3-Q_1$ is the spread of the middle half.
• The fences $Q_1-1.5\text{IQR}$ and $Q_3+1.5\text{IQR}$ flag outliers: look at them, do not just delete them.
• Variance is the mean squared deviation; the standard deviation is its square root, in the data’s units.
• Population: divide by $N$. Sample: divide by $n-1$, because deviations from $\bar{x}$ are a little too small.
• A z-score $\frac{x-\bar{x}}{s}$ counts standard deviations from the mean.
• Bell-shaped: 68–95–99.7. Any shape: at least $1-1/k^2$ within $k$ SDs.

So far each variable was looked at alone. Often the question is how two relate: do students who study longer score higher? A scatter plot draws one point per observation, with one variable across and the other up. If one variable might help explain or predict the other, it goes on the x-axis and is called the explanatory variable; the one it may explain goes on the y-axis and is the response variable.

Describe a scatter plot in three words. Direction: positive if y tends to rise as x rises, negative if it falls. Form: does the cloud follow a straight line (linear), a curve, or no pattern? Strength: how tightly the points hug that form. Also look for outliers, points far from the pattern. The study data is positive, linear and strong. Drag r below to see what the strengths look like.

To put a number on it, draw lines through the means $\bar{x}$ and $\bar{y}$, cutting the plot into four quarters. A point up and to the right of the means has both deviations $x_i-\bar{x}$ and $y_i-\bar{y}$ positive, so their product is positive; down and to the left, both negative, product positive again. In the other two quarters the product is negative. The covariance is the average of these products (dividing by $n-1$ for a sample). Positive: the points sit mostly in the agreeing quarters.