Interpretation
So far, these notes have focused on probability as a mathematical object: we have a probability space obeying a set of axioms, and we can do various things with it. We have not, however, discussed what a probability is. Is it a description of frequencies? Something else?
Under the instrumentalist school of thought, the mathematical definition above is what a probability is. Probability is a measure over a sigma algebra that satisfies Kolmogorov’s axioms, nothing more, nothing less. In this view, all other interpretations of probability are simply applications of probability.
Frequentism defines probability as the long-run behavior of infinite sequences: the probability of an event is the fraction of times it would appear if we repeated the experiment or observation infinitely many independent times. \(\P[H] = 0.5\) because, if we flip a coin infinitely many times, half of the results will be heads.
Subjectivism or subjective Bayesianism defines probability as a consistent description of the beliefs of a rational agent. That is, it describes what the agent currently believes about as-yet-unobserved outcomes. \(\P[H] = 0.5\) because, prior to flipping a coin, the agent believes heads to be just as likely as tails.
There are variants on these theories and other theories as well. They may seem similar, but they are very different in terms of their implications and application. For one thing, under frequentism, probabilities only make sense in the context of repeated (or theoretically repeatable) random events, and we cannot talk about the probability of a fixed but unknown thing, such as a population parameter. Under subjective Bayesianism, because probability is about the agent’s subjective state of belief, we can talk about the probability of a population parameter, because all we are doing is describing the agent’s belief about the parameter’s value.
When I use the term subjective above, I am not using it in a perjorative or negative sense. I am using it in a philosophical sense: it is the agent’s internal state, which it does not necessarily share with other agents. Different rational agents may have different current beliefs, each entirely rational and well-justified, because they have access to different prior information. There is nothing positive or negative about this; it is just a fact.