Bayesian Reasoning

This page is incomplete.

We're often interested in questions about the world which we can't observe directly, or simply haven't yet observed directly.

Bayesianism gives us a principled way to reason about plausible inference. That is, instead of straightforward deduction like

  • X implies Y.
  • We observe X.
  • Therefore Y.

what if we can only observe Y and we want to know about X? We need something like

  • X implies Y.
  • We observe Y.
  • Therefore, X becomes more plausible.

or even

  • X makes Y more plausible.
  • We observe Y.
  • Therefore, X becomes more plausible.

Imagine we could make a list of every possible self-consistent way the world could be. When we discover new information about the world, we go through our list and cross out all the possible worlds that would be inconsistent with what we just observed.

We can bundle these possible worlds together according to whether a certain unobservable or not-yet-observed proposition is true or false. The relative size of these bundles are our priors.

Within each of those bundles, we can further divide our possible worlds according to whether or not we would have to cross them out after observing some particular piece of evidence. The relative size of these sub-bundles are our likelihoods.

What is evidence? Anything you can observe that has a different likelihood where the proposition is true, than where it is false.

Anything you can observe that would have a different probability depending on whether the hypothesis is true.

Likelihoods, E|H

Imagine for a moment that you knew your hypothesis were false. What would that imply about the things you can observe? What do you expect to see? What would surprise you? Now imagine that your hypothesis is true. How would that change the observable world? What do you expect to see?

The probability of H and E|H

Due to our limitations as finite beings, often it's impossible to come up with a fully exhaustive list of hypotheses. In that case, we can simply skip the renormalization step. In other words, we can still know that, for example, hypothesis A is 5 times as plausible as hypothesis B, even though there might be a hypothesis C, which we haven't thought of yet, that assigns a much higher likelihood to our evidence than either A or B.