GUANGTUN BEN ZHU, PhD

Frequenstist vs. Bayesian

 

The frequentist approach focuses on the likelihood P(D|M) and intends to find the model that best describes the data, while the Bayesian approach focuses on the posterior distribution P(M|D) and considers all possibilities.

 

Frequentist:  

 

• To derive the most probable model, assume the measurement (D0) is either the expected value or the maximum likelihood value and find the model that gives

  • <D> = ∫D P(D|M) dD = D0, or

  • P_ML(D|M) = D0

 

• To derive the (Gaussian) confidence intervals (e.g.,  ±1σ upper/lower limits), assume the measurement (D0) is ±1σ away from the expected value or the maximum likelihood value and find the 

  • -1σ lower limit M that gives P(D>D0|M) = 16%

  • +1σ upper limit M that gives P(D<D0)|M) = 16%

 

Bayesian:

 

• Given the measurement (D), find the probability (posterior) distribution of the model P(M|D),

  • P(M|D) = P(D|M)P(M)/P(D)

where P(D) is Bayesian evidence (or fully-marginalized likelihood) and P(M) is the prior.

 

A nice tutorial that I highly recommend is how to fit a straight line by David Hogg et al. (2010). Although it addresses one of the simplest problem, it shows how to treat errors and outliers, and also how to use MCMC to sample the parameter space efficiently.

 

In practice, I tend to use the two approaches for different types of problems. I often use the frequentist approach for some simple and easy tasks (for which you know the frequentist's answer would not be far from the truth), and resort to the Bayesian method for problems in which model parameters and priors are of great importance.

 

An example of the latter (the Bayesian approach) is the so-called halo occupation distribution (HOD) modeling of correlation functions in the cold dark matter cosmological model. As an example, below is the joint-likelihood distribution of the host dark matter halo mass of luminous red galaxies (LRGs, a special type of galaxies that are very useful tracers of the overdensities of the Universe) and the baryon abundance (as traced by Mg+) in their circumgalactic medium, taken from Zhu & SDSS-III (2014).