Posterior Predictive Checks

27 April 2016

We are learning about evaluating goodness-of-fit for probabilistic graphical models using posterior predictive checks in my MIT Bayesian modeling class this week. I collected a few useful references:

1. Gelman/Shalizi philosophical argument against “subjective prior” and in favor of model checking. Blog post version.
2. Blei- Practical notes on how and why to do posterior predictive checks
3. Gelman/Meng/Stern- paper that invented realized discrepancies for predictive checking. A discrepancy is a quantity calculated from the posterior predictive distribution used to evaluate model fit. A realized discrepancy is allowed to depend on latent variables as well as the data.
4. Mimno & Blei- example of applying predictive checks to topic modeling

Perusing these articles, I was surprised to learn that many Bayesians don’t think it is necessary to check the adequacy of a choice of prior distribution, since it is “subjective”. I agree more with the Gelman/Shalizi idea that choosing a prior is just another part of the model and should be checked, just like one would do in the frequentist context by, for example, evaluation on a with-held test set.