- Author:
- Tommi S. Jaakkola,
- Supervisor:
- Michael I. Jordan
Graphical models enhance the representational power of probability models through qualitative characterization of their properties. This also leads to greater efficiency in terms of the computational algorithms that empower such representations. The increasing complexity of these models, however, quickly renders exact probabilistic calculations infeasible. We propose a principled framework for approximating graphical models based on variational methods.
We develop variational techniques from the perspective that unifies and expands their applicability to graphical models. These methods allow the (recursive) computation of upper and lower bounds on the quantities of interest. Such bounds yield considerably more information than mere approximations and provide an inherent error metric for tailoring the approximations individually to the cases considered. These desirable properties, concomitant to the variational methods, are unlikely to arise as a result of other deterministic or stochastic approximations.
The thesis consists of the development of this variational methodology for probabilistic inference, Bayesian estimation, and towards efficient diagnostic reasoning in the domain of internal medicine. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)
Cited By
- Ribeiro F, Leontidis G and Kollias S Introducing routing uncertainty in capsule networks Proceedings of the 34th International Conference on Neural Information Processing Systems, (6490-6502)
- Laufer Y and Gannot S A Bayesian Hierarchical Model for Speech Enhancement 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (46-50)
- López-Cruz P, Bielza C and Larrañaga P (2015). Directional naive Bayes classifiers, Pattern Analysis & Applications, 18:2, (225-246), Online publication date: 1-May-2015.
Tang J, Liu N, Yan J, Shen Y, Guo S, Gao B, Yan S and Zhang M Learning to rank audience for behavioral targeting in display ads Proceedings of the 20th ACM international conference on Information and knowledge management, (605-610)- Oikonomou V, Tripoliti E and Fotiadis D (2010). Bayesian methods for fMRI time-series analysis using a nonstationary model for the noise, IEEE Transactions on Information Technology in Biomedicine, 14:3, (664-674), Online publication date: 1-May-2010.
- Liu Y, Niculescu-Mizil A and Gryc W Topic-link LDA Proceedings of the 26th Annual International Conference on Machine Learning, (665-672)
- Nickisch H and Seeger M Convex variational Bayesian inference for large scale generalized linear models Proceedings of the 26th Annual International Conference on Machine Learning, (761-768)
- Putthividhya D, Attias H and Nagarajan S Independent factor topic models Proceedings of the 26th Annual International Conference on Machine Learning, (833-840)
- Langseth H and Nielsen T (2009). Latent classification models for binary data, Pattern Recognition, 42:11, (2724-2736), Online publication date: 1-Nov-2009.
- Seeger M (2008). Bayesian Inference and Optimal Design for the Sparse Linear Model, The Journal of Machine Learning Research, 9, (759-813), Online publication date: 1-Jun-2008.
- Palmer J, Kreutz-Delgado K and Makeig S Super-Gaussian mixture source model for ICA Proceedings of the 6th international conference on Independent Component Analysis and Blind Signal Separation, (854-861)
- Liu Y, Jin R and Chai J (2006). A statistical framework for query translation disambiguation, ACM Transactions on Asian Language Information Processing (TALIP), 5:4, (360-387), Online publication date: 1-Dec-2006.
- Govaert G and Nadif M (2005). An EM Algorithm for the Block Mixture Model, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27:4, (643-647), Online publication date: 1-Apr-2005.
- Girolami M and Rogers S Hierarchic Bayesian models for kernel learning Proceedings of the 22nd international conference on Machine learning, (241-248)
- Sallans B and Hinton G (2004). Reinforcement Learning with Factored States and Actions, The Journal of Machine Learning Research, 5, (1063-1088), Online publication date: 1-Dec-2004.
- Ueda N and Ghahramani Z (2002). Bayesian model search for mixture models based on optimizing variational bounds, Neural Networks, 15:10, (1223-1241), Online publication date: 1-Dec-2002.
- Wiegerinck W Variational approximations between mean field theory and the junction tree algorithm Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence, (626-633)
- Jordan M, Ghahramani Z, Jaakkola T and Saul L (1999). An Introduction to Variational Methods for Graphical Models, Machine Language, 37:2, (183-233), Online publication date: 1-Nov-1999.
- Murphy K A variational approximation for Bayesian networks with discrete and continuous latent variables Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, (457-466)
- Kearns M and Saul L Large deviation methods for approximate probabilistic inference Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, (311-319)
Recommendations
Variational inference in nonconjugate models
Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the ...
An Introduction to Variational Methods for Graphical Models
This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields). We present a number of examples of graphical models, including the QMR-DT database, ...
Mean-field variational approximate Bayesian inference for latent variable models
The ill-posed nature of missing variable models offers a challenging testing ground for new computational techniques. This is the case for the mean-field variational Bayesian inference. The behavior of this approach in the setting of the Bayesian probit ...