Abstract
We introduce a new sampling scheme for selecting the best alternative out of a given set of systems that are evaluated with respect to their expected performances. We assume that the systems are simulated on a computer and that a joint observation of all systems has a multivariate normal distribution with unknown mean and unknown covariance matrix. In particular, the observations of the systems may be stochastically dependent as is the case if common random numbers are used for simulation.
In each iteration of the algorithm, we allocate a fixed budget of simulation runs to the alternatives. We use a Bayesian set-up with a noninformative prior distribution and derive a new closed-form approximation for the posterior distributions that allows provision of a lower bound for the posterior probability of a correct selection (PCS). Iterations are continued until this lower bound is greater than 1−α for a given α. We also introduce a new allocation strategy that allocates the available budget according to posterior error probabilities. Our procedure needs no additional prior parameters and can cope with different types of ranking and selection tasks.
Our numerical experiments show that our strategy is superior to other procedures from the literature, namely, KN++ and Pluck. In all of our test scenarios, these procedures needed more observation and/or had an empirical PCS below the required 1−α. Our procedure always had its empirical PCS above 1−α, underlining the practicability of our approximation of the posterior distribution.
- Jürgen Branke, Stephen E. Chick, and Christian Schmidt. 2007. Selecting a selection procedure. Management Science 53, 12 (2007), 1916--1932. Google ScholarDigital Library
- Chun-Hung Chen, Hsiao-Chang Chen, and Liyi Dai. 1996. A gradient approach for smartly allocating computing budget for discrete event simulation. In Proceedings of the 1996 Winter Simulation Conference, John M. Charnes, Douglas J. Morrice, Daniel T. Brunner, and James J. Swain (Eds.). IEEE, 398--405. Google ScholarDigital Library
- Chun-Hung Chen, Donghai He, Michael Fu, and Loo Hay Lee. 2008. Efficient simulation budget allocation for selecting an optimal subset. INFORMS Journal on Computing 20, 4 (2008), 579--595.Google ScholarCross Ref
- Chun-Hung Chen and Loo Hay Lee. 2010. Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. World Scientific Publishing Company. Google ScholarDigital Library
- Chun-Hung Chen, Jianwu Lin, Enver Yücesan, and Stephen E. Chick. 2000. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems 10, 3 (2000), 251--270. Google ScholarDigital Library
- Chun-Hung Chen and Enver Yucesan. 2005. An alternative simulation budget allocation scheme for efficient simulation. International Journal of Simulation and Process Modelling 1, 1 (2005), 49--57.Google ScholarCross Ref
- Hsiao-Chang Chen, Liyi Dai, Chun-Hung Chen, and Enver Yücesan. 1997. New development of optimal computing budget allocation for discrete event simulation. In Proceedings of the 1997 Winter Simulation Conference, Sigrun Andradottir, Kevin J. Healy, David H. Withers, and Barry L. Nelson (Eds.). IEEE, 334--341. Google ScholarDigital Library
- Stephen E. Chick and Koichiro Inoue. 2001a. New procedures to select the best simulated system using common random numbers. Management Science 47, 8 (2001), 1133--1149. Google ScholarDigital Library
- Stephen E. Chick and Koichiro Inoue. 2001b. New two-stage and sequential procedures for selecting the best simulated system. Operations Research 49, 5 (2001), 732--743. Google ScholarDigital Library
- Morris H. DeGroot. 2004. Optimal Statistical Decisions. John Wiley 8 Sons.Google Scholar
- Francesca Dominici, Giovanni Parmigiani, and Merlise Clyde. 2000. Conjugate analysis of multivariate normal data with incomplete observations. Canadian Journal of Statistics 28, 3 (2000), 533--550.Google ScholarCross Ref
- Marco Dorigo and Thomas Stützle. 2010. Ant colony optimization: Overview and recent advances. In Handbook of Metaheuristics. Springer, 227--263.Google Scholar
- Richard L. Dykstra. 1970. Establishing the positive definiteness of the sample covariance matrix. The Annals of Mathematical Statistics 41, 6 (1970), 2153--2154.Google ScholarCross Ref
- Peter I. Frazier, Jing Xie, and Stephen E. Chick. 2011. Value of information methods for pairwise sampling with correlations. In Proceedings of the Winter Simulation Conference. Winter Simulation Conference, 3979--3991. Google ScholarDigital Library
- Michael C. Fu, Jian-Qiang Hu, Chun-Hung Chen, and Xiaoping Xiong. 2007. Simulation allocation for determining the best design in the presence of correlated sampling. INFORMS Journal on Computing 19, 1 (2007), 101--111. Google ScholarDigital Library
- Siyang Gao, Hui Xiao, Enlu Zhou, and Weiwei Chen. 2016. Optimal computing budget allocation with input uncertainty. In Proceedings of the 2016 Winter Simulation Conference. IEEE Press, 839--846. Google ScholarDigital Library
- Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2004. Bayesian Data Analysis (2nd ed.). Chapman 8 Hall/CRC.Google Scholar
- Paul Glasserman and David D. Yao. 1992. Some guidelines and guarantees for common random numbers. Management Science 38, 6 (1992), 884--908.Google ScholarDigital Library
- Peter Glynn and Sandeep Juneja. 2004. A large deviations perspective on ordinal optimization. In Proceedings of the 2004 Winter Simulation Conference, Ricki G. Ingalls, Manuel D. Rossetti, Jeffrey S. Smith, and Brett A. Peters (Eds.). Vol. 1. IEEE, 577--585. Google ScholarDigital Library
- Peter Glynn and Sandeep Juneja. 2015. Ordinal optimization-empirical large deviations rate estimators, and stochastic multi-armed bandits. arXiv preprint arXiv:1507.04564 (2015).Google Scholar
- Björn Görder. 2012. Simulationsbasierte Optimierung mit statistischen Ranking- und Selektionsverfahren. Ph.D. Dissertation (in German). TU Clausthal, Lower Saxony, Germany.Google Scholar
- S.-H. Kim and B. L. Nelson. 2006a. Selecting the best system. In Simulation Handbooks in Operations Research and Management Science, S. G. Henderson and B. L. Nelson (Eds.). Elsevier, 501--532.Google Scholar
- Seong-Hee Kim and Barry L. Nelson. 2006b. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research 54, 3 (2006), 475--488. Google ScholarDigital Library
- Soonhui Lee and Barry L. Nelson. 2014. Bootstrap ranking 8 selection revisited. In Proceedings of the 2014 Winter Simulation Conference. IEEE Press, 3857--3868. Google ScholarDigital Library
- Soonhui Lee and Barry L. Nelson. 2016. General-purpose ranking and selection for computer simulation. IIE Transactions 48, 6 (2016), 555--564.Google ScholarCross Ref
- Jun Luo, L. Jeff Hong, Barry L. Nelson, and Yang Wu. 2015. Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Operations Research 63, 5 (2015), 1177--1194.Google ScholarDigital Library
- Barry L. Nelson and Frank J. Matejcik. 1995. Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Management Science 41, 12 (1995), 1935--1945.Google ScholarCross Ref
- Eric C. Ni, Shane G. Henderson, and Susan R. Hunter. 2014. A comparison of two parallel ranking and selection procedures. In Proceedings of the 2014 Winter Simulation Conference. IEEE Press, 3761--3772. Google ScholarDigital Library
- Yijie Peng, Chun-Hung Chen, Michael Fu, and Jian-Qiang Hu. 2013. Efficient simulation resource sharing and allocation for selecting the best. IEEE Trans. Automat. Control 58 (2013), 1017--1023.Google ScholarCross Ref
- Y. Peng, E. K. P. Chong, C.-H. Chen, and M. C. Fu. 2017. Ranking and selection as stochastic control. ArXiv E-prints (Oct. 2017). arxiv:cs.LG/1710.02619Google Scholar
- Huashuai Qu, Ilya O. Ryzhov, and Michael C. Fu. 2012. Ranking and selection with unknown correlation structures. In Proceedings of the Winter Simulation Conference. Winter Simulation Conference, 12. Google ScholarDigital Library
- Huashuai Qu, Ilya O. Ryzhov, Michael C. Fu, and Zi Ding. 2015. Sequential selection with unknown correlation structures. Operations Research 63, 4 (2015), 931--948.Google ScholarDigital Library
- Colin R. Reeves. 2010. Genetic algorithms. In Handbook of Metaheuristics. Springer, 109--139.Google Scholar
- Daniel Russo. 2016. Simple Bayesian algorithms for best arm identification. In 29th Annual Conference on Learning Theory (Proceedings of Machine Learning Research), Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir (Eds.), Vol. 49. PMLR, Columbia University, New York, New York, 1417--1418. http://proceedings.mlr.press/v49/russo16.html.Google Scholar
- Ilya O. Ryzhov. 2015. Approximate Bayesian inference for simulation and optimization. In Modeling and Optimization: Theory and Applications. Springer, 1--28.Google Scholar
- Joseph L. Schafer. 1997. Analysis of Incomplete Multivariate Data. CRC Press.Google Scholar
- Christian Schmidt, Jürgen Branke, and Stephen E. Chick. 2006. Integrating techniques from statistical ranking into evolutionary algorithms. In Applications of Evolutionary Computing, Franz Rothlauf et al. (Eds.). 752--763. Google ScholarDigital Library
- James R. Swisher, Sheldon H. Jacobson, and Enver Yücesan. 2003. Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey. ACM Transactions on Modeling and Computer Simulation 13, 2 (2003), 134--154. Google ScholarDigital Library
- Zijun Wu and Michael Kolonko. 2014. Asymptotic properties of a generalized cross-entropy optimization algorithm. IEEE Transactions on Evolutionary Computation 18, 5 (2014), 658--673.Google ScholarCross Ref
- Jing Xie, Peter I. Frazier, and Stephen E. Chick. 2016. Bayesian optimization via simulation with pairwise sampling and correlated prior beliefs. Operations Research 64, 2 (2016), 542--559.Google ScholarDigital Library
- Qiong Zhang and Yongjia Song. 2016. Moment matching based conjugacy approximation for Bayesian ranking and selection. arXiv preprint arXiv:1610.09400 (2016).Google Scholar
Index Terms
- Ranking and Selection: A New Sequential Bayesian Procedure for Use with Common Random Numbers
Recommendations
Moment-Matching-Based Conjugacy Approximation for Bayesian Ranking and Selection
We study the conjugacy approximation models in the context of Bayesian ranking and selection with unknown correlations. Under the assumption of normal-inverse-Wishart prior distribution, the posterior distribution remains a normal-inverse-Wishart ...
Convergence rate analysis for optimal computing budget allocation algorithms
AbstractOrdinal optimization (OO) is a widely-studied technique for optimizing discrete-event dynamic systems (DEDS). It evaluates the performance of the system designs in a finite set by sampling and aims to correctly make ordinal comparison of the ...
Application of a predictive distribution formula to Bayesian computation for incomplete data models
We consider exact and approximate Bayesian computation in the presence of latent variables or missing data. Specifically we explore the application of a posterior predictive distribution formula derived in Sweeting And Kharroubi (2003), which is a ...
Comments