Wei Xie
Abstract
When we use simulations to estimate the performance of stochastic systems, the simulation is often driven by input models estimated from finite real-world data. A complete statistical characterization of system performance estimates requires quantifying both input model and simulation estimation errors. The components of input models in many complex systems could be dependent. In this paper, we represent the distribution of a random vector by its marginal distributions and a dependence measure: either product-moment or Spearman rank correlations. To quantify the impact from dependent input model and simulation estimation errors on system performance estimates, we propose a metamodel-assisted bootstrap framework that is applicable to cases when the parametric family of multivariate input distributions is known or unknown. In either case, we first characterize the input models by their moments that are estimated using real-world data. Then, we employ the bootstrap to quantify the input estimation error, and an equation-based stochastic kriging metamodel to propagate the input uncertainty to the output mean, which can also reduce the influence of simulation estimation error due to output variability. Asymptotic analysis provides theoretical support for our approach, while an empirical study demonstrates that it has good finite-sample performance.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Multivariate Input Uncertainty in Output Analysis for Stochastic Simulation
- Bruce E. Ankenman, Barry L. Nelson, and Jeremy Staum. 2010. Stochastic kriging for simulation metamodeling. Operations Research 58 (2010), 371--382. Google Scholar
Digital Library
- Eusebio Arenal-Gutiérrez, Carlos Matrán, and Juan A. Cuesta-Albertos. 1996. Unconditional Glivenko-Gantelli-type theorems and weak laws of large numbers for bootstrap. Statistics 8 Probability Letters 26 (1996), 365--375.Google Scholar
- Francois Bachoc. 2013. Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification. Computational Statistics 8 Data Analysis 66 (2013), 55--69.Google Scholar
- Russell R. Barton. 2007. Presenting a more complete characterization of uncertainty: Can it be done? In Proceedings of the 2007 INFORMS Simulation Society Research Workshop. INFORMS Simulation Society, Fontainebleau.Google Scholar
- Russell R. Barton. 2012. Tutorial: Input uncertainty in output analysis. In Proceedings of the 2012 Winter Simulation Conference, C. Laroque, J. Himmelspach, R. Pasupathy, O. Rose, and A. M. Uhrmacher (Eds.). IEEE Computer Society, 67--78. Google ScholarDigital Library
- Russell R. Barton, Barry L. Nelson, and Wei Xie. 2014. Quantifying input uncertainty via simulation confidence intervals. Informs Journal on Computing 26 (2014), 74--87. Google ScholarDigital Library
- Russell R. Barton and Lee W. Schruben. 1993. Uniform and bootstrap resampling of input distributions. In Proceedings of the 1993 Winter Simulation Conference, G. W. Evans, M. Mollaghasemi, E. C. Russell, and W. E. Biles (Eds.). IEEE Computer Society, 503--508. Google ScholarDigital Library
- Russell R. Barton and Lee W. Schruben. 2001. Resampling methods for input modeling. In Proceedings of the 2001 Winter Simulation Conference, B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer (Eds.). IEEE Computer Society, 372--378. Google ScholarDigital Library
- Bahar Biller and Canan G. Corlu. 2011. Accounting for parameter uncertainty in large-scale stochastic simulations with correlated inputs. Operations Research 59 (2011), 661--673. Google ScholarDigital Library
- Bahar Biller and Soumyadip Ghosh. 2006. Multivariate input processes. In Handbooks in Operations Research and Management Science: Simulation, S. Henderson and B. L. Nelson (Eds.). Elsevier, Chapter 5.Google Scholar
- Patrick Billingsley. 1995. Probability and Measure. Wiley-Interscience, New York.Google Scholar
- Marne C. Cario and Barry L. Nelson. 1997. Modeling and Generating Random Vectors with Arbitrary Marginal Distributions and Correlation Matrix. Technical report. Department of Industrial Engineering and Management Sciences, Northwestern University.Google Scholar
- Xi Chen, Bruce E. Ankenman, and Barry L. Nelson. 2012. The effect of common random numbers on stochastic kriging metamodels. ACM Transactions on Modeling and Computer Simulation 22 (2012), 7:1--7:20. Google ScholarDigital Library
- Russell C. H. Cheng and Wayne Holland. 1997. Sensitivity of computer simulation experiments to errors in input data. Journal of Statistical Computation and Simulation 57 (1997), 219--241.Google ScholarCross Ref
- Robert T. Clemen and Terence Reilly. 1999. Correlations and copulas for decision and risk analysis. Management Science 45 (1999), 208--224.Google ScholarCross Ref
- Sourav Das, Tata S. Rao, and Georgi N. Boshnakov. 2012. On the Estimation of Parameters of Variograms of Spatial Stationary Isotropic Random Processes. Research Report No. 2. The University of Manchester.Google Scholar
- Soumyadip Ghosh and Shane G. Henderson. 2002a. Chessboard distributions and random vectors with specified marginals and covariance matrix. Operations Research 50 (2002), 820--834. Google ScholarDigital Library
- Soumyadip Ghosh and Shane G. Henderson. 2002b. Properties of the NORTA method in higher dimensions. In Proceedings of the 2002 Winter Simulation Conference, E. Yűcesan, C. H. Chen, J. L. Snowdon, and J. M. Charnes (Eds.). IEEE Computer Society, 263--269. Google ScholarDigital Library
- Peter Hall. 1988. Rate of convergence in bootstrap approximations. The Annals of Probability 16 (1988), 1665--1684.Google ScholarCross Ref
- Shane G. Henderson, Belinda A. Chiera, and Roger M. Cooke. 2000. Generating dependent quasi-random numbers. In Proceedings of the 2000 Winter Simulation Conference, J. A. Joines, R. R. Barton, K. Kang, and P. A. Fishwick (Eds.). IEEE Computer Society, 527--536. Google ScholarDigital Library
- Nicholas J. Higham. 2002. Computing the nearest correlation matrix -- A problem from finance. IMA Journal on Numerical Analysis 22 (2002), 329--343.Google ScholarCross Ref
- Mark E. Johnson. 1987. Multivariate Statistical Simulation. Wiley, New York. Google ScholarDigital Library
- Donald R. Jones, Matthias Schonlau, and William J. Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13 (1998), 455--492. Google ScholarDigital Library
- Keebom Kang and Bruce Schmeiser. 1990. Graphical methods for evaluation and comparing confidence-interval procedures. Operations Research 38, 3 (1990), 546--553. Google ScholarDigital Library
- Shing T. Li and Joseph L. Hammond. 1975. Generation of pseudorandom numbers with specified univariate distributions and correlation coefficients. IEEE Transactions on Systems, Man, and Cybernetics 5 (September 1975), 557--561.Google ScholarCross Ref
- Jason L. Loeppky, Jerome Sacks, and William J. Welch. 2009. Choosing the sample size of a computer experiment: A practical guide. Technometrics 51 (2009), 366--376.Google ScholarCross Ref
- Tanmoy Mukhopadhyay, Sushanta Chakroborty, Sondipon Adhikari, and Rajib Chowdhury. 2016. A critical assessment of kriging model variants for high-fidelity uncertainty quantification in dynamics of composite shells. Archives on Computational Methods in Engineering (2016). Online version.Google Scholar
- Bruce W. Schmeiser and Ram Lal. 1982. Bivariate gamma random vectors. Operations Research 30, 2 (1982), 355--374.Google ScholarDigital Library
- Jun Shao and Dongsheng Tu. 1995. The Jackknife and Bootstrap. Springer-Verlag.Google Scholar
- Eunhye Song, Barry L. Nelson, and C. Dennis Pegden. 2014. Advanced tutorial: Input uncertainty quantification. In Proceedings of the 2014 Winter Simulation Conference, A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, and J. A. Miller (Eds.). IEEE Computer Society. Google ScholarDigital Library
- A. W. Van Der Vaart. 1998. Asymptotic Statistics. Cambridge University Press, Cambridge, UK.Google Scholar
- William R. Wade. 2010. An Introduction to Analysis (4th ed.). Prentice Hall.Google Scholar
- Wei B. Wu and Jan Mielniczuk. 2010. A new look at measuring dependence. In Dependence in Probability and Statistics, P. Doukhan, G. Lang, D. Surgailis, and G. Teyssière (Eds.). Springer.Google Scholar
- Wei Xie, Barry L. Nelson, and Russell R. Barton. 2014a. A Bayesian framework for quantifying uncertainty in stochastic simulation. Operational Research 62, 6 (2014), 1439--1452.Google ScholarDigital Library
- Wei Xie, Barry L. Nelson, and Russell R. Barton. 2014b. Statistical uncertainty analysis for stochastic simulation with dependent input models. In Proceedings of the 2014 Winter Simulation Conference, A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, and J. A. Miller (Eds.). IEEE Computer Society. Google ScholarDigital Library
- Wei Xie, Barry L. Nelson, and Russell R. Barton. 2015. Statistical uncertainty analysis for stochastic simulation. (2015). Working Paper, Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY.Google Scholar
- Wei Xie, Barry L. Nelson, and Jeremy Staum. 2010. The influence of correlation functions on stochastic kriging metamodels. In Proceedings of the 2010 Winter Simulation Conference, B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, and E. Yucesan (Eds.). IEEE Computer Society, 1067--1078. Google ScholarDigital Library
- Jun Yin, Szu H. Ng, and Kien M. Ng. 2009. A study on the effects of parameter estimation on kriging model’s prediction error in stochastic simulation. In Proceedings of the 2009 Winter Simulation Conference, B. Johansson A. Dunkin M. D. Rossetti, R. R. Hill and R. G. Ingalls (Eds.). IEEE Computer Society, 674--685. Google ScholarDigital Library
Index Terms
- Multivariate Input Uncertainty in Output Analysis for Stochastic Simulation
Recommendations
An Efficient Budget Allocation Approach for Quantifying the Impact of Input Uncertainty in Stochastic Simulation
Simulations are often driven by input models estimated from finite real-world data. When we use simulations to assess the performance of a stochastic system, there exist two sources of uncertainty in the performance estimates: input and simulation ...
Nonparametric confidence intervals for population variance of one sample and the difference of variances of two samples
Confidence intervals for the population variance and the difference in variances of two populations based on the ordinary t-statistics combined with the bootstrap method are suggested. Theoretical and practical aspects of the suggested techniques are ...
Asymptotic Expansions and Bootstrap Approximations in Factor Analysis
We derive asymptotic expansions for the distributions of the normal theory maximum likelihood estimators of unique variances and uniquenesses (standardized unique variances) in the factor analysis model. Asymptotic expansions are given for the ...
Comments