Supporting robust system analysis with the test matrix tool framework

ABSTRACT

We present the Test Matrix Tool (TMT) framework, a simulation-agnostic framework providing end-to-end support for robust analysis of complex systems. The need to execute a large number of simulations is common to many problem environments, even those already reduced by Design of Experiments or similar methodologies. TMT addresses key end-user needs in easing the specification, execution and analysis of simulation workloads in ways that are consistent between specific applications of the framework. The TMT design contributes modular specifications for key data communicated between and within the specification, execution and analysis components. Our TMT implementation is an instantiation of those formats freely available for general use. TMT's data analysis component provides a variety of features data filtering, comparison, transformation and visualization for analytic tasks on any TMT-embedded model. We provide a brief case study as an example of its use in a real-world application.

References

1. Bagrodia, R., Meyer, R., Takai, M., Chen, Y. A., Zeng, X., Martin, J. and Song, H. Y. (1998). Parsec: A parallel simulation environment for complex systems. IEEE Computer, 31(10), pp. 77--85.
2. Bagrodia, R., Yu-an Chen, Gerla, M., Kwan, B., Martin, J., PaInati, P., Walton, S. (1996). Parallel Simulation of a High-Speed Wormhole Routing Network. In Proc. of the Tenth Workshop on Parallel and Distributed Simulation (PADS 96), pp. 47--56.
3. Buyya, R. and Murshed, M. (2003). Gridsim: A toolkit for the modeling and simulation of distributed resource management and scheduling for grid computing. Concurrency and Computation: Practice and Experience, 14(13-15), pp. 1175--1220.
4. Clarkson, E., Navathe, S. and Foley, J.D. (2009). Generalized formal models for faceted user interfaces. In Proc. of the 9th ACM/IEEE Joint Conference on Digital Libraries (JCDL 09), pp. 125--134.
5. Dahmann, J. S., Fujimoto, R. M. and Weatherly, R. M. (1997). The Department of Defense High Level Architecture. In Proc. of the 29th Winter Simulation Conference, pp. 142--149.
6. Fekete, J. D., Van Wijk, J., Stasko, J. and North, C. (2008). The Value of Information Visualization. In Information Visualization, Andreas Kerren, John T. Stasko, Jean-Daniel Fekete, and Chris North (Eds.). Lecture Notes In Computer Science, Vol. 4950. Springer-Verlag pp. 1--18.
7. Flager, F., Welle, B., Bansal, P., Soremekun, G., and Haymaker, J. (2009). Multidisciplinary Process Integration and Design Optimization of a Classroom Building. Journal of Information Technology in Construction (14), pp. 595--612.
8. Fogue, M., Garrido, P., Martinez, F. J., Cano, J. C., Calafate, C. T. and Manzoni, P. (2012). A realistic simulation framework for vehicular networks. In Proc. of the 5th International ICST Conference on Simulation Tools and Techniques, pp. 37--46.
9. Fujimoto, R. (1990). Parallel discrete event simulation. Communications of the ACM 33(10), pp. 30--53.
10. Gamma, E., Helm, R., Johnson, R. and Vlissides, J. (1994). Design Patterns - Elements of Reusable Object-Oriented Software. Addison-Wesley.
11. Himmelspach, J. and Uhrmacher, A.M. (2009) The JAMES II Framework for Modeling and Simulation. In Proc. of the International Workshop on High Performance Computational Systems Biology (HIBI 09), pp.101--102.
12. Inselberg, A. (1985). The plane with parallel coordinates. The Visual Computer 1(2), pp. 69--91.
13. Kandel, S., Paepcke, A., Hellerstein, J. M. and Heer, J. (2012). Enterprise Data Analysis and Visualization: An Interview Study. IEEE Transactions on Visualization and Computer Graphics, 18(12), pp. 2917--2926.
14. Kotkas, V., Ojamaa, A., Grigorenko, P., Maigre, R., Harf, M. and Tyugu, E. (2011). CoCoViLa as a Multifunctional Simulation Platform. In Proc. of the 4th International ICST Conference on Simulation Tools and Techniques, pp. 198--205.
15. Rodrigues, A. F., Hemmert, K. S., Barrett, B. W., Kersey, C., Oldfield, R., Weston, M., Risen, R., Cook, J., Rosenfeld, P., CooperBalls, E. and Jacob, B. (2011). The Structural Simulation Toolkit. ACM SIGMETRICS Performance Evaluation Review, 38(4), pp. 37--42.
16. Savitzky, A. and Golay, M.J.E. (1964). Smoothing and Differentiation of Data by Simplified Least Squares Procedures. Analytical Chemistry 36(8), pp. 1627--1639.
17. Shvachko, K., Kuang, H., Radia, S. and Chansler, R. (2010). The Hadoop Distributed File System. In Proc. of the IEEE Symposium on Mass Storage Systems and Technologies, pp. 1--10.
18. Spence, R. and Tweedie, L. (1998). The Attribute Explorer: Information Synthesis via Exploration. Interacting with Computers, 11(2), pp. 137--146.
19. Wonnacott, P., Bruce, D. (1996). The APOSTLE simulation language: granularity control and performance data. In Proc. of the Tenth Workshop on Parallel and Distributed Simulation (PADS 96), pp. 114--123.
20. Yee, K. P., Swearingen, K., Li, K. and Hearst, M. (2003). Faceted Metadata for Image Search and Browsing. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI 03), pp. 401--408.
21. Yi, J.S., Kang, Y., Stasko, J. and Jacko, J. (2007). Toward a Deeper Understanding of the Role of Interaction in Information Visualization, IEEE Transactions on Visualization and Computer Graphics, 13(6), pp. 1224--1231.
22. Zeigler, B. P., Praehofer, H, and Kim, T.G. (2000) Theory of modeling and simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems. Academic Press.

Index Terms

• Computing methodologies

  • Modeling and simulation

    • Simulation support systems