11 PFLOP/s simulations of cloud cavitation collapse

ABSTRACT

We present unprecedented, high throughput simulations of cloud cavitation collapse on 1.6 million cores of Sequoia reaching 55% of its nominal peak performance, corresponding to 11 PFLOP/s. The destructive power of cavitation reduces the lifetime of energy critical systems such as internal combustion engines and hydraulic turbines, yet it has been harnessed for water purification and kidney lithotripsy. The present two-phase flow simulations enable the quantitative prediction of cavitation using 13 trillion grid points to resolve the collapse of 15'000 bubbles. We advance by one order of magnitude the current state-of-the-art in terms of time to solution, and by two orders the geometrical complexity of the flow. The software successfully addresses the challenges that hinder the effective solution of complex flows on contemporary supercomputers, such as limited memory bandwidth, I/O bandwidth and storage capacity. The present work redefines the frontier of high performance computing for fluid dynamics simulations.

References

1. R. Abgrall and S. Karni. Computations of compressible multifluids. Journal of Computational Physics, 169(2):594--623, 2001.
2. N. Adams and S. Schmidt. Shocks in cavitating flows. In C. F. Delale, editor, Bubble Dynamics and Shock Waves, volume 8 of Shock Wave Science and Technology Reference Library, pages 235--256. Springer Berlin Heidelberg, 2013.
3. A. S. Almgren, J. B. Bell, M. J. Lijewski, Z. Lukić, and E. V. Andel. Nyx: A massively parallel amr code for computational cosmology. The Astrophysical Journal, 765(1):39, 2013.
4. AMD Inc. Software Optimization Guide for the AMD 15h Family, 2011.
5. T. B. Benjamin and A. T. Ellis. The collapse of cavitation bubbles and the pressures thereby produced against solid boundaries. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 260(1110):221--240, 1966.
6. M. Berger and P. Colella. Local adaptive mesh refinement for shock hydrodynamics. Journal of Computational Physics, 82(1):64--84, 1989.
7. M. J. Berger and J. Oliger. Adaptive mesh refinement for hyperbolic partial differential equations. J. Comput. Phys., 53(3):484--512, 1984.
8. M. Berzins, J. Luitjens, Q. Meng, T. Harman, C. A. Wight, and J. R. Peterson. Uintah: a scalable framework for hazard analysis. In Proceedings of the 2010 TeraGrid Conference, TG '10, pages 3:1--3:8. ACM, 2010.
9. J. R. Blake, M. C. Hooton, P. B. Robinson, and R. P. Tong. Collapsing cavities, toroidal bubbles and jet impact. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 355(1724):537--550, 1997.
10. C. E. Brennen. Cavitation and bubble dynamics. Oxford University Press, USA, 1995.
11. C. E. Brennen. An introduction to cavitation fundamentals. Technical report, In: WIMRC Forum 2011 -- Cavitation: Turbo-machinery & Medical Applications, 2011.
12. A. Cohen, I. Daubechies, and P. Vial. Wavelets on the interval and fast wavelet transforms. Applied and Computational Harmonic Analysis, 1(1):54--81, 1993.
13. A. Cucheval and R. Chow. A study on the emulsification of oil by power ultrasound. Ultrasonics Sonochemistry, 15(5):916--920, 2008.
14. J. Dear and J. Field. A study of the collapse of arrays of cavities. J. Fluid Mech, 190(409):172, 1988.
15. N. G. Dickson, K. Karimi, and F. Hamze. Importance of explicit vectorization for cpu and gpu software performance. Journal of Computational Physics, 230(13):5383--5398, 2011.
16. M. O. Domingues, S. M. Gomes, O. Roussel, and K. Schneider. Space-time adaptive multiresolution methods for hyperbolic conservation laws: Applications to compressible euler equations, Sep 2009.
17. D. Donoho. Interpolating wavelet transforms, 1992.
18. S. Faulk, E. Loh, M. L. D. Vanter, S. Squires, and L. Votta. Scientific computing's productivity gridlock: How software engineering can help. Computing in Science Engineering, 11(6):30--39, 2009.
19. R. T. Fisher, L. P. Kadanoff, D. Q. Lamb, A. Dubey, T. Plewa, A. Calder, F. Cattaneo, P. Constantin, I. T. Foster, M. E. Papka, S. I. Abarzhi, S. M. Asida, P. M. Rich, C. C. Glendenin, K. Antypas, D. J. Sheeler, L. B. Reid, B. Gallagher, and S. G. Needham. Terascale turbulence computation using the flash3 application framework on the ibm blue gene/lsystem. IBM Journal of Research and Development, 52(1--2):127--136, 2008.
20. Folk M., Cheng A. and Yates K. Hdf5: A file format and i/o library for high performance computing applications. In Proceedings of Supercomputing, 1999.
21. J. P. Franc and M. Riondet. Incubation time and cavitation erosion rate of work-hardening materials. In The proceeding of the Sixth International Symposium on Cavitation, CAV2006, 2006.
22. B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. MacNeice, R. Rosner, J. W. Truran, and H. Tufo. Flash: An adaptive mesh hydrodynamics code for modeling astrophysical thermonuclear flashes. The Astrophysical Journal Supplement Series, 131(1):273, 2000.
23. J.-l. Gailly and M. Adler. Zlib compression library. 2004.
24. E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design patterns: Abstraction and reuse of object-oriented design. In O. Nierstrasz, editor, ECOOP' 93 --- Object-Oriented Programming, volume 707 of Lecture Notes in Computer Science, pages 406--431. Springer Berlin/Heidelberg, 1993.
25. F. R. Gilmore. The collapse and growth of a spherical bubble in a viscous compressible liquid. Technical Report 26--4, California Institute of Technology, 1952.
26. J. A. Greenough, B. R. De Supinski, R. K. Yates, C. A. Rendleman, D. Skinner, V. Beckner, M. Lijewski, and J. Bell. Performance of a block structured, hierarchical adaptive mesh refinement code on the 64k node ibm bluegene/l computer. Computer, pages 1--12, 2005.
27. W. Gropp, D. Kaushik, D. Keyes, and B. Smith. High-performance parallel implicit CFD. Parallel Computing, 27(4):337--362, 2001.
28. F. Günther, M. Mehl, M. Pögl, and C. Zenger. A cache aware algorithm for pdes on hierarchical data structures based on space filling curves. SIAM Journal on Scientific Computing, 28(5):1634--1650, 2006.
29. F. G. Hammitt. Damage to solids caused by cavitation. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 260(1110):245--255, 1966.
30. I. Hansson, V. Kedrinskii, and K. A. Morch. On the dynamics of cavity clusters. Journal of Physics D: Applied Physics, 15(9):1725, 1982.
31. R. Haring, M. Ohmacht, T. Fox, M. Gschwind, D. Satterfield, K. Sugavanam, P. Coteus, P. Heidelberger, M. Blumrich, R. Wisniewski, A. Gara, G.-T. Chiu, P. Boyle, N. Chist, and C. Kim. The ibm blue gene/q compute chip. Micro, IEEE, 32(2):48--60, 2012.
32. N. A. Hawker and Y. Ventikos. Interaction of a strong shockwave with a gas bubble in a liquid medium: a numerical study. Journal of Fluid Mechanics, 701:59--97, 2012.
33. B. Hejazialhosseini, D. Rossinelli, C. Conti, and P. Koumoutsakos. High throughput software for direct numerical simulations of compressible two-phase flows. Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pages 16:1--16:12, 2012.
34. B. Hejazialhosseini, D. Rossinelli, and P. Koumoutsakos. 3D shock-bubble interactions at mach 3. Physics of Fluids (Gallery of Fluid Motion), 2013.
35. R. Hickling and M. S. Plesset. Collapse and rebound of a spherical bubble in water. Physics of Fluids, 7(1):7--14, 1964.
36. M. Holmström. Solving hyperbolic pdes using interpolating wavelets. SIAM Journal on Scientific Computing, 21(2):405--420, 1999.
37. X. Y. Hu, B. C. Khoo, N. A. Adams, and F. L. Huang. A conservative interface method for compressible flows. Journal of Computational Physics, 219(2):553--578, Dec 10 2006.
38. Y. Hu, A. Cox, and W. Zwaenepoel. Improving fine-grained irregular shared-memory benchmarks by data reordering. In Supercomputing, ACM/IEEE 2000 Conference, pages 33--33, 2000.
39. IBM. A2 Processor User's Manual for Blue Gene/Q, 2012.
40. T. Ikeda, S. Yoshizawa, M. Tosaki, J. S. Allen, S. Takagi, N. Ohta, T. Kitamura, and Y. Matsumoto. Cloud cavitation control for lithotripsy using high intensity focused ultrasound. Ultrasound in Medicine & Biology, 32(9):1383--1397, 2006.
41. Intel Corporation. Intel® 64 and IA-32 Architectures Optimization Reference Manual. Intel Corporation, 2009.
42. G. Jiang and C. Shu. Efficient implementation of weighted ENO schemes. Journal of Computational Physics, 126(1):202--228, Jun 1996.
43. E. Johnsen and T. Colonius. Implementation of WENO schemes in compressible multicomponent flow problems. Journal of Computational Physics, 219(2):715--732, Dec 10 2006.
44. E. Johnsen and T. Colonius. Numerical simulations of non-spherical bubble collapse. Journal of Fluid Mechanics, 629:231--262, 5 2009.
45. E. Johnsen and F. Ham. Preventing numerical errors generated by interface-capturing schemes in compressible multi-material flows. Journal of Computational Physics, 231(17):5705--5717, 2012.
46. D. Kelly. A software chasm: Software engineering and scientific computing. Software, IEEE, 24(6):120--119, 2007.
47. N. K. R. Kevlahan and O. V. Vasilyev. An adaptive wavelet collocation method for fluid-structure interaction at high reynolds numbers. SIAM Journal on Scientific Computing, 26(6):1894--1915, 2005.
48. B.-J. Kim and W. Pearlman. An embedded wavelet video coder using three dimensional set partitioning in hierarchical trees (spiht). In Data Compression Conference, 1997. DCC '97. Proceedings, pages 251--260, 1997.
49. R. T. Knapp. Recent investigations of the mechanics of cavitations and cavitation damage. Trans. ASME, 77, 1955.
50. T. Kodama and K. Takayama. Dynamic behavior of bubbles during extracorporeal shock-wave lithotripsy. Ultrasound in Medicine & Biology, 24(5):723--738, 1998.
51. E. Lauer, X. Y. Hu, S. Hickel, and N. A. Adams. Numerical investigation of collapsing cavity arrays. Physics of Fluids, 24(5):052104, 2012.
52. J. Li, W.-k. Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale. Parallel netcdf: A high-performance scientific i/o interface. In Proceedings of the 2003 ACM/IEEE conference on Supercomputing, SC '03. ACM, 2003.
53. J. F. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin. Flexible io and integration for scientific codes through the adaptable io system (adios). In Proceedings of the 6th international workshop on Challenges of large applications in distributed environments, CLADE '08, pages 15--24. ACM, 2008.
54. J. Mellor-Crummey, D. Whalley, and K. Kennedy. Improving memory hierarchy performance for irregular applications using data and computation reorderings. International Journal of Parallel Programming, 29(3):217--247, 2001.
55. Q. Meng and M. Berzins. Abstract: Uintah hybrid task-based parallelism algorithm. In High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion:, pages 1431--1432, 2012.
56. K. Mørch. Energy considerations on the collapse of cavity clusters. Applied Scientific Research, 38:313--321, 1982.
57. A. Myers. Stanford researchers break million-core supercomputer barrier, January 2013.
58. P. Colella, D. T. Graves, T. J. Ligocki, D. F. Martin, D. Mondiano, D. B. Serafini, and B. Van Straalen. Chombo software package for amr applications design document. Technical report, Lawrence Berkeley National Laboratory, 2003.
59. R. Prosser. Resolution independent lifted interpolating wavelets on an interval. 2009.
60. D. Ranjan, J. H. J. Niederhaus, J. G. Oakley, M. H. Anderson, J. A. Greenough, and R. Bonazza. Experimental and numerical investigation of shock-induced distortion of a spherical gas inhomogeneity. Physica Scripta Volume T, 132(1):014020, Dec. 2008.
61. L. Rayleigh. Viii. on the pressure developed in a liquid during the collapse of a spherical cavity. Philosophical Magazine Series 6, 34(200):94--98, 1917.
62. S. J. Reckinger, D. Livescu, and O. V. Vasilyev. Adaptive wavelet collocation method simulations of Rayleigh-Taylor instability. Physica Scripta, T142, DEC 2010. 2nd International Conference and Advanced School on Turbulent Mixing and Beyond, Abdus Salam Int Ctr Theoret Phys, Trieste, ITALY, JUL 27-AUG 07, 2009.
63. S. M. Reckinger, O. V. Vasilyev, and B. Fox-Kemper. Adaptive volume penalization for ocean modeling. Ocean Dynamics, 62(8):1201--1215, AUG 2012.
64. D. Rossinelli, B. Hejazialhosseini, D. Spampinato, and P. Koumoutsakos. Multicore/multi-gpu accelerated simulations of multiphase compressible flows using wavelet adapted grids. SIAM J. Scientific Computing, 33(2), 2011.
65. O. Roussel, K. Schneider, A. Tsigulin, and H. Bockhorn. A conservative fully adaptive multiresolution algorithm for parabolic pdes. Journal Of Computational Physics, 188(2):493--523, Jul 2003.
66. E. R. Schendel, S. V. Pendse, J. Jenkins, D. A. Boyuka, II, Z. Gong, S. Lakshminarasimhan, Q. Liu, H. Kolla, J. Chen, S. Klasky, R. Ross, and N. F. Samatova. Isobar hybrid compression-i/o interleaving for large-scale parallel i/o optimization. In Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing, HPDC '12, pages 61--72, New York, NY, USA, 2012. ACM.
67. D. P. Schmidt and M. L. Corradini. The internal flow of diesel fuel injector nozzles: A review. International Journal of Engine Research, 2(1):1--22, 2001.
68. Schmidt et al. Assessment of the prediction capabalities of a homogeneous cavitation model fir the collapse characteristics of a vapour-bubble cloud. In WIMRC 3rd International Cavitation Forum, Coventry, U.K., 2011.
69. K. Schneider and O. V. Vasilyev. Wavelet methods in computational fluid dynamics. Annual Review of Fluid Mechanics, 42(1):473--503, 2010.
70. K. Schwaber and M. Beedle. Agile Software Development with Scrum. Prentice Hall PTR, 1st edition, 2001.
71. J. Sermulins, W. Thies, R. Rabbah, and S. Amarasinghe. Cache aware optimization of stream programs. SIGPLAN Not., 40(7):115--126, June 2005.
72. J. M. Shapiro. Embedded image coding using zerotrees of wavelet coefficients. Signal Processing, IEEE Transactions on, 41(12):3445--3462, 1993.
73. I. B. G. team. Design of the ibm blue gene/q compute chip. IBM Journal of Research and Development, 57(1/2):1:1--1:13, 2013.
74. Y. Tomita and A. Shima. Mechanisms of impulsive pressure generation and damage pit formation by bubble collapse. Journal of Fluid Mechanics, 169:535--564, Aug. 1986.
75. Y. Utturkar, J. Wu, G. Wang, and W. Shyy. Recent progress in modeling of cryogenic cavitation for liquid rocket propulsion. Progress in Aerospace Sciences, 41(7):558--608, 2005.
76. O. Vasilyev and C. Bowman. Second-generation wavelet collocation method for the solution of partial differential equations. Journal of Computational Physics, 165(2):660--693, DEC 10 2000.
77. T. Wen, J. Su, P. Colella, K. Yelick, and N. Keen. An adaptive mesh refinement benchmark for modern parallel programming languages. In Proceedings of the 2007 ACM/IEEE conference on Supercomputing, SC '07, pages 1--12. ACM, 2007.
78. B. Wendroff. Approximate Riemann solvers, Godunov schemes and contact discontinuities. In Toro, EF, editor, Godunov Methods: Theory and Applications, pages 1023--1056, 233 Spring St, New York, NY 10013 USA, 2001. London Math Soc, Kluwer Academic/Plenum Publ.
79. S. Williams, A. Waterman, and D. Patterson. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM, 52:65--76, 2009.
80. J. Williamson. Low-Storage Runge-Kutta Schemes. Journal of Computational Physics, 35(1):48--56, 1980.
81. G. Wilson. Where's the real bottleneck in scientific computing? American Scientist, 2006.