HACC: extreme scaling and performance across diverse architectures

Abstract

Supercomputing is evolving toward hybrid and accelerator-based architectures with millions of cores. The Hardware/Hybrid Accelerated Cosmology Code (HACC) framework exploits this diverse landscape at the largest scales of problem size, obtaining high scalability and sustained performance. Developed to satisfy the science requirements of cosmological surveys, HACC melds particle and grid methods using a novel algorithmic structure that flexibly maps across architectures, including CPU/GPU, multi/many-core, and Blue Gene systems. In this Research Highlight, we demonstrate the success of HACC on two very different machines, the CPU/GPU system Titan and the BG/Q systems Sequoia and Mira, attaining very high levels of scalable performance. We demonstrate strong and weak scaling on Titan, obtaining up to 99.2% parallel efficiency, evolving 1.1 trillion particles. On Sequoia, we reach 13.94 PFlops (69.2% of peak) and 90% parallel efficiency on 1,572,864 cores, with 3.6 trillion particles, the largest cosmological benchmark yet performed. HACC design concepts are applicable to several other supercomputer applications.

References

1. Bhattacharya, S., Habib, S., Heitmann, K., Vikhlinin, A. Dark matter Halo profiles of massive clusters: Theory versus observations. Astrophys. J. 766 (2013), 32.
2. Bryan, G.L., Norman, M.L. In 12th Kingston Meeting on Theoretical Astrophysics, Proceedings of Meeting Held in Halifax; Nova Scotia (ASP Conference Series # 123), D.A. Clarke and M. Fall, eds. 1996; see also O'Shea, B.W., Nagamine, K., Springel, V., Hernquist, L., Norman, M.L. Astrophys. J. Supp. 160 (2005), 1.
3. Couchman, H.M.P., Thomas, P.A., Pearce, F.R. Hydra: An adaptive-mesh implementation of P 3M-SPH Astrophys. J. 452, 797 (1995).
4. For a review of cosmological simulation methods, see also Dolag, K., Borgani, S., Schindler, S., Diaferio, A., Bykov, A.M. Space Sci. Rev. 134 (2008), 229.
5. Fryxell, B., et al. FLASH: An adaptive mesh hydrodynamics code for modeling astrophysical thermonuclear flashes. Astrophys. J. Supp. 131 (2000), 273.
6. Gafton, E., Rosswog, S. A fast recursive coordinate bisection tree for neighbour search and gravity. Mon. Not. R. Astron. Soc. 418 (2011), 770.
7. Habib, S., Morozov, V., Finkel, H., Pope, A., Heitmann, K., Kumaran, K., Peterka, T., Insley, J., Daniel, D., Fasel, P., Frontiere, N., Lukić, Z. arXiv:1211.4864, Supercomputing 2012.
8. Habib, S., Pope, A., Finkel, H., Frontiere, N., Heitmann, K., Daniel, D., Fasel, P., Morozov, V., Zagaris, G., Peterka, T., Vishwanath, V., Lukić, Z., Sehrish, S., Liao, W.-K. HACC: Simulating sky surveys on state-of-the-art supercomputing architectures. New Astron. 42 (2016), 49 arXiv:1410.2805 {astro-ph.IM}.
9. Habib, S., Pope, A., Lukić, Z., Daniel, D., Fasel, P., Desai, N., Heitmann, K., Hsu, C.-H., Ankeny, L., Mark, G., Bhattacharya, S., Ahrens, J. Hybrid petacomputing meets cosmology: The Roadrunner Universe project. J. Phys. Conf. Ser. 180 (2009), 012019.
10. Hamming, R.W. Digital Filters. Dover, Publications, Mineola, New York 1998.
11. Heitmann, K., Frontiere, N., Sewell, C., Habib, S., Pope, A., Finkel, H., Rizzi, S., Insley, J., Bhattacharya, S. The Q continuum simulation: Harnessing the power of GPU accelerated supercomputers. J. - Astrophys. J. Supp. 219 (2015), 34 arXiv:1411.3396 {astro-ph.CO}.
12. Heitmann, K., Higdon, D., White, M., Habib, S., Williams, B.J., Lawrence, E., Wagner, C. The Coyote universe. II. Cosmological models and precision emulation of the nonlinear matter power spectrum Astrophys. J. 705 (2009), 156.
13. Heitmann, K., Lukić, Z., Fasel, P., Habib, S., Warren, M.S., White, M., Ahrens, J., Ankeny, L., Armstrong, R., O'Shea, B., Ricker, P.M., Springel, V., Stadel, J., Trac, H. The cosmic code comparison project. Comput. Sci. Dis. 1 (2008), 015003.
14. Heitmann, K., Ricker, P.M., Warren, M.S., Habib, S. Robustness of cosmological simulations. I. Large-scale Structure. Astrophys. J. Supp. 160 (2005), 28.
15. Hockney, R.W., Eastwood, J.W. Computer Simulation Using Particles. Adam Hilger, New York, 1988.
16. Peebles, P.J.E., Structure of the coma cluster of galaxies. Astron. J. 75 (1970), 13.
17. Peebles, P.J.E. The Large-Scale Structure of the Universe. Princeton University Press, Princeton, New Jersey 1980.
18. Pfalzner, S., Gibbon, P. Many-Body Tree Methods in Physics. Cambridge University Press, 1996; see also Barnes, J., Hut, P. Nature 324, 446 (1986); Warren, M.S., Salmon, J.K. Technical Paper, Supercomputing, Cambridge University Press, New York, USA 1993.
19. Pope, A., Habib, S., Lukic, Z., Daniel, D., Fasel, P., Desai, N., Heitmann, K. Comput. Sci. Eng. 12 (2010), 17.
20. The accelerated universe. Power, C., Navarro, J.F., Jenkins, A., Frenk, C.S., White, S.D.M., Springel, V., Stadel, J., Quinn, T. The inner structure of ACDM haloes - I. A numerical convergence study. Mon. Not. R. Astron. Soc. 338 (2003), 14.
21. Shandarin, S.F., Zeldovich, Ya.B. The large-scale structure of the universe: Turbulence, intermittency, structures in a self-gravitating medium. Rev. Mod. Phys. 61 (1989), 185.
22. Springel, V. The cosmological simulation code GADGET-2. Mon. Not. R. Astron. Soc. 364 (2005), 1105.
23. Teyssier, R. Cosmological hydrodynamics with adaptive mesh refinement. A new high resolution code called RAMSES. A&A 385 (2002), 337.
24. White, M. The mass of a halo. Astron. and Astrophys. 367 (2001), 27.
25. White, M., Pope, A., Carlson, J., Heitmann, K., Habib, S., Fasel, P., Daniel, D., Lukić, Z. Particle mesh simulations of the Lyα forest and the signature of Baryon acoustic oscillations in the intergalactic medium. Astrophys. J. 713 (2010), 383.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Performance

• Information systems

  • Information retrieval

    • Search engine architectures and scalability