Scalable adaptive mantle convection simulation on petascale supercomputers

ABSTRACT

Mantle convection is the principal control on the thermal and geological evolution of the Earth. Mantle convection modeling involves solution of the mass, momentum, and energy equations for a viscous, creeping, incompressible non-Newtonian fluid at high Rayleigh and Peclet numbers. Our goal is to conduct global mantle convection simulations that can resolve faulted plate boundaries, down to 1 km scales. However, uniform resolution at these scales would result in meshes with a trillion elements, which would elude even sustained petaflops supercomputers. Thus parallel adaptive mesh refinement and coarsening (AMR) is essential.

We present RHEA, a new generation mantle convection code designed to scale to hundreds of thousands of cores. RHEA is built on ALPS, a parallel octree-based adaptive mesh finite element library that provides new distributed data structures and parallel algorithms for dynamic coarsening, refinement, rebalancing, and repartitioning of the mesh. ALPS currently supports low order continuous Lagrange elements, and arbitrary order discontinuous Galerkin spectral elements, on octree meshes. A forest-of-octrees implementation permits nearly arbitrary geometries to be accommodated. Using TACC's 579 teraflops Ranger supercomputer, we demonstrate excellent weak and strong scalability of parallel AMR on up to 62,464 cores for problems with up to 12.4 billion elements. With RHEA'S adaptive capabilities, we have been able to reduce the number of elements by over three orders of magnitude, thus enabling us to simulate large-scale mantle convection with finest local resolution of 1.5 km.

References

1. D. J. DePaolo, T. E. Cerling, S. R. Hemming, A. H. Knoll, F. M. Richter, L. H. Royden, R. L. Rudnick, L. Stixrude, and J. S. Trefil, "Origin and Evolution of Earth: Research Questions for a Changing Planet," National Academies Press, Committee on Grand Research Questions in the Solid Earth Sciences, National Research Council of the National Academies, 2008.
2. D. Bailey, "The 1997 Petaflops Algorithms Workshop," Computational Science & Engineering, IEEE, vol. 4, no. 2, pp. 82--85, Apr-Jun 1997.
3. D. P. McKenzie, J. M. Roberts, and N. O. Weiss, "Convection in the Earth's mantle: Towards a numerical solution," Journal of Fluid Mechanics, vol. 62, pp. 465--538, 1974.
4. S. Zhong, M. T. Zuber, L. N. Moresi, and M. Gurnis, "Role of temperature-dependent viscosity and surface plates in spherical shell models of mantle convection," Journal of Geophysical Research, vol. 105, pp. 11 063--11 082, 2000.
5. L. N. Moresi, M. Gurnis, and S. Zhong, "Plate tectonics and convection in the Earth's mantle: Toward a numerical simulation," Computing in Science and Engineering, vol. 2, no. 3, pp. 22--33, 2000.
6. M. Gurnis, C. Hall, and L. Lavier, "Evolving force balance during incipient subduction," Geochemistry, Geophysics, Geosystems, vol. 5, no. 7, pp. 1--31, 2004.
7. M. I. Billen, "Modeling the dynamics of subducting slabs," Annu. Rev. Earth Planet. Sci., vol. 36, pp. 325--356, 2008.
8. A. N. Brooks and T. J. R. Hughes, "Streamline upwind Petrov-Galerkin formulations for convection dominated flows with particular emphasis on the incompressible Navier-Stokes equations," Computer Methods in Applied Mechanics and Engineering, vol. 32, pp. 199--259, 1982.
9. T. J. R. Hughes, The Finite Element Method. New York: Dover, 2000.
10. C. Dohrmann and P. Bochev, "A stabilized finite element method for the Stokes problem based on polynomial pressure projections," International Journal for Numerical Methods in Fluids, vol. 46, pp. 183--201, 2004.
11. H. C. Elman, D. J. Silvester, and A. J. Wathen, Finite Elements and Fast Iterative Solvers with Applications in Incompressible Fluid Dynamics. Oxford: Oxford University Press, 2005.
12. C. C. Paige and M. A. Saunders, "Solution of sparse indefinite systems of linear equations," SIAM Journal on Numerical Analysis, vol. 12, no. 4, pp. 617--629, 1975.
13. hypre. High Performance Preconditioners, User Manual, Center for Applied Scientific Computing, Lawrence Livermore National Laboratory, 2007, https://computation.llnl.gov/casc/linear_solvers/.
14. H. De Sterck, U. M. Yang, and J. J. Heys, "Reducing complexity in parallel algebraic multigrid preconditioners," SIAM Journal on Matrix Analysis and Applications, vol. 27, no. 4, pp. 1019--1039, 2006.
15. C. Burstedde, O. Ghattas, G. Stadler, T. Tu, and L. C. Wilcox, "Towards adaptive mesh PDE simulations on petascale computers," in Proceedings of Teragrid '08, 2008.
16. T. Tu, D. R. O'Hallaron, and O. Ghattas, "Scalable parallel octree meshing for terascale applications," in Proceedings of ACM/IEEE SC05, 2005.
17. L. F. Diachin, R. Hornung, P. Plassmann, and A. Wissink, "Parallel adaptive mesh refinement," in Parallel Processing for Scientific Computing, M. A. Heroux, P. Raghavan, and H. D. Simon, Eds. SIAM, 2006, ch. 8.
18. P. Colella, J. Bell, N. Keen, T. Ligocki, M. Lijewski, and B. van Straalen, "Performance and scaling of locally-structured grid methods for partial differential equations," Journal of Physics: Conference Series, vol. 78, pp. 1--13, 2007.
19. A. Calder, B. Curtis, L. Dursi, B. Fryxell, G. Henry, P. MacNeice, K. Olson, P. Ricker, R. Rosner, F. Timmes, H. Tufo, J. Truran, and M. Zingale, "High-performance reactive fluid flow simulations using adaptive mesh refinement on thousands of processors," in Proceedings of ACM/IEEE SCOO, 2000.
20. P. MacNeice, K. M. Olson, C. Mobarry, R. de Fainchtein, and C. Packer, "PARAMESH: A parallel adaptive mesh refinement community toolkit," Computer Physics Communications, vol. 126, pp. 330--354, 2000.
21. A. M. Wissink, D. A. Hysom, and R. D. Hornung, "Enhancing scalability of parallel structured AMR calculations," in Proceedings of the International Conference on Supercomputing 2003 (ICS '03), San Francisco, CA, June 2003, pp. 336--347.
22. D. J. Mavriplis, M. J. Aftosmis, and M. Berger, "High resolution aerospace applications using the NASA Columbia Supercomputer," in Proceedings of ACM/IEEE Supercomputing 2005. Washington, DC, USA: IEEE Computer Society, 2005, p. 61.
23. C. D. Norton, G. Lyzenga, J. Parker, and R. E. Tisdale, "Developing parallel GeoFEST(P) using the PYRAMID AMR library," NASA Jet Propulsion Laboratory, Tech. Rep., 2004.
24. J. E. Flaherty, R. M. Loy, M. S. Shephard, B. K. Szymanski, J. D. Teresco, and L. H. Ziantz, "Adaptive local refinement with octree load balancing for the parallel solution of three-dimensional conservation laws," Journal of Parallel and Distributed Computing, vol. 47, no. 2, pp. 139--152, 1997.
25. J.-F. Remacle and M. Shephard, "An algorithm oriented mesh database," International Journal for Numerical Methods in Engineering, vol. 58, pp. 349--374, 2003.
26. H. Sundar, R. S. Sampath, and G. Biros, "Bottom-up construction and 2:1 balance refinement of linear octrees in parallel," SIAM Journal on Scientific Computing, vol. 30, no. 5, pp. 2675--2708, 2008.
27. W. Bangerth, R. Hartmann, and G. Kanschat, "deal. II --- A general-purpose object-oriented finite element library," ACM Transactions on Mathematical Software, vol. 33, no. 4, p. 24, 2007.
28. B. Kirk, J. W. Peterson, R. H. Stogner, and G. F. Carey, "libMesh: A C++ library for parallel adaptive mesh refinement/coarsening simulations," Engineering with Computers, vol. 22, no. 3--4, pp. 237--254, 2006.
29. L. Demkowicz, J. Kurtz, D. Pardo, M. Paszyński, W. Rachowicz, and A. Zdunek, Computing with hp Finite Elements II. Frontiers: Three-Dimensional Elliptic and Maxwell Problems with Applications. CRC Press, Taylor and Francis, 2007.
30. A. Laszloffy, J. Long, and A. K. Patra, "Simple data management, scheduling and solution strategies for managing the irregularities in parallel adaptive hp-finite element simulations," Parallel Computing, vol. 26, pp. 1765--1788, 2000.
31. J.-L. Fattebert, R. Hornung, and A. Wissink, "Finite element approach for density functional theory calculations on locally-refined meshes," Journal of Computational Physics, vol. 223, pp. 759--773, 2007.
32. S. Zhong, "Dynamics of thermal plumes in three-dimensional isoviscous thermal convection," Geophysical Journal International, vol. 162, pp. 289--300, 2005.
33. C. Burstedde, O. Ghattas, G. Stadler, T. Tu, and L. C. Wilcox, "Parallel scalable adjoint-based adaptive solution for variable-viscosity Stokes flows," Computer Methods in Applied Mechanics and Engineering, 2008, accepted for publication.
34. J. S. Hesthaven and T. Warburton, Nodal Discontinuous Galerkin Methods, ser. Texts in Applied Mathematics. Springer, 2008, no. 54.
35. M. Deville, P. Fischer, and E. Mund, High-Order Methods for Incompressible Fluid Flow, ser. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2002, no. 9.

Index Terms

• Applied computing

  • Physical sciences and engineering

    • Earth and atmospheric sciences

• Computer systems organization

  • Dependable and fault-tolerant systems and networks

• Computing methodologies

  • Modeling and simulation

• General and reference

  • Cross-computing tools and techniques

    • Performance

• Networks

  • Network performance evaluation