Extreme scale multi-physics simulations of the tsunamigenic 2004 sumatra megathrust earthquake

ABSTRACT

We present a high-resolution simulation of the 2004 Sumatra-Andaman earthquake, including non-linear frictional failure on a megathrustsplay fault system. Our method exploits unstructured meshes capturing the complicated geometries in subduction zones that are crucial to understand large earthquakes and tsunami generation. These up-to-date largest and longest dynamic rupture simulations enable analysis of dynamic source effects on the seafloor displacements.

To tackle the extreme size of this scenario an end-to-end optimization of the simulation code SeisSol was necessary. We implemented a new cache-aware wave propagation scheme and optimized the dynamic rupture kernels using code generation. We established a novel clustered local-time-stepping scheme for dynamic rupture. In total, we achieved a speed-up of 13.6 compared to the previous implementation. For the Sumatra scenario with 221 million elements this reduced the time-to-solution to 13.9 hours on 86, 016 Haswell cores. Furthermore, we used asynchronous output to overlap I/O and compute time.

References

1. C. J. Ammon, C. Ji, H.-K. Thio, D. Robinson, S. Ni, V. Hjorleifsdottir, H. Kanamori, T. Lay, S. Das, D. Helmberger, G. Ichinose, J. Polet, and D. J. Wald. 2005. Rupture Process of the 2004 Sumatra-Andaman Earthquake. Science 308, 5725 (2005), 1133--1139.
2. J.-P. Ampuero and Y. Ben-Zion. 2008. Cracks, pulses and macroscopic asymmetry of dynamic rupture on a bimaterial interface with velocity-weakening friction. Geophys. J. Int. 173, 2 (2008), 674--692.
3. D. J. Andrews. 1999. Test of two methods for faulting on finite-difference calculations. B. Seismol. Soc. Am. 89, 4 (1999), 931--937.
4. D. J. Andrews. 2005. Rupture dynamics with energy loss outside the slip zone. J. Geophy. Res. 110, 1 (2005), 1--14.
5. D. J. Andrews, T. C. Hanks, and J. W. Whitney. 2007. Physical limits on ground motion at Yucca Mountain. B. Seismol. Soc. Am. 97, 6 (2007), 1771--1792.
6. H. Aochi and R. Madariaga. 2003. The 1999 Izmit, Turkey, Earthquake: Nonplanar Fault Structure, Dynamic Rupture Process, and Strong Ground Motion. B. Seismol. Soc. Am. 93, 3 (2003), 1249--1266.
7. H. Aochi, R. Madariaga, and E. Fukuyama. 2003. Constraint of fault parameters inferred from nonplanar fault modeling. Geochem. Geophys. 4, 2 (2003). 1020.
8. P. Audet, M. G. Bostock, N. I. Christensen, and S. M. Peacock. 2009. Seismic evidence for overpressured subducted oceanic crust and megathrust fault sealing. Nature 457, 7225 (2009), 76--78.
9. M. Barall. 2009. A grid-doubling finite-element technique for calculating dynamic three-dimensional spontaneous rupture on an earthquake fault. Geophys. J. Int. 178 (2009), 845--859.
10. M. Barall and R. A. Harris. 2014. Metrics for Comparing Dynamic Earthquake Rupture Simulations. Seismol. Res. Lett. 86, 1 (Nov. 2014), 223--235.
11. G. I. Barenblatt. 1962. The Mathematical Theory of Equilibrium Cracks in Brittle Fracture. Advances in Applied Mechanics 7 (1962), 55--129.
12. Q. Bletery, A. Sladen, J. Jiang, and M. Simons. 2016. A Bayesian source model for the 2004 great Sumatra-Andaman earthquake. J. Geophy. Res. Solid Earth 121, 7 (2016), 5116--5135.
13. A. Breuer, A. Heinecke, and M. Bader. 2016. Petascale Local Time Stepping for the ADER-DG Finite Element Method. In 2016 IEEE Int. Parallel and Distrib. Process. Symp. (IPDPS). 854--863.
14. A. Breuer, A. Heinecke, S. Rettenberger, M. Bader, A.-A. Gabriel, and C. Pelties. 2014. Sustained Petascale Performance of Seismic Simulations with SeisSol on SuperMUC. In Supercomputing: 29th Int. Conf. Springer, 1--18.
15. C. Burstedde, O. Ghattas, M. Gurnis, T. Isaac, G. Stadler, T. Warburton, and L. Wilcox. 2010. Extreme-Scale AMR. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. (SC '10). IEEE Computer Society, 1--12.
16. A. P. S. Chauhan, S. C. Singh, N. D. Hananto, H. Carton, F. Klingelhoefer, J.-X. Dessa, H. Permana, N. J. White, and D. Graindorge. 2009. Seismic imaging of forearc backthrusts at northern Sumatra subduction zone. Geophys. J. Int. 179, 3 (Dec. 2009), 1772--1780.
17. Computational Infrastructure for Geodynamics. 2017. SPECFEM3D Cartesian. www.geodynamics.org/cig/software/specfem3d/. (2017).
18. Y. Cui, E. Poyraz, K. B. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, and others. 2013. Physics-based seismic hazard analysis on petascale heterogeneous supercomputers. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. ACM, 70.
19. L. A. Dalguer and S. M. Day. 2007. Staggered-grid split-node method for spontaneous rupture simulation. J. Geophy. Res. 112, 2 (2007), 1--15.
20. L. A. Dalguer, K. Irikura, J. D. Riera, and H. C. Chiu. 2001. The importance of the dynamic source effects on strong ground motion during the 1999 Chi-Chi, Taiwan, earthquake: Brief interpretation of the damage distribution on buildings. B. Seismol. Soc. Am. 91, 5 (2001), 1112--1127.
21. S. M. Day, L. A. Dalguer, N. Lapusta, and Y. Liu. 2005. Comparison of finite difference and boundary integral solutions to three-dimensional spontaneous rupture. J. Geophy. Res. 110 (2005), 1--23.
22. J. de la Puente, J.-P. Ampuero, and M. Käser. 2009. Dynamic rupture modeling on unstructured meshes using a discontinuous Galerkin method. J. Geophy. Res. Solid Earth 114, B10 (2009). B10302.
23. N. DeDontney, J. R. Rice, and R. Dmowska. 2012. Finite Element Modeling of Branched Ruptures Including Off-Fault Plasticity. B. Seismol. Soc. Am. 102, 2 (2012), 541--562.
24. B. Duan. 2012. Dynamic rupture of the 2011 Mw9.0 Tohoku-Oki earthquake: Roles of a possible subducting seamount. J. Geophy. Res. Solid Earth 117, B5 (2012). B05311.
25. B. Duan and S. M. Day. 2008. Inelastic strain distribution and seismic radiation from rupture of a fault kink. J. Geophy. Res. 113, B12 (Dec. 2008), B12311.
26. M. Dumbser and M. Käser. 2006. An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes - II. The three-dimensional isotropic case. Geophys. J. Int. 167 (2006), 319--336.
27. M. Dumbser, M. Käser, and E. F. Toro. 2007. An arbitrary high-order Discontinuous Galerkin method for elastic waves on unstructured meshes V: Local time steppingan p-adaptivity. Geophys. J. Int. 171, 2 (2007), 695--717.
28. M. Dumbser and C.-D. Munz. 2005. Arbitrary high order discontinuous Galerkin schemes. Numerical methods for hyperbolic and kinetic problems (2005), 295--333.
29. E. M. Dunham. 2007. Conditions governing the occurrence of supershear ruptures under slip-weakening friction. Geophys. J. Int. 112, B7 (2007), 1--24.
30. G. P. Ely, S. M. Day, and J.-B. Minster. 2010. Dynamic rupture models for the southern San Andreas fault. B. Seismol. Soc. Am. 100, 1 (2010), 131--150.
31. W. Frings, F. Wolf, and V. Petkov. 2009. Scalable Massively Parallel I/O to Task-local Files. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. (SC '09). ACM, New York, NY, USA, 1--11.
32. A.-A. Gabriel, J.-P. Ampuero, L. A. Dalguer, and P. M. Mai. 2012. The Transition of Dynamic Rupture Modes in Elastic Media Under Velocity-Weakening Friction. J. Geophy. Res. 117, B9 (2012), 0148--0227.
33. A.-A. Gabriel, J.-P. Ampuero, L. A. Dalguer, and P. M. Mai. 2013. Source Properties of Dynamic Rupture Pulses with Off-Fault Plasticity. J. Geophy. Res. (2013).
34. P. Galvez, J.-P. Ampuero, L. A. Dalguer, S. N. Somala, and T. Nissen-Meyer. 2014. Dynamic earthquake rupture modelled with an unstructured 3-D spectral element method applied to the 2011 M9 Tohoku earthquake. Geophys. J. Int. 198, 2 (2014), 1222--1240.
35. P. Galvez, L. A. Dalguer, J.-P. Ampuero, and D. Giardini. 2016. Rupture Reactivation during the 2011 Mw 9.0 Tohoku Earthquake: Dynamic Rupture and Ground-Motion Simulations. B. Seismol. Soc. Am. 106, 3 (2016), 819--831.
36. C. Geuzaine and J. F. Remacle. 2009. Gmsh: A 3-D finite element mesh generator with built-in pre- and post-processing facilities. Internat. J. Numer. Methods Engrg. 79, 11 (2009), 1309--1331.
37. J. L. Hardebeck. 2015. Stress orientations in subduction zones and the strength of subduction megathrust faults. Science 349, 6253 (Sept. 2015), 1213--1216.
38. R. A. Harris, M. Barall, D. J. Andrews, B. Duan, S. Ma, E. M. Dunham, A.-A. Gabriel, Y. Kaneko, Y. Kase, B. T. Aagaard, D. D. Oglesby, J.-P. Ampuero, T. C. Hanks, and N. Abrahamson. 2011. Verifying a Computational Method for Predicting Extreme Ground Motion. Seismol. Res. Lett. 82, 5 (2011), 638--644.
39. R. A. Harris, M. Barall, R. J. Archuleta, E. M. Dunham, B. T. Aagaard, J.-P. Ampuero, H. Bhat, V. Cruz-Atienza, L. A. Dalguer, P. Dawson, S. M. Day, B. Duan, G. P. Ely, Y. Kaneko, Y. Kase, N. Lapusta, Y. Liu, S. Ma, D. D. Oglesby, K. B. Olsen, A. Pitarka, S. Song, and E. Templeton. 2009. The SCEC/USGS Dynamic Earthquake Rupture Code Verification Exercise. Seismol. Res. Lett. 80, 1 (2009), 119--126.
40. R. A. Harris and S. M. Day. 1999. Dynamic 3D simulations of earthquakes on en echelon faults. Geophys. Res. Lett. 26, 14 (1999), 2089--2092.
41. G. P. Hayes, D. J. Wald, and R. L. Johnson. 2012. Slab 1.0: A three-dimensional model of global subduction zone geometries. J. Geophy. Res. 117, B1 (Jan. 2012), B01302.
42. A. Heinecke, A. Breuer, M. Bader, and P. Dubey. 2016. High Order Seismic Simulations on the Intel Xeon Phi Processor (Knights Landing). In High Performance Computing: 31st International Conference, ISC High Performance 2016, Frankfurt, Germany, June 19-23, 2016, Proceedings. Springer, 343--362.
43. A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A.-A. Gabriel, C. Pelties, A. Bode, W. Barth, X.-K. Liao, K. Vaidyanathan, M. Smelyanskiy, and P. Dubey. 2014. Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. IEEE, New Orleans, 3--14. Gordon Bell Finalist.
44. A. Heinecke, G. Henry, M. Hutchinson, and H. Pabst. 2016. LIBXSMM: Accelerating Small Matrix Multiplications by Runtime Code Generation. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. (SC '16). IEEE Press, Piscataway, NJ, USA, 84:1--84:11.
45. K. Hiroo and K. Masayuki. 1993. The 1992 Nicaragua earthquake: a slow tsunami earthquake associated with subducted sediments. Nature 361, 6414 (Feb. 1993), 714--716.
46. Y. Huang, J.-P. Ampuero, and H. Kanamori. 2014. Slip-Weakening Models of the 2011 Tohoku-Oki Earthquake and Constraints on Stress Drop and Fracture Energy. Pure Appl. Geophys. 171, 10 (2014), 2555--2568.
47. M. Hutchinson, A. Heinecke, H. Pabst, G. Henry, M. Parsani, and D. Keyes. 2016. Efficiency of High Order Spectral Element Methods on Petascale Architectures. In High Performance Computing: 31st Int. Conf. Springer, 449--466.
48. T. Ichimura, K. Fujita, P. E. B. Quinay, L. Maddegedara, M. Hori, S. Tanaka, Y. Shizawa, H. Kobayashi, and K. Minami. 2015. Implicit Nonlinear Wave Simulation with 1.08T DOF and 0.270T Unstructured Finite Elements to Enhance Comprehensive Earthquake Simulation. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. (SC '15). ACM, New York, 4:1--4:12.
49. T. Ichimura, K. Fujita, S. Tanaka, M. Hori, M. Lalith, Y. Shizawa, and H. Kobayashi. 2014. Physics-based urban earthquake simulation enhanced by 10.7 BlnDOFX 30 K time-step unstructured FE non-linear seismic wave simulation. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. IEEE, 15--26.
50. M. Käser and M. Dumbser. 2006. An Arbitrary High Order Discontinuous Galerkin Method for Elastic Waves on Unstructured Meshes I: The Two-Dimensional Isotropic Case with External Source Terms. Geophys. J. Int. 166, 2 (2006), 855--877.
51. M. Käser, M. Dumbser, J. de la Puente, and H. Igel. 2007. Anarbitraryhigh-order Discontinuous Galerkin method for elastic waves on unstructured meshes - III. Viscoelastic attenuation. Geophys. J. Int. 168 (2007), 224--242.
52. M. Käser and F. Gallovič. 2008. Effects of complicated 3-D rupture geometries on earthquake ground motion and their implications: a numerical study. Geophys. J. Int. 172, 1 (2008), 276--292.
53. J. E. Kozdon and E. M. Dunham. 2013. Rupture to the trench: Dynamic rupture simulations of the 11 March 2011 Tohoku earthquake. B. Seismol. Soc. Am. 103, 2B (2013), 1275--1289.
54. F. Krüger and M. Ohrnberger. 2005. Spatio-temporal source characteristics of the 26 December 2004 Sumatra earthquake as imaged by teleseismic broadband arrays. Geophys. Res. Lett. 32, 24 (2005), 1--4.
55. G. Laske, G. Masters, Z. Ma, and M. Pasyanos. 2013. Update on CRUST1.0 - A 1-degree Global Model of Earth's Crust, Abstract EGU2013-2658. European Geophysical Union.
56. R. J. LeVeque. 2002. Finite volume methods for hyperbolic problems. Vol. 31. Cambridge University Press.
57. J. Li, W. keng Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale. 2003. Parallel netCDF: A High-Performance Scientific I/O Interface. In Supercomputing, 2003 ACM/IEEE Conference. 39--39.
58. J.-Y. Lin, X. Le Pichon, C. Rangin, J. C. Sibuet, and T. Maury. 2009. Spatial after-shock distribution of the 26 December 2004 great Sumatra-Andaman earthquake in the northern Sumatra area. Geochem. Geophys. 10, 5 (2009).
59. Q. Liu, J. Logan, Y. Tian, H. Abbasi, N. Podhorszki, J. Y. Choi, S. Klasky, R. Tchoua, J. Lofstead, R. Oldfield, M. Parashar, N. Samatova, K. Schwan, A. Shoshani, M. Wolf, K. Wu, and W. Yu. 2014. Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency and Computation: Practice and Experience 26, 7 (2014), 1453--1473.
60. J. Lofstead, R. Oldfield, T. Kordenbrock, and C. Reiss. 2011. Extending Scalability of Collective IO Through Nessie and Staging. In Proceedings of the Sixth Workshop on Parallel Data Storage (PDSW '11). ACM, New York, NY, USA, 7--12.
61. J. C. Lozos. 2016. A case for historic joint rupture of the San Andreas and San Jacinto faults. Science advances 2, 3 (2016), e1500621.
62. S. Ma. 2012. A self-consistent mechanism for slow dynamic deformation and large tsunami generation for earthquakes in the shallow subduction zone. Geophys. Res. Lett. 39 (2012), 1--7.
63. S. Ma and G. C. Beroza. 2008. Rupture dynamics on a bimaterial interface for dipping faults. B. Seismol. Soc. Am. 98, 4 (2008), 1642--1658.
64. S. Ma and E. T. Hirakawa. 2013. Dynamic wedge failure reveals anomalous energy radiation of shallow subduction earthquakes. Earth Planet. Sci. Lett. 375 (2013), 113--122.
65. R. Madariaga. 1983. High frequency radiation from dynamic earthquake fault models. Annales Geophysicae 1, 1 (1983), 17--23.
66. A. J. Meltzner, K. Sieh, M. Abrams, D. C. Agnew, K. W. Hudnut, J.-P. Avouac, and D. H. Natawidjaja. 2006. Uplift and subsidence associated with the great Aceh-Andaman earthquake of 2004. J. Geophy. Res. Solid Earth 111, B2 (2006).
67. G. F. Moore, N. L. Bangs, A. Taira, S. Kuramoto, E. Pangborn, and H. J. Tobin. 2007. Three-Dimensional Splay Fault Geometry and Implications for Tsunami Generation. Science 318, 5853 (2007), 1128--1131.
68. D. D. Oglesby. 2008. Rupture termination and jump on parallel offset faults. B. Seismol. Soc. Am. 98, 1 (2008), 440--447.
69. D. D. Oglesby, R. J. Archuleta, and S. B. Nielsen. 2000. The three-dimensional dynamics of dipping faults. B. Seismol. Soc. Am. 90, 3 (2000), 616--628.
70. C. Pelties, J. de la Puente, J.-P. Ampuero, G. B. Brietzke, and M. Käser. 2012. Three-dimensional dynamic rupture simulation with a high-order discontinuous Galerkin method on unstructured tetrahedral meshes. J. Geophy. Res. Solid Earth 117, B2 (2012). B02309.
71. S. Rettenberger and M. Bader. 2015. Optimizing Large Scale I/O for Petascale Seismic Simulations on Unstructured Meshes. In 2015 IEEE International Conference on Cluster Computing (CLUSTER). IEEE Xplore, Chicago, IL, 314--317.
72. M. Rietmann, M. Grote, D. Peter, and O. Schenk. 2017. Newmark local time stepping on high-performance computing architectures. J. Comput. Phys. 334 (2017), 308--326.
73. M.E. Rognes, R. C. Kirby, and A. Logg. 2010. Efficient Assembly of H (div) and H (curl) Conforming Finite Elements. SIAM Journal on Scientific Computing 31, 6 (2010), 4130--4151.
74. O. Rojas, S. Day, J. Castillo, and L. A. Dalguer. 2008. Modelling of rupture propagation using high-order mimetic finite differences. Geophys. J. Int. 172, 2 (2008), 631--650.
75. D. Roten, Y. Cui, K. B. Olsen, S. M. Day, K. Withers, W. H. Savran, P. Wang, and D. Mu. 2016. High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers. In Proc. Int. Conf. for High Performance Computing, Networking, Storage and Anal. IEEE, 82.
76. D. Roten, K. B. Olsen, H. Magistrale, J. C. Pechmann, and V. M. Cruz-Atienza. 2011. 3D simulations of M7 earthquakes on the Wasatch fault, Utah, part I: Long-period (0-1 Hz) ground motion. B. Seismol. Soc. Am. 101, 5 (2011), 2045--2063.
77. D. M. Saffer and H. J. Tobin. 2011. Hydrogeology and Mechanics of Subduction Zone Forearcs: Fluid Flow and Pore Pressure. Annu. Rev. Earth Planet. Sci. 39 (2011), 157--86.
78. K. Schloegel, G. Karypis, and V. Kumar. 2002. Parallel static and dynamic multi-constraint graph partitioning. Concurrency and Computation: Practice and Experience 14, 3 (2002), 219--240.
79. P. Shearer and R. Bürgmann. 2010. Lessons Learned from the 2004 Sumatra-Andaman Megathrust Rupture. Annu. Rev. Earth Planet. Sci. 38 (2010), 103--131.
80. J.-C. Sibuet, C. Rangin, X. Le Pichon, S. Singh, A. Cattaneo, D. Graindorge, F. Klingelhoefer, J.-Y. Lin, J. Malod, T. Maury, and others. 2007. 26th December 2004 great Sumatra-Andaman earthquake: Co-seismic and post-seismic motions in northern Sumatra. Earth Planet. Sci. Lett. 263, 1-2 (Nov. 2007), 88--103.
81. S. C. Singh, H. Carton, P. Tapponnier, N. D. Hananto, A. P. S. Chauhan, D. Hartoyo, M. Bayly, S. Moeljopranoto, T. Bunting, P. Christie, H. Lubis, and J. Martin. 2008. Seismic evidence for broken oceanic crust in the 2004 Sumatra earthquake epicentral region. Nature Geoscience 1, 11 (Oct. 2008), 777--781.
82. A. H. Stroud. 1971. Approximate calculation of multiple integrals. Prentice-Hall, Englewood Cliffs, N.J.
83. The HDF Group. 2017. HDF5. https://www.hdfgroup.org/HDF5/. (2017).
84. E. F. Toro. 1999. Riemann Solvers and Numerical Methods for Fluid Dynamics, A Practical Introduction. Springer, Berlin, Heidelberg, New York.
85. K. Tsuda, S. Iwase, H. Uratani, S. Ogawa, T. Watanabe, J. Miyakoshi, and J.-P. Ampuero. 2017. Dynamic Rupture Simulations Based on the Characterized Source Model of the 2011 Tohoku Earthquake. Pure Appl. Geophys. (2017), 1--12.
86. C. Uphoff and M. Bader. 2016. Generating high performance matrix kernels for earthquake simulations with viscoelastic attenuation. In 2016 Int. Conf. High Performance Computing Simulation (HPCS). 908--916.
87. V. Vishwanath, M. Hereld, and M. E. Papka. 2011. Toward simulation-time data analysis and I/O acceleration on leadership-class systems. In 2011 IEEE Symposium on Large Data Analysis and Visualization. 9--14.
88. P. Weatherall, K. M. Marks, M. Jakobsson, T. Schmitt, S. Tani, J. E. Arndt, M. Rovere, D. Chayes, V. Ferrini, and R. Wigley. 2015. A new digital bathymetric model of the world's oceans. Earth and Space Science 2, 8 (2015), 331--345.

Index Terms

• Applied computing

  • Physical sciences and engineering

    • Earth and atmospheric sciences