Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures

ABSTRACT

Computational seismology is an area of wide sociological and economic impact, ranging from earthquake risk assessment to subsurface imaging and oil and gas exploration. At the core of these simulations is the modeling of wave propagation in a complex medium. Here we report on the extension of the high-order finite-element seismic wave simulation package SPECFEM3D to support the largest scale hybrid and homogeneous supercomputers. Starting from an existing highly tuned MPI code, we migrated to a CUDA version. In order to be of immediate impact to the science mission of computational seismologists, we had to port the entire production package, rather than just individual kernels. One of the challenges in parallelizing finite element codes is the potential for race conditions during the assembly phase. We therefore investigated different methods such as mesh coloring or atomic updates on the GPU. In order to achieve strong scaling, we needed to ensure good overlap of data motion at all levels, including internode and host-accelerator transfers. Finally we carefully tuned the GPU implementation. The new MPI/CUDA solver exhibits excellent scalability and achieves speedup on a node-to-node basis over the carefully tuned equivalent multi-core MPI solver. To demonstrate the performance of both the forward and adjoint functionality, we present two case studies run on the Cray XE6 CPU and Cray XK6 GPU architectures up to 896 nodes: (1) focusing on most commonly used forward simulations, we simulate seismic wave propagation generated by earthquakes in Turkey, and (2) testing the most complex seismic inversion type of the package, we use ambient seismic noise to image 3-D crust and mantle structure beneath western Europe.

References

1. L. Bautista-Gomez, N. Maruyama, D. Komatitsch, S. Tsuboi, F. Cappello, S. Matsuoka, and T. Nakamura. FTI: high performance fault tolerance interface for hybrid systems. In Proceedings of the Supercomputing 2011 Conference, 2011.
2. L. Carrington, D. Komatitsch, M. Laurenzano, M. M. Tikir, D. Michéa, N. Le Goff, A. Snavely, and J. Tromp. High-frequency simulations of global seismic wave propagation using SPECFEM3D GLOBE on 62K processors. In Proceedings of the Supercomputing 2008 Conference, 2008.
3. C. Cecka, A. J. Lew, and E. Darve. Assembly of finite element methods on graphics processors. International journal for numerical methods in engineering, 85(5):640--669, 2011.
4. C. Chevalier and F. Pellegrini. Pt-scotch: A tool for efficient parallel graph ordering. Parallel Computing, 34(68):318--331, 2008.
5. T. Cubit. Cubit 13.2 users manual. Sandia National Laboratories, Albuquerque, NM, 2012.
6. J.-J. Droux. An algorithm to optimally color a mesh. Computer Methods in Applied Mechanics and Engineering, 104(2):249--260, 1993.
7. D. Kirk, W. H. Wen-mei, and W. Hwu. Programming massively parallel processors: a hands-on approach. Morgan Kaufmann, 2010.
8. D. Komatitsch. Fluid-solid coupling on a cluster of GPU graphics cards for seismic wave propagation. Comptes Rendus de l'Acadmie des Sciences Mécanique, 339:125--135, 2011.
9. D. Komatitsch, G. Erlebacher, D. Göddeke, and D. Michéa. High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster. Joural of Computational Physics, 229(20):7692--7714, 2010.
10. D. Komatitsch, G. Erlebacher, D. Göddeke, and D. Michéa. Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs. Computer Science Research and Development, 25(1-2):75--82, 2010.
11. D. Komatitsch, D. Michéa, and G. Erlebacher. Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA. Journal of Parallel and Distributed Computing, 69(5):451--460, 2009.
12. D. Komatitsch and J. Tromp. Introduction to the spectral element method for three-dimensional seismic wave propagation. Geophysical Journal International, 139(3):806--822, December 1999.
13. D. Komatitsch and J. Tromp. Spectral-element simulations of global seismic wave propagation-I. Validation. Geophysical Journal International, 149(2):390--412, May 2002.
14. D. Komatitsch, S. Tsuboi, J. Chen, and J. Tromp. A 14.6 billion degrees of freedom, 5 teraflop, 2.5 terabyte earthquake simulation on the Earth Simulator. Proceedings of the ACM/IEEE Supercomputing SC'2003 conference, 2003. on CD-ROM.
15. M. Krivelevich. Coloring random graphs--an algorithmic perspective. In Proc. 2nd Colloquium on Mathematics and Computer Science, pages 175--195, 2002.
16. R. Madariaga. Dynamics of an expanding circular fault. Bulletin of the Seismological Society of America, 66(3):639--666, 1976.
17. I. Molinari and A. Morelli. Epcrust: a reference crustal model for the european plate. Geophysical Journal International, 185(1):352--364, 2011.
18. T. Nissen-Meyer, A. Fournier, and F. A. Dahlen. A 2-D spectral-element method for computing spherical-earth seismograms--I. Moment-tensor source. Geophysical Journal International, 168:1067--1093, 2007b.
19. D. Peter, D. Komatitsch, Y. Luo, R. Martin, N. Le Goff, E. Casarotti, P. Le Loher, F. Magnoni, Q. Liu, C. Blitz, T. Nissen-Meyer, P. Basini, and J. Tromp. Forward and adjoint simulations of seismic wave propagation on fully unstructured hexahedral meshes. Geophysical Journal International, 186(2):721--739, August 2011.
20. N. Pulido, A. Ojeda, K. Atakan, and T. Kubo. Strong ground motion estimation in the sea of marmara region (turkey) based on a scenario earthquake. Tectonophysics, 391(1):357--374, 2004.
21. J. F. Schaefer, L. Boschi, T. W. Becker, and E. Kissling. Radial anisotropy in the European mantle: Tomographic studies explored in terms of mantle flow. Geophysical Research Letters, 38(23):L23304, December 2011.
22. N.M. Shapiro, M. Campillo, L. Stehly, and M. H. Ritzwoller. High-resolution surface-wave tomography from ambient seismic noise. Science, 307(5715):1615, 2005.
23. A. Tarantola. Inverse Problem Theory. Society for Industrial and Applied Mathematics, Philadelphia, 2005.
24. J. Tromp, Y. Luo, S. Hanasoge, and D. Peter. Noise cross-correlation sensitivity kernels. Geophysical Journal International, 183(2):791--819, 2010.
25. J. Tromp, C. Tape, and Q. Liu. Seismic tomography, adjoint methods, time reversal and banana-doughnut kernels. Geophysical Journal International, 160:195--216, 2005.
26. H. M. Tufo and P. F. Fischer. Terascale spectral element algorithms and implementations. Proceedings of the ACM/IEEE Supercomputing SC'1999 conference, 1999. on CD-ROM.
27. H. Zhu, Y. Luo, T. Nissen-Meyer, C. Morency, and J. Tromp. Elastic imaging and time-lapse migration based upon adjoint methods. Geophysics, 74:WCA167--WCA177, 2009.

• Applied computing

  • Physical sciences and engineering

• Computer systems organization

• General and reference

  • Cross-computing tools and techniques

• Networks