ASAGI: A Parallel Server for Adaptive Geoinformation

ABSTRACT

We present ASAGI, an open-source library with a simple interface to access Cartesian material and geographic datasets in massively parallel simulations with dynamically adaptive mesh refinement (AMR). ASAGI distributes geographic datasets over all compute nodes storing only a portion of the dataset on each node. An automatic replication mechanism copies the data between nodes to assure fast local access even after load migration in the application. We demonstrate ASAGI's preparedness for up-to-petascale simulations in three use cases. We simulate a Tsunami on 512 cores and a porous media flow on up to 8,192 cores of SuperMUC with the AMR framework sam(oa)2. We also run an earthquake simulation with SeiSol on 65,536 cores. For all applications, ASAGI provides large complex 3D material datasets required for the realistic scenarios. The NUMA-awareness of ASAGI turned out to be especially useful for the hybrid MPI+OpenMP parallelization of both codes.

References

1. Conventions for the standardization of NetCDF files, 1995. http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html, Accessed: 2016-02-11.
2. The GEBCO_08 Grid, version 20100927, 2010. http://www.gebco.net.
3. J. Dinan, P. Balaji, D. Buntinas, D. Goodell, W. Gropp, and R. Thakur. An implementation and evaluation of the MPI 3.0 one-sided communication interface. Concurrency and Computation: Practice and Experience, 2016.
4. A. Gabriel and C. Pelties. Simulating large-scale earthquake dynamic rupture scenarios on natural fault zones using the ADER-DG method. In EGU General Assembly Conference Abstracts, volume 16 of EGU General Assembly Conference Abstracts, page 10572, May 2014. poster abstract.
5. P. Galvez, J.-P. Ampuero, L. A. Dalguer, S. N. Somala, and T. Nissen-Meyer. Dynamic earthquake rupture modelled with an unstructured 3-D spectral element method applied to the 2011 M9 Tohoku earthquake. Geophysical Journal International, 198(2):1222--1240, 2014.
6. D. L. George and R. J. LeVeque. Finite volume methods and adaptive refinement for global tsunami propagation and local inundation. Science of Tsunami Hazards, 24:319--328, 2006.
7. A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A.-A. Gabriel, C. Pelties, A. Bode,W. Barth, X.-K. Liao, K. Vaidyanathan, M. Smelyanskiy, and P. Dubey. Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis SC14, pages 3--14, New Orleans, LA, USA, Nov. 2014. IEEE.
8. A. Humphrey, D. Sunderland, T. Harman, and M. Berzins. Radiative heat transfer calculation on 16384 GPUs using a reverse Monte Carlo ray tracing approach with adaptive mesh refinement. In The 17th IEEE International Workshop on Parallel and Distributed Scientific and Engineering Computing (PDSEC 2016), 2016. accepted.
9. M. D. Kohler, H. Magistrale, and R. W. Clayton. Mantle heterogeneities and the SCEC reference three-dimensional seismic velocity model version 3. Bulletin of the Seismological Society of America, 93(2):757--774, 2003.
10. O. Meister and M. Bader. 2D adaptivity for 3D problems: Parallel SPE10 reservoir simulation on dynamically adaptive prism grids. Journal of Computational Science, 9:101--106, 2015.
11. MPI Forum. MPI: A message passing interface standard. Version 2.0, May 1998.
12. A. K. Patra, A. Bauer, C. Nichita, E. B. Pitman, M. Sheridan, M. Bursik, B. Rupp, A. Webber, A. Stinton, L. Namikawa, et al. Parallel adaptive numerical simulation of dry avalanches over natural terrain. Journal of Volcanology and Geothermal Research, 139(1):1--21, 2005.
13. A. Plesch, C. Tape, R. Graves, P. Small, G. Ely, and J. Shaw. Updates for the CVM-H including new representations of the offshore Santa Maria and San Bernardino basin and a new Moho surface. In 2011 Southern California Earthquake Center Annual Meeting, Proceedings and Abstracts, volume 21, page 214, 2011.
14. S. Popinet. Quadtree-adaptive tsunami modelling. Ocean Dynamics, 61(9):1261--1285, 2011.
15. S. Popinet. Adaptive modelling of long-distance wave propagation and fine-scale flooding during the Tohoku tsunami. Natural Hazards and Earth System Sciences, 12(4):1213--1227, 2012.
16. J. Rudi, A. C. I. Malossi, T. Isaac, G. Stadler, M. Gurnis, P. W. J. Staar, Y. Ineichen, C. Bekas, A. Curioni, and O. Ghattas. An extreme-scale implicit solver for complex PDEs: Highly heterogeneous flow in earth's mantle. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '15, pages 5:1--5:12, New York, NY, USA, 2015. ACM.
17. O. Sahni, K. E. Jansen, C. A. Taylor, and M. S. Shephard. Automated adaptive cardiovascular flow simulations. Engineering with Computers, 25(1):25--36, 2008.
18. SeisSol, 2016. http://www.seissol.org/.
19. E. Strohmaier, J. Dongarra, H. Simon, and M. Meuer. Top500 list, November 2015. http://www.top500.org.
20. M. P. Süss and J. H. Shaw. P wave seismic velocity structure derived from sonic logs and industry reflection data in the Los Angeles basin, California. Journal of Geophysical Research: Solid Earth, 108(B3), 2003. 2170.
21. A. S. Tanenbaum. Modern operating systems (2. ed.). Prentice Hall, 2001.
22. K. Unterweger, R. Wittmann, P. Neumann, T. Weinzierl, and H.-J. Bungartz. Integration of FULLSWOF2D and PeanoClaw: adaptivity and local time-stepping for complex overland flows. In Recent trends in computational engineering - CE2014: optimization, uncertainty, parallel algorithms, coupled and complex problems., pages 181--195. Springer, 2015.

• Applied computing

  • Physical sciences and engineering