Anton 2: raising the bar for performance and programmability in a special-purpose molecular dynamics supercomputer

ABSTRACT

Anton 2 is a second-generation special-purpose supercomputer for molecular dynamics simulations that achieves significant gains in performance, programmability, and capacity compared to its predecessor, Anton 1. The architecture of Anton 2 is tailored for fine-grained event-driven operation, which improves performance by increasing the overlap of computation with communication, and also allows a wider range of algorithms to run efficiently, enabling many new software-based optimizations. A 512-node Anton 2 machine, currently in operation, is up to ten times faster than Anton 1 with the same number of nodes, greatly expanding the reach of all-atom biomolecular simulations. Anton 2 is the first platform to achieve simulation rates of multiple microseconds of physical time per day for systems with millions of atoms. Demonstrating strong scaling, the machine simulates a standard 23,558-atom benchmark system at a rate of 85 μs/day---180 times faster than any commodity hardware platform or general-purpose supercomputer.

References

1. Acellera, "ACEMD: High-throughput molecular dynamics with NVIDIA Kepler GPUs", http://www.slideshare.net/CanOzdoruk/gtc-expressacemdwebinar, retrieved April 2, 2014.
2. Sadaf R. Alam, Pratul K. Agarwal, Melissa C. Smith, Jeffrey S. Vetter and David Caliga, "Using FPGA Devices to Accelerate Biomolecular Simulations", IEEE Computer, Volume 40, Issue 3, March, 2007, pp. 66--73.
3. Yoshimichi Andoh, Noriyuki Yoshii, Kazushi Fujimoto, Keisuke Mizutani, Hidekazu Kojima, Atsushi Yamada, Susumu Okazaki, Kazutomo Kawaguchi, Hidemi Nagao, Kensuke Iwahashi, Fumiyasu Mizutani, Kazuo Minami, Shin-ichi Ichikawa, Hidemi Komatsu, Shigeru Ishizuki, Yasuhiro Takeda and Masao Fukushima, "MODYLAS: A Highly Parallelized General-Purpose Molecular Dynamics Simulation Program for Large-Scale Systems with Long-Range Forces Calculated by Fast Multipole Method (FMM) and Highly Scalable Fine-Grained New Parallel Processing Algorithms", Journal of Chemical Theory and Computation, Volume 9, Issue 7, July 9, 2013, pp. 3201--3209.
4. Anton Arkhipov, Yibing Shan, Rahul Das, Nicholas F. Endres, Michael P. Eastwood, David E. Wemmer, John Kuriyan and David E. Shaw, "Architecture and Membrane Interactions of the EGF Receptor", Cell, Volume 152, Issue 3, January 31, 2013, pp. 557--569.
5. Abhinav Bhatelé, Sameer Kumar, Chao Mei, James C. Phillips, Gengbin Zheng, Laxmikant V. Kalé, "Overcoming Scaling Challenges in Biomolecular Simulations across Multiple Platforms", 22^nd IEEE International Parallel and Distributed Processing Symposium, Miami, FL, April 14--18, 2008, pp. 1--12.
6. Kevin J. Bowers, Edmond Chow, Huafeng Xu, Ron O. Dror, Michael P. Eastwood, Brent A. Gregersen, John L. Klepeis, Istvan Kolossvary, Mark A. Moraes, Federico D. Sacerdoti, John K. Salmon, Yibing Shan and David E. Shaw, "Scalable Algorithms for Molecular Dynamics Simulations on Commodity Clusters", 2006 ACM/IEEE Conference on Supercomputing, Tampa, FL, November 11--17, 2006.
7. Richard A. Buckingham, "The Classical Equation of State of Gaseous Helium, Neon and Argon", Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, Volume 168, Number 933, October 25, 1938, pp. 264--283.
8. Nicolas Calimet, Manuel Simoes, Jean-Pierre Changeux, Martin Karplus, Antoine Taly and Marco Cecchini, "A gating mechanism of pentameric ligand-gated ion channels", Proceedings of the National Academy of Sciences of the United States of America, Volume 110, Number 42, October 15, 2013, pp. 3987--3996.
9. David A. Case, Thomas E. Cheatham III, Tom Darden, Holger Gohlke, Ray Luo, Kenneth M. Merz Jr., Alexey Onufriev, Carlos Simmerling, Bing Wang and Robert J. Woods, "The Amber Biomolecular Simulation Programs", Journal of Computational Chemistry, Volume 26, Issue 16, December, 2005, pp. 1668--1688.
10. Edmond Chow, Charles A. Rendleman, Kevin J. Bowers, Ron O. Dror, Douglas H. Hughes, Justin Gullingsrud, Federico D. Sacerdoti and David E. Shaw, "Desmond Performance on a Cluster of Multicore Processors", Technical Report DESRES/TR-2008-01, https://www.deshawresearch.com/publications/Desmond%20Performance%20on%20a%20Cluster%20of%20Multicore%20Processors.pdf, retrieved February 8, 2014.
11. Giovanni Ciccotti and Jean-Paul Ryckaert, "Molecular Dynamics Simulation of Rigid Molecules", Computer Physics Reports, Volume 4, Issue 6, September--October, 1986, pp. 356--392.
12. Tom Darden, Darrin York and Lee Pederson, "Particle mesh Ewald: An N·log(N) method for Ewald sums in large systems", Journal of Chemical Physics, Volume 98, Number 12, June 15, 1993, pp. 10089--10092.
13. A. G. Donchev, V. D. Ozrin, M. V. Subbotin, O. V. Tarasov and V. I. Tarasov, "A quantum mechanical polarizable force field for biomolecular interactions", Proceedings of the National Academy of Sciences, Volume 102, Number 22, May 31, 2005, pp. 7829--7834.
14. Ron O. Dror, J. P. Grossman, Kenneth M. Mackenzie, Brian Towles, Edmond Chow, John K. Salmon, Cliff Young, Joseph A. Bank, Brannon Batson, Martin M. Deneroff, Jeffrey S. Kuskin, Richard H. Larson, Mark A. Moraes and David E. Shaw, "Exploiting 162-Nanosecond End-to-End Communication Latency on Anton", 2010 International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, LA, November 13--19, 2010.
15. Ron O. Dror, Robert M. Dirks, J. P. Grossman, Huafeng Xu and David E. Shaw, "Biomolecular Simulation: A Computational Microscope for Molecular Biology", Annual Review of Biophysics, Volume 41, June, 2012, pp. 429--452.
16. Ron O. Dror, Hillary F. Green, Celine Valant, David W. Borhani, James R. Valcourt, Albert C. Pan, Daniel H. Arlow, Meritxell Canals, J. Robert Lane, Raphaël Rahmani, Jonathan B. Baell, Patrick M. Sexton, Arthur Christopoulos and David E. Shaw, "Structural basis for modulation of a G-protein-coupled receptor by allosteric drugs", Nature, Volume 503, Number 7475, November 14, 2013, pp. 295--299.
17. Paul P. Ewald, "Die Berechnung optischer und elektrostatischer Gitterpotentiale", Annalen der Physik, Volume 369, Issue 3, 1921, pp. 253--287.
18. Blake G. Fitch, Aleksandr Rayshubskiy, Maria Eleftheriou, T. J. Christopher Ward, Mark Giampapa, Yuri Zhestkov, Michael C. Pitman, Frank Suits, Alan Grossfield, Jed Pitera, William Swope, Ruhong Zhou, Scott Feller and Robert S. Germain, "Blue Matter: Strong Scaling of Molecular Dynamics on Blue Gene/L", Lecture Notes in Computer Science, Volume 3992, 2006, pp. 846--854.
19. The Green 500, "The Green500 List -- November 2013", http://www.green500.org/lists/2013/11/top/green500_top_201311.xls, retrieved April 15, 2014.
20. J. P. Grossman, Jeffrey S. Kuskin, Joseph A. Bank, Michael Theobald, Ron O. Dror, Douglas J. Ierardi, Richard H. Larson, U. Ben Schafer, Brian Towles, Cliff Young and David E. Shaw, "Hardware Support for Fine-Grained Event-Driven Computation in Anton 2", 18^th International Conference on Architectural Support for Programming Languages and Operating Systems, Houston, TX, March 16--20, 2013, pp. 549--560.
21. Matt J. Harvey, Giovanni Giupponi and Gianni De Fabritiis, "ACEMD: Accelerated molecular dynamics simulations in the microsecond timescale", Journal of Chemical Theory and Computation, Volume 5, Issue 6, June 9, 2009, pp. 1632--1639.
22. Berk Hess, Carsten Kutzner, David van der Spoel and Erik Lindahl, "GROMACS 4: Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation", Journal of Chemical Theory and Computation, Volume 4, Issue 3, February 2008, pp. 435--447.
23. Morten Ø. Jensen, Vishwanath Jogini, David W. Borhani, Abba E. Leffler, Ron O. Dror and David E. Shaw, "Mechanism of Voltage Gating in Potassium Channels", Science, Volume 336, Number 6078, April 13, 2012, pp. 229--233.
24. Server Kasap and Khaled Benkrid, "Parallel Processor Design and Implementation for Molecular Dynamics Simulations on a FPGA-Based Supercomputer", Journal of Computers, Volume 7, Number 6, June, 2012, pp. 1312--1328.
25. Md. Ashfaquzzaman Khan, Matt Chiu and Martin C. Herbordt, "FPGA-Accelerated Molecular Dynamics", in Wim Vanderbauwhede and Khaled Benkrid (eds.), High-Performance Computing Using FPGAs, Springer Science+Business Media, LLC, 2013, pp. 105--135.
26. Sameer Kumar, Yanhua Sun and Laxmikant V. Kalé, "Acceleration of an Asynchronous Message Driven Programming Paradigm on IBM Blue Gene/Q", 27^th IEEE International Parallel and Distributed Processing Symposium, Boston, MA, May 20--24, 2013, pp. 689--699.
27. Jeffrey S. Kuskin, Cliff Young, J. P. Grossman, Brannon Batson, Martin M. Deneroff, Ron O. Dror and David E. Shaw, "Incorporating Flexibility in Anton, a Specialized Machine for Molecular Dynamics Simulation", 14^th Annual International Symposium on High-Performance Computer Architecture, Salt Lake City, UT, February 16--20, 2008, pp. 343--354.
28. Carsten Kutzner, Rossen Apostolov, Berk Hess and Helmut Grubmüller, "Scaling of the GROMACS 4.6 molecular dynamics code on SuperMUC", unpublished, 2014.
29. Richard H. Larson, John K. Salmon, Ron O. Dror, Martin M. Deneroff, Cliff Young, J. P. Grossman, Yibing Shan, John L. Klepeis and David E. Shaw, "High-Throughput Pairwise Point Interactions in Anton, a Specialized Machine for Molecular Dynamics Simulation", 14^th Annual International Symposium on High-Performance Computer Architecture, Salt Lake City, UT, February 16--20, 2008, pp. 331--342.
30. Jing Li, Saher A. Shaikh, Giray Enkavi, Po-Chao Wen, Zhijian Huang and Emad Tajkhorshid, "Transient formation of water-conducting states in membrane transporters", Proceedings of the National Academy of Sciences of the United States of America, Volume 110, Number 19, May 7, 2013, pp. 7696--7701.
31. Erik Lindahl, "Evolutions & Revolutions in Peta- and Exascale Biomolecular Simulation", Conference on Scientific Computing, Paphos, Cyprus, December 3--6, 2013.
32. Kresten Lindorff-Larsen, Stefano Piana, Ron O. Dror and David E. Shaw, "How Fast-Folding Proteins Fold", Science, Volume 334, Number 6055, 2011, pp. 517--520.
33. Siewert J. Marrink, H. Jelger Risselada, Serge Yefimov, D. Peter Tieleman and Alex H. de Vries, "The MARTINI Force Field: Coarse Grained Model for Biomolecular Simulations", Journal of Physical Chemistry B, Volume 111, Issue 27, July 12, 2007, pp. 7812--7824.
34. Tetsu Narumi, Ryutaro Susukita, Takahiro Koishi, Kenji Yasuoka, Hideaki Furusawa, Atsushi Kawai and Toshikazu Ebisuzaki, "1.34 Tflops Molecular Dynamics Simulation for NaCl with a Special-Purpose Computer: MDM", 2000 ACM/IEEE Conference on Supercomputing, Dallas, TX, November 5--10, 2000.
35. Jared Ostmeyer, Sudha Chakrapani, Albert C. Pan, Eduardo Perozo and Benoît Roux, "Recovery from slow inactivation in K+ channels is controlled by water molecules", Nature, Volume 501, Issue 7465, September 5, 2013, pp. 121--124.
36. James C. Phillips, Rosemary Braun, Wei Wang, James Gumbart, Emad Tajkhorshid, Elizabeth Villa, Christophe Chipot, Robert D. Skeel, Laxmikant V. Kalé and Klaus Schulten, "Scalable Molecular Dynamics with NAMD", Journal of Computational Chemistry, Volume 26, Issue 16, December, 2005, pp. 1781--1802.
37. Stefano Piana, Kresten Lindorff-Larsen and David E. Shaw, "Protein folding kinetics and thermodynamics from atomistic simulation", Proceedings of the National Academy of Sciences of the United States of America, Volume 109, Number 44, October 30, 2012, pp. 17845--17850.
38. Stefano Piana, Kresten Lindorff-Larsen and David E. Shaw, "Atomic-level description of ubiquitin folding", Proceedings of the National Academy of Sciences of the United States of America, Volume 110, Number 15, April 9, 2013, pp. 5915--5920.
39. Steve Plimpton, "Fast Parallel Algorithms for Short-Range Molecular Dynamics", Journal of Computational Physics, Volume 117, Issue 1, March 1, 1995, pp. 1--19.
40. Cristian Predescu, Ross A. Lippert, Adam K. Lerer, Brian Towles, J. P. Grossman, Robert M. Dirks and David E. Shaw, "The u-series: A separable, Gaussian-based decomposition for electrostatics computation", unpublished, 2014.
41. Daniele P. Scarpazza, Douglas J. Ierardi, Adam K. Lerer, Kenneth M. Mackenzie, Albert C. Pan, Joseph A. Bank, Edmond Chow, Ron O. Dror, J. P. Grossman, Daniel Killebrew, Mark A. Moraes, Cristian Predescu, John K. Salmon and David E. Shaw, "Extending the generality of molecular dynamics simulations on a special-purpose machine", 27^th IEEE International Parallel and Distributed Processing Symposium, Boston, MA, May 20--24, 2013, pp. 933--945.
42. Ronald Scrofano, Maya B. Gokhale, Frans Trouw and Viktor K. Prasanna, "Accelerating Molecular Dynamics Simulations with Reconfigurable Computers", IEEE Transactions on Parallel and Distributed Systems, Volume 19, Number 6, June, 2008, pp. 764--778.
43. Yibing Shan, John L. Klepeis, Michael P. Eastwood, Ron O. Dror and David E. Shaw, "Gaussian split Ewald: A fast Ewald mesh method for molecular simulation", Journal of Chemical Physics, Volume 122, Number 5, February 1, 2005, pp. 054101:1--13.
44. David E. Shaw, "A Fast, Scalable Method for the Parallel Evaluation of Distance-Limited Pairwise Particle Interactions", Journal of Computational Chemistry, Volume 26, Issue 13, October, 2005, pp. 1318--1328.
45. David E. Shaw, Martin M. Deneroff, Ron O. Dror, Jeffrey S. Kuskin, Richard H. Larson, John K. Salmon, Cliff Young, Brannon Batson, Kevin J. Bowers, Jack C. Chao, Michael P. Eastwood, Joseph Gagliardo, J. P. Grossman, C. Richard Ho, Douglas J. Ierardi, István Kolossváry, John L. Klepeis, Timothy Layman, Christine McLeavey, Mark A. Moraes, Rolf Mueller, Edward C. Priest, Yibing Shan, Jochen Spengler, Michael Theobald, Brian Towles and Stanley C. Wang, "Anton, a Special-Purpose Machine for Molecular Dynamics Simulation", 34^th Annual International Symposium on Computer Architecture, San Diego, CA, June 9--13, 2007, pp. 1--12.
46. David E. Shaw, Ron O. Dror, John K. Salmon, J. P. Grossman, Kenneth M. Mackenzie, Joseph A. Bank, Cliff Young, Martin M. Deneroff, Brannon Batson, Kevin J. Bowers, Edmond Chow, Michael P. Eastwood, Douglas J. Ierardi, John L. Klepeis, Jeffrey S. Kuskin, Richard H. Larson, Kresten Lindorff-Larsen, Paul Maragakis, Mark A. Moraes, Stefano Piana, Yibing Shan and Brian Towles, "Millisecond-Scale Molecular Dynamics Simulations on Anton", 2009 Conference on High Performance Computing, Networking, Storage and Analysis, Portland, OR, November 14--20, 2009.
47. David E. Shaw, Paul Maragakis, Kresten Lindorff-Larsen, Stefano Piana, Ron O. Dror, Michael P. Eastwood, Joseph A. Bank, John M. Jumper, John K. Salmon, Yibing Shan and Willy Wriggers, "Atomic-Level Characterization of the Structural Dynamics of Proteins", Science, Volume 330, Number 6002, October 15, 2010, pp. 341--346.
48. Yanhua Sun, Gengbin Zheng, Chao Mei, Eric J. Bohm, James C. Phillips and Laxmikant V. Kalé, "Optimizing Fine-grained Communication in a Biomolecular Simulation Application on Cray XK6", 2012 International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, November 10--16, 2012.
49. Johan Sund, Martin Andér and Johan Åqvist, "Principles of stop-codon reading on the ribosome", Nature, Volume 465, Issue 7300, June 17, 2010, pp. 947--950.
50. Makoto Taiji, Junichiro Makino, Akihiro Shimizu, Ryo Takada, Toshikazu Ebisuzaki and Daiichiro Sugimoto, "MD-GRAPE: a parallel special-purpose computer system for classical molecular dynamics simulations", 6^th Joint EPS-APS International Conference on Physics Computing, Lugano, Switzerland, August 22--26, 1994, pp. 609--612.
51. Makoto Taiji, Noriyuki Futatsugi, Tetsu Narumi, Atsushi Suenaga, Yousuke Ohno, Naoki Takada and Akihiko Konagaya, "Protein Explorer: A Petaflops Special-Purpose Computer System for Molecular Dynamics Simulations", 2003 ACM/IEEE Conference on Supercomputing, Phoenix, AZ, November 15--21, 2003.
52. Brian Towles, J. P. Grossman, Brian L. Greskamp and David E. Shaw, "Unifying on-chip and inter-node switching within the Anton 2 network", 2014 International Symposium on Computer Architecture, Minneapolis, MN, June 14--18, 2014, pp. 1--12.
53. Mike Wu and Ross Walker, "Amber 12 NVIDIA GPU acceleration support", http://ambermd.org/gpus/benchmarks.htm, retrieved April 2, 2014.
54. Cliff Young, Joseph A. Bank, Ron O. Dror, J. P. Grossman, John K. Salmon and David E. Shaw, "A 32×32×32, spatially distributed 3D FFT in four microseconds on Anton", 2009 Conference on High Performance Computing, Networking, Storage and Analysis, Portland, OR, November 14--20, 2009.
55. Gongpu Zhao, Juan R. Perilla, Ernest L. Yufenyuy, Xin Meng, Bo Chen, Jiying Ning, Jinwoo Ahn, Angela M. Gronenborn, Klaus Schulten, Christopher Aiken and Peijun Zhang, "Mature HIV-1 capsid structure by cryo-electron microscopy and all-atom molecular dynamics", Nature, Volume 497, Issue 7451, May 30, 2013, pp. 643--646.

Index Terms

• Computer systems organization

  • Architectures

    • Distributed architectures

      • Grid computing

    • Parallel architectures

      • Multicore architectures

    • Serial architectures

      • Superscalar architectures

  • Dependable and fault-tolerant systems and networks

• Computing methodologies

  • Modeling and simulation

    • Simulation types and techniques

      • Massively parallel and high-performance simulations

• General and reference

  • Cross-computing tools and techniques

    • Performance

• Networks

  • Network performance evaluation

• Software and its engineering

  • Software organization and properties

    • Software system structures

      • Distributed systems organizing principles

        • Grid computing