Addressing the Challenges of I/O Variability in Post-Petascale HPC Simulations

Abstract:
Million-core supercomputers have become a reality in 2012 with LLNL's Sequoia supercomputer. Following Moore's law, Exascale machines (capable of 10^18 floating point operations per second) are expected by 2018. Such an immense computational power is used in many research areas, including earth sciences, biology, climate, or cosmology, where large-scale simulations are conducted to understand physical phenomena better. But larger simulations on larger machines lead to the production of larger amounts of data that need to be efficiently stored and processed in order to retrieve scientific insights. In this talk, we present some of the new approaches to data management that we proposed during our PhD. We first introduce the Damaris approach, which leverages the multicore nature of recent machines to offload data-management tasks into dedicated cores. We study in particular how Damaris can be used to hide the variability in I/O (Input/Output) performance, and improve the scalability as well as the I/O throughput of HPC applications. We then present how Damaris was extended to provide in situ visualization capabilities to simulations in a way that does not impact their performance. Finally we study the effect of multi-application I/O contention on the performance of parallel storage systems.