Abstract
In the era of rapid experimental expansion data analysis needs are rapidly outpacing the capabilities of small institutional clusters and looking to integrate HPC resources into their workflow. We propose one way of reconciling on-demand needs of experimental analytics with the batch managed HPC resources within a system that dynamically moves nodes between an on-demand cluster configured with cloud technology (OpenStack) and a traditional HPC cluster managed by a batch scheduler (Torque). We evaluate this system experimentally both in the context of real-life traces representing two years of a specific institutional need, and via experiments in the context of synthetic traces that capture generalized characteristics of potential batch and on-demand workloads. Our results for the real-life scenario show that our approach could reduce the current investment in on-demand infrastructure by 82% while at the same time improving the mean batch wait time almost by an order of magnitude (8x).
Original language | English (US) |
---|---|
Title of host publication | Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 |
Publisher | Institute of Electrical and Electronics Engineers Inc. |
Pages | 493-503 |
Number of pages | 11 |
ISBN (Electronic) | 9781538683842 |
DOIs | |
State | Published - Jul 2 2018 |
Event | 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 - Dallas, United States Duration: Nov 11 2018 → Nov 16 2018 |
Publication series
Name | Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 |
---|
Conference
Conference | 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 |
---|---|
Country/Territory | United States |
City | Dallas |
Period | 11/11/18 → 11/16/18 |
Bibliographical note
Publisher Copyright:© 2018 IEEE.
Keywords
- Computers and information processing
- Distributed computing
- Grid computing
- Metacomputing