Affiliations: Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL USA. [email protected] | Computer Science Department, University of Chicago, Chicago, IL USA. [email protected] | Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL USA. [email protected] | Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL USA. [email protected] | Computation Institute, University of Chicago & Argonne National Laboratory, Chicago, IL USA. [email protected] | Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL USA. [email protected] | Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL USA. [email protected]
Note: [] Address for correspondence: Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL USA
Abstract: Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper layer of many loosely coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program. “Many-task” programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tightly coupled parallelism at the lower level through multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and intertask data movement is a significant performance challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems.
