Scientific workflow scheduling in non-dedicated heterogeneous multicluster with advance reservations

Subtitle:

Article type: Research Article

Authors: Zhang, Jinghui^* | Luo, Junzhou | Dong, Fang

Affiliations: School of Computer Science and Engineering, Southeast University, Nanjing, Jiangsu, China

Correspondence: [*] Corresponding author: Jinghui Zhang, School of Computer Science and Engineering, Southeast University, Nanjing 211189, Jiangsu, China. E-mail: [email protected]

Abstract: Scientific workflow structured as Parallel Task Graphs (PTG) exhibits both data and task parallelism, and arises in scientific as well as in industrial domains. Efficient scheduling of such workflow on a multicluster platform has been a long-standing challenge. Most of previous work on PTG scheduling primarily focused on dedicated multicluster. In this paper, a novel scheduling algorithm known as the Moldable Task Duplication (MTD) is applied to non-dedicated heterogeneous multicluster platform with advance reservations. A novel method for the calculation of dynamic critical path that handles the availability fluctuation of multicluster and the moldability of scientific workflow's data-parallel tasks is proposed. A moldable task duplication strategy with migration of pre-duplicated predecessor tasks is developed to fully exploit the flexibility of data-parallel tasks. Simulations spanning a broad range of scientific workflow and multicluster platform settings are performed in order to verify the proposed approach. The numerical results show that MTD can achieve better average PTG makespan than previous methods in the literature.

Keywords: Scientific workflow, heterogeneous multicluster, scheduling, advance reservation, task duplication, task migration

DOI: 10.3233/ICA-150489

Journal: Integrated Computer-Aided Engineering, vol. 22, no. 3, pp. 261-280, 2015