Affiliations: Information Sciences Institute, University of Southern California, Marina Del Rey, CA, USA. E-mails: {gideon, deelman, vahi, gmehta}@isi.edu
Note: [] Corresponding author: Ewa Deelman, Information Sciences Institute, University of Southern California, 4676 Admiralty Way, Suite 1001, Marina Del Rey, CA 90292, USA. Tel.: +1 310 448 8408; Fax: +1 310 823 6714; E-mail: [email protected].
Abstract: The development of grid and workflow technologies has enabled complex, loosely coupled scientific applications to be executed on distributed resources. Many of these applications consist of large numbers of short-duration tasks whose runtimes are heavily influenced by delays in the execution environment. Such applications often perform poorly on the grid because of the large scheduling overheads commonly found in grids. In this paper we present a provisioning system based on multi-level scheduling that improves workflow runtime by reducing scheduling overheads. The system reserves resources for the exclusive use of the application, and gives applications control over scheduling policies. We describe our experiences with the system when running a suite of real workflow-based applications including in astronomy, earthquake science, and genomics. Provisioning resources with Corral ahead of the workflow execution, reduced the runtime of the astronomy application by up to 78% (45% on average) and of a genome mapping application by an order of magnitude when compared to traditional methods. We also show how provisioning can benefit applications both on a small local cluster as well as a large-scale campus resource.
Keywords: Scientific workflows, grid computing, distributed computing, high-throughput computing, web services, pilot jobs, glideins
