Fundamenta Informaticae - Volume 128, issue 3

Authors: Hidders, Jan | Missier, Paolo | Sroka, Jacek | Van den Bussche, Jan

Article Type: Other

DOI: 10.3233/FI-2013-945

Citation: Fundamenta Informaticae, vol. 128, no. 3, pp. i-iii, 2013

Authors: Abouelhoda, Mohamed | Issa, Shady | Ghanem, Moustafa

Article Type: Research Article

Abstract: Cloud-based scientific workflow systems can play an important role in the development of cost effective bioinformatics analysis applications. So far, most efforts for supporting cloud computing in such workflow systems have focused on simply porting them to the cloud environment. The next due steps are to optimize these systems to exploit the advantages of the cloud computing model, basically in terms of managing resource elasticity and the associated business model. In this paper, we introduce new advancements in designing scalable and cost-effective workflows in the cloud using the Tavaxy workflow system, focusing on genome analysis applications. We provide an overview …of the system and describe its key cloud features including the configuration and execution of complete workflows and/or specific sub-workflows in the cloud. Taking real world examples, we demonstrate the key elasticity management features of the system. These features are designed to support two common scenarios: (1) minimizing workflow execution time under budget constraints and (2) minimizing budget spend under workflow deadline constraints. We evaluate the effectiveness of our approach by conducting experiments on the Amazon EC2 cloud with dynamic pricing and variable heterogeneous resource allocation. Show more

Keywords: Workflow systems, cloud computing, Tavaxy, bioinformatics, Next Generation Sequencing data

DOI: 10.3233/FI-2013-946

Citation: Fundamenta Informaticae, vol. 128, no. 3, pp. 255-280, 2013

Authors: Płóciennik, Marcin | Żok, Tomasz | Altintas, Ilkay | Wang, Jianwu | Crawl, Daniel | Abramson, David | Imbeaux, Frederic | Guillerminet, Bernard | Lopez-Caniego, Marcos | Plasencia, Isabel Campos | Pych, Wojciech | Ciecieląg, Pawel | Palak, Bartek | Owsiak, Michał | Frauel, Yann | ITM-TF Contributors

Article Type: Research Article

Abstract: The Kepler scientific workflow system enables creation, execution and sharing of workflows across a broad range of scientific and engineering disciplines while also facilitating remote and distributed execution of workflows. In this paper, we present and compare different approaches to distributed execution of workflows using the Kepler environment, including a distributed data-parallel framework using Hadoop and Stratosphere, and Cloud and Grid execution using Serpens, Nimrod/K and Globus actors. We also present real-life applications in computational chemistry, bioinformatics and computational physics to demonstrate the usage of different distributed computing capabilities of Kepler in executable workflows. We further analyze the differences of …each approach and provide a guidance for their applications. Show more

Keywords: Kepler, scientific workflow, distributed execution

DOI: 10.3233/FI-2013-947

Citation: Fundamenta Informaticae, vol. 128, no. 3, pp. 281-302, 2013

Authors: Simitsis, Alkis | Wilkinson, Kevin | Dayal, Umeshwar

Article Type: Research Article

Abstract: To remain competitive, enterprises are evolving in order to quickly respond to changing market conditions and customer needs. In this new environment, a single centralized data warehouse is no longer sufficient. Next generation business intelligence involves data flows that span multiple, diverse processing engines, that contain complex functionality like data/text analytics, machine learning operations, and that need to be optimized against various objectives. A common example is the use of Hadoop to analyze unstructured text and merging these results with relational database queries over the data warehouse. We refer to these multi-engine analytic data flows as hybrid flows. Currently, it …is a cumbersome task to create and run hybrid flows. Custom scripts must be written to dispatch tasks to the individual processing engines and to exchange intermediate results. So, designing correct hybrid flows is a challenging task. Optimizing such flows is even harder. Additionally, when the underlying computing infrastructure changes, existing flows likely need modification and reoptimization. The current, ad-hoc design approach cannot scale as hybrid flows become more commonplace. To address this challenge, we are building a platform to design and manage hybrid flows. It supports the logical design of hybrid flows in which implementation details are not exposed. It generates code for the underlying processing engines and orchestrates their execution. But the key enabling technology in the platform is an optimizer that converts the logical flow to an executable form that is optimized for the underlying infrastructure according to user-specified objectives. In this paper, we describe challenges in designing the optimizer and our solutions. We illustrate the optimizer through a real-world use case. We present a logical design and optimized designs for the use case. We show how the performance of the use case varies depending on the system configuration and how the optimizer is able to generate different optimized flows for different configurations. Show more

Keywords: Optimization, Data Flows, Databases, Map-Reduce

DOI: 10.3233/FI-2013-948

Citation: Fundamenta Informaticae, vol. 128, no. 3, pp. 303-335, 2013

Authors: Wozniak, Justin M. | Armstrong, Timothy G. | Maheshwari, Ketan | Lusk, Ewing L. | Katz, Daniel S. | Wilde, Michael | Foster, Ian T.

Article Type: Research Article

Abstract: Efficiently utilizing the rapidly increasing concurrency of multi-petaflop computing systems is a significant programming challenge. One approach is to structure applications with an upper layer of many loosely coupled coarse-grained tasks, each comprising a tightly-coupled parallel function or program. “Many-task” programming models such as functional parallel dataflow may be used at the upper layer to generate massive numbers of tasks, each of which generates significant tightly coupled parallelism at the lower level through multithreading, message passing, and/or partitioned global address spaces. At large scales, however, the management of task distribution, data dependencies, and intertask data movement is a significant performance …challenge. In this work, we describe Turbine, a new highly scalable and distributed many-task dataflow engine. Turbine executes a generalized many-task intermediate representation with automated self-distribution and is scalable to multi-petaflop infrastructures. We present here the architecture of Turbine and its performance on highly concurrent systems. Show more

Keywords: dataflow language, Swift, ADLB, MPI, Turbine

DOI: 10.3233/FI-2013-949

Citation: Fundamenta Informaticae, vol. 128, no. 3, pp. 337-366, 2013