A Framework for Distributed Data-Parallel Execution in the Kepler Scientific Workflow System

Distributed Data-Parallel (DDP) patterns such as MapReduce have become increasingly popular as solutions to facilitate data-intensive applications, resulting in a number of systems supporting DDP workows. Yet, applications or workows built using these patterns are usually tightly-coupled with the underlying DDP execution engine they select. We present a framework for distributed data-parallel execution in the Kepler scientic workow system that enables users to easily switch between different DDP execution engines. We describe a set of DDP actors based on DDP patterns and directors for DDP workow executions within the presented framework. We demonstrate how DDP workows can be easily composed in the Kepler graphic user interface through the reuse of these DDP actors and directors and how the generated DDP workows can be executed in different distributed environments. Via a bioinformatics usecase, we discuss the usability of the proposed framework and validate its execution scalability