English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Scientific applications in HPC environment are more com-plex and moredata-intensive nowadays. Scientists usually rely on workflow system to managethe complexity: simply define multiple processing steps into a single scriptand let the work-flow systems compile it and schedule all tasks accordingly.Numerous workflow systems have been proposed and widely used, like Galaxy,Pegasus, Taverna, Kepler, Swift, AWE, etc., to name a few examples.  Traditionally, scientific workflow systems work with parallel file systems,like Lustre, PVFS, Ceph, or other forms of remote shared storage systems. Assuch, the data (including the intermediate data generated during workflowexecution) need to be transferred back and forth between compute nodes andstorage systems, which introduces a significant performance bottleneck on I/Ooperations. Along with the enlarging perfor-mance gap between CPU and storagedevices, this bottleneck is expected to be worse.  Recently, we have introduced a new concept of Compute-on-Data-Path to allowtasks and data binding to be more efficient to reduce the data movement cost.To workflow systems, the key is to exploit the data locality in HPC storagehierarchy: if the datasets are stored in compute nodes, near the workflowtasks, then the task can directly access them with better performance with lessnetwork usage. Several recent studies have been done regarding building such ashared storage system, utilizing compute node resources, to serve HPC workflowswith locality, such as Hercules [1] and WOSS [2] etc. In this research, wefurther argue that providing a compute-node side storage system is notsufficient to fully exploit data locality. A cross-layer solution combiningstorage system, compiler, and runtime is necessary. We take Swift/T [3], aworkflow system for data-intensive applications, as a prototype platform todemonstrate such a cross-layer solution

A Cross-Layer Solution in Scientific Workflow System for Tackling Data Movement Challenge