The Circulate architecture: Avoiding workflow bottlenecks caused by centralised orchestration

Adam Barker*, Jon B. Weissman, Jano I. van Hemert

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

As the number of services and the size of data involved in workflows increases, centralised orchestration techniques are reaching the limits of scalability. In the classic orchestration model, all data passes through a centralised engine, which results in unnecessary data transfer, wasted bandwidth and the engine to become a bottleneck to the execution of a workflow. This paper presents and evaluates the Circulate architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Circulate could be realised within any existing workflow framework, in this paper, we focus on WS-Circulate, a Web services based implementation. Taking inspiration from the Montage workflow, a number of common workflow patterns (sequence, fan-in and fan-out), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 24 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using the Circulate architecture increases as the complexity of a workflow grows.
Original languageEnglish
Pages (from-to)221-235
Number of pages15
JournalCluster Computing
Volume12
DOIs
Publication statusPublished - 2009

Keywords / Materials (for Non-textual outputs)

  • decentralised orchestration
  • web services
  • workflow
  • workflow optimisation

Fingerprint

Dive into the research topics of 'The Circulate architecture: Avoiding workflow bottlenecks caused by centralised orchestration'. Together they form a unique fingerprint.

Cite this