TY - GEN
T1 - Eliminating the middleman
T2 - 17th International Symposium on High Performance Distributed Computing 2008, HPDC'08
AU - Barker, Adam
AU - Weissman, Jon B.
AU - Van Hemert, Jano
PY - 2008
Y1 - 2008
N2 - Efficiently executing large-scale, data-intensive workflows such as Montage must take into account the volume and pattern of communication. When orchestrating data-centric workflows, centralised servers common to standard workflow systems can become a bottleneck to performance. However, standards-based workflow systems that rely on centralisation, e.g., Web service based frameworks, have many other benefits such as a wide user base and sustained support. This paper presents and evaluates a light-weight hybrid architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Furthermore our architecture is standards compliment, flexible and is a non-disruptive solution; service definitions do not have to be altered prior to enactment. Our architecture could be realised within any existing workflow framework, in this paper, we focus on a Web service based framework. Taking inspiration from Montage, a number of common workflow patterns (sequence, fan-in and fan-out), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 2-4 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using our hybrid architecture increases as the complexity of a workflow grows.
AB - Efficiently executing large-scale, data-intensive workflows such as Montage must take into account the volume and pattern of communication. When orchestrating data-centric workflows, centralised servers common to standard workflow systems can become a bottleneck to performance. However, standards-based workflow systems that rely on centralisation, e.g., Web service based frameworks, have many other benefits such as a wide user base and sustained support. This paper presents and evaluates a light-weight hybrid architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Furthermore our architecture is standards compliment, flexible and is a non-disruptive solution; service definitions do not have to be altered prior to enactment. Our architecture could be realised within any existing workflow framework, in this paper, we focus on a Web service based framework. Taking inspiration from Montage, a number of common workflow patterns (sequence, fan-in and fan-out), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 2-4 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using our hybrid architecture increases as the complexity of a workflow grows.
KW - Decentralised orchestration
KW - Workflow optimisation
UR - http://www.scopus.com/inward/record.url?scp=57349123480&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=57349123480&partnerID=8YFLogxK
U2 - 10.1145/1383422.1383430
DO - 10.1145/1383422.1383430
M3 - Conference contribution
AN - SCOPUS:57349123480
SN - 9781595939975
SN - 9781595939975
T3 - Proceedings of the 17th International Symposium on High Performance Distributed Computing 2008, HPDC'08
SP - 55
EP - 64
BT - Proceedings of the 17th International Symposium on High Performance Distributed Computing 2008, HPDC'08
Y2 - 23 June 2008 through 27 June 2008
ER -