Muralidharan, Servesh and Casey, Kevin and Gregg, David (2013) A Parallel Runtime Framework for Communication Intensive Stream Applications. In: 2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications. IEEE, pp. 1179-1187. ISBN 9780769550220

Preview

Download (238kB) | Preview

Abstract

Stream applications are often limited in their performance by their underlying communication system. A typical implementation relies on the operating system to handle the majority of network operations. In such cases, the communication stack, which was not designed to handle tremendous amounts of data, acts as a bottleneck and restricts the performance of the application. In this paper, we propose a parallel runtime framework that integrates the communication operations with stream applications, and provides a common parallel processing engine that can execute both the communication and computation operations in parallel on multicore processors. We place an emphasis on the low-level details required to implement such a framework, but also provide some guidelines on how an application programmer can employ the framework. Our runtime system uses a set of operations represented as filters to perform the relevant computations on the data stream. Filters that handle the application specific operations are categorized as computation filters and those that transform data to and from network devices are classified as communication filters. Computation filters are designed by the user and are specific to the application. Communication filters are provided by the runtime system and are built using system software that allows direct access to network hardware. Such system software allows the network operations to be performed by the runtime system in parallel, leading to better communication performance. Applications that are designed for this framework are built by constructing application specific computation filters and then connecting them to the communication filters provided by the runtime system. This abstracts the low-level programming of network adapters and protocols by the application developer, making it easier to build stream applications that take advantage of the improved communication performance. Moreover, by dynamically replicating and statically scheduling such filters on the given multicore architecture, it is possible for the runtime system to process multiple data streams in parallel. We are able to parallelize stream applications and achieve speedups of more than a factor of eight in all the applications we tested. The results show that our system scales to as many parallel processes as there are cores on our computer, and achieves speedups of more than a factor of ten in some cases compared to sequential implementations.

Item Type:	Book Section
Keywords:	Streaming model; data parallelism; clusters;
Academic Unit:	Faculty of Science and Engineering > Computer Science Faculty of Science and Engineering > Research Institutes > Hamilton Institute
Item ID:	10175
Identification Number:	https://doi.org/10.1109/TrustCom.2013.142
Depositing User:	Hamilton Editor
Date Deposited:	05 Nov 2018 15:44
Publisher:	IEEE
Refereed:	Yes
Funders:	Irish Research Council for Science Engineering and Technology (IRCSET), IBM Ireland
URI:	https://ieeexplore.ieee.org
Use Licence:	This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

Repository Staff Only(login required)

Item control page

Downloads

Origin of downloads