Distributed stream processing systems need to satisfy application QoS requirements despite the shared, dynamic, and large-scale environment. Designed and implemented in Synergy techniques for composing distributed stream processing applications with QoS demands. Synergy enhances QoS provision and reduces resource load by efficiently reusing both data streams and processing components. To achieve this goal Synergy provides a set of fully distributed algorithms to discover and evaluate the reusability of available data streams and processing components when instantiating new stream applications. For QoS provision, Synergy performs QoS impact projection to examine whether the shared processing can cause QoS violations on currently running applications. Projection is based on queueing theoretical models for both regular and bursty traffic. For initial component deployment Synergy employs a decentralized replica placement protocol that aims to maximize availability, while respecting resource constraints, and making performance-aware placement decisions.
Managing the load of the nodes of a large-scale, dynamic distributed stream processing system in real-time and without centralized supervision is challenging. Designed and implemented in Synergy a self-managing resource monitoring architecture for identifying and relieving overloaded nodes. Monitoring responsibilities are shared among all nodes using a completely decentralized DHT-based architecture. Nodes proactively predict application QoS violations using a statistical forecasting framework based on auto- and cross-correlation, as well as linear regression. To alleviate hot-spots, nodes autonomously migrate the execution of stream processing components using a non-disruptive migration protocol. Synergy's performance evaluation over PlanetLab involves the implementation of a network traffic monitoring application operating on real streaming data.
Managing large-scale distributed real-time applications. Satisfying end-to-end QoS requirements in large-scale distributed real-time applications is challenging, due to the unpredictability and heterogeneity of the environment. Focusing on media streaming and transcoding as an example, co-designed in C++ a decentralized architecture that enables nodes to collaborate to offer composite applications. A resource allocation algorithm distributes the processing and communication loads fairly, while satisfying application end-to-end QoS demands. Nodes adapt to variable resource conditions by optimizing their resource usage locally, selecting from discrete QoS output levels. Multiple nodes participating in a composite application coordinate their QoS through feedback from the service receiver.
Efficiently locating data or services in a fully decentralized, self-organizing, unstructured overlay network is challenging, due to the large scale and the lack of global view. Designed in Java a content-driven routing mechanism that guides queries to peers most probable to provide results. Intelligently routing queries is based on Bloom filter synopses of peers' content. Adaptive dissemination algorithms propagate synopses according to peers' interactions. A decentralized trust management architecture enables peers to evaluate reputation information and avoid lying and colluding.
Designed, implemented in Java, evaluated, and documented a replication middleware for distributed, multi-tier, server architectures. Proposed and incorporated in the middleware an efficient, distributed strong consistency protocol. Quantified the server replication and data partitioning performance benefits, as well as the consistency overhead, using the TPC-W transactional web commerce benchmark.
Designed, implemented in C++, and documented an event-driven software prototype of a collaborative spam filter that employs a distributed protocol to defend against sybil attacks.
Designed, implemented in C++, and documented a logging mechanism used for asynchronous replication in a distributed disk array.
Designed and implemented in C a protocol for dynamic memory page migration across the nodes of a Software Distributed Shared Memory System. Dynamic page migration improves performance by increasing locality and adaptability, while remaining transparent to the application programmer.
Analyzed the H.323 protocol family, used for multimedia applications (VoIP) in packet switched networks, and summarized the results in a technical report, including detailed protocol description and performance evaluation of applications under IPv6 in Solaris.