Abstract
This chapter describes Aneka-Federation, a decentralized and distributed system that combines enterprise Clouds, overlay networking, and structured peer-to-peer techniques to create scalable wide-area networking of compute nodes for high-throughput computing. The Aneka-Federation integrates numerous small scale Aneka Enterprise Cloud services and nodes that are distributed over multiple control and enterprise domains as parts of a single coordinated resource leasing abstraction. The system is designed with the aim of making distributed enterprise Cloud resource integration and application programming flexible, efficient, and scalable. The system is engineered such that it: enables seamless integration of existing Aneka Enterprise Clouds as part of single wide-area resource leasing federation; self-organizes the system components based on a structured peer-to-peer routing methodology; and presents end-users with a distributed application composition environment that can support variety of programming and execution models. This chapter describes the design and implementation of a novel, extensible and decentralized peer-to-peer technique that helps to discover, connect and provision the services of Aneka Enterprise Clouds among the users who can use different programming models to compose their applications. Evaluations of the system with applications that are programmed using the Task and Thread execution models on top of an overlay of Aneka Enterprise Clouds have been described here.
TopIntroduction
Wide-area overlays of enterprise Grids (Luther, Buyya, Ranjan, & Venugopal, 2005; Andrade, Cirne, Brasileiro, & Roisenberg, 2003; Butt, Zhang, & Hu, 2003; Mason & Kelly, 2005) and Clouds (Amazon Elastic Compute Cloud, 2008; Google App Engine, 2008; Microsoft Live Mesh, 2008; Buyya, Yeo, Venugopal, 2008) are an appealing platform for the creation of high-throughput computing resource pools and cross-domain virtual organizations. An enterprise Cloud1 is a type of computing infrastructure that consists of a collection of inter-connected computing nodes, virtualized computers, and software services that are dynamically provisioned among the competing end-user’s applications based on their availability, performance, capability, and Quality of Service (QoS) requirements. Various enterprise Clouds can be pooled together to form a federated infrastructure of resource pools (nodes, services, virtual computers). In a federated organisation: (i) every participant gets access to much larger pools of resources; (ii) the peak-load handling capacity of every enterprise Cloud increases without having the need to maintain or administer any additional computing nodes, services, and storage devices; and (iii) the reliability of a enterprise Cloud is enhanced as a result of multiple redundant clouds that can efficiently tackle disaster condition and ensure business continuity.
Emerging enterprise Cloud applications and the underlying federated hardware infrastructure (Data Centers) are inherently large, with heterogeneous resource types that may exhibit temporal resource conditions. The unique challenges in efficiently managing a federated Cloud computing environment include:
- •
Large scale: composed of distributed components (services, nodes, applications, users, virtualized computers) that combine together to form a massive environment. These days enterprise Clouds consisting of hundreds of thousands of computing nodes are common (Amazon Elastic Compute Cloud, 2008; Google App Engine, 2008; Microsoft Live Mesh, 2008) and hence federating them together leads to a massive scale environment;
- •
Resource contention: driven by the resource demand pattern and a lack of cooperation among end-user’s applications, particular set of resources can get swamped with excessive workload, which significantly undermines the overall utility delivered by the system; and
- •
Dynamic: the components can leave and join the system at will.
The aforementioned characteristics of the infrastructure accounts to significant development, system integration, configuration, and resource management challenges. Further, the end-users follow a variety of programming models to compose their applications. In other words, in order to efficiently harness the computing power of enterprise Cloud infrastructures (Chu, Nandiminti, Jin, Venugopal, & Buyya, 2007; Amazon Elastic Compute Cloud, 2008; Google App Engine, 2008; Microsoft Live Mesh, 2008), software services that can support high level of scalability, robustness, self-organization, and application composition flexibility are required.
Key Terms in this Chapter
Enterprise Cloud: An enterprise Cloud is a type of computing infrastructure that consists of a collection of inter-connected computing nodes, virtualized computers, and software services that are dynamically provisioned among the competing end-user’s applications based on their availability, performance, capability, and Quality of Service (QoS) requirements.
Resource Discovery: Resource discovery activity involve searching for the appropriate service, node, or data type that match the requirements of applications such as file sharing, Grid applications, Cloud applications etc. The resource discovery methods can be engineered based on various network models including centralized, decentralized, and hierarchical with varying degree of scalability, fault-tolerance, and network performance.
Decentralized Systems: A distributed Cloud system configuration is considered to be decentralized if none of the components in the system are more important than the others, in case that one of the component fails, then it is neither more nor less harmful to the system than caused by the failure of any other component in the system.
Distributed Hash Table (DHT): A DHT is a data structure that associates unique index with a data. Entries in the DHTs are stored as (index, data) pair. A data can be looked up within a logarithmic overlay routing hops and messages bound if the corresponding index is known. DHTs are self-managing in their behavior as they can dynamically adapt to leave, join and failure of nodes or services in the system. Recently, DHTs have been applied to build Internet scale systems that involve hundreds of thousands of components (node, service, data, and file).
Aneka-Federation: The Aneka-Federation integrates numerous small scale Aneka Enterprise Cloud services and nodes that are distributed over multiple control and enterprise domains as parts of a single coordinated resource leasing abstraction.
Overlay Networking: A logical inter-connection of services, nodes, devices, sensors, instruments, and data hosts at application layer (under TCP/IP model) over an infrastructure of physical network routing systems such as the Internet or Local Area Network (LAN). In overlays, the routing and forwarding of messages between services is done on the basis of their relationship in the logical space, while the messages are actually transported through the physical links.
Multi-Dimensional Queries: Complex web services, Grid resource characteristics, and Cloud services are commonly represented by a number of attributes such as service type, hardware (processor type, speed), installed software (libraries, operating system), service name, security (authentication and authorization control); efficiently discovering the aforementioned services with deterministic guarantees in decentralized and scalable manner requires lookup queries to encapsulate search values for each attribute (search dimension) . The search is resolved by satisfying the constraints for the values expressed in each dimension, hence resulting in multi-dimensional queries that search for values in a virtual space that has multiple dimensions (x, y, z…).