With huge advancement of Cloud computing and networking technologies, a wide spectrum of Cloud services have been developed by various providers and utilized by numerous organizations as indispensable ingredients of their information systems. Cloud service performance has a significant impact on performance of the future information infrastructure.
Thorough evaluation on Cloud service performance is crucial and beneficial, In the business world, for an innovative idea to be useful, to both service providers and consumers; thus forming an active research area. Some key technologies for Cloud computing, such as virtualization and the Service-Oriented Architecture (SOA), bring in special challenges to service performance evaluation. A tremendous amount of effort has been put by the research community to address these challenges and exciting progress has been made.
Introduction
The rapid advances in Cloud computing technologies in the past decade have enabled a wide spectrum of Cloud services. Cloud services are services made available to users on demand via networks from data centers operated by Cloud computing providers. A Cloud service can dynamically scale to meet the needs of its users, and because the service provider supplies the hardware and software necessary for the service, there is no need for a user to provision or deploy its own resources or allocate IT staff to manage the service. The Service-Oriented Architecture (SOA) , which encapsulates computational resources through abstract interfaces to decouple services from their implementations, plays a key role in Cloud service provisioning. Typical Cloud service models include Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), and Software-as-a-Service (SaaS).
Cloud computing and networking technologies, a wide variety of Cloud services have been adopted as dispensable ingredients in the information systems of numerous organizations; therefore Cloud service performance has a significant impact on the performance of the entire future information infrastructure. Cloud service providers need to have deep insights about the relationship between service performance and available resources in order to fully utilize their infrastructures while meeting users’ performance requirements. Cloud service consumers also want effective methods for evaluating the performance that can be guaranteed by various Cloud services, especially different offers of the same service function, in order to make decisions on optimal service selection and composition. Evaluation on Cloud service performance is crucial and beneficial for both service providers and consumers.
Therefore, developing effective methods for evaluating Cloud service performance becomes a very important research problem that has attracted extensive attention from both academia and industry. Researchers have taken different types of approaches to tackle this challenging problem from various perspectives, for example the perspectives of system modeling, system design, and network protocol. Among the work on Cloud performance evaluation, approaches from the system modeling perspective have formed an active area with much progress recently. However, the numerous published works on cloud performance are scattered in different sections of the literature mainly due to the diverse methodologies that they took. A comprehensive survey that provides a big picture of Cloud performance evaluation from the system modeling perspective and reflects the state of the art of related technologies will be very beneficial to both researchers and practitioners.
Measurement-based evaluation on cloud service performance
Representative work on measurement-based cloud performance evaluation
Cloud infrastructures can only become viable alternatives for traditional enterprise information systems if they provide a proper level of performance. Therefore researchers started evaluating performance of Cloud services as soon as such services became available in order to provide a guideline to potential users for making decisions about Cloud service adoption. Many of the evaluations were based on performance measurements conducted on a Cloud infrastructure as a testbed.
A general procedure for measurement-based service performance evaluation on a Cloud testbed is comprised of the following steps. First the researchers need to specify the purpose and scope of the evaluation, and then identify the features/aspects of the Cloud services that are to be evaluated. The next step is to determine the performance metrics that will be analyzed and select appropriate benchmarks applications for testing. Then an experimental environment should be setup and testing experiments can be performed.
Challenges and opportunities
We now look at measurement-based Cloud service performance evaluations using public commodity Cloud infrastructures as testbeds. The representative works reviewed in the last section covered the performance of the most popular Cloud services, for example Amazon IaaS clouds, therefore offer useful insights and practical guidelines to both service providers and consumers regarding service performance evaluation. On the other hand, testing experiments conducted using public Cloud infrastructures as testbeds are facing some challenges, which also offer opportunities for future research.
First, such evaluations are only performed using the currently available Cloud services with the configuration settings made by service providers. The SOA principle in Cloud provisioning makes internal implementations of Cloud infrastructures transparent to service consumers. Although such transparency greatly enhances the usability and flexibility of Cloud services, it makes measurement-based performance evaluations more difficult. In most of the experiments reported in the literature, researchers had no control of the commercial Cloud configuration they used as their testbed. Therefore, whether the results obtained from these experiments with certain settings are applicable to more general scenarios of Cloud service provisioning needs further investigation.
Secondly, the lack of knowledge and control of infrastructure configuration for service deployments limits the researchers’ ability to study the impact of resource management inside Cloud infrastructures on service performance through measurement-based evaluations. Service users often want to evaluate the achievable performance of new services to support their decision making a service selection. However, it is difficulty for a service customer to use a measurement-based method to obtain insights about performance behaviors of new Cloud services (or service options) until the customer accepts the service offer and starts using the service.
Also, measurement-based evaluation results obtained from commercial Clouds could become invalid and useless soon after the evaluations. Cloud service providers constantly upgrade hardware and software infrastructures to enhance their current Cloud services as well as add new Cloud service offerings. Decoupling of service capabilities and service implementations in Cloud computing makes such upgrades transparent to service consumers, who may use the same services that are hosted on completely different implementations. For example, some earlier performance evaluations on Amazon EC2 found that it was insufficient for typical scientific computing applications. Then Amazon deployed the CC platform and storage-optimized instance family tailored for high-performance computing in EC2, which made the results obtained from previous tests using regular instances invalid.