This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.
The implementation phase of this research mostly engaged with the server-grouping mechanism. The total system was implemented on the load balancer because the proposed solution is all about how to distribute the workload across servers. We implemented the system based on the service-oriented architecture (SOA). We use a web service as the common application for all clients and a set of axis2 servers hosting the web service to process the incoming requests. We implemented our load balancing solution on top of Apache Synapse mediation framework which acts as a high performance load balancer. Apache Synapse is implemented using Java programming language. We implemented the other artifacts such as services and client simulators also in Java. It is necessary to understand the architecture of Synapse in order to understand the implementation of this system.
5.1 Apache Synapse Mediation Framework
Apache Synapse is a simple, lightweight and extremely high performance open source mediation engine distributed under the Apache Software License 2.0. Synapse also acts as an Enterprise Service Bus (ESB) as it has the capability to communicate over many applications with different transport layer protocols. Synapse is used as the underlying framework in WSO2 Load Balancer and WSO2 Enterprise Service Bus (ESB), which are commercial products being offered. EBay, the huge internet auction house, uses WSO2 ESB to achieve the highly reliable transaction processing environment required for processing huge number of transactions per second.
Typically, Synapse is deployed between the clients and the implementation of the service where it can mediate the request stream in between. Synapse accepts messages on behalf of the actual service and performs the configured actions. The in-built facilities include authentication, validation, logging, routing based on the message content etc. Synapse decides the destination endpoint for the request and directs it to the actual server. The other capabilities of Synapse include load balancing, throttling, cashing and detecting timeouts and transport failures.
Synapse can keep up the performance of cloud services as it has implemented a non-blocking HTTP/S transport implementation to handle thousands of concurrent requests using very little resources and threads. This implementation is capable of connection throttling to control the rate at which large messages are read and processed, and thus can handle heavy concurrent loads of large messages using constant memory. The figure 0 shows the high-level architecture of Synapse mediation framework.
Sequence A End point
End point Sequence B
Response to Client from Synapse
Response to Synapse from Service
Request to Service from Synapse
Client Request to Synapse
5.1.1 Message Receivers
The 'Synapse Message Receivers' (Synapse MR) receives the messages, that clients send, from the underlying transport mechanism. When a message receiver receives a message, Synapse injects the message into the Synapse environment (figure 0). The Synapse environment manages the rest of the work including mediation and sending for the service. The 'Synapse Callback Receivers' receives the responses for outgoing messages from the services. They tie the responses with the relevant requests made, in order to maintain the proper context for the out flow mediation.
Mediators are the basic message processing units in Synapse. A mediator takes an input message, performs some action on it and gives an output message. Mediators can be linked together and arrange chains of mediators in order to create complex message flows. They are called sequences. As we can see in the figure 0, there are two main sequences in the Synapse Environment, 'In' sequence and 'Out' sequence which are also called 'sequence A' and 'sequence B' respectively. All incoming requests go through 'In' sequence when they are injected into the Synapse environment. Each request that goes through a sequence is altered by each of the mediator attached to the sequence. Mediators can manipulate message content (payload), properties and headers and if needed, they can execute additional tasks such as database lookup, service invocation and script execution as well. A Synapse Message Context (SMC) is created for each message that goes through the mediators. This context holds the message, message context properties and other parameters of the environment. If the mediators need to share any information regarding the message between them, they have to use Synapse Message Context properties, as Synapse is stateless. When a message has gone through a sequence of mediators, Synapse sends it to an endpoint. The 'Out' sequence (Sequence B in figure 0) contains the out flow mediators which process the response messages received by the 'Callback Receivers'.
An endpoint is the logical representation of an actual endpoint. An actual endpoint refers to a physical machine, which hosts a service or an application. However, there can be endpoints that represent a collection of endpoints. They are wrapped with some logic to distribute the messages over the collection. Those are usually called 'Load Balance Endpoints'. When an endpoint receives a message, it may do some modifications to the Synapse Message Context and hand over the message to the Sender with the reference of which the message should be forwarded to. If it is a request message, Sender would send the message to an actual endpoint or of it is a response; back to the client. The endpoints can me defined in the Synapse Configuration File.
5.1.4 Load balancing support on Synapse
To perform load balancing, synapse uses a Load Balance Endpoint. A Load Balance Endpoint contains a set of child endpoints. It routes user requests to the child endpoints according to the given algorithm. The default load-balancing algorithm in Synapse is Round Robin. However, we can write additional algorithms, include them in Synapse, and specify which algorithm to be used in the Synapse Configuration File. The requests which have gone through the 'In' sequence are usually sent to the Load Balance Endpoint. Then for each request it calls for the distribution algorithm to get the next endpoint to send the message. When the Load Balance Endpoint gets an endpoint reference (EPR), it hands the message over to the 'Sender' with the EPR to actually send the message to the required server.
Synapse Tasks are used to execute programmed activities periodically. Synapse can schedule tasks to be triggered at the given trigger mechanism. We could use triggers such as a count and an interval, for the task to be executed given number of times in specified interval. We can also set the task to be executed continuously in regular intervals specified or just for a single time. Each task is executed in a separate thread so it does not directly affect the load balancing process or message mediation. The relevant configurations should be provided in the Synapse Configuration File.
5.1.6 Synapse Configuration Language
Apache Synapse loads its most of the configuration from the Synapse Configuration File. This helps users to change configurations easily by hand editing. All configurations, which include what need to be done with each incoming message until the response message is sent back to the client, must be defined in the Synapse Configuration File using Synapse Configuration language. Specifically they include 'In Sequence' and 'Out Sequence' with the list of mediators, set of endpoints with the IP addresses ports and the service names, tasks to be executed periodically and error handling information etc.
5.2 Implementation of the Solution
To implement our load balancing solution on Synapse, we introduce new entities in the synapse architecture. We introduce two mediators, the 'Arrival Rate' mediator and the 'Service Rate' mediator in the two main sequences, 'In' sequence and 'Out' sequence, respectively and two Tasks, 'Adjust Partitions' task and 'Message On Flight Counter' task and the load distribution logic.
5.2.1 Monitoring Arrival Rates
We use the 'Arrival Rate' mediator to calculate the arrival rates of user-requests class-wise. The 'Arrival Rate' mediator is included in the 'In' sequence thus all incoming user-requests are processed by this mediator. In multi-tenant environment the client requests need to contain the 'tenant ID' in the header in order to load that tenant's configuration for the service. The 'Arrival Rate' mediator tracks the tenant ID of each request and check for the SLA information of the relevant tenant. Based on the SLA data all incoming requests are classified in different service classes. The SLA data is fed to the system using a configuration file. The 'Arrival Rate' mediator maintains message counters for each class separately. We set the 'Adjust Partitions' task to be triggered periodically in ten-second intervals. Each time, the task checks for the values of message counters and divide them by the number of seconds of the interval (ten seconds) to obtain the average arrival rates (per second) of each service class. The 'Adjust Partitions' task sets the message counters to zero each time so they would be able to count the messages in the next observation period.
5.2.2 Monitoring Service Times
Here we introduce another mediator named 'Service Rate' mediator which is included in the 'Out' Sequence. It is important to have two mediators in both 'In' sequence and 'out' sequence to calculate the response times of requests. We maintain lists for each class to hold information about all the messages which are in the system (which are distributed to servers and not yet received the response) in a comment work space provided by Synapse which is named Synapse Configuration Context. The 'Arrival Rate' mediator gets the time of arrival of each request and creates a unique ID for every request. These request-specific data are bound together as a single item for each request and inserted into the relevant list. The request-ID which is created and the service class ID is inserted into the Operation Context of the message as they are needed when the response is received at the 'Service Rate' mediator. The 'Service Rate' mediator finds the time when it receives the response from the server, get service class from the Operation Context and check for the arrival time of the request in the relevant list at Synapse Configuration Context. Using those time values we can calculate the service time of each request. To find the average, we sum up all the service time values through the ten second observation period, separately for each class and divide those by the number of messages served during the period. The number of messages arrived is not equal to the number of messages served. Because of that we maintain another set of message-counters at the Service Rate mediator to keep track of the number of messages served within the observation period class wise. To perform the calculation for finding the averages after each observation period, we use the 'Adjust Partitions' task.
5.2.3 Request Distribution
To rout the requests to the servers the load balancer need to keep track of each server available. We maintain an endpoints list, which contains all endpoint objects relevant to each physical server. This list was originally introduced by Synapse. The number of endpoints and the endpoint details are loaded at the startup from the Configuration file. In order to put the idea of 'Elastic Server-Groups' into practice we split the endpoints list into separate lists which would correspond to each class. These endpoint lists would represent the server-groups, in the load balancer. All incoming messages are sent to the 'Load Balance Endpoint' after the going through 'In' sequence. The 'Arrival Rate' mediator attaches the 'service class ID' to the message headers replacing the 'tenant ID' so that the Load Balance Endpoint can directly get the service class of each request. Referring the service class, we distribute the messages over the relevant server group in 'Round-Robin' manner.
We adjust the sizes of the groups (number of endpoints in each list) at the end of each observation period to achieve our goal. For the calculation, we use the arrival rates and service rates during the observation period. However, at the startup we need to create separate lists for the classes without that information. We only know the total number of servers available. Therefore we assume the arrival rates and the service raters are equal in all classes and do the calculation in our method to obtain some meaningful values for the initial sizes of the groups. Anyway it is better than having the sizes similar for all groups initially. Anyway, these sizes last only for ten seconds and after that the 'Adjust Partitions' task starts adjusting these sizes regularly according to the demands.
5.2.4 Adjusting Server-Groups On Demand
The 'Adjust Partitions' task is responsible of maintaining the sizes of the groups. It fires after each observation period and get the average arrival rates and the average service times of each class as explained in sections 5.2.1 and 5.2.2 respectively. Each time, the task retrieves the 'total number of servers' from Synapse. It is important because if there were some auto scaling function enabled, then the total number of servers would change. However, in this thesis we talk about a scenario where the total number of servers is fixed. Nevertheless, we have shown, later in this thesis, that our solution can be easily coupled with auto scaling.
As we discussed, we initially obtain the preferred numbers of average concurrent threads for each class. We feed them to the system using another configuration file. The 'Adjust Partitions' task reads the file and get the information and then it finds the ratios of these values as discussed in the section 4.8. With that information, we have completed gathering all necessary attributes to find out the most appropriate way of dividing the servers across the groups. Then we carry out with the calculation presented in section 4.8 and find the set of sizes of the server-groups. Not surprisingly, the mathematical function gives the sizes as decimal numbers. Therefore, we cannot use with those values for the group-sizes, as the servers are discrete objects. Therefore, we round off the values to be integers that fit into the total number of servers.
The rounding off process cannot be carried out in the general way, where we check whether the first decimal digit is larger than five (> 5) and if so, change it to the next integer or remove the decimal part otherwise. The reason is we need to keep the summation of all values unchanged even after the values are rounded off. To achieve that we first retrieve the largest integer lower than each value by just assigning each floating-point number to an integer variable. Then we get a set of integers for the group sizes but the summation of these integers would be less than the 'total number of servers', unless the original values were already integers. The difference between the 'total number of servers' and the summation of the integers should be an integer less than the 'number of classes' and we shall call it 'D' for ease of exposition. We check the deviation made by each integer that we obtained, from the respective original floating-point value. We order the set of integers according to this deviation, in the way that the integer, which makes the highest deviation, comes first in the list. Then we iterate the list from the beginning, each time incrementing value by one and decrementing 'D' by one, until 'D' would become zero. At the end, only a subset of the integers would be incremented. This way we get a rounded set of integers for the group-sizes that would add up to the 'total number of servers'.
We need to update the number of endpoints in the endpoint-lists relevant for each class, with these new group-sizes. The new sizes may be same as the previous sizes. If so, we do not perform any modification to the endpoint-lists. Otherwise, some groups should be expanded and some should be shrunken. We first remove servers from the groups which should be shrunken and assign each those to the groups which should be expanded appropriately. This process is carried out regularly after each time frame. Since we have multi-tenancy, regular server re-assignment does not need to perform costly virtual machine migrations. Each server being multi-tenant makes all servers identical to the load balancing perspective. This group-updating process does not add any latency to the user-requests because the 'Adjust Partitions' task runs in a separate thread. However, this process would use few threads of the thread pool of Synapse.