Patent Application
20100153928
High-performance network services are present in private, commercial, government, and other organizations. As used in this application, “network services” includes internet services as well as services available on other networks such as wireless wide area networks, telephone networks, local area networks, wide area networks, etc. These network services may provide interaction with management software, online transactions, or access to government programs. Developing and maintaining high-performance network services is highly demanding of developers and system administrators. While a common conceptual model describes a network service as a single pipe which takes user input and outputs a result, the actual development of a network service is far more complicated.
The complexity regarding development of high-performance network services results, in part, from the array of components used to implement and deploy the network service and their interactions. An otherwise simple service may find itself executing in a complicated environment. This environment may encompass a cluster of servers, each server having variable reliability and performance characteristics, responding to a dynamically changing, large-scale real-world load.
Development of network services that operate with high-performance and reliability take into account this range of environmental factors through careful planning and design of complex software, hardware and network architectures and infrastructure. This planning is time consuming, difficult, and ultimately results in a network service based upon a static assessment of the environment. An undesirable development choice regarding the environment may result in a system which is inoperative, slow, or prone to failures. Furthermore, modifying the network service after deployment to adapt to a change in the environment or to further optimize the network service may be expensive and challenging given the intricate implications of environmental factors and the complexity of the architectures and infrastructures of the service.
As described above, development and optimization of a network service is complicated and time intensive. To provide an optimized and high performance network service, developers and administrators must take into account a variety of factors in a complicated and dynamic environment. Furthermore, changes in traffic over time may necessitate varying the configuration to maintain optimal responses to end users.
This application describes a method of developing network service applications without detailed consideration by a developer as to the environmental implications, such as the implications of workload distributions, on the service architecture and infrastructure. The application also describes ways to automate optimization of existing network services. A directed acyclic service graph describing components of the network service may be used to build a network service. A service graph may also be generated by analyzing an existing network service. During execution of the network service in a network service runtime module, runtime events are stored, and provided to an optimizer module. The optimizer module simulates variations on the service graph including, for example, different cache policies. In that case, during cache policy simulation, the optimizer module uses a hill climber algorithm or other algorithm to find optimal cache policies. The network service runtime module may then incorporate the optimal cache policies.
The disclosure is made with reference to the accompanying figures. In the figures, the left most digit of each reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical terms.
Development of network services involves taking into account a wide range of factors, which is time consuming, difficult, and ultimately results in a network service based upon a static assessment of the environment by the developer. This application describes a method of developing network service applications without detailed consideration by a developer as to underlying infrastructure, as well as the optimization of network services. While this disclosure is described in terms of a network service application, the teachings are also applicable to development and optimization of other applications as well.
A developer 108 may have an idea for a network service. Developer 108 produces a service graph 110, discussed in more detail below, which describes the network service. This graph may be a directed acyclic graph (DAG), that is, connections (edges) between components in the graph have a particular direction and do not form a loop. In one implementation, components in the graph may represent either application level components or high-level control flow statements while connections (edges) within the graph represent dataflow. Service graph 110 is then loaded into network service runtime module 106 for execution.
A user 112 may provide input 114 to the network service runtime module 106. For example, the user 112 may click a button on a webpage, enter text for a search, etc. The network service runtime module 106 may then produce an output 116 which may be presented to user 112, via a webpage, email, text message, etc.
During execution, the network service runtime module 106 outputs runtime events 118. These runtime events comprise session information indicating what session the event is associated with, a traversed edge (i.e., a path taken among the components), a time at which the event took place, a datum such as size, among others.
The runtime events 118 are then processed by an optimizer module 120. The optimizer module 120 takes runtime events, simulates them, updates a simulation state using a hill climber algorithm, and produces an optimized caching policy 122. While optimized caching policies are described in this application, optimization may additionally or alternatively comprise changing a service graph layout, replacing or swapping a component or other element for an alternate component or element to improve overall performance, etc. The optimized caching policy may then be input to the network service runtime module 106 for execution. The change from the service graph 110 to an optimized service graph incorporating the optimized caching policy 122 may be made while the network service runtime module is handling transactions (online) or when the network service runtime module is quiescent (offline).
After validating the user account, database server 410 sends a payment validation request 412 via a data link to e-commerce server 414 to validate payment. E-commerce server 414 validates payment, then sends an approval 416 to provide content to content server 418 via a data link. Content server 418 then provides content 420 to web server 404 via a data link, which then serves content 422 to user 112 via the network 406. Content server 416 and e-commerce server 412 are physically located in data center 102(B) in this illustration.
In this simple example, a single user request has accessed several components located across four servers and two data centers. To conventionally build the network service and optimize performance for even this simple application calls for consideration by the developer of these factors. Factors which may comprise the multiple servers, data links, different data center environments in this example, as well as those factors described above in
Each element may comprise a staleness-parameter delta (Δ). This staleness-parameter may be used when a component computes some function of its input that may change over time. For example, a component processing weather information may use the staleness-parameter to indicate when an output from that element is possibly stale. The possibly-stale return value of a component is given by
fΔ(t,x) (Equation 1)
and satisfies the following staleness-condition
∀t,x fΔ(t,x)=f(t′,x)f or some t′,t−Δ≦t′≦t. (Equation 2)
Assume that Δ is several orders of magnitude larger than the time to compute f itself. Thus, when reasoning about staleness of components, it may be assumed that all components produce their results instantaneously. Also, the component is allowed “perfect freedom” in choosing a return value, within the bounds of its staleness condition. “Perfect freedom” indicates that the component is not otherwise restricted from choosing a return value. The staleness-condition does not guarantee any correlation between different components and for different inputs to the same component. For example, in a component which returns weather information, it is valid for weather in city A to be stale while the weather in city B is not.
A cache may be introduced around a sub-graph (e.g., one or more nodes in the graph). The service graph method may ensure that the cached values satisfy the staleness-condition for every node in the sub-graph. In particular, when introducing a cache around component f followed by a component g, the method may ensure that
∀t,x(gΔ,fc)=g(t1,f(t2,x)) (Equation 3)
where t−Δ≦t1 and t−ε≦t2≦t.
A service graph thus comprises a collection of these elements, joined by edges indicating data flow. Thus, a user may create a service graph describing a network application.
Component A 604 resides on server 104(X) within data center 102(A). Components B 606 and C 608 reside on server 104(Y) within data center 102(B). Component D resides on server 104(Z) and is also within data center 102(B). In this and the following illustrations, broken lines indicate physical locations of elements within particular data centers and servers. However, placement of elements within particular physical locations is illustrative, and is not necessarily part of the service graph.
Source 602 connects to component A 604, which connects to a duplication element 702. In this illustration, both component A 604 and duplication element 702 reside on server 104(X) within data center 102(A), as indicated by broken lines. Duplication element 702 in turn connects to two components within data center 102(B): component Cache(B,D) 704 located on server 104(L) and component B 606 located on server 104(Y). Component B 606 connects to component C 608, which also resides on server 104(Y). Component C 608 connects to component D 610, which resides on server 104(Z). Component Cache(B,D) 704 and component D 610 connect to reduction element 708, which resides on server 104(M) within data center 102(B). Reduction element 708 receives input and emits an output to sink 612. The output of reduction element 708 may be, for example, a first result presented by either Cache(B,D) 704 or component D 610, or reduction element 708 may otherwise compare and determine which input to emit.
In the event of a cache miss (e.g., a particular piece of information is requested from a cache, and the cache is unable to fill the request), the original service functionality provided by component B 606, C 608 and D 610 will execute due to the duplication of input data by the duplication component 702. The output of component D 610 will furthermore be added to the cache (B, D) 704 by an implicit duplication and data flow mechanism not shown in this figure.
A type signature of each node's input and outputs determine how that node may connect to other nodes. For a cache location to be valid, when the output type and input type of two components connected together must match. Furthermore, all input and output edges of components, including join, duplication and reduction nodes, must be connected to other components For example, if a join-m node cannot receive m inputs, it is not a valid graph.
Returning to duplication element 806, duplication element 806 also emits data containing the IP address and zip code 822 to component 824. Component 824 looks up the weather based on zip code, and emits weather information 826 to reduction component 820.
Reduction component 820 determines an input to use, either weather information 826 or weather information 816. For example, if component 824 returns the weather information 826 to reduction component 820 before the edge traversing components 810 and 814, then reduction 820 may emit that information (i.e., the first information received). Alternatively, if the weather information 826 is determined to be stale, it may be discarded and reduction component 820 may utilize weather information 816 emitted by component 814 (i.e., the most up to date information may be used). Reduction component 820 emits weather 828 data to component 830, which parses the data. Parse data component 830 then emits a report string 832 to sink 834, including the requested weather information.
The service graph enables relatively simple construction and re-plumbing of a network service, including the addition of caching for sub-graphs. Determining what sub-graphs or individual components to cache may be aided by the optimization methods described next.
Simulator module 1010 tests the policy 1008 using runtime traces and service statistics 1014 to determine a utility value 1016, which is returned to the hill climber module 1006. The utility value may comprise a number of cache hits, resource costs, latency effects, assessment of complexity, assessment of reliability, etc. Higher utility values indicate a more useful result such as a cache which has a high number of hits, a lower response time after placement of a cache, a service graph having a minimum number of nodes, etc. A hill climber module 1006 utilizes this utility value and proceeds to use the hill climbing algorithm to determine a maximum utility value. Caching policies are iterated using the hill climber module to determine a policy which produces maximum utility from a space of tested policies.
After the initial set of k policies is created 1104 in the cache policy generator from the top k of the N candidate initial policies, at 1106, data from a statistics aggregator module is loaded. At 1108, an initial policy is selected and simulated in a simulator module. At 1110, the simulator module outputs a utility of a policy under test to a hill climber module. At 1112, the hill climber module increments the policy being simulated by a multiple of a mean observed output datum size (MOODS). Examining an entire space of possible caching policies to determine one with greatest utility is computational intensive, particularly as the size of the service graph grows. Accordingly, rather than generating every possible cache value as a policy to simulate, cache values and their resulting policies may be incremented in steps determined by the MOODS.
The assumption underlying use of the MOODS is that the difference between allocation B bytes to a cache and B+1 bytes to a cache is likely to be negligible unless the values stored in the cache are very small. Hence, the cache policy generator may be configured to generate policies using multiples of the MOODS. As the hill climber module progresses, successor states are generated for simulation. For example, one method of generating a successor state is, for each pair of cache locations, to modify the original state by adding a MOODS of bytes to the first cache and removing the same number of bytes from the second cache.
At 1114, the incremented policy is simulated in the simulator. At 1116, the simulator outputs the utility of the incremented policy to the hill climber module. As described above, a cache may be added to a service graph during generation of policies for simulation. Similarly, caches may be evicted when their utility is below a specified threshold, or when other cache eviction policies such as a least-frequently used policy are met. In a least-frequently used policy, caches with the lowest number of hits may be removed.
At 1118, a determination is made if a maxima of utility has been reached. If not, then the process returns to 1112 and increments the simulation policy in the hill climber module. Otherwise, once a maxima for the policy has been reached at 1118, another determination is made, at 1120, whether all k initial policies have not been tested. If not, at 1122 the process proceeds to select the next initial policy from the set of top k policies and proceed to 1108 to simulate the selected initial policy in the simulator. Otherwise, if at 1120 all initial k policies have been tested, then at 1124, the process outputs the policy with the greatest simulated utility.
At 1206, user requests are processed using the network service runtime module. At 1208, runtime events are output to an optimizer module. At 1210, the optimizer module simulates caching policy alternatives, as described below, to develop an optimized caching policy. At 1212, the network service runtime module is updated to use the optimized caching policy.
As described in this application, modules may be implemented using software, hardware, firmware, or a combination of these. In the illustrated examples, the modules are implemented using software including instructions stored on a computer-readable storage medium or otherwise in memory and executable on a processor.
Although specific details of exemplary methods are described with regard to the figures and other flow diagrams presented herein, it should be understood that certain acts shown in the figures need not be performed in the order described, and may be modified, and/or may be omitted entirely, depending on the circumstances. Moreover, the acts and methods described may be implemented by a computer, processor or other computing device based on instructions stored on one or more computer-readable storage media. The computer-readable storage media (CRSM) may be any available physical media that can be accessed by a computing device to implement the instructions stored thereon. CRSM may include, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.
