Patent Application
20080104609
The invention relates generally to distributed interactive simulations and, more particularly, to a system and method for load balancing distributed simulations in virtual interactive environments.
Computer systems running virtual interactive environments (such as, for example, video games or flight simulators) may be run in client/server environments. The client and server may communicate via a computer network (such as, for example, a proprietary network, or the Internet). In such a client/server environment, the processing of the simulation of the virtual interactive environment may be performed by the server. In that case, the server may distribute processing requests to a plurality of processors through a dispatcher.
Historically, the dispatcher controls the distribution of processing among the processors according to a centralized management scheme, sometimes referred to as the “state model.” The dispatcher tracks and manages the workload of each processor, distributing processing requests approximately equally among the processors, such that unused processing capabilities and response time to the client are both minimized.
Some interactive virtual environments, though, in particular massively multi-user interactive virtual environments, have become so complex that the processing demands may exceed the capabilities of the dispatcher, such that a bottleneck forms at the dispatcher. These circumstances may have two undesirable effects. First, processing capability may be underutilized; and second, response time to the client could be varying.
Accordingly, a need has developed for overcoming the above problems.
In a first aspect of the invention, a device comprises two or more nodes for processing a simulation of a virtual interactive environment. The two or more nodes comprising at least one component to determine workload amongst at least a first node and a second node the two or more nodes. The at least one component further delegates work to the second node when the workload on the first node exceeds a predetermined boundary, and accepts work from the second node when the workload on the second node is within the predetermined boundary.
In a second aspect of the invention, an apparatus comprising at least two nodes for processing a simulation of a virtual interactive environment comprises a module for transferring work amongst each of the nodes when workload on a node exceeds a predetermined boundary. The apparatus also includes a module for accepting work amongst at least one of the nodes when the workload is within the predetermined boundary on the at least one node.
In a third aspect of the invention, a method comprises balancing processing requests between at least two nodes. The first node of the at least two nodes determines a workload on the first node and transfers work to a second node when the workload on the first node exceeds a profiled boundary. The method further includes accepting work from the second node when the workload on the first node is within the profiled boundary.
In another aspect of the invention, a computer program product comprises a computer useable medium including a computer readable program. The computer readable program when executed on a computer causes the computer to distribute a workload among more than one node, and causes a first node to: determine workload on the first node; delegate work to a second node when the workload on the first node exceeds a predetermined threshold; and receive work from the second node when the workload on the first node is below the predetermined threshold.
The invention relates to load balancing distributed simulations in virtual interactive environments. More particularly, the invention relates to highly interactive virtual worlds for online games and other distributed simulation scenarios. For example, the invention can be configured for virtual worlds typically simulated for massively multi-player online games, warfare training, and a variety of other distributed interactive simulations composed of massive amounts of object data and state that must be updated for potentially millions of users.
In embodiments, the invention is configured to distribute processing requests according to a territorial model, such that any bottlenecks that previously existed at the dispatch level, is reduced or eliminated. In embodiments, nodes are assigned to a sector or portions of a virtual interactive environment. The nodes are configured to process data and state information associated with the assigned sector or portion of the interactive environment. As objects move in/out of zones (e.g., pre-determined boundaries or geographic locations/features), the data and state information is transferred between the nodes. In one implementation, the nodes are constantly monitored for workload and the size of each zone assigned to a particular node is dynamically adjusted to balance the workload; that is, the processing of simulations within the zones may be adjusted according to workload allocations. In this schema the zones could varying in size as the simulation executes. In addition, in embodiments, the data is kept centrally, but an indexing scheme is used to transfer ownership of object data from one node to another as well as object state. Nodes may be executing programs specifically required to do performance monitoring and to determine if there exists an unbalanced system workload. In a Cell Broadband Engine (or Cell BE) implementation, it is contemplated that data, state, and behavior are all defined for each object. In this type of embodiment, code objects (e.g. SPU-lets on a Cell BE) would be moved with each object from one node to another in a cell implementation.
With reference to the accompanying drawings,
In any event, the computing device 115 can comprise any general purpose computing article of manufacture capable of executing computer program code installed thereon (e.g., a personal computer, server, handheld device, etc.). However, it is understood that the computing device 115 is only representative of various possible equivalent computing devices that may perform the processes described herein. To this extent, in other embodiments, the functionality provided by computing device 115 can be implemented by a computing article of manufacture that includes any combination of general and/or specific purpose hardware and/or computer program code. In each embodiment, the program code and hardware can be created using standard programming and engineering techniques, respectively.
Similarly, the computer infrastructure 110 is only illustrative of various types of computer infrastructures for implementing the invention. For example, in one embodiment, the computer infrastructure 110 comprises two or more computing devices (e.g., a client and a server, or a server cluster) that communicate over any type of communications link, such as a network, a shared memory, or the like, to perform the process described herein. Further, while performing the process described herein, one or more computing devices in the computer infrastructure 110 can communicate with one or more other computing devices external to computer infrastructure 110 using any type of communications link. In either case, the communications link can comprise any combination of various types of wired and/or wireless links; comprise any combination of one or more types of networks (e.g., the Internet, a wide area network, a local area network, a virtual private network, etc.); and/or utilize any combination of various types of transmission techniques and protocols. As discussed herein, the manager 120 enables computer infrastructure 110 to delegate or accept a processing request 155.
Referring still to
By way of example,
Upon receiving user input regarding an object in the virtual interactive environment, the client 210 may send a request for processing to the server 220. The server 220 will send the request for processing to the node that is associated with the zone in which the object exists. For example, in the flight simulation application discussed above, if the user input is regarding an object representing a plane on the airstrip, then the server 220 will send the request for processing to the first node 230, as it is associated with the first zone 310, which represents the airstrip.
According to yet another aspect of the present invention, the method includes communication between the nodes such that when a node is operating beyond its optimal maximum workload, it may delegate some processing requests to another node. Likewise, when a node is operating below its optimal minimum workload, it may accept some processing requests from another node. To this end, each node will self-monitor, for example, by measuring its frame processing or simulation rate, in known methodologies. The communication between the nodes may be linear, that is each node may communicate only with its immediate neighbors, or may be universal, that is, each node may communicate with all other nodes. Additional schemes may be employed where groups of nodes are balanced and then work is moved between groups and then each group is subsequently rebalanced. This could be a binary balancing scheme or some other technique. A more balanced state of the workload may be achieved over time as each node communicates with the other nodes and the work is distributed as zones are adjusted.
In embodiments, the territorial limits of the nodes may be dynamically readjusted. For example, in the flight simulation application discussed above, four users may be engaging in a dog fight over the mountain range, while no users are active in the airstrip. In these circumstances, the first zone 310 (originally comprising the airstrip) could expand to also cover the northern half of the mountain range. The second zone 320 (originally comprising the mountain range) could correspondingly contract to cover only the southern half of the mountain range. Depending upon the extent of the processing requests, the third zone 330 may or may not expand to accept some of the processing requests occurring over the mountain range. In any event, each node may adjust to include other territorial bounds or other conditions, depending on the loads placed on each of the nodes.
In embodiments, when an object in the simulated environment crosses a dividing line between one zone and another, one node will delegate future processing requests regarding that object to another node. Such delegation will include all data, state, and behavior information regarding the object. In this way, little kernels of code, or SPU-lets, move with each object from one node to another in a cell implementation.
According to yet another aspect of the present invention, the method includes dynamically and periodically balancing the workloads among the nodes in real time. Thus, in one example, the each of the nodes may determine workload amongst each other and delegate work amongst each other when the workload exceeds or is below a determined threshold. When the workload is above a threshold (or beyond a profiled boundary), one node may delegate work to another node. Alternatively or concurrently, a node may accept simulation processing when its workload is below a threshold (or within a profiled boundary).
Additionally, the method includes dynamically and periodically adjusting the territorial limits of each zone in real time. That is, in an embodiment, the distribution of the workload may be continually assessed and adjustments made as necessary to optimally distribute the workload, effectively minimizing unused processing capability and minimizing response time to the client. This may be, for example, accomplished by a component of the node or nodes configured to change the geographic or physical features within the virtual interactive environment according workload allocations. By way of example, when a first node is overworked, a second node may assist the first node within the same zone or geographic features.
Administrative requests or other interfacing with the network or the client 210, such as, for example, subscription, login, or other housekeeping events, may be managed by an event handler. The functionality and interoperability of the event handler with other components are well known to those of skill in the art. In one embodiment, such an event handler may reside on the server, but not within any of the compute nodes. If so, the event handler will transfer processing requests to an appropriate node after handling such administrative requests. Alternatively, such an event handler may reside within each node.
While the invention has been described in terms of embodiments, those skilled in the art will recognize that the invention can be practiced with modifications and in the spirit and scope of the appended claims.
