Patent Application
20080091403
While embodiments of this invention can take many different forms, specific embodiments thereof are shown in the drawings and will be described herein in detail with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention, as well as the best mode of practicing same, and is not intended to limit the invention to the specific embodiment illustrated.
An architecture which embodies the invention passes entity objects between computers in a discrete simulation to overcome computational limitations that result from the use of networked independent discrete event simulations.
The receipt of the entity from one simulation triggers events in another simulation. This methodology enables a distributed simulation approach which not only enhances scalability, but also enables independently functioning simulations to communicate with and affect the outcome of each other in a synchronized manner.
In one aspect of the invention the execution of the simulation on the multiple simulators is controlled externally with a synchronization handshake mechanism.
A participating simulation player need only adhere to predetermined network data formats and message send/receive protocols. As a result new players, with internally different structures can readily be added to the network.
In another aspect of the invention, synchronized multi-processor discrete-event simulation communication of entity or event data takes place over a network. Two types of messaging: entity data messages and processor synchronization messages can be sent. The receipt and issuance of synchronization messages keeps the different processors in time step with each other. The receipt of entity data from one simulation triggers events in another simulation. In accordance with the invention, this distributed simulation approach not only enhances scalability, but also enables specialized simulations to communicate and affect the outcome of each other while maintaining time integrity.
Embodiments of the invention can be applied to the simulation of a schoolhouse training system, but could equally be applied to the simulation of other operational systems, such as a manufacturing or transportation system. The present method can be use to simulate other complex systems without limitation.
Different discrete-event simulations can be coupled together in a distributed multi-processor environment by entity or event transfer and message synchronization via network packets. The approach is time-interval based and uses one of the processors to host a main controller node. Simulation processor nodes execute independently, export entities via network packets, and “report” via network packets that they have completed the established time interval.
The simulation may be partitioned into multiple simulators for many reasons, such as to better represent the real system being simulated or for computer resource usage. In some cases, the actual system includes networked machines in different buildings. The entities are work items being sent from machine to machine. In other cases, the simulation software does not support a large enough system, or the processing time would have been prohibitive if all the simulation capabilities were implemented in one computer.
An exemplary entity/event data format is depicted in Table 1. Each entity or event is packaged by the issuing simulator with a unique ID, a type specification, and a variable number of attributes. Note that although a timestamp filed is provided, the present methodology does not use this field for synchronization. It can be used by any simulation requiring this information.
Synchronization is maintained by network packets (e.g., UDP packets) that indicate it is time to simulate the next time interval. Time intervals can be configurable to be by minute, hour, day or week, or some other time period appropriate to the system being simulated. These intervals can be of higher or lower fidelity depending on the nature of the simulations.
In accordance herewith, client-server communications from a controller are used to configure and initialize the simulations players. The main controller also manages the receipt and issuing of all synchronization messages which are sent via broadcast messages to all simulations. After all players report that they have completed the current time interval, they are given the “go ahead” by the main controller to start the next time interval.
When each simulator player receives the start frame message all entity data from the simulation must have been sent to the appropriate simulation player. As each simulation player reads packets from the port, the packets are stored in the player's process queue in time order. The player proceeds to execute its next interval frame processing, issues its Frame Complete message, and waits for the start of the next frame.
This type of timing scheme is advantageous in that it avoids the need for time-costly prediction “Time Warp” or rollback algorithms. Deadlock situations are also avoided because all messages are received in the same time interval.
Each participating simulator starts its simulation at the same simulated time, Ts, and runs only for the simulated time specified by P. At time Ts+P, the simulator stops processing, and sends a message to a central controller that says that it has completed its time frame processing.
While it was processing, and/or at the completion of its processing period, P, the simulator may have been sending entity packets and other information to other simulators that are participating in the distributed simulation. In a method which embodies the present invention, unlike in the rollback method, all the other simulators will delay processing this information until the start of the next period.
Deadlock and rollback issues are avoided by never letting any simulation proceed further into the future than any other simulation. There is a predetermined duration (time frame) into the future during which each simulation can execute before it must stop and wait for all the other associated simulations to catch up. Thus, there is a discretization of time that limits the processing of the simulations into a maximum duration, P, during which time, no simulation will affect the processing of another simulation.
Network messaging bandwidth is efficiently used since there is no need to transmit cancellation messages or null “anti-deadlock” messages as employed by other distributed discrete-event methods. Network transmissions are dedicated towards moving the simulation execution forward in time.
In yet another aspect of the invention, independent discrete-event simulations do not need to be written in any particular programming language. The independent simulations only have to adhere to pre-defined network entity/event data formats and network message send/receive protocols. Synchronization is managed by an external Controller module.
Entities/Events may queue up at the entrance to the other simulators, but will not affect the other simulators during the current period. Those of skill in the art will understand that the period, P, needs to be determined so as to get proper simulation of the system. P may take on time periods of microseconds to months, or whatever is appropriate for the simulation at hand, just so that the underlying system being simulated does not change too much during that time, as determined by a systems analysis of the actual system to be simulated.
When all simulators have transmitted the ‘done with current period’ signal to the main controller, the main controller will issue the ‘start’ signal to all the simulators. The distributed simulators will then read in and process the new information that had arrived, and process another period of simulation, stopping again, after a simulated time period P has elapsed.
Embodiments of the invention establish a synchronized networking framework for simulators, enabling the rapid extension of the functionality of existing simulators, and reducing the time required to run an extensive simulation.
In a disclosed embodiment, entities are transmitted instead of events, although events could also be transmitted. Known discrete simulations transmit events that may cause roll-back to be required in other co-simulations to account for events occurring in other co-simulations. Embodiments of the invention require no rollbacks. The processing time is discretized to plurality of steps and the entities are collected in prior to commencing the next processing period.
No entity proxies are required in order to represent entities being worked on at other simulators, since the entity can only exist in one simulator at a time.
Embodiments of the present method provide substantial scalability potential. For example, in an implementation using common desktop computers and computer networks, overall system computing power can be raised by the addition of more computers to the network, thus avoiding the requirement to modify the performance of any one of the simulators already coupled to the network.
Dynamic Networked entities/events (DNEs) 112 and 114 enter and leave the individual simulators, and wait in the Input Queues 102 and 122 until a Go command is received 110 and 130 at which time they enter into the Simulations, 104 and 124 for processing. When the DNEs have been processed in the respective simulations they leave asynchronously 106 and 126, and proceed to the next simulation as ordered by a processing schedule, to wait in the appropriate queue until the next time period of processing. The overall processing is conducted in a synchronized manner as controlled by the Controller 140.
There is a separate Done message sent by each simulator, 108, and 128 to the controller 140. Only when all the Done messages have been received 142 and 144 will the Controller 140 issue the Go signal 146 to all the simulators, entering at 110, and 130, to begin simulation for the next time frame. Example implementations of the transmission and reception of the Done and Go signals could involve the use of web services or UDP sockets, if permitted.
The use of multiple simulators allows entity pipelines to be realized to govern a large scale flow of entities through different simulators or processes sharing the same system. While multiple entities may share the entry point for a given process, the entities leaving that process do not necessarily have to go to a shared “next process”. This dynamic characteristic of the entity representation allows different plumbing of entity paths through the simulations. The “next location” of an entity may also be changed in a given simulation, if, for instance, the simulation flags the entity as defective, it can change the next address to be a waste heap, or if the quality is exemplary, it can go to an advanced processing location, as in a faster measured time spec on a microprocessor chip entity.
For the overall simulation the entry of different types of entities into the system can be controlled with an Entity Generation Unit 134. Unit 134 time-sequences when entities enter, what types of entities that they are, what processing pipelines they go to, and where their first simulator is to be. The entity generation unit, 134 can be implemented on a separate computer, as long as it has access to the network on which the processing is occurring. The Entity Generation Unit 134 sends DNEs 136 into the system, to their appropriate simulator destinations. These DNEs travel through the network in the same manner as DNEs that leave the simulators after processing.
In the case of a simulator, each process or processor, 312, 318, 322 can be given a multicast address 310, 320, 324 and it subscribes on the multicast network to receive all data traffic intended for that address. When a given process has events or entities to send to another simulator, the entity or event information is packaged into a multicast message and transmitted to the receiving process's multicast address. The system does not need to know in what processor the destination simulator process resides. Multicast Ethernet-based processing will automatically cause the entity or event packet to reach the simulator process correctly.
Simulator 100, in the case of
During processing, the information that the DNE contains may be altered, thus indicating the dynamic nature of the DNE. The DNE would then be given the address of the next process to enter in the system simulation, and this may even be a terminal process where the DNE exits the system, such as a finished manufactured product, for example. If more processes must process the DNE, then the multicast address of the next process is coded into the DNE multicast representation of the entity or event and it is sent out onto the network 314, 106, 114 for further processing.
Simulator 100 also includes a Simulation Timer 332, to measure how long to conduct the execution of the processes simulated in simulator 100. There is a common, pre-determined, time duration or interval that all simulators in the networked simulation will use in conjunction with their internal simulation timers.
When the Go command 110 is received at the simulator, all the DNEs received to that point will enter the Simulation 104. Until the duration is complete, the simulation will execute, sending DNE packets when they become ready for issue. Once the end of the simulation interval has been reached, all simulation execution in this simulator will stop, and the Done signal 108 will be transmitted from Simulator 100.
The process starts with Simulator 1 issuing the Simulator 1 Ready signal 108 to the controller input 142 indicating that it has completed initialization and is ready to participate in the simulation. The Controller 140 is set to expect both Simulator 1 and Simulator 2 to participate in the simulation. Therefore, the Controller waits until Simulator 2 sends its Simulator Ready signal 128 to the Controller input 144 before it issues the Go command 146 to the multiple simulators inputs 110, 130.
After receiving the Go commands, both simulators begin processing their simulations, for the pre-determined simulation time frame duration. The simulation duration is a time frame in simulated time not in real, wall-clock time. For instance, the simulator might be processing 24 simulated hours in one time frame, but in wall-clock time, it may only take 1 minute to perform the processing.
Some simulators may be hosted on faster computers and can execute their simulation time frame faster, as seen in
When each simulator has completed its simulation time frame processing, it sends the Done signal 128, 108, to the Controller inputs 144, 142. When the Controller 140 determines that it has received the Done signal from each of the simulators participating in the simulation, then the Controller will issue the next Go signal 146, to all the simulator inputs 110, 130. They each load whatever DNE's they have received as a result the previous simulation period, and process the next simulation period 154 and 156. When the simulation period is complete at each simulator, the respective Done signal, 108 and 128 is sent to the Controller input 144, 142. The processing continues in this manner until the system has reached the maximum simulation time or is stopped in some other manner.
When the simulation initialization is complete, the simulator subscribes on the network to each of the addresses or multicast groups it is using for its processes 404. Then the simulator issues the Simulator X Ready signal, 406, to show that it is ready to simulate, and exits to wait for the Go signal 408. The Controller 140 adds the simulator to the list of simulation players at this point, and waits until all needed simulators are online and ready before issuing the Go signal.
The DNEs are stored until the Go signal is received from the controller. The Go signal starts the next processing cycle 504. DNEs enter their processes, or processors and have work performed on them. DNEs already in the processes continue being processed. When processing is complete on a DNE 506, the DNE is prepared for transmission to a particular “next address” on the network. The “next address” and the DNE are encoded into a network entity/event message and transmitted.
The entity/event will be stored at its new location until the start of the next processing cycle before processing begins anew on it. Processing continues in the simulator 508 until the end of the current simulation period.
All system processing is complete at the end of each simulation period. All transmitted entities/events are sent out on the network system, but do not enter their appropriate destination simulation processes until the start of the next simulation period.
The simulation period P must be made short enough so that the system processes the entities and events in a natural time frame. It may be made as small or as large as desired, based on how quickly the system changes, the nature of the simulations, and how fast the information coming from other simulators must be regarded in the simulation system.
One application of DNE is a multi-site training analysis & design system. In that system, students represent the networked entities. The entities are the DNEs that move from course to course, lesson to lesson, and student pipeline to pipeline, competing with other entities at the training site for training-related resources as they become available.
When a course is simulated at a single location or site, students attending that course will stay confined to simulator running at a single site. When multiple sites are involved, the choices are to update the software simulation to handle multiple sites, or to distribute the effort across multiple computers. In an embodiment of the present invention, a set of networked computers can be used, one for each training site. Each location site and the site's associated resources can be are assigned to a unique networked computer. In order for each instance of the schoolhouse simulation to interact with another, the DNE methodology can be used to synchronize the sites while maintaining the same level of precision and performance achieved when the simulation was implemented in only one computer.
In the training system being simulated, students finishing one training course generally are scheduled to be sent to a follow-on course, often at a different training site. The present method enables students to exit one course and be automatically directed to the entrance queue of another course, whether that course was located at the same site or another site.
In the above-noted training, analysis design system, each networked computer runs its simulation, representing all of the courses with their associated resources that are located at a single training site, and simulates and records the demand and usage of resources at that particular site. To simulate a training system involving multiple sites, multiple computers are networked together. Synchronization of each site is maintained by a master system Controller. The Controller manages simulator initializations and message handling based on the specified rules of the network communication method described earlier.
Synchronization is a portion of the system that enables the distributed system to work correctly, since otherwise, the variations in speed between the various computers on the network would cause the different sites to diverge in simulation time. So a synchronization period is established, for instance at the end of a training week of simulated processing. At the start of the week, new students from another site are brought into the site/computer. Then the week of simulated training occurs, and any students that graduate or fail out of the courses are released into the network to go to their next course, reassigned course, or wastage point. There the student entity packet waits, until the beginning of the next week, when processing begins again. After each simulator finishes its week of processing, it stops and waits for the next Go signal. The Controller monitors all the computers for completion, then it transmits a Go message to all computers in the network, and they process another simulated week of training.
From the foregoing, it will be observed that numerous variations and modifications may be effected without departing from the spirit and scope of the invention. It is to be understood that no limitation with respect to the specific apparatus illustrated herein is intended or should be inferred. It is, of course, intended to cover by the appended claims all such modifications as fall within the scope of the claims.
