Patent Grant
9652310
A portion of the disclosure of this patent document may contain command formats and other computer language listings, all of which are subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
This application relates to asynchronous messaging.
Asynchronous message processing is a very common pattern in computer science. Techniques involving Message Oriented Middleware, such as IBM's MQ Series or open source products such as Pivotal's (formerly VMWare's) RabbitMQ, provide great flexibility in building distributed information system architectures wherein data processing can happen in a distributed environment involving multiple simultaneous programs sending and receiving data over a message bus.
In order to increase the throughput of message processing, typically, in order to achieve processing of a large volume of event messages (e.g., stock trade data in a financial domain or alert/alarm processing from devices and other components in an IT infrastructure), the traditional best practice is to use scale out architectures, wherein a multiplicity of processing nodes are co-deployed and work in parallel to process the stream of incoming messages. This pattern is very typical in computer science and very common in Message Oriented Middleware centric architectures. A description of this pattern, commonly called the “worker queue pattern” in the context of message oriented middleware, can be found: http://www.rabbitmq.com/tutorials/tutorial-two-java.html, which is incorporated herein by reference in its entirety.
Example embodiments of the present invention provide a method and a system for using consistent-hashing to ensure proper sequencing of message processing in a scale-out environment. The method includes receiving a stream of a plurality of messages and performing a consistent-hash on each message of the plurality of messages. Each message then may be distributed toward a respective downstream worker of a plurality of downstream workers for processing according to results of the hash.
The above and further advantages of the present invention may be better under stood by referring to the following description taken into conjunction with the accompanying drawings in which:
A significant problem with the worker queue pattern arises when the event messages are not independent. For example, in the IT infrastructure monitoring domain, if a stream of messages depicting alerts/alarms from the infrastructure contains arbitrary sequences of alerts related to the same resource, it is critical that the sequence be maintained in exact order throughout the processing steps. With parallel asynchronous processing, such as appears in scale-out architectures using the worker queue pattern, it is very difficult to guarantee that messages related to a particular resource are processed exactly in the sequence they are generated.
For example, consider a sequence of events in which each event has two salient information items: the event “name” (reflecting the semantic of the message) and the event resource (reflecting the entity about which the event refers). In the case of alert/alarm processing, the event “name” might be “New Alert” (reflecting that this is a new situation for the corresponding entity), “Update Alert” (reflecting a change to an event previous in the event stream) or “Clear (Delete) Alert” (referring to an alert previous in the alert stream). The event resource in this example may be an identifier for the IT infrastructure component for which that alert/alarm event was created. It should be understood that these resources may be physical infrastructure devices (e.g., router, switch, host, array), virtual devices (e.g., virtual machines), or software components (e.g., databases, reverse web serving proxies, etc.). The stream of events may combine events for different resources, as well as a set of events related to the same resource. As described above, the sequence of these events matters to maintain the “New Alert”, “Update Alert” (1 or more times), “Delete Alert” semantic (i.e., if any of the messages is processed out of order, the entire system may enter an inconsistent state, rendering the solution unreliable).
In a traditional scale out architecture without the queuing system 120, where multiple workers are processing the event stream 100 in parallel, it is quite possible that message sequences can be processed out of order. If the three workers 170 each process the stream 100 in parallel and treat the stream 100 as a queue (i.e., the worker queue pattern), message sequencing can get out of order because the processing cannot be guaranteed to preserve sequence of events (i.e., each of the workers 170 would read from the same shared queue, process messages, and update the database, as is intended in the worker queue pattern). For example, in this hypothetical without the queuing system 120 of example embodiments of the present invention, the problem is in maintaining the sequence of events Em with respect to a resource Rn. For example, if worker 11701 dequeues and processes message 101 E1-R1, worker 21702 dequeues and processes message 102 E1-R2, and worker 31703 dequeues and processes message 103 E2-R2, and presuming worker 21702 takes longer to process message 102 E1-R2 than worker 31703 takes to process message 103 E2-R2 (e.g., it gets blocked because of other demands on the hardware upon which it is executing), worker 31703 would update the database 190 with the results of processing message 103 E2-R2 before the database 190 is updated with the results of worker 21702 processing message 102 E1-R2 (i.e., the events are processed out of order and the semantic of sequence preservation is lost).
Accordingly, example embodiments of the present invention overcome these and other deficiencies by using consistent-hashing to ensure proper sequencing of message processing in a scale-out environment.
As illustrated in
Accordingly, consistent-hashing guarantees that for any resource (e.g., any value of Rn), messages will always be delivered to the same queue 150. Advantageously, the relative processing speeds of the workers 170 does not affect the sequence of processing of the events 101-112 because events for the same resource R1, are always delivered to the same queue 150 and, therefore, the same worker 170 for processing. In other words, referring to the hypothetical posted above, it is not possible for message 103 E2-R2 to be processed before message 102 E1-R2. Therefore, in example embodiments of the present invention, scale out architectures may be built safely that process a message stream 100 in a specific sequence order, which is critical for scale out solutions to IT infrastructure alert/alarm solutions and management applications, such as Smarts by EMC Corporation of Hopkinton, Mass.
As illustrated in
The methods and apparatus of this invention may take the form, at least partially, of program code (i.e., instructions) embodied in tangible non-transitory media, such as floppy diskettes, CD-ROMs, hard drives, random access or read only-memory, or any other machine-readable storage medium. When the program code is loaded into and executed by a machine, such as the computer of
The logic for carrying out the method may be embodied as part of the aforementioned system, which is useful for carrying out a method described with reference to embodiments shown. For purposes of illustrating the present invention, the invention is described as embodied in a specific configuration and using special logical arrangements, but one skilled in the art will appreciate that the device is not limited to the specific configuration but rather only by the claims included with this specification.
Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present implementations are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8838817 | Biswas | Sep 2014 | B1 |
| 20050166187 | Das | Jul 2005 | A1 |
