Patent Application
20110179028
The Internet comprises a plurality of websites/webpages that provide highly interactive and immersive applications, along with transactional-based applications, to users. Social networking, online games, online environments, and conferencing/chatting, for example, allow users to interact with each other and virtual environments. Further, transaction based applications, such as online product/service purchasing, account set-up, and financial service, typically require that users enter information to the web-application (web-app).
These web-based interactions/transactions typically generate data that is stored off (e.g., off the web) so that it can be later processed or accessed by another or the same application. For example, where a user updates their personal status message (e.g., in a social network or micro-blogging site), the data generated by entering the update can be stored off in a data store and later retrieved by a feed distribution service. Here, the feed distribution service may distribute the PSM update to subscribed contacts of the user.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Data generated by a web-event, which is to be stored off to a data store, is typically sent to a web-based asynchronous queue service (e.g., where data is put in then taken out asynchronously). The asynchronous queue service then stores the data off in a data store. Data from a web-event is typically generated in sequence, such as updates to a feed, and is subsequently processed or retrieved in the order in which it is generated. This type of storage and retrieval is commonly referred to as first-in-first-out (FIFO), such that data is aggregated, stored, and retrieved in a FIFO manner.
Web-apps that generate data to be aggregated and stored off can generate large amounts of data, and a plurality of web-apps, such as found on the Internet, exponentially increase the amount of data that may need to be stored. If a large amount of data is aggregated in a FIFO queue, it becomes very difficult to store it on one storage device (e.g., server). Therefore, asynchronous queue services typically employ distributed data stores to store the aggregated data, where the data is stored and organized over a plurality of storage devices. However, when aggregated data is distributed over a distributed data store it gives up the FIFO aggregation structure. Therefore, when data is retrieved or processed from a distributed data store it may not be retrieved in a FIFO manner, which can lead to improper and/or slow processing of the data.
One or more techniques and/or systems are disclosed that provide for aggregation of data from a web-event, regardless of an amount of data generated, where the data can be stored in a FIFO queue on a distributed data store. For example, a web-event can generate data and the data can be tagged with an aggregation queue. In this example, the data can be aggregated into a FIFO queue using the aggregation key, and the FIFO queue can be stored off in a distributed data store.
In one embodiment, web-based data, such as generated by a web-event, is aggregated and stored in a distributed data store, so that it can be retrieved in a first-in, first-out (FIFO) manner. A unique aggregation key is generated for data generated from a web-based event, if the data is to be added to the distributed data store, where the aggregation key corresponding merely to the data generated from the web-based event. The data from the web based event is aggregated in a FIFO queue and stored in a same partition of the distributed data store, based upon the aggregation key.
To the accomplishment of the foregoing and related ends, the following description and annexed drawings set forth certain illustrative aspects and implementations. These are indicative of but a few of the various ways in which one or more aspects may be employed. Other aspects, advantages, and novel features of the disclosure will become apparent from the following detailed description when considered in conjunction with the annexed drawings.
The claimed subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the claimed subject matter. It may be evident, however, that the claimed subject matter may be practiced without these specific details. In other instances, structures and devices are shown in block diagram form in order to facilitate describing the claimed subject matter.
Typically, the user types a message into the feed input service 102, such as FEED1 listing the user's status as “waiting for grades coming out today.” The feed is forwarded to a web-based 106 asynchronous queue service 108, which may be one of a plurality of queue services that asynchronously put data in and take data out of a work queue, where the data comprising the feed is tagged. The tagged data can then be stored off in a distributed data storage service 110 for processing.
As an example, a user may update their PSM rapidly (e.g., several times over a couple of minutes), as shown in the example environment 100. Here, the user's FEED2 stated that “I passed, yeah,” however they may have quickly realized that they did not and immediately updated the feed to say “oops, I read it wrong.” The feeds are sent to the work queue service 108 (e.g., an asynchronous queue service), over the web 106, and stored off in the database service 110. A feed distribution service 104, such as a friend status updater, can detect that user1's status has changed and pickup the feeds from the work queue service 108.
The work queue service 108 retrieves the feeds from the database service 110, and forwards them to the feed distribution service 104, where they can be sent out and/or displayed to those subscribed to the feed distribution service 104 for user1. In this example, feeds are retrieved from by the distribution service 104 in the order in which they are sent out by the work queue service 108. As shown in the example, it is typically very important that the feeds be retrieved by the feed distribution service 104 in a same order in which they were input by the feed input service 102. For example, it would be a little confusing if user1's first output feed said “oops I read it wrong,” and the second output feed said “I passed, yeah.” This example has a completely different meaning than the original feed order, described above.
That is, when a web-event generates some data, and the generated data is to be stored off in the distributed data store, an aggregation key is generated for that data to be stored from that event, which is specific to that data, for example. In one embodiment, the data can be stored off, along with a unique aggregation key for the web-event that identifies the data from that event. As an example, a web-event may comprise an online transaction, such as a product purchase. In this embodiment, for example, an aggregation key can be generated for the purchase information entered by the purchaser (e.g., account information, address, etc.), where the key is merely associated with that transaction.
At 206, in the exemplary method 200, data from the web based event is aggregated in a FIFO queue, in a same partition of the distributed data store, using the aggregation key. That is, the distributed data store, used to store off the web-event generated data, is partitioned into a plurality of data storage partitions. In one embodiment, partitions are virtual storage blocks, in as much as the partitions do not represent physical barriers, but merely virtually created combinations of available storage using a directory structure.
As an illustrative example, all of the data generated from a single web-based event, such as the online transaction, is stored in a same partition. It is aggregated together in the partition, in this example, if it has the same unique key for the transaction. Further, the data from the transaction is aggregated in a FIFO queue, where data that is put into the data store first is taken from the data store first, for example, when requested for retrieval. This can be of particular importance, for example, where an order of the data from the web-event is relevant, such as a PSM update, as described above. Further, for example, a FIFO queue typically represents an asynchronous queue, where data is put in and taken out at different times (e.g., asynchronously).
In one embodiment, a first partition of the distributed data store is designated as a storage location to store aggregated data associated with a first aggregation key; and a second partition of the distributed data store is designated as a storage location to store aggregated data associated with a second aggregation key. That is, in one embodiment, when a first web-event generates data, a first aggregation key is generated for that first data, and when a second web-event generates data, a second aggregation key is generated for that second data.
As an example, a first partition of the distributed data store is designated to store the first data, identified by the first aggregation key, and a second partition of the data store is designated to store the second data, identified by the second aggregation key. In these embodiments, this example is merely intended to illustrate how data associated with a particular aggregation key is stored in a same partition, and different aggregation keys can be assigned to different partitions. It will be appreciated that, in some embodiments, data associated with different aggregation keys can be stored in a same partition. However, data associated with a first aggregation key is stored merely in a first partition, and is not distributed over a plurality of partitions.
Having aggregated the data, the exemplary method 200, of
At 306, if the data generated by the web event is not to be stored, for example, where the web-service does not need to store the data for later processing, the data is not stored, at 318. On the other hand, in this embodiment, if the web-service wants the data to be stored off for later processing, a processing loop runs from 308 to 314. For each of the data generated by the web-event, such as for each part of a transaction, an aggregation key is generated (e.g., a key unique to the web-event), at 310.
The aggregation key is not limited to any particular format, merely that is can be used to specifically identify the data associated with a particular web-event. In some embodiments, the aggregation key may comprise: a randomly generated string, such as a string of alpha-numeric symbols; or a hash (hash value) generated by hashing the data (e.g., running the data through a hash function); or a sequentially generated string (e.g., a next sequential value is assigned as a key for a next set of data).
At 312, the aggregation key is attached to the data. It will be appreciated that attaching the aggregation key to the data is not limited to a particular embodiment. The attachment of the aggregation key to the data is intended to allow the data to be identified by the key. For example, the key can be attached as metadata to the data by adding the metadata sequence to the data sequence, such as a prefix or suffix. As another example, the key may be linked in data table, where if one looks up the key in the table the data linked to the key can be identified.
At 314, another data item generated from the web-event can be run through the loop beginning at 308. After respective data generated by the web-event have the aggregation key attached, the data can be stored to a partition in the distributed data store, at 316. Here, the data that has the same aggregation key is stored to the same partition. As an example, all the data generated by the web-event is given the same aggregation key, and, using that aggregation key, all the data from that web-event is aggregated together in the same partition, and not split-up between different partitions. In one embodiment, an operation can be performed on the distributed data store, for example, by the web queue service, where an add operation is used to store data to the partition of the distributed operation in the FIFO queue.
Further, in this embodiment, the aggregated data is stored in the partition of the distributed data store in a first-in-first-out (FIFIO) queue. That is, for example, data that is generated first by the web-event (e.g., a first PSM update) is stored first in the FIFO queue, and subsequent data generated by the web-event (e.g., subsequent PSM updates) are stored second, third, etc. in the FIFO queue. In this way, in this example, when data is later taken from the data store, the data that is first in the FIFO queue is taken first, such that the first in data is the first out data.
At 402, the second web service connects to the work queue service (e.g., 108), such as by connecting over the Internet (e.g., 106). In one embodiment, the web queue service may comprise a service oriented application executed on one or more computers, which may be used to generated the aggregation key and aggregate the data, such as for storage in the distributed data store. In this embodiment, the service oriented application may be built from an API in a software development framework. As an example, a SOAP-based service application (originally an acronym for “Simple Object Access Protocol,” however common art usage of SOAP no longer applies to the original acronym) can be created using a framework API (e.g., Windows Communication Foundation). In this example, the second web service can also contact the work queue service.
At 404, the second web service (e.g., 104) requests data from the web queue service (e.g., 108). As an example, the second web-service may be a feed distribution service (e.g., distributing a user's feed to the user's connection), which, when notified that a user's feed has been updated, requests the feed data from the web queue service. As a further example, a user may be listening to a new music album, recently purchased, and has settings that notify his subscribing contacts whenever the user is listening to music. The second web service, as a feed distribution service, contacts the web queue service and requests the data that identifies the music.
At 406 in the exemplary portion of the method 400, a data retrieval command can be sent to the data store using the aggregation key. For example, the web queue service (e.g., 108) can send a command that performs a take operation on the data store, intended to retrieve data from the FIFO queue of the partition in the distributed data store (e.g., 110).
Here, upon request, data that has a same aggregation key can be retrieved from the distributed data store in a FIFO manner, at 408. As described above, the data having the same aggregation key attached is aggregated together in one partition of the data store. Therefore, for example, all of the data associated with the web-event for which the aggregation key was generated can be retrieved at a same time, by using the command that identifies the aggregation key. Further, in this example, the data will be retrieved in a FIFO manner, as it is stored in a FIFO queue in the partition. In this way, in this example, the data that went in first will come out first.
At 410, the retrieved data is sent to the second web service in a FIFO manner. In this way, as an example, the second web service can broadcast to the subscribing contacts that “the user is listening to the new album.” Further, other information associated with the web event can be distributed by the second web service, for example, in a manner consistent with when it was generated, which may avoid potentially confusing situations, as described in
In one embodiment, the service oriented application (e.g., the web queue service 108 of
In another embodiment, the distributed data store may be web-based, such that the storage volumes and hierarchical directory structures merely utilize the Internet for connecting and retrieving data. In another embodiment, the data store and queue services may be operating together as a same service, and/or one a same device. It will be appreciated that those skilled in the art may devise alternate configurations for these services, and the methods and systems described herein are not limited to any particular embodiment.
Further, in one aspect, the distributed data store can be comprise of a plurality of data storage devices, where related data can be stored on different devices, even if it is in a same partition.
Respective volumes 512, 514 can be made up of one or more portions of physical storage devices 516, 518, such as physical storage disks disposed in one or more data storage devices. Volumes (e.g., 512, 514) can span a portion of a disk, a cluster of disks, or portions of different disks, for example, enabling a distributed data store to define an overall logical arrangement of file storage. Disks (e.g., 516, 518) may comprise any type of mass storage devices, such as magnetic disk drives, flash memory, and any other similar media adapted to store information.
In one embodiment a volume can comprise data stored as one or more files 504, 506, 508, 510 that reside in a hierarchical directory structure within the volume, such as a partition 502. Therefore, for example, while data from a same web-event, having a same aggregation key, can be aggregated in a FIFO queue (e.g., 504) on a partition (e.g., 502), because the partition may be disposed on a volume (e.g., 512) that is comprised on one or more storage devices (e.g., 516), the aggregated data may be store in different physical locations. However, in this embodiment, because the partition organizes the data hierarchically, the aggregated data can be quickly identified and retrieved in a FIFO manner using the aggregation key.
A system may be devised that provides for related data, such as generated by a web event, to be aggregated on a distributed data store. The system can provide for related data generated by a web-based event to be aggregated, and stored in a FIFO queue. In this way, the data can be retrieved from this distributed data store in a FIFO manner, even while the system comprises a highly distributed data storage protocol, which is also highly scalable.
The work queue service component 604 comprises an aggregation key generation component 605 that generates an aggregation key corresponding merely to data generated from a same web-based event, for data that is intended to be added to the distributed data store 602. For example, a user may interact with a web-based service 654 (e.g., an web-app running on a website, such as a feed input service) over the Internet 652, such that the interaction provides a web event that generates data to be stored in the data store for later processing, such as by another web-based service 654 (or the same one).
The work queue service component 604 further comprises a queue storing component 608 that aggregates the data generated by the web-event in a FIFO queue in a same partition of the distributed data store based on the aggregation key. That is, for example, the aggregation key generator 606 generates a unique key for the data generated by the web event, and the queue storing component 608 aggregates the web event data together using the aggregation key. In this example, the aggregated data is stored in a FIFO queue together in one partition of the distributed data store 602.
In one aspect, the web-based work queue service 604 may comprise a service-oriented application that is running on the one or more computers 650 utilizing a web-based environment. Here, the service-oriented application, can be built from an application programming interface (API) in a software development framework, and use a SOAP interface to communicate with the one or more services 724, 726.
In this embodiment 700, the web-based work queue service component 604 comprises an aggregation key attachment component 720 that attaches the aggregation key to respective web-based data from a same web-based event. For example, when the work queue service 604 detects a web-event, such as undertaken by the web-based service data generator 724, and an aggregation key is generated for respective data by the aggregation key generator 606, the aggregation key attacher 720 can attach the key to the data.
The aggregation key attacher 720 is not limited to any particular embodiment in attaching the key. For example, the key may be appended before or after the data, such as a prefix or suffix; or the key may be linked to the data using a look up table (e.g., an XML lookup table). Further, the aggregation key is not limited to any particular embodiment. For example, the aggregation key may comprise a string generated for the web-based event that corresponds merely to the web-based event, such as a randomly generated alpha-numeric string, or a sequentially generated string. As another example, the aggregation key may comprise a hash (hash value) generated by hashing the data from the web-based event (e.g., running the data through a hash function).
In the exemplary embodiment 700, the web-based work queue service component 604 comprises a data retrieval component 722 that can provide data to a web-based service, such as the web-based service data user 726, upon request. The data retrieval component 722 can retrieve and provide the data in a FIFO manner from the partitioned queue on the distributed data store 602.
Still another embodiment involves a computer-readable medium comprising processor-executable instructions configured to implement one or more of the techniques presented herein. An exemplary computer-readable medium that may be devised in these ways is illustrated in
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
As used in this application, the terms “component,” “module,” “system”, “interface”, and the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.
Although not required, embodiments are described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions may be distributed via computer readable media (discussed below). Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. Typically, the functionality of the computer readable instructions may be combined or distributed as desired in various environments.
In other embodiments, device 912 may include additional features and/or functionality. For example, device 912 may also include additional storage (e.g., removable and/or non-removable) including, but not limited to, magnetic storage, optical storage, and the like. Such additional storage is illustrated in
The term “computer readable media” as used herein includes computer storage media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions or other data. Memory 918 and storage 920 are examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by device 912. Any such computer storage media may be part of device 912.
Device 912 may also include communication connection(s) 926 that allows device 912 to communicate with other devices. Communication connection(s) 926 may include, but is not limited to, a modem, a Network Interface Card (NIC), an integrated network interface, a radio frequency transmitter/receiver, an infrared port, a USB connection, or other interfaces for connecting computing device 912 to other computing devices. Communication connection(s) 926 may include a wired connection or a wireless connection. Communication connection(s) 926 may transmit and/or receive communication media.
The term “computer readable media” may include communication media. Communication media typically embodies computer readable instructions or other data in a “modulated data signal” such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” may include a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
Device 912 may include input device(s) 924 such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, and/or any other input device. Output device(s) 922 such as one or more displays, speakers, printers, and/or any other output device may also be included in device 912. Input device(s) 924 and output device(s) 922 may be connected to device 912 via a wired connection, wireless connection, or any combination thereof. In one embodiment, an input device or an output device from another computing device may be used as input device(s) 924 or output device(s) 922 for computing device 912.
Components of computing device 912 may be connected by various interconnects, such as a bus. Such interconnects may include a Peripheral Component Interconnect (PCI), such as PCI Express, a Universal Serial Bus (USB), firewire (IEEE 1394), an optical bus structure, and the like. In another embodiment, components of computing device 912 may be interconnected by a network. For example, memory 918 may be comprised of multiple physical memory units located in different physical locations interconnected by a network.
Those skilled in the art will realize that storage devices utilized to store computer readable instructions may be distributed across a network. For example, a computing device 930 accessible via network 928 may store computer readable instructions to implement one or more embodiments provided herein. Computing device 912 may access computing device 930 and download a part or all of the computer readable instructions for execution. Alternatively, computing device 912 may download pieces of the computer readable instructions, as needed, or some instructions may be executed at computing device 912 and some at computing device 930.
Various operations of embodiments are provided herein. In one embodiment, one or more of the operations described may constitute computer readable instructions stored on one or more computer readable media, which if executed by a computing device, will cause the computing device to perform the operations described. The order in which some or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated by one skilled in the art having the benefit of this description. Further, it will be understood that not all operations are necessarily present in each embodiment provided herein.
Moreover, the word “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous over other aspects or designs. Rather, use of the word exemplary is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims may generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.
Also, although the disclosure has been shown and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, resources, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the disclosure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms “includes”, “having”, “has”, “with”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising.”
