Patent Grant
11507540
The field relates generally to multi-cloud computing environments, and more particularly to techniques for providing a file system and message bus for processing of training data sets in such multi-cloud computing environments.
Enterprises typically utilize a cloud computing environment to manage their data. One or more cloud platforms that are part of this environment may be within the exclusive control and management of the enterprise, and therefore are considered “private clouds” (or private cloud platforms). On the other hand, the cloud computing environment can include one or more cloud platforms that can be used by multiple enterprises, and not necessarily controlled or managed by any of the multiple enterprises but rather are controlled and managed by one or more third-party cloud providers. Such cloud platforms are typically considered “public clouds” (or public cloud platforms). Examples of public clouds include, but are not limited to, Amazon Web Services® (AWS), Google Compute Engine® (GCE), and Windows Azure® Services platforms. More typically, enterprises may choose to host their data (as well as applications, workloads, services, etc.) on a combination of private and public clouds that form a multi-cloud computing environment. One or more clouds may also be referred to as a “data center.” Alternatively, multiple data centers could be implemented in a single cloud.
It is common that an enterprise may desire to deploy and execute a model (e.g., a training model) that is targeted towards a data set (e.g., training data set). By way of example only, the model could be an analytic model that is run on some data set. For example, in a retail use case, the training model could be an algorithm that receives customer purchase data (training data) and tests revenue improvement techniques. Further, in a medical use case, the training data could be physiological data obtained from one or more individuals wearing physiological monitor/sensors. The model that runs on such physiological data could be an algorithm used by an insurance carrier to make insurance policy decisions. Still further, in a traffic control use case, an algorithm (training model) could be run on global positioning system (GPS) coordinates for vehicles (training data) to test ways to ease traffic congestion. In all of these illustrative use cases, a training model is run against a set of training data.
While the enterprise may typically store and maintain the training model and the training data set in a private cloud environment, it is typically desirable to execute the model against the data set in a public cloud environment. However, managing the deployment of the model and data set to the public cloud environment for execution can present many challenges.
Embodiments of the invention provide a decentralized file system and message bus architecture for processing training data sets in multi-cloud computing environments.
For example, in one embodiment, a method comprises the following steps. In a multi-cloud computing environment comprising a plurality of cloud platforms, wherein one cloud platform of the plurality of cloud platforms is a source of a model and a data set and further wherein the model is to be executed against the data set on one or more of the other cloud platforms of the plurality of cloud platforms, the method maintains a decentralized architecture comprising a file system and a message bus, wherein the file system comprises a plurality of decentralized file system nodes corresponding to the plurality of cloud platforms, and the message bus comprises a plurality of decentralized message bus nodes corresponding to the plurality of cloud platforms. Further, the method manages sharing of the model and the data set via at least a portion of the decentralized file system nodes and manages messaging related to execution of the model against the data set via at least a portion of the decentralized message bus nodes.
Advantageously, illustrative embodiments provide improved techniques for managing the deployment of a training model and training data set to a public cloud environment for execution.
These and other features and advantages of the invention will become more readily apparent from the accompanying drawings and the following detailed description.
Illustrative embodiments will be described herein with reference to exemplary information processing systems (referred to as training systems) and associated host devices, storage devices and other processing devices. It is to be appreciated, however, that embodiments are not restricted to use with the particular illustrative system and device configurations shown. Accordingly, the terms “information processing system” and “training system” as used herein are intended to be broadly construed, so as to encompass, for example, processing systems comprising cloud computing and storage systems, as well as other types of processing systems comprising various combinations of physical and virtual computing resources. An information processing system or training system may therefore comprise, for example, a cloud infrastructure hosting multiple tenants that share cloud computing resources. Such systems are considered examples of what are more generally referred to herein as cloud computing environments. A cloud computing environment with multiple cloud platforms is referred to as a “multi-cloud computing environment.” As mentioned above, enterprises may desire to deploy and execute a training model against some training data on such a multi-cloud computing environment. The training system is referred to as a “multi-cloud training system.” The term “enterprise” as used herein is intended to be broadly construed, and may comprise, for example, one or more businesses, one or more corporations or any other one or more entities, groups, or organizations. An “entity” as illustratively used herein may be a person or system.
In order for each of the workers 122-1, 122-2, 122-3 and 122-4 to run the training model 112 against the training data set 114, the training model 112 and the training data set 114 must be transferred to the public cloud environment 120 (as graphically illustrated in
The workers 122-1, 122-2, 122-3 and 122-4 could also experience significantly more inefficiency when they are distributed across multiple clouds. The decision to run workers across cloud platforms could occur for any number of reasons including, but not limited to:
(i) A given cloud may not have enough compute, network, and or storage resources for all workers.
(ii) The model owner may wish to configure the workers across multiple clouds for availability/performance reasons.
(iii) The model owner may wish to configure the workers across multiple clouds for evaluation purposes (evaluating the performance of multiple clouds, e.g., a bake-off).
(iv) The model owner may not wish to have all the data processed on one cloud for security reasons.
(v) The model owner may wish to quickly deploy all workers to a different cloud for cost and/or speed purposes.
In addition, an enterprise may wish to achieve fluidity in their choice of public cloud providers.
As shown in
As described above, there are a variety of ways to transfer training data sets and training models from a master node of a private cloud to the workers in a given public cloud. One approach is to transfer the training data set and the training model to each worker, while another is to transfer the training data set/training model (once) to a shared location available to all the workers.
The following problems arise with respect to transferring the training model and training data set. Note that many of these problems are common to workers that run within the context of one cloud as well as across multiple clouds.
Wait Time for Worker Start
Each worker must wait for the receipt of the entire training model and entire training data set before they can start. This increases the overall completion time of the analytic system (e.g., the training window takes longer to complete).
Data Duplication
Should the system transfer identical copies of the training data set to each worker, a penalty will be paid for: (a) the cost of multiple network transfers; and (b) the cost of duplicate data storage.
Latency for Shared Storage
Should the system place the training data set on a shared storage location available to all the workers, this shared location may not have ideal latency characteristics, especially in environments where workers are distributed across multiple clouds.
Preservation of Worker Results
As workers complete their runs and return results (e.g., parameters) to a master node, the record of these results is often lost. Preservation of these results on a per-worker basis can be used for a variety of purposes, not the least of which is to serve as a permanent record of how quickly the worker came up with their results, and what those results were.
Visibility of Cross-Worker Results
Worker results may be visible to the master node but not to other workers. This prevents slower workers from recognizing delays in their processing (as opposed to the faster workers) and also prohibits them from taking corrective action based on this knowledge.
Specialized Application Programming Interfaces (APIs) for Multi-Cloud Targets
When dealing with multiple cloud provider targets, each target platform has their own nuances for file transfer and collection of results, these nuances become limiting and often require specialized coding, which can slow down implementation and/or introduce errors into the process.
Multi-Cloud Pipelining of Models
The output of one public cloud analytic model may then be fed into a model (different or the same) running on a different cloud.
Illustrative embodiments overcome the above and other challenge in existing training systems by providing a decentralized file system and message bus architecture.
As shown in
Further, as shown in
It is to be appreciated that one or more computing devices at each of the cloud platforms may implement each dFS node and each dMB node.
In one or more illustrative embodiments, the decentralized file system and message bus architecture 400 can be implemented using a decentralized ledger or blockchain network (or RabbitMQ-type framework) and a decentralized data sharing system such as the Interplanetary File System (IPFS) protocol. That is, in illustrative embodiments, the dMB nodes and the dFS nodes in
As used herein, the terms “blockchain,” “digital ledger” and “blockchain digital ledger” may be used interchangeably. As is known, the blockchain or digital ledger protocol is implemented via a distributed, decentralized computer network of compute nodes (e.g., dMB nodes in
In the case of a “bitcoin” implementation of a blockchain distributed ledger, the blockchain contains a record of all previous transactions that have occurred in the bitcoin network. The bitcoin system was first described in S. Nakamoto, “Bitcoin: A Peer to Peer Electronic Cash System,” 2008, the disclosure of which is incorporated by reference herein in its entirety.
A key principle of the blockchain is that it is trusted. That is, it is critical to know that data in the blockchain has not been tampered with by any of the compute nodes in the computer network (or any other node or party). For this reason, a cryptographic hash function is used. While such a hash function is relatively easy to compute for a large data set, each resulting hash value is unique such that if one item of data in the blockchain is altered, the hash value changes. However, it is realized that given the constant generation of new transactions and the need for large scale computation of hash values to add the new transactions to the blockchain, the blockchain protocol rewards compute nodes that provide the computational service of calculating a new hash value. In the case of a Bitcoin network, a predetermined number of bitcoins are awarded for a predetermined amount of computation. The compute nodes thus compete for bitcoins by performing computations to generate a hash value that satisfies the blockchain protocol. Such compute nodes are referred to as “miners.” Performance of the computation of a hash value that satisfies the blockchain protocol is called “proof of work.” While bitcoins are one type of reward, blockchain protocols can award other measures of value (monetary or otherwise) to successful miners.
It is to be appreciated that the above description represents an illustrative implementation of the blockchain protocol and that embodiments of the invention are not limited to the above or any particular blockchain protocol implementation. As such, other appropriate processes may be used to securely maintain and add to a set of data in accordance with embodiments of the invention. For example, distributed ledgers such as, but not limited to, R3 Corda, Ethereum, and Hyperledger may be employed in alternative embodiments.
As mentioned above, a data sharing system such as the IPFS protocol may be employed in the decentralized file system and message bus architecture 400. IPFS is an open-source protocol that provides a decentralized method of storing and sharing files relying on a content-addressable, peer-to-peer hypermedia distribution. The compute nodes in an IPFS network form a distributed file system. The IPFS protocol was developed to replace the HyperText Transfer Protocol (HTTP) of the Internet which relies on location addressing (i.e., using Internet Protocol (IP) addresses to identify the specific computing resource that is hosting a desired data set). As such, the subject data set must be retrieved from the computing resource where it originated or some computing resource within the content delivery network (CDN) each time the data set is requested.
IPFS operates by operatively coupling computing resources with the same system of files via a system of nodes (e.g., dFS nodes in
In one example, the IFPS system is further described in J. Benet, “IPFS—Content Addressed, Versioned, P2P File System,” 2014, the disclosure of which is incorporated by reference herein in its entirety. However, illustrative embodiments are not limited to this particular data sharing system and alternative systems may be employed.
Returning now to
Cloud-Neutral Presentation of Data, Models, and Results
By using the architecture depicted in
This feature solves a number of problems including, but not limited to, the amount of time it would take to transfer the entire file to one (or more) of the cloud providers. It also addresses many of the manageability issues that go along with these file transfers (e.g., account addresses, permissions, etc.).
Cloud Neutral Message Bus for Running Models
Once a training model and a training data set have been loaded into a dFS node of dFS 410, the owner that wishes to run the model can simply insert a message into the decentralized message bus 420 (
Instant Launch and Completion/Results
Once a command has been issued to the dMB 420, the target cloud platform instantly begins running the training model against the training data set. This significantly shortens model iteration time for a number of reasons:
(i) The remote cloud does not have to wait for the training model, and the entire training data set, to be transferred over.
(ii) The local user does not have to step through the manual processes involved with transferring the entirety of both files, maintaining/keeping/deleting previous versions, dealing with user accounts, etc.
(iii) Publishing of results is similarly instant; the cloud provider can simply notify the decentralized message bus that the run is complete. Neither the local user or the remote cloud has to manually transfer the results back to the central location.
Reduced Data Transfers/Costs
Training data sets can be large, and the transfer of the entire training data set to a public cloud provider will come with a cost. This is especially wasteful since an analytic training model will likely only access a portion of the training data set (and not the entire file).
Accordingly, in one or more illustrative embodiments of the decentralized file system, only the requested segments of the file are transferred. In addition, once these segments cross over to the public cloud to a specific worker, that worker keeps (or “pins”) the segment locally so that it can be shared with other workers. The optimization of only transferring the data (once) that is needed to the public cloud not only speeds up the overall run time but it also reduces the bill that the public cloud provider will send when counting bytes for data transfer.
If any of the workers 222-1, 222-2, 222-3 and 222-4 in public cloud platform 220 request a data segment that has already been stored in the dFS node 412-1 (e.g., worker 222-1 requesting “Range1” in
Efficient Pipelining
As multi-stage analytic jobs execute across multiple cloud platforms, the intermediate published results can simply be placed in a dFS folder and a message signaled to the next cloud via the dMB. This eliminates the bottleneck of pushing results back to a central arbiter (e.g., master node of the private cloud platform 210), which in turn pushes these results (and additional data) to the next processing cloud in the pipeline.
More particularly,
At least portions of the decentralized file system and message bus architecture shown in
As is apparent from the above, one or more of the processing modules or other components of the decentralized file system and message bus architecture shown in
The processing platform 1000 in this embodiment comprises a plurality of processing devices, denoted 1002-1, 1002-2, 1002-3, . . . 1002-N, which communicate with one another over a network 1004.
The network 1004 may comprise any type of network, including by way of example a global computer network such as the Internet, a WAN, a LAN, a satellite network, a telephone or cable network, a cellular network, a wireless network such as a WiFi or WiMAX network, or various portions or combinations of these and other types of networks.
As mentioned previously, some networks utilized in a given embodiment may comprise high-speed local networks in which associated processing devices communicate with one another utilizing Peripheral Component Interconnect Express (PCIe) cards of those devices, and networking protocols such as InfiniBand, Gigabit Ethernet or Fibre Channel.
The processing device 1002-1 in the processing platform 1000 comprises a processor 1010 coupled to a memory 1012.
The processor 1010 may comprise a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other type of processing circuitry, as well as portions or combinations of such circuitry elements.
The memory 1012 may comprise random access memory (RAM), read-only memory (ROM) or other types of memory, in any combination. The memory 1012 and other memories disclosed herein should be viewed as illustrative examples of what are more generally referred to as “processor-readable storage media” storing executable program code of one or more software programs.
Articles of manufacture comprising such processor-readable storage media are considered embodiments of the present disclosure. A given such article of manufacture may comprise, for example, a storage array, a storage disk or an integrated circuit containing RAM, ROM or other electronic memory, or any of a wide variety of other types of computer program products. The term “article of manufacture” as used herein should be understood to exclude transitory, propagating signals. Numerous other types of computer program products comprising processor-readable storage media can be used.
Also included in the processing device 1002-1 of the example embodiment of
The other processing devices 1002 of the processing platform 1000 are assumed to be configured in a manner similar to that shown for processing device 1002-1 in the figure.
Again, this particular processing platform is presented by way of example only, and other embodiments may include additional or alternative processing platforms, as well as numerous distinct processing platforms in any combination, with each such platform comprising one or more computers, servers, storage devices or other processing devices.
For example, other processing platforms used to implement embodiments of the disclosure can comprise different types of virtualization infrastructure, in place of or in addition to virtualization infrastructure comprising virtual machines. Such virtualization infrastructure illustratively includes container-based virtualization infrastructure configured to provide Docker containers or other types of Linux containers (LXCs).
The containers may be associated with respective tenants of a multi-tenant environment, although in other embodiments a given tenant can have multiple containers. The containers may be utilized to implement a variety of different types of functionality within the system. For example, containers can be used to implement respective cloud compute nodes or cloud storage nodes of a cloud computing and storage system. The compute nodes or storage nodes may be associated with respective cloud tenants of a multi-tenant environment. Containers may be used in combination with other virtualization infrastructure such as virtual machines implemented using a hypervisor.
As another example, portions of a given processing platform in some embodiments can comprise converged infrastructure such as VxRail™, VxRack™ or Vblock® converged infrastructure commercially available from VCE, the Virtual Computing Environment Company, now the Converged Platform and Solutions Division of Dell EMC. For example, portions of a system of the type disclosed herein can be implemented utilizing converged infrastructure.
It should therefore be understood that in other embodiments different arrangements of additional or alternative elements may be used. In many embodiments, at least a subset of these elements may be collectively implemented on a common processing platform, or each such element may be implemented on a separate processing platform.
Also, in other embodiments, numerous other arrangements of computers, servers, storage devices or other components are possible in the decentralized file system and message bus architecture. Such components can communicate with other elements of the system over any type of network or other communication media.
As indicated previously, in some embodiments, components of the decentralized file system and message bus architecture as disclosed herein can be implemented at least in part in the form of one or more software programs stored in memory and executed by a processor of a processing device. For example, at least portions of the dFS, the dMB, or other system components are illustratively implemented in one or more embodiments in the form of software running on a processing platform comprising one or more processing devices.
It should again be emphasized that the above-described embodiments of the disclosure are presented for purposes of illustration only. Many variations and other alternative embodiments may be used. For example, the disclosed techniques are applicable to a wide variety of other types of systems. Also, the particular configurations of system and device elements, associated processing operations and other functionality illustrated in the drawings can be varied in other embodiments. Moreover, the various assumptions made above in the course of describing the illustrative embodiments should also be viewed as exemplary rather than as requirements or limitations of the embodiments. Numerous other alternative embodiments within the scope of the appended claims will be readily apparent to those skilled in the art.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6992978 | Humblet | Jan 2006 | B1 |
| 9967334 | Ford | May 2018 | B2 |
| 10075298 | Struttmann | Sep 2018 | B2 |
| 10210115 | MacKenzie | Feb 2019 | B2 |
| 10310760 | Dreier | Jun 2019 | B1 |
| 10389602 | Chang | Aug 2019 | B2 |
| 10496926 | Florissi | Dec 2019 | B2 |
| 10509684 | Florissi | Dec 2019 | B2 |
| 10515097 | Florissi | Dec 2019 | B2 |
| 10567234 | Todd | Feb 2020 | B1 |
| 10587413 | Todd | Mar 2020 | B1 |
| 10698879 | Todd | Jun 2020 | B1 |
| 10824740 | Wu | Nov 2020 | B2 |
| 10833960 | Ranchal | Nov 2020 | B1 |
| 10841079 | Todd | Nov 2020 | B1 |
| 10841237 | Todd | Nov 2020 | B2 |
| 10855758 | O'Connell | Dec 2020 | B1 |
| 10922304 | Todd | Feb 2021 | B1 |
| 10924361 | Todd | Feb 2021 | B2 |
| 10956377 | Todd | Mar 2021 | B2 |
| 11010265 | Wu | May 2021 | B2 |
| 11025626 | Todd | Jun 2021 | B1 |
| 11038891 | Wu | Jun 2021 | B2 |
| 11128463 | Thompson | Sep 2021 | B1 |
| 11164115 | Todd | Nov 2021 | B1 |
| 11334456 | Natanzon | May 2022 | B1 |
| 11341009 | Wu | May 2022 | B1 |
| 20110137991 | Russell | Jun 2011 | A1 |
| 20120266156 | Spivak | Oct 2012 | A1 |
| 20150347542 | Sullivan | Dec 2015 | A1 |
| 20160055045 | Souza | Feb 2016 | A1 |
| 20160147952 | Garcia | May 2016 | A1 |
| 20160171682 | Abedini | Jun 2016 | A1 |
| 20160300142 | Feller | Oct 2016 | A1 |
| 20160357778 | MacKenzie | Dec 2016 | A1 |
| 20160378450 | Fu | Dec 2016 | A1 |
| 20170013046 | Flynn | Jan 2017 | A1 |
| 20170257432 | Fu | Sep 2017 | A1 |
| 20170357532 | Miraftabzadeh | Dec 2017 | A1 |
| 20180114334 | Desai | Apr 2018 | A1 |
| 20180165386 | Soundiramourthy | Jun 2018 | A1 |
| 20180184232 | Maitra | Jun 2018 | A1 |
| 20180240191 | Aronson | Aug 2018 | A1 |
| 20180264347 | Tran | Sep 2018 | A1 |
| 20180308072 | Smith | Oct 2018 | A1 |
| 20180375840 | Moy | Dec 2018 | A1 |
| 20190026839 | Drangmeister | Jan 2019 | A1 |
| 20190065353 | Tammariello | Feb 2019 | A1 |
| 20190079744 | Bosch | Mar 2019 | A1 |
| 20190087239 | Adibowo | Mar 2019 | A1 |
| 20190102389 | LeGault | Apr 2019 | A1 |
| 20190238486 | Zizka | Aug 2019 | A1 |
| 20190318012 | Danilov | Oct 2019 | A1 |
| 20190325044 | Gray | Oct 2019 | A1 |
| 20190325155 | Wu | Oct 2019 | A1 |
| 20190327180 | Todd | Oct 2019 | A1 |
| 20190349426 | Smith | Nov 2019 | A1 |
| 20190358515 | Tran | Nov 2019 | A1 |
| 20190361917 | Tran | Nov 2019 | A1 |
| 20190372762 | Shainski | Dec 2019 | A1 |
| 20200019626 | Todd | Jan 2020 | A1 |
| 20200021444 | Young | Jan 2020 | A1 |
| 20200023812 | Hassani | Jan 2020 | A1 |
| 20200034548 | Wu | Jan 2020 | A1 |
| 20200058381 | Patel | Feb 2020 | A1 |
| 20200089509 | Arya | Mar 2020 | A1 |
| 20200117690 | Tran | Apr 2020 | A1 |
| 20200137064 | Wu | Apr 2020 | A1 |
| 20200201827 | Chacko | Jun 2020 | A1 |
| 20200219088 | Dhingra | Jul 2020 | A1 |
| 20200225655 | Cella | Jul 2020 | A1 |
| 20200233760 | Wu | Jul 2020 | A1 |
| 20200236168 | Todd | Jul 2020 | A1 |
| 20200242200 | Todd | Jul 2020 | A1 |
| 20200302496 | Todd | Sep 2020 | A1 |
| 20200313982 | Todd | Oct 2020 | A1 |
| 20200348662 | Cella | Nov 2020 | A1 |
| 20200351323 | Gong | Nov 2020 | A1 |
| 20210014065 | Gourisetti | Jan 2021 | A1 |
| 20210019303 | Gangwar | Jan 2021 | A1 |
| 20210056141 | Wojcik | Feb 2021 | A1 |
| 20210157312 | Cella | May 2021 | A1 |
| 20210294805 | Damodaran | Sep 2021 | A1 |
| 20210366586 | Ryan | Nov 2021 | A1 |
| 20210374125 | Kumar | Dec 2021 | A1 |
| 20210374154 | Kumar | Dec 2021 | A1 |
| 20220067738 | Fang | Mar 2022 | A1 |
| 20220067752 | Fang | Mar 2022 | A1 |
| 20220108262 | Cella | Apr 2022 | A1 |
| 20220129257 | Touati | Apr 2022 | A1 |
| 20220141198 | Lundgren | May 2022 | A1 |
| 20220156665 | Beth | May 2022 | A1 |
| 20220207119 | Andon | Jun 2022 | A1 |
| Entry |
|---|
| Satoshi Nakamoto, “Bitcoin: A Peer-to-Peer Electronic Cash System,” www.bitcoin.org, 2008. |
| Juan Benet, “IPFS-Content Addressed, Versioned, P2P File System,” arxiv.org/abs/1407.3561, 2014. |
