The field of the invention is data processing, or, more specifically, methods, apparatus, and products for emulating a remote direct memory access (‘RDMA’) link between controllers in a storage array.
Modern storage systems can include many storage devices that are accessed via multiple controllers. Communication between the controllers may be facilitated through the use of special purpose adapters, cables, and other computer hardware. Such adapters, cables, and other computer hardware, however, are often expensive and consume valuable resources such as space, power, and the like. Furthermore, incorporating the controllers into a single form factor can make the inclusion of such adapters, cables, and other computer hardware impossible.
Methods, apparatuses, and products for emulating a remote direct memory access (‘RDMA’) link between controllers in a storage array, including: inserting, into a buffer utilized by a direct memory access (‘DMA’) engine of a first storage array controller, a data transfer descriptor describing data stored in memory of the first storage array controller and a location to write the data to memory of the second storage array controller; retrieving, in dependence upon the data transfer descriptor, the data stored in memory of the first storage array controller; and writing the data into the memory of the second storage array controller in dependence upon the data transfer descriptor.
The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular descriptions of example embodiments of the invention as illustrated in the accompanying drawings wherein like reference numbers generally represent like parts of example embodiments of the invention.
Example methods, apparatuses, and products for emulating an RDMA link between controllers in a storage array in accordance with the present invention are described with reference to the accompanying drawings, beginning with
The computing devices (164, 166, 168, 170) in the example of
The local area network (160) of
The example storage arrays (102, 104) of
Each storage array controller (106, 112) may be implemented in a variety of ways, including as a Field Programmable Gate Array (‘FPGA’), a Programmable Logic Chip (‘PLC’), an Application Specific Integrated Circuit (‘ASIC’), or computing device that includes discrete components such as a central processing unit, computer memory, and various adapters. Each storage array controller (106, 112) may include, for example, a data communications adapter configured to support communications via the SAN (158) and the LAN (160). Although only one of the storage array controllers (112) in the example of
Each NVRAM device (148, 152) may be configured to receive, from the storage array controller (106, 112), data to be stored in the storage devices (146). Such data may originate from any one of the computing devices (164, 166, 168, 170). In the example of
The NVRAM devices may be implemented with computer memory in the form of high bandwidth, low latency random access memory (‘RAM’). In such an embodiment, each NVRAM device is referred to as ‘non-volatile’ because each NVRAM device may receive or include a unique power source that maintains the state of the RAM after main power loss to the NVRAM device (148, 152). Such a power source may be a battery, one or more capacitors, or the like. During the power loss, the NVRAM device (148, 152) may be configured to write the contents of the RAM to a persistent storage, such as the storage devices (146, 150).
A ‘storage device’ as the term is used in this specification refers to any device configured to record data persistently. The term ‘persistently’ as used here refers to a device's ability to maintain recorded data after loss of a power source. Examples of storage devices may include mechanical, spinning hard disk drives, solid-state drives (“Flash drives”), and the like.
The storage array controllers (106, 112) of
The storage array controllers (106, 112) of
The buffer utilized by each DMA engine may be embodied as a data structure used to store descriptors that generally describe data transfer operations to be performed by the DMA engine. For example, the buffer in the first storage array controller (106) may be used to store a data transfer descriptor that describes data stored in memory of the first storage array controller (106) and a location to write the data to memory of the second storage array controller (112). In such an example, the DMA engine may operate by retrieving descriptors from the buffer and performing the data transfer operations described by the descriptors.
The storage array controllers (106, 112) of
The DMA engine within the first storage array controller (106) may retrieve the data stored in memory of the first storage array controller (106) in dependence upon the data transfer descriptor. As described above, the data transfer descriptor may include information that describes data stored in memory of the first storage array controller (106). Such information that describes data stored in memory of the first storage array controller (106) can include, for example, the address of the data stored in memory of the first storage array controller (106), the size of the data stored in memory of the first storage array controller (106), and so on. The DMA engine within the first storage array controller (106) may be configured to utilize such information to retrieve the data stored in memory of the first storage array controller (106), for example, by reading an amount of data that is equal to the size of the data as described by the data transfer descriptor from the address of the data as described by the data transfer descriptor.
The storage array controllers (106, 112) of
The DMA engine within the first storage array controller (106) may write the data into the memory of the second storage array controller (112) by performing a direct memory access of the memory in the second storage array controller (112). In such an example, the data may be written into the memory of the second storage array controller (112) in dependence upon the data transfer descriptor. As described above, the data transfer descriptor can include information describing a location to write the data to memory of the second storage array controller (112). The DMA engine of the first storage array controller (106) may utilize such information by writing the data into the memory location in the second storage array controller (112) that is described in the data transfer descriptor.
Although the example illustrated in
Emulating an RDMA link between controllers in a storage array in accordance with embodiments of the present invention is generally implemented with computers. In the system of
The storage array controller (202) of
The storage array controller (202) of
Stored in RAM (214) is an operating system (246). Examples of operating systems useful in storage array controllers (202) configured for emulating an RDMA link between controllers in a storage array according to embodiments of the present invention include UNIX™, Linux™, Microsoft Windows™, and others as will occur to those of skill in the art. Also stored in RAM (236) is an array management module (248), a module of computer program instructions useful in emulating an RDMA link between controllers in a storage array according to embodiments of the present invention. The array management module (248) may be configured to perform steps such as: inserting, into a buffer utilized by a direct memory access (‘DMA’) engine of the storage array controller (202), a data transfer descriptor describing data stored in memory of the storage array controller (202) and a location to write the data to memory of a second storage array controller; retrieving, in dependence upon the data transfer descriptor, the data stored in memory of the storage array controller (202); writing the data into the memory of the second storage array controller in dependence upon the data transfer descriptor; and other steps that will be described in greater detail below as being performed by the storage array controller generally. Readers will appreciate that while the array management module (248) and the operating system (246) in the example of
The storage array controller (202) of
The storage array controller (202) of
The storage array controller (202) of
The storage array controller (202) of
Readers will recognize that these components, protocols, adapters, and architectures are for illustration only, not limitation. Such a storage array controller may be implemented in a variety of different ways, each of which is well within the scope of the present invention.
For further explanation,
In the example depicted in
In the example depicted in
For further explanation,
The example method depicted in
The buffers (408, 426) utilized by the DMA engines (418, 432) may be embodied as data structures used to store descriptors that generally describe data transfer operations to be performed by the DMA engines (418, 432). For example, the buffer (408) in the first storage array controller (402) may be used to store a data transfer descriptor (406) that describes data (412) stored in memory (414) of the first storage array controller (402) and a location to write the data (412) to memory (428) of the second storage array controller (424). In such an example, the DMA engines (418, 432) may operate by retrieving descriptors from the buffers (408, 426) and performing the data transfer operations described by the descriptors.
The example method depicted in
In the example method depicted in
The example method depicted in
In the example method depicted in
In the example method depicted in
In the example method depicted in
In the example method depicted in
In the example method depicted in
In the example method depicted in
For further explanation,
The example method depicted in
In the example method depicted in
The example method depicted in
In the example method depicted in
The example method depicted in
Example embodiments of the present invention are described largely in the context of a fully functional computer system for emulating an RDMA link between controllers in a storage array according to embodiments of the present invention. Readers of skill in the art will recognize, however, that the present invention also may be embodied in a computer program product disposed upon computer readable storage media for use with any suitable data processing system. Such computer readable storage media may be any storage medium for machine-readable information, including magnetic media, optical media, or other suitable media. Examples of such media include magnetic disks in hard drives or diskettes, compact disks for optical drives, magnetic tape, and others as will occur to those of skill in the art. Persons skilled in the art will immediately recognize that any computer system having suitable programming means will be capable of executing the steps of the method of the invention as embodied in a computer program product. Persons skilled in the art will recognize also that, although some of the example embodiments described in this specification are oriented to software installed and executing on computer hardware, nevertheless, alternative embodiments implemented as firmware or as hardware are well within the scope of the present invention.
It will be understood from the foregoing description that modifications and changes may be made in various embodiments of the present invention without departing from its true spirit. The descriptions in this specification are for purposes of illustration only and are not to be construed in a limiting sense. The scope of the present invention is limited only by the language of the following claims.
This application is a continuation application of U.S. patent application Ser. No. 14/817,168, filed Aug. 3, 2015.
Number | Name | Date | Kind |
---|---|---|---|
5706210 | Kumano et al. | Jan 1998 | A |
5799200 | Brant et al. | Aug 1998 | A |
5933598 | Scales et al. | Aug 1999 | A |
6012032 | Donovan et al. | Jan 2000 | A |
6085333 | DeKoning et al. | Jul 2000 | A |
6643641 | Snyder | Nov 2003 | B1 |
6647514 | Umberger et al. | Nov 2003 | B1 |
6789162 | Talagala et al. | Sep 2004 | B1 |
7089272 | Garthwaite et al. | Aug 2006 | B1 |
7107359 | Burton | Sep 2006 | B1 |
7107389 | Inagaki et al. | Sep 2006 | B2 |
7146521 | Nguyen | Dec 2006 | B1 |
7334124 | Pham et al. | Feb 2008 | B2 |
7437530 | Rajan | Oct 2008 | B1 |
7493424 | Bali et al. | Feb 2009 | B1 |
7669029 | Mishra et al. | Feb 2010 | B1 |
7689609 | Lango et al. | Mar 2010 | B2 |
7743191 | Liao | Jun 2010 | B1 |
7899780 | Shmuylovich et al. | Mar 2011 | B1 |
8042163 | Karr et al. | Oct 2011 | B1 |
8086585 | Brashers et al. | Dec 2011 | B1 |
8190816 | Balasubramanian | May 2012 | B2 |
8271700 | Annem et al. | Sep 2012 | B1 |
8387136 | Lee et al. | Feb 2013 | B2 |
8437189 | Montierth et al. | May 2013 | B1 |
8465332 | Hogan et al. | Jun 2013 | B2 |
8527544 | Colgrove et al. | Sep 2013 | B1 |
8566546 | Marshak et al. | Oct 2013 | B1 |
8578442 | Banerjee | Nov 2013 | B1 |
8613066 | Brezinski et al. | Dec 2013 | B1 |
8620970 | English et al. | Dec 2013 | B2 |
8751463 | Chamness | Jun 2014 | B1 |
8762642 | Bates et al. | Jun 2014 | B2 |
8769622 | Chang et al. | Jul 2014 | B2 |
8800009 | Beda, III et al. | Aug 2014 | B1 |
8812860 | Bray | Aug 2014 | B1 |
8832216 | Bugge | Sep 2014 | B2 |
8850546 | Field et al. | Sep 2014 | B1 |
8898346 | Simmons | Nov 2014 | B1 |
8909854 | Yamagishi et al. | Dec 2014 | B2 |
8931041 | Banerjee | Jan 2015 | B1 |
8949863 | Coatney et al. | Feb 2015 | B1 |
8984602 | Bailey et al. | Mar 2015 | B1 |
8990905 | Bailey et al. | Mar 2015 | B1 |
9124569 | Hussain et al. | Sep 2015 | B2 |
9134922 | Rajagopal et al. | Sep 2015 | B2 |
9209973 | Aikas et al. | Dec 2015 | B2 |
9250823 | Kamat et al. | Feb 2016 | B1 |
9300660 | Borowiec et al. | Mar 2016 | B1 |
9405725 | Makhervaks | Aug 2016 | B2 |
9444822 | Borowiec et al. | Sep 2016 | B1 |
9507532 | Colgrove et al. | Nov 2016 | B1 |
20020013802 | Mori et al. | Jan 2002 | A1 |
20030145172 | Galbraith et al. | Jul 2003 | A1 |
20030191783 | Wolczko et al. | Oct 2003 | A1 |
20030225961 | Chow et al. | Dec 2003 | A1 |
20040010612 | Pandya | Jan 2004 | A1 |
20040080985 | Chang et al. | Apr 2004 | A1 |
20040111573 | Garthwaite | Jun 2004 | A1 |
20040153844 | Ghose et al. | Aug 2004 | A1 |
20040193814 | Erickson et al. | Sep 2004 | A1 |
20040260967 | Guha et al. | Dec 2004 | A1 |
20050160416 | Jamison | Jul 2005 | A1 |
20050188246 | Emberty et al. | Aug 2005 | A1 |
20050216800 | Bicknell et al. | Sep 2005 | A1 |
20060015771 | Van Gundy et al. | Jan 2006 | A1 |
20060129817 | Borneman et al. | Jun 2006 | A1 |
20060161726 | Lasser | Jul 2006 | A1 |
20060230245 | Gounares et al. | Oct 2006 | A1 |
20060239075 | Williams et al. | Oct 2006 | A1 |
20070022227 | Miki | Jan 2007 | A1 |
20070028068 | Golding et al. | Feb 2007 | A1 |
20070055702 | Fridella et al. | Mar 2007 | A1 |
20070109856 | Pellicone et al. | May 2007 | A1 |
20070150689 | Pandit et al. | Jun 2007 | A1 |
20070168321 | Saito et al. | Jul 2007 | A1 |
20070220227 | Long | Sep 2007 | A1 |
20070294563 | Bose | Dec 2007 | A1 |
20070294564 | Reddin et al. | Dec 2007 | A1 |
20080005587 | Ahlquist | Jan 2008 | A1 |
20080077825 | Bello et al. | Mar 2008 | A1 |
20080162674 | Dahiya | Jul 2008 | A1 |
20080195833 | Park | Aug 2008 | A1 |
20080270678 | Cornwell et al. | Oct 2008 | A1 |
20080282045 | Biswas et al. | Nov 2008 | A1 |
20090077340 | Johnson et al. | Mar 2009 | A1 |
20090100115 | Park et al. | Apr 2009 | A1 |
20090198889 | Ito et al. | Aug 2009 | A1 |
20100052625 | Cagno et al. | Mar 2010 | A1 |
20100211723 | Mukaida | Aug 2010 | A1 |
20100246266 | Park et al. | Sep 2010 | A1 |
20100257142 | Murphy et al. | Oct 2010 | A1 |
20100262764 | Liu et al. | Oct 2010 | A1 |
20100325345 | Ohno et al. | Dec 2010 | A1 |
20100332754 | Lai et al. | Dec 2010 | A1 |
20110072290 | Davis et al. | Mar 2011 | A1 |
20110125955 | Chen | May 2011 | A1 |
20110131231 | Haas et al. | Jun 2011 | A1 |
20110167221 | Pangal et al. | Jul 2011 | A1 |
20120023144 | Rub | Jan 2012 | A1 |
20120054264 | Haugh et al. | Mar 2012 | A1 |
20120079318 | Colgrove et al. | Mar 2012 | A1 |
20120131253 | McKnight et al. | May 2012 | A1 |
20120303919 | Hu et al. | Nov 2012 | A1 |
20120311000 | Post et al. | Dec 2012 | A1 |
20130007845 | Chang et al. | Jan 2013 | A1 |
20130031414 | Dhuse et al. | Jan 2013 | A1 |
20130036272 | Nelson | Feb 2013 | A1 |
20130071087 | Motiwala et al. | Mar 2013 | A1 |
20130145447 | Maron | Jun 2013 | A1 |
20130191555 | Liu | Jul 2013 | A1 |
20130198459 | Joshi et al. | Aug 2013 | A1 |
20130205173 | Yoneda | Aug 2013 | A1 |
20130219164 | Hamid | Aug 2013 | A1 |
20130227201 | Talagala et al. | Aug 2013 | A1 |
20130290607 | Chang et al. | Oct 2013 | A1 |
20130311434 | Jones | Nov 2013 | A1 |
20130318297 | Jibbe et al. | Nov 2013 | A1 |
20130332614 | Brunk et al. | Dec 2013 | A1 |
20140020083 | Fetik | Jan 2014 | A1 |
20140074850 | Noel et al. | Mar 2014 | A1 |
20140082715 | Grajek et al. | Mar 2014 | A1 |
20140086146 | Kim et al. | Mar 2014 | A1 |
20140090009 | Li et al. | Mar 2014 | A1 |
20140096220 | Da Cruz Pinto et al. | Apr 2014 | A1 |
20140101434 | Senthurpandi et al. | Apr 2014 | A1 |
20140164774 | Nord et al. | Jun 2014 | A1 |
20140173232 | Reohr et al. | Jun 2014 | A1 |
20140195636 | Karve et al. | Jul 2014 | A1 |
20140201512 | Seethaler et al. | Jul 2014 | A1 |
20140201541 | Paul et al. | Jul 2014 | A1 |
20140208155 | Pan | Jul 2014 | A1 |
20140215590 | Brand | Jul 2014 | A1 |
20140229654 | Goss et al. | Aug 2014 | A1 |
20140230017 | Saib | Aug 2014 | A1 |
20140258526 | Le Sant et al. | Sep 2014 | A1 |
20140282983 | Ju et al. | Sep 2014 | A1 |
20140285917 | Cudak et al. | Sep 2014 | A1 |
20140325262 | Cooper et al. | Oct 2014 | A1 |
20140351627 | Best et al. | Nov 2014 | A1 |
20140373104 | Gaddam et al. | Dec 2014 | A1 |
20140373126 | Hussain et al. | Dec 2014 | A1 |
20150026387 | Sheredy et al. | Jan 2015 | A1 |
20150074463 | Jacoby et al. | Mar 2015 | A1 |
20150089569 | Sondhi et al. | Mar 2015 | A1 |
20150095515 | Krithivas et al. | Apr 2015 | A1 |
20150113203 | Dancho et al. | Apr 2015 | A1 |
20150121137 | McKnight et al. | Apr 2015 | A1 |
20150134920 | Anderson et al. | May 2015 | A1 |
20150149822 | Coronado et al. | May 2015 | A1 |
20150193169 | Sundaram et al. | Jul 2015 | A1 |
20150378888 | Zhang et al. | Dec 2015 | A1 |
20160098323 | Mutha et al. | Apr 2016 | A1 |
20160350009 | Cerreta et al. | Dec 2016 | A1 |
20160352720 | Hu et al. | Dec 2016 | A1 |
20160352830 | Borowiec et al. | Dec 2016 | A1 |
20160352834 | Borowiec et al. | Dec 2016 | A1 |
20170039150 | Dreier | Feb 2017 | A1 |
Number | Date | Country |
---|---|---|
0725324 | Aug 1996 | EP |
WO-2012087648 | Jun 2012 | WO |
WO-2013071087 | May 2013 | WO |
WO-2014110137 | Jul 2014 | WO |
WO-2016015008 | Dec 2016 | WO |
WO-2016190938 | Dec 2016 | WO |
WO-2016195759 | Dec 2016 | WO |
WO-2016195958 | Dec 2016 | WO |
WO-2016195961 | Dec 2016 | WO |
Entry |
---|
Paul Sweere, Creating Storage Class Persistent Memory with NVDIMM, Published in Aug. 2013, Flash Memory Summit 2013, <http://ww.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130814—T2—Sweere.pdf>, 22 pages. |
PCMAG, Storage Array Definition, Published May 10, 2013. <http://web.archive.org/web/20130510121646/http://www.pcmag.com/encyclopedia/term/52091/storage-array>, 2 pages. |
Google Search of “storage array define” performed by the Examiner on Nov. 4, 2015 for U.S. Appl. No. 14/725,278, Results limited to entries dated before 2012, 1 page. |
Techopedia, What is a disk array, techopedia.com (online), Jan. 13, 2012, 1 page, URL: web.archive.org/web/20120113053358/http://www.techopedia.com/definition/1009/disk-array. |
Webopedia, What is a disk array, webopedia.com (online), May 26, 2011, 2 pages, URL: web/archive.org/web/20110526081214/http://www.webopedia.com/TERM/D/disk—array.html. |
Li et al., Access Control for the Services Oriented Architecture, Proceedings of the 2007 ACM Workshop on Secure Web Services (SWS '07), Nov. 2007, pp. 9-17, ACM New York, NY. |
Hota et al., Capability-based Cryptographic Data Access Control in Cloud Computing, International Journal of Advanced Networking and Applications, col. 1, Issue 1, Aug. 2011, 10 pages, Eswar Publications, India. |
Faith, dictzip file format, GitHub.com (online), accessed Jul. 28, 2015, 1 page, URL: github.com/fidlej/idzip. |
Wikipedia, Convergent Encryption, Wikipedia.org (online), accessed Sep. 8, 2015, 2 pages, URL: en.wikipedia.org/wiki/Convergent—encryption. |
Storer et al., Secure Data Deduplication, Proceedings of the 4th ACM International Workshop on Storage Security and Survivability (StorageSS'08), Oct. 2008, 10 pages, ACM New York, NY. USA, DOI: 10.1145/1456469.1456471. |
ETSI, Network Function Virtualisation (NFV); Resiliency Requirements, ETSI GS NFCV-REL 001, V1.1.1, Jan. 2015, 82 pages, etsi.org (online), URL: www.etsi.org/deliver/etsi—gs/NFV-REL/001—099/001/01.01.01—60/gs—NFV-REL001v010101p.pdf. |
Microsoft, Hybrid for SharePoint Server 2013—Security Reference Architecture, Microsoft (online), Oct. 2014, 53 pages, URL: hybrid.office.com/img/Security—Reference—Architecture.pdf. |
Microsoft, Hybrid Identity, Microsoft (online), Apr. 2014, 36 pages, URL: www.aka.ms/HybridIdentityWp. |
Microsoft, Hybrid Identity Management, Microsoft (online), Apr. 2014, 17 pages, URL: download.microsoft.com/download/E/A/E/EAE57CD1-A80B-423C-96BB-142FAAC630B9/Hybrid—Identity—Datasheet.pdf. |
Bellamy-McIntyre et al., OpenID and the Enterprise: A Model-based Analysis of Single Sign-On Authentication, 15th IEEE International Enterprise Distributed Object Computing Conference (EDOC), Aug. 29, 2011, pp. 129-138, IEEE Computer Society, USA, DOI: 10.1109/EDOC.2011.26, ISBN: 978-1-4577-0362-1. |
Kong, Using PCI Express as the Primary System Interconnect in Multiroot Compute, Storage, Communications and Embedded Systems, White Paper, IDT.com (online), Aug. 28, 2008, 12 pages, URL: www.idt.com/document/whp/idt-pcie-multi-root-white-paper. |
Hu et al., Container Marking: Combining Data Placement, Garbage Collection and Wear Levelling for Flash, 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunications Systems, Jul. 25-27, 2011, 11 pages, ISBN: 978-0-7695-4430-4, DOI: 10.1109/MASCOTS.2011.50. |
International Search Report and Written Opinion, PCT/US2016/015006, dated Jul. 18, 2016, 12 pages. |
International Search Report and Written Opinion, PCT/US2016/015008, dated May 4, 2016, 12 pages. |
International Search Report and Written Opinion, PCT/US2016/020410, dated Jul. 8, 2016, 12 pages. |
International Search Report and Written Opinion, PCT/US2016/032084, dated Jul. 18, 2016, 12 pages. |
International Search Report and Written Opinion, PCT/US2016/016333, dated Jun. 8, 2016, 12 pages. |
International Search Report and Written Opinion, PCT/US2016/032052, dated Aug. 30, 2016, 17 pages. |
International Search Report and Written Opinion, PCT/US2016/035492, dated Aug. 17, 2016, 10 pages. |
International Search Report and Written Opinion, PCT/US2016/036693, dated Aug. 29, 2016, 10 pages. |
International Search Report and Written Opinion, PCT/US2016/038758, dated Oct. 7, 2016, 10 pages. |
International Search Report and Written Opinion, PCT/US2016/040393, dated Sep. 22, 2016, 10 pages. |
International Search Report and Written Opinion, PCT/US2016/044020, dated Sep. 30, 2016, 11 pages. |
International Search Report and Written Opinion, PCT/US2016/044874, dated Oct. 7, 2016, 11 pages. |
International Search Report and Written Opinion, PCT/US2016/044875, dated Oct. 5, 2016, 13 pages. |
International Search Report and Written Opinion, PCT/US2016/044876, dated Oct. 21, 2016, 12 pages. |
International Search Report and Written Opinion, PCT/US2016/044877, dated Sep. 29, 2016, 13 pages. |
Number | Date | Country | |
---|---|---|---|
Parent | 14817168 | Aug 2015 | US |
Child | 15697802 | US |