Locally providing cloud storage array services

Information

  • Patent Grant
  • 10027757
  • Patent Number
    10,027,757
  • Date Filed
    Monday, January 30, 2017
    7 years ago
  • Date Issued
    Tuesday, July 17, 2018
    6 years ago
Abstract
Locally providing cloud storage array services for a storage array of a data center when the storage array is not connected to a remote cloud-based storage array services provider includes initiating, by a primary storage array, one or more cloud storage array services and locally providing the cloud storage array services. Such local providing of the cloud storage array services also includes generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics and presenting the metadata to a user through a local area network.
Description
TECHNICAL FIELD

The field of technology is data processing, or, more specifically, methods, apparatus, and products for providing cloud storage array services for a storage array of a data center when the storage array is not connected to a remote cloud-based storage array services provider.


BACKGROUND ART

Data centers may include many computing components including servers, network devices, and storage arrays. As the need for storage of large amounts of data and efficient access to that data increases, storage array technology is advancing. Such storage arrays may provide persistent storage for any number of computing devices in a data center. Given that many data centers provide services to many different users, various considerations must be taken into account to provide efficient, useful, and secure access to and administration of storage arrays. Various management tools, reporting services, and the like may be implemented for storage array through cloud-based service providers. In some implementations, however, such cloud-based service providers may be unavailable, either by design or through a loss of connection.


SUMMARY

Methods, apparatus, and products for locally providing cloud storage array services for a storage array of a data center when the storage array is not connected to a remote cloud-based storage array services provider are disclosed in this specification. Providing such cloud storage array services in accordance with embodiments of the present invention includes: initiating, by a primary storage array, one or more cloud storage array services; and locally providing the cloud storage array services including: generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics; and presenting the metadata to a user through a local area network.


The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular descriptions of exemplary embodiments of the invention as illustrated in the accompanying drawings wherein like reference numbers generally represent like parts of exemplary embodiments of the invention.





BRIEF DESCRIPTION OF THE DRAWINGS


FIG. 1 sets forth a block diagram of a system configured for providing storage array services first in the cloud then locally according to embodiments of the present invention.



FIG. 2 sets forth a block diagram of several example computers useful for providing cloud storage array services according first in the cloud then locally according to embodiments of the present invention.



FIG. 3 sets forth a block diagram of an example storage controller of a storage array useful for locally providing cloud storage array services.



FIG. 4 sets forth an example block diagram of a system that locally provides cloud storage array services in accordance with embodiments of the present invention.



FIG. 5 sets forth a flow chart illustrating an exemplary method of locally providing cloud storage array services according to embodiments of the present invention



FIG. 6 sets forth a flow chart illustrating another method of locally providing cloud storage array services in accordance with embodiments of the present invention.



FIG. 7 sets forth a flow chart illustrating another method of locally providing cloud storage array services in accordance with embodiments of the present invention.





SUMMARY OF INVENTION

Exemplary methods, apparatus, and products for providing cloud storage array services for a storage array of a data center when the storage array is not connected to a remote cloud-based storage array services provider in accordance with the present invention are described with reference to the accompanying drawings, beginning with FIG. 1. The term ‘cloud’ as used in this specification refers to systems and computing environments that provide services to user devices through the sharing of computing resources through a network. Generally, the user device is unaware of the exact computing resources utilized by the cloud system to provide the services. Although in many cases such ‘cloud’ environments or systems are accessible via the Internet, readers of skill in the art will recognize that any system that abstracts the use of shared resources to provide services to a user through any network may be considered a cloud-based system.



FIG. 1 sets forth a block diagram of a system configured for locally providing cloud storage array services according to embodiments of the present invention. The system of FIG. 1 includes a number of computing devices (164, 166, 168, 170). Such computing devices may be implemented in a number of different ways. For example, a computing device may be a server in a data center, a workstation, a personal computer, a notebook, or the like.


The computing devices (164-170) in the example of FIG. 1 are coupled for data communications to a number of storage arrays (102, 104) through a storage area network (SAN) (158) as well as a local area network (160) (LAN). The SAN (158) may be implemented with a variety of data communications fabrics, devices, and protocols. Example fabrics for such a SAN may include Fibre Channel, Ethernet, Infiniband, SAS (Serial Attached Small Computer System Interface), and the like. Example data communications protocols for use in such a SAN (158) may include ATA (Advanced Technology Attachment), Fibre Channel Protocol, SCSI, iSCSI, HyperSCSI, and others. Readers of skill in the art will recognize that a SAN is just one among many possible data communications couplings which may be implemented between a computing device and a storage array. Any other such data communications coupling is well within the scope of embodiments of the present invention.


The local area network (160) of FIG. 1 may also be implemented with a variety of fabrics and protocols. Examples of such fabrics include Ethernet (802.3), wireless (802.11), and the like. Examples of such data communications protocols include TCP (Transmission Control Protocol), UDP (User Datagram Protocol), IP (Internet Protocol), HTTP (HyperText Transfer Protocol), WAP (Wireless Access Protocol), HDTP (Handheld Device Transport Protocol), SIP (Session Initiation Protocol), RTP (Real Time Protocol) and others as will occur to those of skill in the art.


The example storage arrays (102, 104) of FIG. 1 provide persistent data storage for the computing devices. Each storage array (102, 104) includes a storage controller (106, 112). The storage controller is a module of automated computing machinery comprising computer hardware, computer software, or a combination of computer hardware and software. The storage controller may be configured to carry out various storage-related tasks. Such tasks may include writing data received from a computing device to storage, erasing data from storage, retrieving data from storage to provide to a computing device, monitoring and reporting of disk utilization and performance, performing RAID (Redundant Array of Independent Drives) or RAID-like data redundancy operations, compressing data, encrypting data, and so on.


Each storage controller (106, 112) may be implemented in a variety of ways, including as an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Chip), an ASIC (Application Specific Integrated Circuit), or computing device that includes discrete components such as a central processing unit, computer memory, and various adapters. Each storage controller (106, 112) may, for example, include a data communications adapter configured to support communications via the SAN (158) and the LAN (160). For clarity, only one of the storage controllers (112) in the example of FIG. 1 is depicted as coupled to the LAN (160) for data communications. Readers should understand that both storage controllers (106, 112) are independently coupled to the LAN (160). Each storage controller (106, 112) may also, for example, include an I/O controller or the like that couples the storage controller (106, 112) for data communications, through a midplane (114), to a number of storage devices (146, 150), and a number of write buffer (148, 152) devices.


Each write buffer device (148, 152) may be configured to receive, from the storage controller (106, 112), data to be stored in the storage devices (146). Such data may originate from any one of the computing devices (164-170). In the example of FIG. 1, writing data to the write buffer device may be carried out more quickly than writing data to the storage device. The storage controller (106, 112) may be configured to effectively utilize the write buffer devices (148, 152) as a quickly accessible redundant buffer for data destined to be written to storage. In this way, if the storage device to which the data is to be written fails or if the write does not complete, the write buffer device may maintain the data to be written during a retry of the write or during failover of the storage device to another location. That is, the write buffer device may provide redundancy for the storage devices.


A ‘storage device’ as the term is used in this specification refers to any device configured to record data persistently. The term ‘persistently’ as used here refers to a device's ability to maintain recorded data after loss of a power source. Examples of storage devices may include mechanical, spinning hard disk drives, Solid-state drives (“Flash drives”), and the like.


In addition to being coupled to the computing devices through the SAN (158), the storage arrays may also be coupled to the computing devices through the LAN (160) and to one or more cloud service providers through the Internet (172). One example cloud service in FIG. 1 is a storage array services provider (176). The storage array service provider (176) may be configured to provide various storage array services (184) such as reporting of storage array performance characteristics, configuration control of the storage arrays, and the like. The storage array services provider may rely on modules executing on the storage array itself to gather or process such data.


In some instances, the storage arrays (102, 104) may be disconnected from the storage array services provider (176). Various reasons may exist for such a disconnect. For example, a loss of data communications connection between the storage array and the storage array services provider (176) may occur. In high-security data centers, as another example, the storage arrays (102, 104) may never be connected to the storage array services provider at all. In such situations, users may still desire the services provided by the storage array services provider. To that end, the system of FIG. 1 may be configured, according to embodiments of the present invention, to locally provide cloud storage array services for a storage array of a data center when the storage array is not connected to a remote cloud-based storage array services provider. Such local providing of the cloud storage array services may be carried out by a storage array services module (182) of one of the storage arrays (102, 104), referred to here as the ‘primary’ storage array (180). The storage array services module (182) is a module of automated computing machinery comprising computer hardware, computer software, or a combination of computer hardware and software. The storage array services module (182) may locally provide the cloud storage array services by, initiating one or more cloud storage array services (184) and locally providing the cloud storage array services. Providing the cloud storage array services locally may include generating, by the cloud storage array services (184), metadata describing one or more real-time storage array characteristics and presenting the metadata to a user through a local area network (160). ‘Metadata’ as the term is used in this specification refers to data that describes various characteristics of the storage array. Such data may take on various forms in dependence upon the types of cloud storage array services executing on the storage array. Some examples of such services may include a service that tracks and reports the rate and type of access of the storage devices of the storage array, a service that tracks and reports throughput of the storage array, a service that monitors and reports memory wear leveling of the storage devices, a service that reports capacity utilization of the storage devices, a service that reports deduplication statistics, a service that reports RAID (Redundant Array of Independent Drives) statistics, a service that reports logical volume statistics, a service that monitors and reports power consumption of the storage array, and other services as will occur to readers of skill in the art.


Prior to presentation, the metadata may be processed and analyzed. Such analysis processing and analysis may also be included in the presentation of the metadata to the user through the LAN (160). That is, ‘presenting the metadata’ is not limited to presenting the metadata in its original form. The metadata may be formatted, processed, analyzed, and otherwise altered for purposes of presentation.


As mentioned above, one example in which cloud storage array services are provided locally in accordance with embodiments of the present invention includes when a loss of data communications occurs. To that end, the primary storage array (180), through the storage array services module (182) may be configured to detect a data communications connection between the primary storage array (180) and a remote cloud-based storage array services provider (176). Here, a ‘heartbeat’ signal may be used to ensure that the data communications connection between the primary storage array (180) and the cloud-based storage array services provider (176) is maintained. Such a ‘heartbeat’ may be a periodic ping of the storage array services provider. As long as the ping is returned, the storage array services module (182) may infer that the data communications connection between the storage array services provider (176) and the primary storage array (180) has not been lost. In such an embodiment, the cloud-based storage array services provider (176) may provide the cloud storage array services remotely, rather than the storage array services module (182).


Further, while the data communications connection between the primary storage array (180) and the storage array services provider (176) is active, the storage array services module (182) may periodically retrieve, from the remote cloud-based storage array services provider (176), a current configuration of the cloud storage array services and the a current configuration of the plurality of storage arrays (102, 104) in the data center. The term ‘current configuration of the cloud storage array services’ may include any data related to the execution of the services including, for example, a list of the types of services to be executed, the executable computer program instructions of the services themselves, updates to the executable computer program instructions of the services, data used to seed the services, data describing the format of presenting the results of the services, and so on. The term ‘current configuration of the plurality of storage arrays’ refers to any data describing each of the storage arrays of the data center including: storage capabilities of each storage array; data communications capabilities, endpoints, and present connectivity of each storage array; one or more identifiers of the storage array or the array's components; a listing of components of the storage array; an identification of logical volumes maintained by the storage array; and the like. By periodically updating these current configurations, the storage array services module (182) may be primed to locally provide the cloud storage array services upon a loss of data communications between the primary storage array (180) and the storage array services provider (176).


The storage array services module (182) may later detect loss of the data communications connection between the primary storage array and the remote cloud-based storage array services provider. The storage array services module (182) may detect such a loss of communications when a predefined number of pings of the heartbeat signal are not returned. Responsive to detecting the loss of communications between the primary storage array (180) and the storage array services provider (176), the storage array services provider may then initiate the cloud storage array services (184).


In embodiments in which multiple storage arrays are included in the data center, each storage array may be independently coupled for data communications to the cloud-based storage array services provider (176). In such embodiments, each storage array is generally unaware of the other storage arrays at least for purposes of participating in storage array services offered by the cloud-based storage array services provider (176). In such an embodiment, the cloud-based storage array services provider may expose a REST (Representational State Transition) API (Application Programming Interface), or the like, to the storage arrays to carry out data communications between a storage array service and a storage array. REST APIs generally utilize HTTP commands and a domain as the basis for data communications to between two endpoints. To that end, upon a loss of communications with the cloud-based storage array services provider (176), the primary storage array (180) may initiate the one or more cloud storage array services (184) by altering a local DNS (Domain Name Service) record such that the domain originally utilized by the cloud-based storage array services provider (176) redirects to an IP address of the primary storage array. In this way, any communications that would normally be addressed to the cloud-based storage array services provider from any storage array in the data center are now redirected to the primary storage array (180).


In such an embodiment in which multiple storage arrays are located within a single data center, generating metadata describing one or more real-time storage array characteristics may also include generating metadata describing one or more real-time storage array characteristics of each of the plurality of the storage arrays; and aggregating, for presentation to the user, the metadata of each of the plurality of storage arrays.


Although the example of FIG. 1 depicts the primary storage array (180) as the host of the cloud storage array services (184), other storage arrays may host the services in various circumstances. In this way, initiating the one or more cloud storage array services may also include selecting, by the primary storage array through the storage array services module (182) one of the plurality of storage arrays in the data center to host the one or more cloud storage array services in dependence upon workload characteristics of each of the plurality of storage arrays. The term ‘workload characteristics’ here refers to any metric that may be utilized to infer availability of processing capabilities. Examples of such workload characteristics may include processing capabilities of each storage array (CPU speed, memory bus speed, and the like), a history of storage access operations over a predefined period of time, status of a job queue directed to accessing a volume hosted on a storage array, and so on as will occur to readers of skill in the art. That is, a storage array with less workload than all other storage arrays may be utilized to host the cloud storage array services (184) so that primary data storage operations are unaffected.


In embodiments in which workload characteristics are utilized to select a host for the cloud-based storage array services, the primary storage array (180) may also be configured to detect a change in workload characteristics of each of the plurality of storage arrays. Such a change may be detected in a variety of ways including, for example, by detecting a failure of a storage array, by periodically retrieving workload characteristics, and so on. Upon detecting the change, the primary storage array (180) may then re-select one of the storage arrays to host the one or more cloud storage array services in dependence upon the changed workload characteristics. In this way, hosting of the services may be dynamically reassigned when workload changes amongst the storage arrays.


In embodiments in which a storage array other than the primary storage array (180) is selected to host the cloud-based storage array services, the primary storage array may operate as a proxy for the selected storage array. In such an embodiment, presenting the metadata to a user through a local area network may include presenting the metadata through the proxy of the primary storage array. The primary storage array (180) may operate as a proxy in different ways. In one way, the primary storage array receives only the final aggregated results for presentation to the user through the local area network (160). That is, the selected storage array actually hosting the cloud-based storage array services (184) may perform the services, collecting metadata from all storage arrays, aggregate the metadata into a single result and pass that result on to the primary storage array for presentation to a user. In another example, the selected storage array may execute the cloud-based storage array services and each storage array may pass its metadata directly to the primary storage array. In such an embodiment, the primary storage array may aggregate and process the metadata of each storage array prior to presenting the aggregated and processed metadata to a user through the LAN (160).


The arrangement of computing devices, storage arrays, cloud-based service providers, networks and other devices making up the exemplary system illustrated in FIG. 1 are for explanation, not for limitation. Systems useful according to various embodiments of the present invention may include different configurations of servers, routers, switches, computing devices, and network architectures, not shown in FIG. 1, as will occur to those of skill in the art.


Locally providing cloud storage array services in accordance with embodiments of the present invention is generally implemented with computers. In the system of FIG. 1, for example, all the computing devices (164-170), storage arrays (102, 104), and storage array services provider (176) may be implemented to some extent at least as computers. For further explanation, therefore, FIG. 2 sets forth a block diagram several example computers useful for locally providing cloud storage array services according to embodiments of the present invention. The example computers in FIG. 2 include a primary storage array (202), a client-side computer (204), and a storage array services provider (176). For clarity, only one computer, the primary storage array (202), in the example of FIG. 2 is depicted with a number of various computing components which are described below in greater detail. Readers will understand that the client-side user computer (204) and storage array services provider (176) may include similar components that operate in a similar manner.


The primary storage array (202) of FIG. 2 includes at least one computer processor (210) or ‘CPU’ as well as random access memory (214) (RAM′) which is connected through a high speed memory bus and bus adapter (212) to processor (210) and to other components of the primary storage array (202). Stored in RAM (214) is a storage array services module (182), a module of computer program instructions that, when executed, causes the primary storage array (202) of FIG. 2 to locally provide cloud storage array services when the storage array (202) is not connected to a remote cloud-based storage array services provider (172) by initiating one or more cloud storage array services (184) and locally providing the cloud storage array services including: generating, by the cloud storage array services, metadata (226) describing one or more real-time storage array characteristics; and presenting the metadata to a user through a local area network.


Here, the user may receive or view the metadata (226) through a client-side array services module (228) stored in RAM (238) of a client-side user computer (204). In the example of FIG. 2, the client-side user computer (204) is depicted as being coupled through the SAN (158) to the storage array (202) for use in storing data in the storage array. Alternatively, the client-side user computer (204) may be a device that is not connected to the storage array (202) for purposes of storing data but rather may be connected only via a LAN for purposes of receiving results of the cloud storage array services. Consider, for example, a client-side user computer that is implemented as a mobile device that is carried by a system administrator. Although the mobile device is not coupled to the storage array for purposes of data storage, the system administrator may utilize the mobile device to request and receive results from various reporting services that indicate storage array performance characteristics, errors, alerts, and the like.


In some embodiments, prior to the storage array services module (182) initiating cloud storage array services (184) locally, such cloud storage array services may be provided remotely via the storage array services provider (176). In such an embodiment, the cloud storage array services (184) may be stored in RAM (24) of the storage array services provider (176) and be accessible via an API exposed by the storage array services provider via data communications over the Internet (172) or some other wide-area network.


Turning back to the components of the primary storage array (202), also stored in RAM (214) is an operating system (234). Examples of operating systems useful in primary storage arrays configured for locally providing cloud storage array services according to embodiments of the present invention include UNIX™, Linux™, Microsoft Windows™, and others as will occur to those of skill in the art. The operating system (234), the storage array services module (182), the cloud storage array services (184) and the metadata (226) in the example of FIG. 2 are shown in RAM (168), but many components of such software typically are stored in non-volatile memory also, such as, for example, on a disk drive (224). Likewise, the modules depicted in RAM (238, 240) of the client-side user computer (204) and the storage array services provider (176) may be stored in non-volatile memory.


The primary storage array (202) of FIG. 2 also includes disk drive adapter (222) coupled through an expansion bus and bus adapter (212) to the processor (210) and other components of the primary storage array (202). Disk drive adapter (222) connects non-volatile data storage to the primary storage array (202) in the form of disk drive (224). Disk drive adapters may be implemented in a variety of ways including as SATA (Serial Advanced Technology Attachment) adapters, PATA (Parallel ATA) adapters, Integrated Drive Electronics (‘IDE’) adapters, Small Computer System Interface (SCSI′) adapters, and others as will occur to those of skill in the art. Non-volatile computer memory also may be implemented as an optical disk drive, electrically erasable programmable read-only memory (so-called ‘EEPROM’ or ‘Flash’ memory), RAM drives, and so on, as will occur to those of skill in the art.


The example primary storage array (202) of FIG. 2 includes one or more input/output (′I/O′) adapters (216). I/O adapters implement user-oriented input/output through, for example, software drivers and computer hardware for controlling output to display devices such as computer display screens, as well as user input from user input devices (220) such as keyboards and mice. The example primary storage array (202) of FIG. 2 also includes a video adapter (208), which is an example of an I/O adapter specially designed for graphic output to a display device (206) such as a display screen or computer monitor. Video adapter (208) is connected to the processor (210) through a high speed video bus.


The exemplary primary storage array (202) of FIG. 2 includes a communications adapter (218) for data communications with other computers (204, 176) through the Internet (172). Such data communications may be carried out through data communications networks such as IP data communications networks, and in other ways as will occur to those of skill in the art. Communications adapters implement the hardware level of data communications through which one computer sends data communications to another computer, directly or through a data communications network. Examples of such communications adapters useful include modems for wired dial-up communications, Ethernet (IEEE 802.3) adapters for wired data communications, and 802.11 adapters for wireless data communications.


The example of FIG. 2 depicts a storage array implemented as one type of example computer. Readers of skill in the art will immediately recognize, however, that such a storage array useful for locally providing cloud storage array services in accordance with embodiments of the present invention may be implemented in a variety of different ways and include a variety of different components. To that end and for further explanation, FIG. 3 sets forth a block diagram of an example storage controller (106) of a primary storage array (202) configured for locally providing cloud storage array services in accordance with embodiments of the present invention. The example storage controller includes a computer processor (314). The computer processor is coupled to RAM (214) through a DDR4 (Double-Data Rate 4) bus. Stored in RAM (214) is a storage array services module (182) that operates as described above to locally provide one or more cloud storage array services (184) including the generation and presentation, to a user, of metadata (226) describing real-time storage array characteristics.


The processor (314) is also coupled for data communications through PCIe (Peripheral Component Interface express) links (308, 310, 312, 322) to several Fibre Channel host bus adapters (302, 304), an Ethernet adapter (306), and a PCIe switch (324). The Fibre Channel host bus adapters (308, 310) may couple the storage controller to a storage area network, such the SAN (158) depicted in the example of FIGS. 1 and 2. The Ethernet adapter (306) may couple the storage controller to a local area network such as the LAN (160) depicted in the example of FIGS. 1 and 2. The PCIe switch (324) may provide data communications across other PCI links through the midplane to PCI endpoints, such as storage devices or write buffer devices. Likewise, the processor (314) is also coupled through a SAS (Serial Attached SCSI) host bus adapter (316) to a SAS expander (320). The SAS expander may provide SAS connections between the computer processor (314) and other devices through the midplane.


Readers of skill in the art will recognize that these components, protocols, adapters, and architectures are for illustration only, not limitation. Such a storage controller may be implemented in a variety of different ways. Each such way is well within the scope of the present invention.


For further explanation, FIG. 5 sets forth a flow chart illustrating an exemplary method for locally providing cloud storage array services according to embodiments of the present invention. The method of FIG. 5 includes initiating (402), by a primary storage array (202), one or more cloud storage array services (184). Initiating (402) one or more cloud storage array services (184) may include deploying and executing one or more modules of computer program instructions.


The method of FIG. 5 also includes locally providing (404) the cloud storage array services (184). In the method of FIG. 5, locally providing (404) the cloud storage array services (184) is carried out by generating (406), by the cloud storage array services (184), metadata (226) describing one or more real-time storage array characteristics; and presenting (408) the metadata to a user through a local area network. Generating (406) such metadata may be carried out in various ways depending upon the type of service performed. In embodiments in which a service reports throughput of a storage array, the service may monitor a number of data storage write operations completed over a predefined period time. Such a number is metadata describing real-time storage array characteristic.


Presenting (408) metadata (226) locally to a user may be carried out by sending the data through the local area network (160) to a client-side array services module for presentation in a GUI (Graphical User Interface) (410). A client-side array services module (228) is a module of automated computing machinery comprising computer hardware, computer software, or a combination of computer hardware and software that is configured to receive and present in a GUI metadata from cloud storage array services. One example of a client-side array services module (228) may be a web browser and the GUI (410) may be a webpage hosted by the primary storage array (202). In another example, the client-side array services module (228) may be implemented as an application for a mobile device. These are but a few of many possible implementations of a client-side array services module (228) that may be configured to receive and present metadata to a user.


For further explanation, FIG. 5 sets forth a flow chart illustrating another method of locally providing cloud storage array services in accordance with embodiments of the present invention. The method of FIG. 5 is similar to the method of FIG. 4 including, as it does: initiating (402) one or more cloud storage array services (184) and locally providing (404) the cloud storage array services including: generating (406) metadata; and presenting (408) the metadata to a user through a local area network (160).


The method of FIG. 5 differs from the method of FIG. 5, however, in that the method of FIG. 5 also includes detecting (502), by the primary storage array (202) prior to initiating (402) the cloud storage array services (184), a data communications connection (512) between the primary storage array (202) and a remote cloud-based storage array services provider (176). Detecting (502) a data communications connection (512) between the primary storage array (202) and the remote cloud-based storage array services provider (176) may be carried out in a variety of manners. In one example, the primary storage array (202) may periodically ping the cloud-based storage array services provider (176). Each ping is returned by the cloud-based storage array services provider as long as a data communications connection over the wide area network (WAN) (512) is active.


While the data communications connection is active, the method of FIG. 5 includes providing (506), by the remote cloud-based storage array services provider (176), the cloud storage array services. The cloud-based storage array services provider (176) may provide many different types of cloud storage array services (184) in many different ways. In one example, the cloud-based storage array services provider (176) may expose an API to the primary storage array (and other storage arrays) where the API enables the performance of the cloud storage array services.


Also while the data communications connection between the primary storage array (202) and the remote cloud based storage array services provider (176) is active, the method of FIG. 5 may optionally include periodically retrieving (504), by the primary storage array from the remote cloud-based storage array services provider, a current configuration (508) of the cloud storage array services and a current configuration (508) of the plurality of storage arrays. The configuration of cloud storage array services may be updated over time by developers of the cloud storage array services and the configuration of the storage arrays in the data center may updated by system administrators of the storage arrays or data center.


The method of FIG. 5 also includes detecting (508), by the primary storage array (202), loss of the data communications connection (514) between the primary storage array (202) and the remote cloud-based storage array services provider (176). Continuing with the heartbeat example described above, upon a predefined number of un-returned pings, the primary storage array (202) may infer a loss of data communications connection between the primary storage array (202) and the cloud-based storage area service provider (176). In an embodiment in which the a data communication connection was previous active then is lost, the primary storage array (202) may be configured to initiate (402) the one or more cloud storage array services by the primary storage array only in response to detecting the loss of the data communications connection between the primary storage array and the remote, cloud-based storage array services provider.


For further explanation, FIG. 6 sets forth a flow chart illustrating another method of locally providing cloud storage array services in accordance with embodiments of the present invention. The method of FIG. 6 is similar to the method of FIG. 4 including, as it does: initiating (402) one or more cloud storage array services (184) and locally providing (404) the cloud storage array services including: generating (406) metadata; and presenting (408) the metadata to a user through a local area network (160).


The method of FIG. 6 differs from the method of FIG. 5, however, in that in the method of FIG. 6, the data center includes a plurality of storage arrays (602, 604), including the primary storage array (202). Also in the method of FIG. 5 generating (406) metadata (226) describing one or more real-time storage array characteristics includes: generating (606) metadata describing one or more real-time storage array characteristics of each of the plurality of the storage arrays (602, 604, 402) and aggregating (608), for presentation to the user, the metadata (226) of each of the plurality of storage arrays. In this way, cloud storage array services may be performed for a plurality of storage arrays in a single data center and all results are aggregated for presentation of a user.


For further explanation, FIG. 7 sets forth a flow chart illustrating another method of locally providing cloud storage array services in accordance with embodiments of the present invention. The method of FIG. 7 is similar to the method of FIG. 4 including, as it does: initiating (402) one or more cloud storage array services (184) and locally providing (404) the cloud storage array services including: generating (406) metadata; and presenting (408) the metadata to a user through a local area network (160).


The method of FIG. 6 differs from the method of FIG. 5, however, in that in the method of FIG. 6, the data center includes a plurality of storage arrays (702, 704), including the primary storage array (202). The method of FIG. 7 differs from the method of FIG. 5 in that in the method of FIG. 7 initiating (402), by the primary storage array, one or more cloud storage array services (402) includes selecting (706), by the primary storage array (202) in dependence upon workload characteristics of each of the plurality of storage arrays (702, 704), one of the plurality of storage arrays to host the one or more cloud storage array services (184). Selecting a storage array to host the cloud storage array services in dependence upon workload characteristics may include retrieving from each of the storage arrays one or more metrics useful for inferring availability of processing capabilities and selecting the storage array with the greatest inferred availability of processing capabilities.


The method of FIG. 7 also includes detecting (708), by the primary storage array (202), a change in workload characteristics of each of the plurality of storage arrays and re-selecting (710) one of the plurality of storage arrays to host the one or more cloud storage array services in dependence upon the changed workload characteristics. The primary storage array (202) may be configured to periodically request workload characteristics from each of the plurality of storage arrays and determine if a change in the workload characteristics necessitates a change in selection of the storage array hosting the cloud storage array services (184).


The method of FIG. 7 also includes operating (714), by the primary storage array (202), as a proxy for the selected storage array (716) when the selected storage array (716) is not the primary storage array (202). Operating (714) as a proxy may take various forms. As mentioned above, the primary storage array (202) may operate as a proxy by receiving a final aggregated result of the performance of the cloud storage array services or may operate as a proxy by receiving, from each storage array independently, metadata streams generated as a result of the execution of the cloud storage array services by the selected storage array (716). In the latter example, the primary storage array (202) may process, analyze, and aggregated the various independent streams of metadata. To that end, presenting (408) the metadata to a user through a local area network in the method of FIG. 7 is carried out by presenting (712) the metadata through the proxy of the primary storage array (714).


Exemplary embodiments of the present invention are described largely in the context of a fully functional computer system. Readers of skill in the art will recognize, however, that the present invention also may be embodied in a computer program product disposed upon computer readable media for use with any suitable data processing system. Such computer readable storage media may be any transitory or non-transitory media. Examples of such media include storage media for machine-readable information, including magnetic media, optical media, or other suitable media. Examples of such media also include magnetic disks in hard drives or diskettes, compact disks for optical drives, magnetic tape, and others as will occur to those of skill in the art. Persons skilled in the art will immediately recognize that any computer system having suitable programming means will be capable of executing the steps of the method of the invention as embodied in a computer program product. Persons skilled in the art will recognize also that, although some of the exemplary embodiments described in this specification are oriented to software installed and executing on computer hardware, nevertheless, alternative embodiments implemented as firmware, as hardware, or as an aggregation of hardware and software are well within the scope of embodiments of the present invention.


It will be understood from the foregoing description that modifications and changes may be made in various embodiments of the present invention without departing from its true spirit. The descriptions in this specification are for purposes of illustration only and are not to be construed in a limiting sense. The scope of the present invention is limited only by the language of the following claims.

Claims
  • 1. A method of locally providing cloud storage array services for a storage array of a data center, wherein: the data center comprises a plurality of storage arrays and each storage array is coupled only to one or more local networks;the method comprising:initiating, on at least one of the plurality of storage arrays, one or more cloud storage array services; andlocally providing the cloud storage array services including: generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics; andpresenting the metadata to a user through at least one of the one or more local networks.
  • 2. The method of claim 1, wherein generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics further comprises: generating metadata describing one or more real-time storage array characteristics of each of the plurality of the storage arrays; andaggregating, for presentation to the user, the metadata of each of the plurality of storage arrays.
  • 3. The method of claim 1, wherein initiating the one or more cloud storage array services further comprises: selecting, in dependence upon workload characteristics of each of the plurality of storage arrays, one of the plurality of storage arrays to host the one or more cloud storage array services.
  • 4. The method of claim 3, further comprising: detecting a change in workload characteristics of at least one of the plurality of storage arrays; andre-selecting one of the plurality of storage arrays to host the one or more cloud storage array services in dependence upon the changed workload characteristics.
  • 5. The method of claim 3, further comprising: operating, by a storage array that is not hosting the one or more cloud storage array services, as a proxy for the storage array that is hosting the one or more cloud storage array services; andpresenting the metadata to a user through at least one of the one or more local networks further comprises presenting the metadata through the proxy of the storage array that is hosting the one or more cloud storage array services.
  • 6. An apparatus for locally providing cloud storage array services for a storage array of a data center, wherein: the data center comprises a plurality of storage arrays and each storage array is coupled only to one or more local networks;the apparatus comprising a computer processor and a computer memory operatively coupled to the computer processor, where the computer memory includes computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:initiating, on at least one of the plurality of storage arrays, one or more cloud storage array services; andlocally providing the cloud storage array services including: generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics; andpresenting the metadata to a user through at least one of the one or more local networks.
  • 7. The apparatus of claim 6, wherein generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics further comprises: generating metadata describing one or more real-time storage array characteristics of each of the plurality of the storage arrays; andaggregating, for presentation to the user, the metadata of each of the plurality of storage arrays.
  • 8. The apparatus of claim 6, wherein initiating the one or more cloud storage array services further comprises: selecting, in dependence upon workload characteristics of each of the plurality of storage arrays, one of the plurality of storage arrays to host the one or more cloud storage array services.
  • 9. The apparatus of claim 8, further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of: detecting a change in workload characteristics of at least one of the plurality of storage arrays; andre-selecting one of the plurality of storage arrays to host the one or more cloud storage array services in dependence upon the changed workload characteristics.
  • 10. The apparatus of claim 8, further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of: operating, by a storage array that is not hosting the one or more cloud storage array services, as a proxy for the storage array that is hosting the one or more cloud storage array services; andpresenting the metadata to a user through at least one of the one or more local networks further comprises presenting the metadata through the proxy of the storage array that is hosting the one or more cloud storage array services.
  • 11. A computer program product for locally providing cloud storage array services for a storage array of a data center, wherein: the data center comprises a plurality of storage arrays and each storage array is coupled only to one or more local networks;the computer program product disposed upon a non-transitory computer readable medium, the computer program product comprising computer program instructions that, when executed, cause a computer to carry out the steps of:initiating, on at least one of the plurality of storage arrays, one or more cloud storage array services; andlocally providing the cloud storage array services including: generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics; andpresenting the metadata to a user through at least one of the one or more local networks.
  • 12. The computer program product of claim 11, wherein generating, by the cloud storage array services, metadata describing one or more real-time storage array characteristics further comprises:generating metadata describing one or more real-time storage array characteristics of each of the plurality of the storage arrays; andaggregating, for presentation to the user, the metadata of each of the plurality of storage arrays.
  • 13. The computer program product of claim 11, wherein initiating the one or more cloud storage array services further comprises:selecting, in dependence upon workload characteristics of each of the plurality of storage arrays, one of the plurality of storage arrays to host the one or more cloud storage array services.
  • 14. The computer program product of claim 13, further comprising computer program instructions that, when executed, cause the computer to carry out the steps of: detecting a change in workload characteristics of at least one of the plurality of storage arrays; andre-selecting one of the plurality of storage arrays to host the one or more cloud storage array services in dependence upon the changed workload characteristics.
CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of and claims priority from U.S. patent application Ser. No. 14/721,571, filed on May 26, 2015.

US Referenced Citations (170)
Number Name Date Kind
5706210 Kumano et al. Jan 1998 A
5799200 Brant et al. Aug 1998 A
5933598 Scales et al. Aug 1999 A
6012032 Donovan et al. Jan 2000 A
6085333 DeKoning et al. Jul 2000 A
6643641 Snyder Nov 2003 B1
6647514 Umberger et al. Nov 2003 B1
6789162 Talagala et al. Sep 2004 B1
7089272 Garthwaite et al. Aug 2006 B1
7107389 Inagaki et al. Sep 2006 B2
7146521 Nguyen Dec 2006 B1
7334124 Pham et al. Feb 2008 B2
7437530 Rajan Oct 2008 B1
7493424 Bali et al. Feb 2009 B1
7669029 Mishra et al. Feb 2010 B1
7689609 Lango et al. Mar 2010 B2
7743191 Liao Jun 2010 B1
7899780 Shmuylovich et al. Mar 2011 B1
8042163 Karr et al. Oct 2011 B1
8086585 Brashers et al. Dec 2011 B1
8271700 Annem et al. Sep 2012 B1
8387136 Lee et al. Feb 2013 B2
8437189 Montierth et al. May 2013 B1
8465332 Hogan et al. Jun 2013 B2
8527544 Colgrove et al. Sep 2013 B1
8566546 Marshak et al. Oct 2013 B1
8578442 Banerjee Nov 2013 B1
8613066 Brezinski et al. Dec 2013 B1
8620970 English et al. Dec 2013 B2
8751463 Chamness Jun 2014 B1
8762642 Bates et al. Jun 2014 B2
8769622 Chang et al. Jul 2014 B2
8800009 Beda, III et al. Aug 2014 B1
8812860 Bray Aug 2014 B1
8850546 Field et al. Sep 2014 B1
8898346 Simmons Nov 2014 B1
8909854 Yamagishi et al. Dec 2014 B2
8931041 Banerjee Jan 2015 B1
8949863 Coatney et al. Feb 2015 B1
8984602 Bailey et al. Mar 2015 B1
8990905 Bailey et al. Mar 2015 B1
9112936 Poletto et al. Aug 2015 B1
9124569 Hussain et al. Sep 2015 B2
9134922 Rajagopal et al. Sep 2015 B2
9152578 Saad Oct 2015 B1
9197696 Jakatdar et al. Nov 2015 B1
9209973 Aikas et al. Dec 2015 B2
9250823 Kamat et al. Feb 2016 B1
9300660 Borowiec et al. Mar 2016 B1
9330106 Piasecki et al. May 2016 B2
9444822 Borowiec et al. Sep 2016 B1
9507532 Colgrove et al. Nov 2016 B1
9521200 Borowiec et al. Dec 2016 B1
9626127 Glickman Apr 2017 B2
9716746 Garg Jul 2017 B2
9716755 Borowiec et al. Jul 2017 B2
20020013802 Mori et al. Jan 2002 A1
20030145172 Galbraith et al. Jul 2003 A1
20030191783 Wolczko et al. Oct 2003 A1
20030225961 Chow et al. Dec 2003 A1
20040080985 Chang et al. Apr 2004 A1
20040111573 Garthwaite Jun 2004 A1
20040153844 Ghose et al. Aug 2004 A1
20040193814 Erickson et al. Sep 2004 A1
20040260967 Guha et al. Dec 2004 A1
20050160416 Jamison Jul 2005 A1
20050188246 Emberty et al. Aug 2005 A1
20050216800 Bicknell et al. Sep 2005 A1
20060015771 Vana Gundy et al. Jan 2006 A1
20060129817 Borneman et al. Jun 2006 A1
20060161726 Lasser Jul 2006 A1
20060230245 Gounares et al. Oct 2006 A1
20060239075 Williams et al. Oct 2006 A1
20070022227 Miki Jan 2007 A1
20070028068 Golding et al. Feb 2007 A1
20070055702 Fridella et al. Mar 2007 A1
20070109856 Pellicone et al. May 2007 A1
20070150689 Pandit et al. Jun 2007 A1
20070168321 Saito et al. Jul 2007 A1
20070220227 Long Sep 2007 A1
20070294563 Bose Dec 2007 A1
20070294564 Reddin et al. Dec 2007 A1
20080005587 Ahlquist Jan 2008 A1
20080077825 Bello et al. Mar 2008 A1
20080162674 Dahiya Jul 2008 A1
20080195833 Park Aug 2008 A1
20080270678 Cornwell et al. Oct 2008 A1
20080282045 Biswas et al. Nov 2008 A1
20090077340 Johnson et al. Mar 2009 A1
20090100115 Park et al. Apr 2009 A1
20090198889 Ito et al. Aug 2009 A1
20100052625 Cagno et al. Mar 2010 A1
20100199042 Bates Aug 2010 A1
20100211723 Mukaida Aug 2010 A1
20100246266 Park et al. Sep 2010 A1
20100257142 Murphy et al. Oct 2010 A1
20100262764 Liu et al. Oct 2010 A1
20100325345 Ohno et al. Dec 2010 A1
20100332754 Lai et al. Dec 2010 A1
20110072290 Davis et al. Mar 2011 A1
20110125955 Chen May 2011 A1
20110131231 Haas et al. Jun 2011 A1
20110167221 Pangal Jul 2011 A1
20110258461 Bates Oct 2011 A1
20120023144 Rub Jan 2012 A1
20120054264 Haugh et al. Mar 2012 A1
20120079318 Colgrove et al. Mar 2012 A1
20120131253 McKnight et al. May 2012 A1
20120173822 Testardi Jul 2012 A1
20120303919 Hu et al. Nov 2012 A1
20120311000 Post et al. Dec 2012 A1
20130007845 Chang et al. Jan 2013 A1
20130031414 Dhuse et al. Jan 2013 A1
20130036272 Nelson Feb 2013 A1
20130071087 Motiwala et al. Mar 2013 A1
20130117506 Glickman May 2013 A1
20130145447 Maron Jun 2013 A1
20130191555 Liu Jul 2013 A1
20130198459 Joshi et al. Aug 2013 A1
20130205173 Yoneda Aug 2013 A1
20130219164 Hamid Aug 2013 A1
20130227201 Talagala et al. Aug 2013 A1
20130262638 Kumarasamy Oct 2013 A1
20130262801 Sancheti Oct 2013 A1
20130290607 Chang et al. Oct 2013 A1
20130311434 Jones Nov 2013 A1
20130318297 Jibbe et al. Nov 2013 A1
20130332614 Brunk et al. Dec 2013 A1
20140020083 Fetik Jan 2014 A1
20140074850 Noel et al. Mar 2014 A1
20140082715 Grajek et al. Mar 2014 A1
20140086146 Kim et al. Mar 2014 A1
20140090009 Li et al. Mar 2014 A1
20140096220 Da Cruz Pinto et al. Apr 2014 A1
20140101434 Senthurpandi et al. Apr 2014 A1
20140164774 Nord et al. Jun 2014 A1
20140173232 Reohr et al. Jun 2014 A1
20140195636 Karve et al. Jul 2014 A1
20140201512 Seethaler et al. Jul 2014 A1
20140201541 Paul et al. Jul 2014 A1
20140208155 Pan Jul 2014 A1
20140215590 Brand Jul 2014 A1
20140229654 Goss et al. Aug 2014 A1
20140230017 Saib Aug 2014 A1
20140258526 Le Sant et al. Sep 2014 A1
20140282983 Ju et al. Sep 2014 A1
20140285917 Cudak et al. Sep 2014 A1
20140308098 Lert Oct 2014 A1
20140325262 Cooper et al. Oct 2014 A1
20140351627 Best et al. Nov 2014 A1
20140373104 Gaddam et al. Dec 2014 A1
20140373126 Hussain et al. Dec 2014 A1
20150026387 Sheredy et al. Jan 2015 A1
20150032817 Garg Jan 2015 A1
20150052392 Mickens Feb 2015 A1
20150074463 Jacoby et al. Mar 2015 A1
20150089569 Sondhi et al. Mar 2015 A1
20150095515 Krithivas et al. Apr 2015 A1
20150113203 Dancho et al. Apr 2015 A1
20150121137 McKnight et al. Apr 2015 A1
20150134920 Anderson et al. May 2015 A1
20150149822 Coronado et al. May 2015 A1
20150193169 Sundaram et al. Jul 2015 A1
20150378888 Zhang et al. Dec 2015 A1
20160098323 Mutha et al. Apr 2016 A1
20160170823 Miller Jun 2016 A1
20160350009 Cerreta et al. Dec 2016 A1
20160352720 Hu et al. Dec 2016 A1
20160352830 Borowiec Dec 2016 A1
20160352834 Borowiec et al. Dec 2016 A1
Foreign Referenced Citations (9)
Number Date Country
0725324 Aug 1996 EP
WO-2012087648 Jun 2012 WO
WO-2013071087 May 2013 WO
WO-2014110137 Jul 2014 WO
WO-2016015008 Dec 2016 WO
WO-2016190938 Dec 2016 WO
WO-2016195759 Dec 2016 WO
WO-2016195958 Dec 2016 WO
WO-2016195961 Dec 2016 WO
Non-Patent Literature Citations (32)
Entry
Paul Sweere, Creating Storage Class Persistent Memory with NVDIMM, Published in Aug. 2013, Flash Memory Summit 2013, <http://ww.flashmemorysummit.com/English/Collaterals/Proceedings/2013/20130814_T2_Sweere.pdf>, 22 pages.
PCMAG, Storage Array Definition, Published May 10, 2013. <http://web.archive.org/web/20130510121646/http://www.pcmag.com/encyclopedia/term/52091/storage-array>, 2 pages.
Google Search of “storage array define” performed by the Examiner on Nov. 4, 2015 for U.S. Appl. No. 14/725,278, Results limited to entries dated before 2012, 1 page.
Techopedia, What is a disk array, techopedia.com (online), Jan. 13, 2012, 1 page, URL: web.archive.org/web/20120113053358/http://www.techopedia.com/definition/1009/disk-array.
Webopedia, What is a disk array, webopedia.com (online), May 26, 2011, 2 pages, URL: web/archive.org/web/20110526081214/http://www.webopedia.com/TERM/D/disk_array.html.
Li et al., Access Control for the Services Oriented Architecture, Proceedings of the 2007 ACM Workshop on Secure Web Services (SWS '07), Nov. 2007, pp. 9-17, ACM New York, NY.
Hota et al., Capability-based Cryptographic Data Access Control in Cloud Computing, International Journal of Advanced Networking and Applications, col. 1, Issue 1, Aug. 2011, 10 pages, Eswar Publications, India.
Faith, dictzip file format, GitHub.com (online), accessed Jul. 28, 2015, 1 page, URL: github.com/fidlej/idzip.
Wikipedia, Convergent Encryption, Wikipedia.org (online), accessed Sep. 8, 2015, 2 pages, URL: en.wikipedia.org/wiki/Convergent_encryption.
Storer et al., Secure Data Deduplication, Proceedings of the 4th ACM International Workshop on Storage Security and Survivability (StorageSS'08), Oct. 2008, 10 pages, ACM New York, NY. USA, DOI: 10.1145/1456469.1456471.
ETSI, Network Function Virtualisation (NFV); Resiliency Requirements, ETSI GS NFCV-REL 001, V1.1.1, Jan. 2015, 82 pages, etsi.org (online), URL: www.etsi.org/deliver/etsi_gs/NFV-REL/001_099/001/01.01.01_60/gs_NFV-REL001v010101p.pdf.
Microsoft, Hybrid for SharePoint Server 2013—Security Reference Architecture, Microsoft (online), Oct. 2014, 53 pages, URL: hybrid.office.com/img/Security_Reference_Architecture.pdf.
Microsoft, Hybrid Identity, Microsoft (online), Apr. 2014, 36 pages,URL: www.aka.ms/HybridIdentityWp.
Microsoft, Hybrid Identity Management, Microsoft (online), Apr. 2014, 17 pages, URL: download.microsoft.com/download/E/A/E/EAE57CD1-A80B-423C-96BB-142FAAC630B9/Hybrid_Identity_Datasheet.pdf.
Bellamy-McIntyre et al., OpenID and the Enterprise: A Model-based Analysis of Single Sign-On Authentication, 15th IEEE International Enterprise Distributed Object Computing Conference (EDOC), Aug. 29, 2011, pp. 129-138, IEEE Computer Society, USA, DOI: 10.1109/EDOC.2011.26, ISBN: 978-1-4577-0362-1.
Kong, Using PCI Express as the Primary System Interconnect in Multiroot Compute, Storage, Communications and Embedded Systems, White Paper, IDT.com (online), Aug. 28, 2008, 12 pages, URL: www.idt.com/document/whp/idt-pcie-multi-root-white-paper.
Hu et al., Container Marking: Combining Data Placement, Garbage Collection and Wear Levelling for Flash, 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunications Systems, Jul. 25-27, 2011, 11 pages, ISBN: 978-0-7695-4430-4, DOI: 10.1109/MASCOTS.2011.50.
International Search Report and Written Opinion, PCT/US2016/015006, dated Jul. 18, 2016, 12 pages.
International Search Report and Written Opinion, PCT/US2016/015008, dated May 4, 2016, 12 pages.
International Search Report and Written Opinion, PCT/US2016/020410, dated Jul. 8, 2016, 12 pages.
International Search Report and Written Opinion, PCT/US2016/032084, dated Jul. 18, 2016, 12 pages.
International Search Report and Written Opinion, PCT/US2016/016333, dated Jun. 8, 2016, 12 pages.
International Search Report and Written Opinion, PCT/US2016/032052, dated Aug. 30, 2016, 17 pages.
International Search Report and Written Opinion, PCT/US2016/035492, dated Aug. 17, 2016, 10 pages.
International Search Report and Written Opinion, PCT/US2016/036693, dated Aug. 29, 2016, 10 pages.
International Search Report and Written Opinion, PCT/US2016/038758, dated Oct. 7, 2016, 10 pages.
International Search Report and Written Opinion, PCT/US2016/040393, dated Sep. 22, 2016, 10 pages.
International Search Report and Written Opinion, PCT/US2016/044020, dated Sep. 30, 2016, 11 pages.
International Search Report and Written Opinion, PCT/US2016/044874, dated Oct. 7, 2016, 11 pages.
International Search Report and Written Opinion, PCT/US2016/044875, dated Oct. 5, 2016, 13 pages.
International Search Report and Written Opinion, PCT/US2016/044876, dated Oct. 21, 2016, 12 pages.
International Search Report and Written Opinion, PCT/US2016/044877, dated Sep. 29, 2016, 13 pages.
Continuations (1)
Number Date Country
Parent 14721571 May 2015 US
Child 15419551 US