The present disclosure relates generally to network-attached devices, more particularly, to a system and method for supporting multi-path and/or multi-mode NVMe over fabrics (NVMeoF) devices.
Non-volatile memory express (NVMe) and NVMe over fabrics (NVMeoF) (or NVMf in short) are new emerging technologies. NVMe is a standard that defines a register-level interface for host software to communicate with a non-volatile memory subsystem (e.g., a solid-state drive (SSD)) over a peripheral component interconnect express (PCIe) bus.
NVMeoF defines a common architecture that supports an NVMe block storage protocol over a wide range of storage networking fabrics such as Ethernet, Fibre Channel, InfiniBand, and other network fabrics. For an NVMeoF-based system, an X86-based central processing unit (CPU) on a motherboard is no longer required to move data between an initiator (e.g., host software) and a target device (i.e., an NVMeoF device) because the target device is capable of moving data by itself. The term, fabric, represents a network topology in which network nodes can pass data to each other through a variety of interconnecting protocols, ports, and switches. For example, Ethernet-attached SSDs may attach directly to a fabric, and in this case the fabric is the Ethernet.
The physical connection of the NVMe is based on a PCIe bus. A typical Ethernet SSD has a U.2 connector to interface with a system via a mid-plane over the PCIe bus. In the case of the four-lane PCIe bus (PCIe x4), the two Ethernet ports consume only two lanes of the four-lane PCIe signals, and the remaining two lanes of the PCIe X4 signals remain unused.
According to one embodiment, a system includes a fabric switch including a motherboard, a baseboard management controller (BMC), a network switch configured to transport network signals, and a PCIe switch configured to transport PCIe signals; a midplane; and a plurality of device ports. Each of the plurality of device ports is configured to connect a storage device to the motherboard of the fabric switch over the midplane and carry the network signals and the PCIe signals over the midplane. The storage device is configurable in multiple modes based on a protocol established over a fabric connection between the system and the storage device.
According to another embodiment, an NVMeoF includes: a PCIe module; a network engine; and a connector configured to connect to a switch motherboard over a midplane and carry PCIe signals over the midplane. The PCIe module transports PCIe signals to the switch over the PCIe bus, and the network engine transport network signals to the switch over Serial Attached SCSI (SAS) pins of the connector.
According to yet another embodiment, a system includes: a switch and a plurality of NVMeoF devices. Each NVMeoF device is configured to be coupled to the switch using a connector. The connector is configured to transport the PCIe signals to the switch over a PCIe bus and transport network signals to the switch over a network bus.
The above and other preferred features, including various novel details of implementation and combination of events, will now be more particularly described with reference to the accompanying figures and pointed out in the claims. It will be understood that the particular systems and methods described herein are shown by way of illustration only and not as limitations. As will be understood by those skilled in the art, the principles and features described herein may be employed in various and numerous embodiments without departing from the scope of the present disclosure.
The accompanying drawings, which are included as part of the present specification, illustrate the presently preferred embodiment and together with the general description given above and the detailed description of the preferred embodiment given below serve to explain and teach the principles described herein.
The figures are not necessarily drawn to scale and elements of similar structures or functions are generally represented by like reference numerals for illustrative purposes throughout the figures. The figures are only intended to facilitate the description of the various embodiments described herein. The figures do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims.
Each of the features and teachings disclosed herein can be utilized separately or in conjunction with other features and teachings to provide a system and method for supporting multi-path and/or multi-mode NVMeoF devices. Representative examples utilizing many of these additional features and teachings, both separately and in combination, are described in further detail with reference to the attached figures. This detailed description is merely intended to teach a person of skill in the art further details for practicing aspects of the present teachings and is not intended to limit the scope of the claims. Therefore, combinations of features disclosed above in the detailed description may not be necessary to practice the teachings in the broadest sense, and are instead taught merely to describe particularly representative examples of the present teachings.
In the description below, for purposes of explanation only, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details are not required to practice the teachings of the present disclosure.
Some portions of the detailed descriptions herein are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the below discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems, computer servers, or personal computers may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. It will be appreciated that a variety of programming languages may be used to implement the teachings of the disclosure as described herein.
Moreover, the various features of the representative examples and the dependent claims may be combined in ways that are not specifically and explicitly enumerated in order to provide additional useful embodiments of the present teachings. It is also expressly noted that all value ranges or indications of groups of entities disclose every possible intermediate value or intermediate entity for the purpose of an original disclosure, as well as for the purpose of restricting the claimed subject matter. It is also expressly noted that the dimensions and the shapes of the components shown in the figures are designed to help to understand how the present teachings are practiced, but not intended to limit the dimensions and the shapes shown in the examples.
The present disclosure describes a system that can support both the NVMe and NVMeoF protocols, and various types of fabric-attached SSDs (eSSDs). In some embodiments, an eSSD refers to an SSD that can support the NVMeoF protocols. When configured to support the NVMeoF standard, the system can support various fabrics including not only Ethernet, but also, Fibre Channel, InfiniBand, and other network fabrics. For the convenience of illustration, the following examples and embodiments can show an Ethernet-attached NVMeoF devices. However, it is noted that any other type of NVMeoF devices can be used without deviating from the scope of the present disclosure.
The present system provides a single platform and common building blocks that can support both single and dual pathing systems compatible with both NVMe and NVMeoF devices. According to one embodiment, the common building blocks that support single pathing and dual pathing NVMe and NVMeoF devices include a mid-plane, a chassis, a fan assembly. The present system can scale linearly by adding more similar devices and/or chassis. The present system may also include other building blocks including, but not limited to, a full-width and a half-width switch boards, and an X86 motherboard.
The fabric-attached SSD (eSSD) disclosed herein is a single common device that can be used in multiples systems compatible with NVMe and NVMeoF standards. In this sense, the fabric-attached SSD is also referred to as a multi-mode NVMeoF device. The present system provides a platform that can supporting various types of NVMe and NVMeoF devices in non-high availability (non-HA) mode (i.e., single-path input/output (I/O/)) or HA mode (i.e., multi-path I/O) with minimum hardware changes.
According to one embodiment, the multi-mode NVMeoF device can support either NVMe or NVMeoF standard by detecting product information from a known location. For example, the product information used for self-configuration is stored in the chassis is a vital product data (VPD). During the start-up, the multi-mode NVMeoF device can retrieve the VPD from the chassis and configure itself based on the VPD. However, it is noted that the multi-mode NVMeoF device can be configured in various manners without deviating from the scope of the present disclosure. For example, the multi-mode NVMeoF device can be configured by a control command over the PCIe bus issued by a BMC of the switch to which the multi-mode NVMeoF device is connected.
According to one embodiment, the multi-mode NVMeoF device can be configured in a single port NVMe mode, a dual port NVMe mode, a single port NVMeoF mode, and a dual port NVMeoF mode. Table 1 shows example use of the U.2 connector according to the configuration of the multi-mode NVMeoF device. When configured as an NVMe device, the multi-mode NVMeoF device can be configured in either the single port NVMe mode or the dual port NVMe mode. In the single port NVMe mode, the PCIe lanes 0-3 of the U.2 connector are used to carry PCIe signals. In the dual port NVMe mode, the PCIe lanes are split into 2 by 2 lanes; the PCIe lanes 0 and 1 are used for the first port, and the PCIe lanes 2 and 3 are used for the second port. When configured as an NVMeoF device, the multi-mode NVMeoF device can be configured in either the single port NVMeoF mode or the dual port NVMeoF mode. In the single port NVMeoF mode, the PCIe lanes are split into 2 by 2 lanes but only the PCIe lanes 0 and 1 are used to carry PCIe signals, and the PCIe lanes 2 and 3 are not used. The first pair of the SAS port 0 is used for the Ethernet port 0 (first port), and the SAS port 1 is not used. In the dual port NVMeoF mode, the PCIe lanes are split into 2 by 2 lanes, and the PCIe lanes 0 and 1 are used as a control plane for the first Ethernet port, and the PCIe lanes 2 and 3 are used as a control plane for the second Ethernet port. The first pair of the SAS port 0 is used for the Ethernet port 0 (first port), and the SAS port 1 is used for the Ethernet port 1 (second port).
If the product information is stored in a chassis, the two lanes (in a single port mode) or four lanes (in a dual port mode) of the PCIe bus on the U.2 connector are driven by a PCIe engine. In this case, the multi-mode NVMeoF device can disable the Ethernet engine(s), and the NVMe protocols and functionalities are supported or enabled. If the product information is stored in an NVMeoF chassis, the Ethernet ports use only PCIe lanes 2 and 3, or Serial Attached SCSI (SAS) pins depending on the design of the multi-mode NVMeoF device.
The present multi-mode NVMeoF device can operate in two distinct modes, namely, an NVMe mode and an NVMeoF mode. In the NVMe mode, the multi-mode NVMeoF device behaves as an NVMe device. The PCIe pins of the U.2 connector can be connected to the PCIe x4 module 111. The PCIe bus can be shared by data and control.
In one embodiment, in the NVMeoF mode, the multi-mode NVMeoF device can be configured in a single-path mode or a dual-path mode. In the single path mode, one PCIe x2 is used for control plane and is connected to one motherboard. In the dual-path mode, two PCIe x2 are used for control plane and are connected to two motherboards.
In another embodiment, the NVMeoF device can use SAS pins for the Ethernet ports in the NVMeoF mode. In the non-HA NVMeoF mode, the two lanes of the PCIe bus are used for standard features through a control plane. In the dual-port HA mode, the four lanes of the PCIe bus are split into two X2 lanes and used for port A and port B, respectively. The existing PCIe software and driver may be used as unmodified for the multi-mode NVMeoF device.
Because the multi-mode NVMeoF device can operate both in the NVMe and NVMeoF modes, the cost for developing and deploying the devices can be reduced because the same devices can be used in the NVMe mode and the NVMeoF mode. For the similar reason, the multi-mode NVMeoF device can have a faster time to the market. The multi-mode NVMeoF device can be used in various products and chassis. The two lanes of the PCIe bus are reserved for standard features through a control plane. A CPU, a baseboard management controller (BMC), and other devices can use the two lanes of the PCIe bus as a control plane to communicate to each NVMeoF device inside the chassis at no additional cost. The NVMe mid-plane can be used as unmodified, and there is no need for a new connector on the NVMeoF device due to the additional new pins.
According to one embodiment, the NVMeoF device 101 can be configured to as an NVMe device. In the NVMe mode, a mode selector 160 can configure the NVMeoF device 101 to use all of the four lanes (in a single port mode) or only two lanes (in a dual port mode) of the four lanes of the PCIe bus to carry PCIe signals. The PCI x4 bus is connected to a midplane, and the PCIe bus is shared between data and control signals.
According to another embodiment, the NVMeoF device 101 can be configured as an NVMeoF device. In the NVMeoF mode, the mode selector 160 can configure the NVMeoF device 101 to use the two lanes of the PCIe X2 bus 151 to carry PCIe signals. The NVMeoF device 101 can further configure the remaining two lanes of the PCIe bus to carry Ethernet signals over the two Ethernet ports 153 and 154. In the NVMeoF mode, the two PCIe X2 lanes are directly transported to the PCIe X4 module 111, and signals over the remaining two PCIe X2 lanes are carried over the Ethernet ports 153 and 154 and buffered in the buffer 122 to be transported to the Ethernet NIC 112 of the NVMeoF device 101. The operational mode of the NVMeoF device 101 can be self-configured or externally set. For example, the NVMeoF device 101 can self-configure its operational mode using a physical pin (e.g., a presence pin on the chassis of the switch motherboard) or by an in-band command from a BMC (e.g., BMC 201 of
The NVMeoF device 101 can push various signals and perform various services over the PCIe ports 151 and 152 using the unused PCI X2 bus over the U.2 connector. Examples of the signals that can be pushed include, but are not limited to, health status information, field-replaceable unit (FRU) information, and sensor information of the NVMEOF device 101. Examples of the services that can be pushed over the PCIe ports include, but are not limited to, discovery services to a BMC or a CPU that is local to the switchboard and download services for a new NVMeoF device firmware for performing a firmware upgrade.
The NVMeoF device 101 can push some device-specific information directly to a BMC of the switch motherboard over the PCI X2 bus 151 over a control plane established between the switch motherboard and the NVMeoF device 101. Examples of such device-specific information that can be carried over the control plane include, but are not limited to, discovery information and FRU information of the NVMEOF device 101. This can reduce the burden of the BMC for polling the status of the NVMeoF device 101. The device-specific information may be communicated between the NVMeoF device 101 and the BMC using a new device command. The NVMeoF device 101 can support high availability (HA) multipath I/O with only the two PCIe lanes 151 and 152 of the PCIe X2 bus.
According to one embodiment, the NVMeoF device 301 can use the unused SAS pins of the U.2 connector 321 for Ethernet signals instead of using the PCIe lanes 153 and 154 as shown in
Several eSSDs can be plugged into device ports of the switch. For example, each of the eSSDs is connected to the switch using a U.2 connector. Each eSSD can connect to both the switch motherboard 501A and the switch motherboard 501B. In the present example, the eSSDs plugged into the switch 500 are configured as an NVMeoF device requiring connectivity to the switch 500 over the midplane 561 via the PCIe bus and Ethernet ports.
According to one embodiment, the Ethernet signals between the switch 500 and the eSSDs can use SAS pins S2, S3, S5, and S6 for the primary Ethernet port 553 to the switch motherboard 501A. The Ethernet signals can also use S9, S10, S12, and S13 for the secondary Ethernet port 554 to the switch motherboard 501B. E25 pin of each U.2 connector can be used to enable the dual port configuration. PCIe signals can be carried over to PCIe buses 551 and 552 between the respective switch motherboards 501A and 501B and each of the eSSDs. The eSSD can self-configure its operational mode using a physical pin (e.g., a presence pin on the chassis of the switch) or by an in-band command from a BMC of the switch motherboard.
According to one embodiment, the switch 500 can support 10G Ethernet, and the midplane 561 is a common midplane that can support both a HA mode and a non-HA mode. Depending on the system configuration, signal integrity may need to be tested to ensure that the common midplane 561 can support for both configurations. If the signal integrity is not sufficient, the system can have two midplanes including the first midplane for the HA mode and the second midplane for the non-HA mode.
According to one embodiment, a system includes a fabric switch including a motherboard, a baseboard management controller (BMC), a network switch configured to transport network signals, and a PCIe switch configured to transport PCIe signals; a midplane; and a plurality of device ports. Each of the plurality of device ports is configured to connect a storage device to the motherboard of the fabric switch over the midplane and carry the network signals and the PCIe signals over the midplane. The storage device is configurable in multiple modes based on a protocol established over a fabric connection between the system and the storage device.
The storage device may have a U.2 connector.
The storage device may support both NVMe and NVMeoF protocols.
The midplane may support both a high availability (HA) mode and a non-HA mode.
The network signals may be carried over unused pins of the connector.
The network signals may provide discovery services or download services for a new firmware of the storage device.
The network signals may include device-specific information including one or more of health status information, field-replaceable unit (FRU) information, and sensor information of the storage device, and the device-specific information may be transported to the BMC over the midplane via PCIe lanes.
The storage device may be configured to operate in a HA mode or a non-HA mode.
According to another embodiment, an NVMeoF includes: a PCIe module; a network engine; and a connector configured to connect to a switch motherboard over a midplane and carry PCIe signals over the midplane. The PCIe module transports PCIe signals to the switch over the PCIe bus, and the network engine transport network signals to the switch over Serial Attached SCSI (SAS) pins of the connector.
The connector may be a U.2 connector.
The network signals may include device-specific information including one or more of health status information, FRU information, and sensor information of the NVMeoF device.
The device-specific information may be carried to a BMC of the switch over the midplane.
The network signals may provide discovery services or download services for a new firmware of the NVMeoF device.
The switch may include two switch boards including a primary Ethernet port and a secondary Ethernet port.
SAS pins S2, S3, S5, and S6 may be used for the primary Ethernet port, and SAS pins S9, S10, S12, and S13 may be used for the secondary Ethernet port.
The NVMeoF device may be configured to operate in a HA mode or a non-HA mode.
According to yet another embodiment, a system includes: a switch and a plurality of NVMeoF devices. Each NVMeoF device is configured to be coupled to the switch using a connector. The connector is configured to transport the PCIe signals to the switch over a PCIe bus and transport network signals to the switch over a network bus.
The connector may be a U.2 connector.
The PCIe signals may be transported over two PCIe lanes of the PCIe bus, and the network signals may be transported over the remaining two PCIe lanes of the PCIe bus.
The network signals may be transported over SAS pins.
The network signals may include device-specific information including one or more of health status information, FRU information, and sensor information of each NVMeoF device.
The network signals may provide discovery services or download services for a new firmware of each NVMeoF device.
Each NVMeoF device of the NVMeoF devices may be configured to operate in a HA mode or a non-HA mode.
The above example embodiments have been described hereinabove to illustrate various embodiments of implementing a system and method for supporting multi-path and/or multi-mode NVMeoF devices. Various modifications and departures from the disclosed example embodiments will occur to those having ordinary skill in the art. The subject matter that is intended to be within the scope of the invention is set forth in the following claims.
This application is a continuation application of U.S. patent application Ser. No. 16/692,997 filed Nov. 22, 2019, which is a divisional application of U.S. patent application Ser. No. 16/211,923 filed Dec. 6, 2018, now issued to U.S. Pat. No. 10,540,311, which is a continuation application of U.S. patent application Ser. No. 15/403,088 filed Jan. 10, 2017, now issued to U.S. Pat. No. 10,210,123, which claims the benefits of and priority to U.S. Provisional Patent Application Serial Nos. 62/366,622 filed Jul. 26, 2016 and entitled “SELF-CONFIGURING SSD MULTI-PROTOCOL SUPPORT IN HOST-LESS ENVIRONMENT,” and 62/420,355 filed Nov. 10, 2016 and entitled “MODULAR SYSTEM ARCHITECTURE FOR SUPPORTING MULTI-PATH AND/OR MULTI-MODE NMVE OVER FABRICS DEVICES,” and the disclosures of which are incorporated herein by reference in their entirety.
Number | Name | Date | Kind |
---|---|---|---|
6295567 | Bassman et al. | Sep 2001 | B1 |
6345303 | Knauerhase et al. | Feb 2002 | B1 |
6427198 | Berglund et al. | Jul 2002 | B1 |
6611863 | Banginwar | Aug 2003 | B1 |
6662119 | Mitchell | Dec 2003 | B1 |
7107253 | Sumner et al. | Sep 2006 | B1 |
7120759 | Chiu et al. | Oct 2006 | B2 |
7143153 | Black et al. | Nov 2006 | B1 |
7249173 | Nicolson | Jul 2007 | B2 |
7512585 | Agarwal et al. | Mar 2009 | B2 |
7536486 | Sadovsky et al. | May 2009 | B2 |
7620854 | Kuttan et al. | Nov 2009 | B2 |
7873700 | Pawlowski et al. | Jan 2011 | B2 |
7882393 | Grimes et al. | Feb 2011 | B2 |
7944812 | Carlson et al. | May 2011 | B2 |
8065347 | Demeyer et al. | Nov 2011 | B1 |
8180862 | Baker et al. | May 2012 | B2 |
8396981 | Lee et al. | Mar 2013 | B1 |
8599863 | Davis | Dec 2013 | B2 |
8667224 | Yu et al. | Mar 2014 | B1 |
8832327 | Lin | Sep 2014 | B1 |
8943234 | Voorhees | Jan 2015 | B1 |
8949517 | Cohen et al. | Feb 2015 | B2 |
8953644 | Chandra et al. | Feb 2015 | B2 |
8998636 | Gomez et al. | Apr 2015 | B2 |
9037786 | Asnaashari et al. | May 2015 | B2 |
9047222 | Chandra et al. | Jun 2015 | B2 |
9092321 | Salessi | Jul 2015 | B2 |
9244865 | Hutchison et al. | Jan 2016 | B2 |
9244877 | Yang et al. | Jan 2016 | B2 |
9253275 | Bhogal et al. | Feb 2016 | B2 |
9280357 | Shaver et al. | Mar 2016 | B2 |
9280504 | Ben-Michael et al. | Mar 2016 | B2 |
9389805 | Cohen et al. | Jul 2016 | B2 |
9400749 | Kuzmin et al. | Jul 2016 | B1 |
9460042 | Iskandar et al. | Oct 2016 | B2 |
9465756 | Bennett | Oct 2016 | B2 |
9648148 | Rimmer et al. | May 2017 | B2 |
9653124 | Heyd et al. | May 2017 | B2 |
9734093 | Khemani et al. | Aug 2017 | B2 |
9734106 | Kotzur et al. | Aug 2017 | B2 |
9785346 | Yost | Oct 2017 | B2 |
9785355 | Huang | Oct 2017 | B2 |
9785356 | Huang | Oct 2017 | B2 |
9811481 | Bhatia et al. | Nov 2017 | B2 |
9830082 | Srinivasan et al. | Nov 2017 | B1 |
9842084 | Friedman et al. | Dec 2017 | B2 |
9904330 | Schuette et al. | Feb 2018 | B2 |
9906596 | Sikdar | Feb 2018 | B2 |
9934173 | Sakalley et al. | Apr 2018 | B1 |
9934183 | Brassac et al. | Apr 2018 | B2 |
9959240 | Mundt | May 2018 | B2 |
9965367 | Shih | May 2018 | B2 |
9990313 | Monji et al. | Jun 2018 | B2 |
10019388 | Long | Jul 2018 | B2 |
10063638 | Huang | Aug 2018 | B2 |
10108450 | Pinto et al. | Oct 2018 | B2 |
10114778 | Worley et al. | Oct 2018 | B2 |
10162784 | Bassett | Dec 2018 | B2 |
10206297 | Breakstone et al. | Feb 2019 | B2 |
10223313 | Shih | Mar 2019 | B2 |
10223316 | Mataya | Mar 2019 | B2 |
10235313 | Lee et al. | Mar 2019 | B2 |
10255215 | Breakstone et al. | Apr 2019 | B2 |
10275356 | Chou et al. | Apr 2019 | B2 |
10289517 | Beerens | May 2019 | B2 |
10289588 | Chu | May 2019 | B2 |
10318443 | Su | Jun 2019 | B2 |
10346041 | Olarig et al. | Jul 2019 | B2 |
10372648 | Qiu | Aug 2019 | B2 |
10372659 | Olarig et al. | Aug 2019 | B2 |
10394723 | Yang | Aug 2019 | B2 |
10452576 | Stuhlsatz | Oct 2019 | B2 |
10467163 | Malwankar et al. | Nov 2019 | B1 |
10467170 | McKnight | Nov 2019 | B2 |
10474589 | Raskin | Nov 2019 | B1 |
10560550 | Xue et al. | Feb 2020 | B1 |
10592144 | Roberts et al. | Mar 2020 | B2 |
10733137 | Kachare et al. | Aug 2020 | B2 |
10866911 | Qiu et al. | Dec 2020 | B2 |
10901927 | Fischer et al. | Jan 2021 | B2 |
10929327 | Schrempp et al. | Feb 2021 | B1 |
10942666 | Pydipaty et al. | Mar 2021 | B2 |
11113046 | Bowen et al. | Sep 2021 | B1 |
11126352 | Olarig et al. | Sep 2021 | B2 |
11347740 | Moshe et al. | May 2022 | B2 |
20020087887 | Busam et al. | Jul 2002 | A1 |
20020095491 | Edmonds et al. | Jul 2002 | A1 |
20020123365 | Thorson et al. | Sep 2002 | A1 |
20030058818 | Wilkes et al. | Mar 2003 | A1 |
20040073912 | Meza | Apr 2004 | A1 |
20040111590 | Klein | Jun 2004 | A1 |
20040147281 | Holcombe et al. | Jul 2004 | A1 |
20040153844 | Ghose et al. | Aug 2004 | A1 |
20050025125 | Kwan | Feb 2005 | A1 |
20050060442 | Beverly et al. | Mar 2005 | A1 |
20050120157 | Chen et al. | Jun 2005 | A1 |
20060059287 | Rivard et al. | Mar 2006 | A1 |
20060095625 | Wootten et al. | May 2006 | A1 |
20060098681 | Cafiero et al. | May 2006 | A1 |
20060136621 | Tung et al. | Jun 2006 | A1 |
20060202950 | Lee et al. | Sep 2006 | A1 |
20070077553 | Bentwich | Apr 2007 | A1 |
20080003845 | Hong et al. | Jan 2008 | A1 |
20080288708 | Hsueh | Nov 2008 | A1 |
20090073896 | Gillingham et al. | Mar 2009 | A1 |
20090077478 | Gillingham et al. | Mar 2009 | A1 |
20090217188 | Alexander et al. | Aug 2009 | A1 |
20090222733 | Basham et al. | Sep 2009 | A1 |
20090259364 | Vollmer et al. | Oct 2009 | A1 |
20100077067 | Strole | Mar 2010 | A1 |
20100100858 | Schipper | Apr 2010 | A1 |
20100106836 | Schreyer et al. | Apr 2010 | A1 |
20100169512 | Matton et al. | Jul 2010 | A1 |
20110131380 | Rallens et al. | Jun 2011 | A1 |
20110151858 | Lai | Jun 2011 | A1 |
20120056728 | Erdmann et al. | Mar 2012 | A1 |
20120102580 | Bealkowski | Apr 2012 | A1 |
20120207156 | Srinivasan et al. | Aug 2012 | A1 |
20120311654 | Dougherty, III et al. | Dec 2012 | A1 |
20130117503 | Nellans et al. | May 2013 | A1 |
20130117766 | Bax et al. | May 2013 | A1 |
20130179624 | Lambert et al. | Jul 2013 | A1 |
20130198311 | Tamir et al. | Aug 2013 | A1 |
20130198312 | Tamir et al. | Aug 2013 | A1 |
20130242991 | Basso et al. | Sep 2013 | A1 |
20130282953 | Orme et al. | Oct 2013 | A1 |
20130304979 | Zimmer et al. | Nov 2013 | A1 |
20130311795 | Cong et al. | Nov 2013 | A1 |
20130318371 | Hormuth | Nov 2013 | A1 |
20130325998 | Hormuth et al. | Dec 2013 | A1 |
20140032641 | Du | Jan 2014 | A1 |
20140052928 | Shimoi | Feb 2014 | A1 |
20140122746 | Shaver et al. | May 2014 | A1 |
20140195711 | Bhatia et al. | Jul 2014 | A1 |
20140258679 | McGee | Sep 2014 | A1 |
20140281458 | Ravimohan et al. | Sep 2014 | A1 |
20140317206 | Lomelino et al. | Oct 2014 | A1 |
20140330995 | Levy et al. | Nov 2014 | A1 |
20140344431 | Hsu et al. | Nov 2014 | A1 |
20150006758 | Holtman et al. | Jan 2015 | A1 |
20150039815 | Klein | Feb 2015 | A1 |
20150067188 | Chakhaiyar | Mar 2015 | A1 |
20150086017 | Taylor et al. | Mar 2015 | A1 |
20150106660 | Chumbalkar et al. | Apr 2015 | A1 |
20150120874 | Kim | Apr 2015 | A1 |
20150120971 | Bae et al. | Apr 2015 | A1 |
20150138900 | Choi | May 2015 | A1 |
20150178095 | Balakrishnan et al. | Jun 2015 | A1 |
20150181760 | Stephens | Jun 2015 | A1 |
20150205541 | Nishtala et al. | Jul 2015 | A1 |
20150234815 | Slik | Aug 2015 | A1 |
20150254088 | Chou et al. | Sep 2015 | A1 |
20150255130 | Lee et al. | Sep 2015 | A1 |
20150261434 | Kagan et al. | Sep 2015 | A1 |
20150286599 | Hershberger | Oct 2015 | A1 |
20150301757 | Iwata et al. | Oct 2015 | A1 |
20150301964 | Brinicombe et al. | Oct 2015 | A1 |
20150304423 | Satoyama et al. | Oct 2015 | A1 |
20150324312 | Jacobson et al. | Nov 2015 | A1 |
20150331473 | Jreji et al. | Nov 2015 | A1 |
20150350096 | Dinc et al. | Dec 2015 | A1 |
20150370661 | Swanson et al. | Dec 2015 | A1 |
20150370665 | Cannata et al. | Dec 2015 | A1 |
20150376840 | Shih | Dec 2015 | A1 |
20150381734 | Ebihara et al. | Dec 2015 | A1 |
20160004879 | Fisher et al. | Jan 2016 | A1 |
20160062936 | Brassac et al. | Mar 2016 | A1 |
20160077841 | Lambert et al. | Mar 2016 | A1 |
20160085718 | Huang | Mar 2016 | A1 |
20160092390 | Grothen et al. | Mar 2016 | A1 |
20160094619 | Khan et al. | Mar 2016 | A1 |
20160127468 | Malwankar et al. | May 2016 | A1 |
20160127492 | Malwankar et al. | May 2016 | A1 |
20160146754 | Prasad et al. | May 2016 | A1 |
20160188313 | Dubal et al. | Jun 2016 | A1 |
20160246754 | Rao et al. | Aug 2016 | A1 |
20160259597 | Worley et al. | Sep 2016 | A1 |
20160283428 | Guddeti | Sep 2016 | A1 |
20160306723 | Lu | Oct 2016 | A1 |
20160306768 | Mataya | Oct 2016 | A1 |
20160328344 | Jose et al. | Nov 2016 | A1 |
20160328347 | Worley | Nov 2016 | A1 |
20160337272 | Berman | Nov 2016 | A1 |
20160366071 | Chandran et al. | Dec 2016 | A1 |
20170018149 | Shih | Jan 2017 | A1 |
20170038804 | Shows et al. | Feb 2017 | A1 |
20170063965 | Grenader | Mar 2017 | A1 |
20170068268 | Giriyappa et al. | Mar 2017 | A1 |
20170068628 | Calciu et al. | Mar 2017 | A1 |
20170068630 | Iskandar et al. | Mar 2017 | A1 |
20170168943 | Chou | Jun 2017 | A1 |
20170185554 | Fricker | Jun 2017 | A1 |
20170187629 | Shalev et al. | Jun 2017 | A1 |
20170206034 | Fetik | Jul 2017 | A1 |
20170262029 | Nelson et al. | Sep 2017 | A1 |
20170269871 | Khan et al. | Sep 2017 | A1 |
20170270001 | Suryanarayana et al. | Sep 2017 | A1 |
20170270060 | Gupta et al. | Sep 2017 | A1 |
20170286305 | Kalwitz | Oct 2017 | A1 |
20170317901 | Agrawal et al. | Nov 2017 | A1 |
20170344259 | Freyensee et al. | Nov 2017 | A1 |
20170344294 | Mishra et al. | Nov 2017 | A1 |
20170357299 | Shabbir et al. | Dec 2017 | A1 |
20170357515 | Bower, III et al. | Dec 2017 | A1 |
20180004695 | Chu et al. | Jan 2018 | A1 |
20180019896 | Paquet et al. | Jan 2018 | A1 |
20180032463 | Olarig et al. | Feb 2018 | A1 |
20180032469 | Olarig et al. | Feb 2018 | A1 |
20180032471 | Olarig | Feb 2018 | A1 |
20180052745 | Marripudi et al. | Feb 2018 | A1 |
20180074717 | Olarig et al. | Mar 2018 | A1 |
20180074984 | Olarig et al. | Mar 2018 | A1 |
20180095904 | Bunker et al. | Apr 2018 | A1 |
20180101492 | Cho et al. | Apr 2018 | A1 |
20180131633 | Li | May 2018 | A1 |
20180173652 | Olarig et al. | Jun 2018 | A1 |
20180210517 | Yun | Jul 2018 | A1 |
20180227369 | DuCray et al. | Aug 2018 | A1 |
20180267925 | Rees | Sep 2018 | A1 |
20180275919 | Chirumamilla et al. | Sep 2018 | A1 |
20180307650 | Kachare et al. | Oct 2018 | A1 |
20180373609 | Beerens | Dec 2018 | A1 |
20190042424 | Nair | Feb 2019 | A1 |
20190087268 | Koltsidas et al. | Mar 2019 | A1 |
20190104632 | Nelson et al. | Apr 2019 | A1 |
20190286584 | Olarig et al. | Sep 2019 | A1 |
20190339888 | Sasidharan et al. | Nov 2019 | A1 |
20200042217 | Roberts et al. | Feb 2020 | A1 |
20200117663 | Moshe et al. | Apr 2020 | A1 |
20200293916 | Li | Sep 2020 | A1 |
20210342281 | Olarig et al. | Nov 2021 | A1 |
20220188002 | Olarig | Jun 2022 | A1 |
20220206693 | Jung et al. | Jun 2022 | A1 |
Number | Date | Country |
---|---|---|
1641568 | Jul 2005 | CN |
101847429 | May 2012 | CN |
104025063 | Sep 2014 | CN |
104202197 | Dec 2014 | CN |
104572516 | Apr 2015 | CN |
104615577 | May 2015 | CN |
105260275 | Jan 2016 | CN |
105912275 | Aug 2016 | CN |
103412769 | Nov 2017 | CN |
2290497 | Feb 2011 | EP |
2843557 | Apr 2015 | EP |
H04257050 | Sep 1992 | JP |
2001290752 | Oct 2001 | JP |
2010146525 | Jul 2010 | JP |
2011048534 | Mar 2011 | JP |
2012506184 | Mar 2012 | JP |
2013041390 | Feb 2013 | JP |
2014241545 | Dec 2014 | JP |
2015049742 | Mar 2015 | JP |
2015191649 | Nov 2015 | JP |
2015194005 | Nov 2015 | JP |
20158532985 | Nov 2015 | JP |
2016037501 | Mar 2016 | JP |
2016045968 | Apr 2016 | JP |
WO2015194005 | Apr 2017 | JP |
20090106469 | Oct 2009 | KR |
20120135205 | Dec 2012 | KR |
20150047785 | May 2015 | KR |
20150071898 | Jun 2015 | KR |
20160074659 | Jun 2016 | KR |
201445325 | Dec 2014 | TW |
2014209764 | Dec 2014 | WO |
2015049742 | Sep 2015 | WO |
2015191649 | Dec 2015 | WO |
2016037501 | Mar 2016 | WO |
2016085016 | Jun 2016 | WO |
Entry |
---|
SSD Form Factor Work Group, Enterprise SSD Form Factor 1.0a, 2012, SSD Form Factor Work Group, pp. 1-55. (Year: 2012). |
“What is the New U.2 SSD Connection?” OC3D News, Jul. 2016, https://www.overstsock3d.nrt/news/storage/what_is_the_u_2_ssd_connection/1. |
Advisory Action for U.S. Appl. No. 15/256,495, dated Feb. 1, 2019. |
Corrected Notice of Allowability for U.S. Appl. No. 15/256,495, dated Mar. 18, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 15/256,495, dated May 13, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Aug. 11, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Aug. 30, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Dec. 10, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Jun. 15, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Oct. 27, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/202,079, dated Jul. 12, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/202,079, dated Jul. 22, 2020. |
Corrected Notice of Allowability for U.S. Appl. No. 16/202,079, dated Sep. 15, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/424,474, dated Aug. 18, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/424,474, dated Feb. 22, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/424,474, dated Mar. 29, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/692,997, dated Jun. 18, 2020. |
Corrected Notice of Allowability for U.S. Appl. No. 16/844,995, dated Dec. 9, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/857,172, dated Aug. 20, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/921,923, dated Apr. 1, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/921,923, dated Aug. 24, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/921,923, dated Jul. 14, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/921,923, dated May 26, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/950,624, dated Jul. 16, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 16/950,624, dated Jun. 10, 2021. |
Fang, Chin, “Using NVMe Gen3 PCIe SSD Cards in High-density Servers for High-performance Big Data Transfer Over Multiple Network Channels”, SLAC National Accelerator Laboratory, Stanford University, Stanford, California, Feb. 7, 2015, 17 pages. |
NVM Express over Fabrics specification Revision 1.0; NVM Express Inc.; Jun. 5, 2016. (Year: 2016). |
OC3D, “What is the New U.2 SSD Connection?,” (https://www.overclock3d.net/news/storage/what_is_the_new_u_2_ssd_connection/1), Jul. 2016, retrieved Apr. 12, 2021, 5 pages. |
Wikipedia, “SATA Express”, (https://en.wikipedia.org/wiki/SATA_Express), retrieved May 2018, 6 pages. |
Wikipedia, “SATA Express,” (https://en.wikipedia.org/wiki/SATA_Express), retrieved Apr. 12, 2021, 6 pages. |
Wikipedia, “U.2”, (https://en.wikipedia.org/wiki/U.2), retreived May 2018, 3 pages. |
Wikipedia, “U.2,” (https://en.wikipedia.org/wiki/U.2), retrieved Apr. 12, 2021, 2 pages. |
Notice of Allowance for U.S. Appl. No. 15/345,509, dated Feb. 8, 2022. |
Notice of Allowance for U.S. Appl. No. 16/202,079, dated Mar. 17, 2022. |
Notice of Allowance for U.S. Appl. No. 17/099,776, dated Feb. 15, 2022. |
Office Action for U.S. Appl. No. 16/844,995, dated Feb. 22, 2022. |
Office Action for U.S. Appl. No. 17/063,507, dated Mar. 17, 2022. |
Final Office Action for U.S. Appl. No. 15/256,495, dated Dec. 4, 2019. |
Final Office Action for U.S. Appl. No. 15/256,495, dated Oct. 19, 2018. |
Final Office Action for U.S. Appl. No. 15/345,509, dated Feb. 21, 2019. |
Final Office Action for U.S. Appl. No. 15/411,962, dated Dec. 20, 2018. |
Final Office Action for U.S. Appl. No. 16/211,923, dated Aug. 19, 2019. |
Final Office Action for U.S. Appl. No. 16/424,474, dated May 1, 2020. |
Final Office Action for U.S. Appl. No. 16/692,997, dated Mar. 26, 2020. |
Final Office Action for U.S. Appl. No. 16/844,995, dated Mar. 29, 2021. |
Final Office Action for U.S. Appl. No. 17/063,507, dated Nov. 22, 2021. |
Notice of Allowance for U.S. Appl. No. 15/256,495, dated Mar. 5, 2020. |
Notice of Allowance for U.S. Appl. No. 15/345,507, dated Feb. 19, 2019. |
Notice of Allowance for U.S. Appl. No. 15/345,509, dated May 13, 2021. |
Notice of Allowance for U.S. Appl. No. 15/403,088, dated Oct. 22, 2018. |
Notice of Allowance for U.S. Appl. No. 15/411,962, dated Mar. 18, 2019. |
Notice of Allowance for U.S. Appl. No. 16/202,079, dated Jan. 27, 2021. |
Notice of Allowance for U.S. Appl. No. 16/202,079, dated Jun. 1, 2020. |
Notice of Allowance for U.S. Appl. No. 16/202,079, dated May 14, 2021. |
Notice of Allowance for U.S. Appl. No. 16/211,923, dated Sep. 13, 2019. |
Notice of Allowance for U.S. Appl. No. 16/421,458, dated Apr. 15, 2020. |
Notice of Allowance for U.S. Appl. No. 16/424,474, dated Apr. 30, 2021. |
Notice of Allowance for U.S. Appl. No. 16/424,474, dated Jul. 15, 2020. |
Notice of Allowance for U.S. Appl. No. 16/692,997, dated Jun. 1, 2020. |
Notice of Allowance for U.S. Appl. No. 16/844,995, dated Sep. 29, 2021. |
Notice of Allowance for U.S. Appl. No. 16/857,172, dated May 3, 2021. |
Notice of Allowance for U.S. Appl. No. 16/921,923, dated Feb. 18, 2021. |
Notice of Allowance for U.S. Appl. No. 16/950,624, dated May 10, 2021. |
Office Action for U.S. Appl. No. 15/256,495, dated Jun. 14, 2019. |
Office Action for U.S. Appl. No. 15/256,495, dated Mar. 29, 2018. |
Office Action for U.S. Appl. No. 15/345,507, dated Dec. 3, 2018. |
Office Action for U.S. Appl. No. 15/345,509, dated Apr. 29, 2020. |
Office Action for U.S. Appl. No. 15/345,509, dated Nov. 29, 2019. |
Office Action for U.S. Appl. No. 15/345,509, dated Sep. 10, 2018. |
Office Action for U.S. Appl. No. 15/345,509, dated Sep. 28, 2020. |
Office Action for U.S. Appl. No. 15/403,088, dated Jun. 7, 2018. |
Office Action for U.S. Appl. No. 15/411,962, dated Aug. 10, 2018. |
Office Action for U.S. Appl. No. 16/202,079, dated Aug. 22, 2019. |
Office Action for U.S. Appl. No. 16/202,079, dated Dec. 9, 2021. |
Office Action for U.S. Appl. No. 16/202,079, dated Mar. 4, 2020. |
Office Action for U.S. Appl. No. 16/211,923, dated May 6, 2019. |
Office Action for U.S. Appl. No. 16/421,458, dated Dec. 30, 2019. |
Office Action for U.S. Appl. No. 16/424,474, dated Feb. 3, 2020. |
Office Action for U.S. Appl. No. 16/424,474, dated Oct. 15, 2019. |
Office Action for U.S. Appl. No. 16/692,997, dated Dec. 19, 2019. |
Office Action for U.S. Appl. No. 16/844,995, dated Sep. 4, 2020. |
Office Action for U.S. Appl. No. 16/857,172, dated Oct. 8, 2020. |
Office Action for U.S. Appl. No. 16/921,923, dated Oct. 28, 2020. |
Office Action for U.S. Appl. No. 16/950,624, dated Jan. 25, 2021. |
Office Action for U.S. Appl. No. 17/022,075, dated Oct. 15, 2021. |
Office Action for U.S. Appl. No. 17/063,507, dated Aug. 6, 2021. |
Office Action for U.S. Appl. No. 17/099,776, dated Sep. 24, 2021. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Jun. 10, 2022. |
Final Office Action for U.S. Appl. No. 17/022,075, dated May 25, 2022. |
Supplemental Notice of Allowability for U.S. Appl. No. 17/099,776, dated May 16, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Jun. 27, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 16/202,079, dated Jul. 7, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 17/099,776, dated Jun. 20, 2022. |
Final Office Action for U.S. Appl. No. 17/063,507, dated Jun. 27, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Aug. 8, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 16/202,079, dated Aug. 17, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 17/099,776, dated Jul. 21, 2022. |
Office Action for U.S. Appl. No. 17/408,365, dated Aug. 2, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 15/345,509, dated Sep. 7, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 16/202,079, dated Aug. 31, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 17/022,075, dated Sep. 6, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 17/099,776, dated Aug. 24, 2022. |
Notice of Allowance for U.S. Appl. No. 16/844,995, dated Aug. 26, 2022. |
Notice of Allowance for U.S. Appl. No. 17/022,075, dated Aug. 23, 2022. |
Notice of Allowance for U.S. Appl. No. 17/063,507, dated Oct. 7, 2022. |
Office Action for U.S. Appl. No. 17/376,145, dated Sep. 30, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 16/844,995, dated Oct. 20, 2022. |
Corrected Notice of Allowability for U.S. Appl. No. 17/099,776, dated Oct. 19, 2022. |
Office Action for U.S. Appl. No. 17/230,989, dated Oct. 31, 2022. |
Number | Date | Country | |
---|---|---|---|
20210019272 A1 | Jan 2021 | US |
Number | Date | Country | |
---|---|---|---|
62420355 | Nov 2016 | US | |
62366622 | Jul 2016 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16211923 | Dec 2018 | US |
Child | 16692997 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16692997 | Nov 2019 | US |
Child | 17063501 | US | |
Parent | 15403088 | Jan 2017 | US |
Child | 16211923 | US |