Patent Grant
11256627
Embodiments described herein relate generally to computer systems, and more specifically, to techniques for implementing a buffer cache for a persistent file system in non-volatile memory.
Database systems are typically stored on persistent disk media. Disk I/O latency tends to be a performance bottleneck for on-disk database systems. As a result, on-disk database systems tend to be designed to minimize the effect of I/O latency. Such database systems do not address how to take advantage of non-volatile memory when the entire database fits in non-volatile memory.
A database may be implemented entirely in volatile memory, such as dynamic random-access memory (DRAM). However, because volatile memory is not persistent, it cannot guarantee durability of the data. Thus, such database systems are not suitable for running a database independently of persistent storage.
Byte-addressable non-volatile memory, such as non-volatile random-access memory (NVRAM). NVRAM is random-access memory that retains stored information, even when power is turned off. The latency for this new class of non-volatile memory is expected to be slightly slower than, but within the same order of magnitude of DRAM.
On a disk-based storage system, a buffer cache is typically used for disk buffering. In disk buffering, a buffer cache stored in DRAM is used to mediate data transfer. Accessing the buffer cache in DRAM is faster than reading from disk. Typically, when an I/O request is issued, the operating system searches the buffer cache for the requested data. If the data is in the buffer cache, the I/O request is satisfied without accessing the disk. If the data is not in the buffer cache, the data is copied from disk to the buffer cache on a block by block basis. The I/O operations are then performed on the data in the buffer cache rather than on the disk. Typically, a write operation is initially performed on copies of data blocks in the buffer cache. When the write operation is successfully performed on the data blocks in the buffer cache, the modified data blocks are copied to disk, thereby completing the write operation.
A database system that stores at least a portion of the database in DRAM (a “DRAM-enabled database system”) cannot be directly adapted to store the same data in NVRAM due to fundamental differences in the behavior of DRAM and NVRAM. For example, because DRAM-enabled database systems cannot rely on volatile memory for persistence, these database systems maintain, in persistent storage, a copy of the database, or data usable to construct a persistent copy of the database. If a process in a DRAM-enabled database system makes changes to a database stored in DRAM, the volatile memory can be left in an unpredictable state if the process crashes before the changes are complete. However, because of the persistently stored data, the DRAM-enabled database system can recover from this crash without affecting the atomicity, consistency, isolation and durability (ACID) properties of the database.
If the DRAM and the persistent storage in a DRAM-enabled database system is directly replaced with NVRAM, this scenario would cause a failure. More specifically, if the database were stored completely in NVRAM, when a process that directly operates on the database crashes before changes to the database are complete, the database would be left in an unpredictable state.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
In the drawings:
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, that the embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring embodiments.
Techniques are described herein for implementing a persistent file system in non-volatile memory. Buffer headers stored in DRAM are used to implement access control to data that is persistently stored in NVRAM. The buffer headers store mappings that refer to one or more extents and/or data blocks. When a buffer header is associated with a process, the process is granted access to the referenced data.
Direct mappings reference extents and/or data blocks in NVRAM. When a buffer header stores a direct mapping that includes a reference to an extent in NVRAM, a process that is associated with the buffer header can access the referenced extent in NVRAM.
DRAM-copy mappings reference copies of the extents and/or data blocks in DRAM. When a buffer header stores a DRAM-copy mapping that includes a reference to a copy of data from the extent in DRAM, a process that is associated with the buffer header can access the copy of the extent, which is stored in DRAM. In some embodiments, the extent is copied to a data buffer in DRAM, and the DRAM-copy mapping includes a reference to the data buffer.
The techniques described herein may be used to implement a file system that is persistently stored in NVRAM, and/or a database system comprising a database that is persistently stored in NVRAM. The entire database and other persistent data related to the database may be stored in NVRAM. Furthermore, the techniques described herein may be used in distributed systems, such as to implement a distributed file system and/or a distributed database system.
In a database system, these techniques leverage database mechanisms to support the ACID properties of the database, while taking advantage of non-volatile memory to avoid I/O and to minimize the duplication of data in DRAM. When a database process reads data directly from one or more data blocks in NVRAM, the database system avoids the overhead associated with an I/O operation.
File System
The techniques described herein may be used to implement a file system in non-volatile memory.
As used herein, the term “file system” refers to a combination of software components and/or hardware components for storing and retrieving data in a data file. For example, the term may refer to the hierarchical structure of stored data, the hardware that stores the data, the software application that is used to carry out the scheme, the architecture of the software and/or the hardware, and any combination thereof. In some embodiments, the file system includes software that makes up a portion of an operating system for a computer system, such as computer system 100.
The computer system 100 includes DRAM 102, and NVRAM 104. The computer system 100 persistently stores one or more files in file system 106 in NVRAM 104. The NVRAM 104 includes a plurality of extents. Each file is stored on one or more extents in NVRAM 104. As used herein, the term DRAM refers to any form of volatile random access memory, and the term NVRAM refers to any form of non-volatile random access memory.
As used herein, the term “extent” refers to a set of contiguous data blocks in NVRAM. In some embodiments, data blocks in an extent are logically contiguous (i.e. occupy contiguous addresses within address space) but can be physically spread out on disk, such as due to RAID striping, file system implementations, or other factors. The computer system 100 may be set up such that each extent may include the same number of data blocks or a varying number of data blocks. A file may be stored on zero extents, one extent, or multiple extents. A file may be stored on multiple non-contiguous extents.
In some embodiments, the computer system 100 provides direct access and DRAM-copy access to the files in file system 106. As used herein, the term “direct access” refers to access to the data blocks in NVRAM 104. As used herein, the term “DRAM-copy access” refers to access to a copy, stored in DRAM 102, of data blocks in NVRAM 104. Thus, depending on the type of access granted (e.g. read, write, or other permissions), a process that is granted direct access can perform the type of access on the NVRAM directly, while a process that is granted DRAM-copy access cannot. Direct access and DRAM-copy access shall be described in greater detail hereafter.
Buffer Cache
In traditional disk-based database systems, a buffer cache is used to hold data blocks, or portions thereof, that are read from data files on disk. Instead of implementing a file system 106 comprising data stored on disk, computer system 100 implements a file system 106 comprising data stored in NVRAM 104.
Computer system 100 implements a buffer cache 110 that facilitates I/O operations for data stored in NVRAM 104. The buffer cache 110 includes buffer headers 112 and data buffers 114. The buffer headers 112 are used to store mappings that provide access to one or more extents and/or data blocks in NVRAM 104, or one or more copies thereof in DRAM 102. In some embodiments, when a DRAM copy, of one or more extents and/or data blocks in NVRAM 104, is generated, the DRAM copy is stored in data buffers 114.
The buffer headers are allocated in DRAM 102 of the computer system 100. The computer system 100 grants a process access to the file system 106 by storing metadata in a buffer header and associating a buffer header with the process. For example, buffer header H1 is associated with process P1 and buffer header H2 is associated with process P2, thereby granting access to a specified portion of the file system 106. In some embodiments, the buffer headers 112 are pre-allocated in DRAM 102, and access is granted by associating an existing, unused buffer header with the process to which access is granted.
In some embodiments, the computer system 100 grants access to the file system 106 by storing a mapping in the buffer header. When a process is granted direct access to an extent in NVRAM 104, a mapping to the extent in NVRAM 104 is stored in a buffer header that is associated with the process. When a process is granted DRAM-copy access to an extent, a DRAM-copy mapping to a copy of the extent in DRAM 102 is stored in a buffer header that is associated with the process. Direct mappings and DRAM-copy mappings shall be described in greater detail hereafter.
Direct Access and Direct Mapping
As used herein, the term “direct access” refers to access (e.g. read, write, or other types of access) to one or more data blocks in an extent in NVRAM 104 that stores the corresponding data. Direct access of one or more extents in NVRAM 104, one or more data blocks in NVRAM, or a portion thereof, may be granted to one or more processes 108 executing in DRAM 102.
Access of the file system 106 stored in NVRAM 104, including direct access, may be managed by an operating system of the computer system 100, a file system application executing on the computer system 100, other software executing on the computer system 100, one or more hardware components of the computer system 100 interacting with such software and/or any combination thereof. When the computer system 100 grants direct access to the file system 106, the mechanism to grant direct access may be implemented in software and/or hardware, which may include an operating system, a file system, other software, and/or hardware operating in the computer system 100.
To grant a particular process direct access to data in the file system 106 stored in NVRAM 104, the computer system 100 stores a direct mapping 150 in a particular buffer header of the buffer cache 110, and the particular buffer header is associated with the particular process. The direct mapping 150 includes a reference to the portion of NVRAM 104 to be accessed, such as one or more extents, one or more data blocks, or a portion thereof. For example, the direct mapping 150 may be a pointer into persistent memory (e.g. NVRAM 104). The direct mapping 150 may be a reference to a particular block within an extent. For example, the direct mapping 150 may include a reference to the extent with an offset. In some embodiments, the direct mapping 150 includes a memory address in NVRAM 104 and/or additional information, such as authentication information, permissions, other security information, an offset within the extent, a length, or any other information usable to locate data within the NVRAM 104 of the computer system 100.
Dram-Copy Access and Dram-Copy Mapping
As used herein, the term “DRAM-copy access” refers to access (e.g. read, write, or other types of access) to a copy of one or more data blocks in an extent in NVRAM 104, where the copy is stored in DRAM 102. Thus, although the corresponding data is persistently stored in NVRAM 104, the process is not granted access to the NVRAM 104 itself. DRAM-copy access of one or more extents in NVRAM 104, one or more data blocks in NVRAM, or a portion thereof, may be granted to one or more processes 108 executing in DRAM 102.
DRAM-copy access may be managed by an operating system of the computer system 100, a file system application executing on the computer system 100, or other software executing on the computer system 100. When the computer system 100 grants DRAM-copy access to the file system 106, the mechanism to grant direct access may be implemented in software and/or hardware, which may include an operating system, a file system, or other software operating on the computer system 100.
To grant a process DRAM-copy access to data in the file system 106 stored in NVRAM 104, the computer system 100 stores a DRAM-copy mapping 152 in a particular buffer header of the buffer cache 110, and the buffer header is associated with the process to which DRAM-copy access is granted. The DRAM-copy mapping 152 includes a reference to a portion of DRAM 102 that stores a copy of the data to be accessed. For example, the DRAM-copy mapping 152 may be a pointer into volatile memory (e.g. DRAM 102). In some embodiments, the DRAM-copy mapping 152 may include a memory address in DRAM 102 and/or additional information, such as authentication information, permissions, other security information, a memory offset, a length, or any other information usable to locate data within the DRAM 102 of the computer system 100.
In some embodiments, the DRAM-copy of the corresponding data may be maintained in the buffer cache 110. For example, one or more data buffers 114 may be used to store the DRAM-copy of the corresponding data. Although the DRAM-copy of the corresponding data is described in some embodiments as a copy of an extent, other units of data may be stored in one or more data buffers 114, such as a portion of an extent, one or more data blocks, a portion thereof, or another unit of data other than entire extents. In some embodiments, the data buffers 114 of the buffer cache 110 in DRAM 102 includes a pool of pre-allocated data buffers 114 that are sized based on the size of an extent in NVRAM 104.
Permissions
In some embodiments, the computer system 100 may control access by granting permissions. As used herein, the term “permission” refers to authorization that enables the grantee to access specific resources in a particular manner. Permissions may include a type of access, such as open, read, write, delete, modify, execute, or special permissions, including customized permissions. The computer system 100 may also control the type of access, such as direct access and DRAM-copy access. As used herein, the term “read” refers to access, permissions, and/or operations that cannot modify the corresponding data. As used herein, the term “write” refers to access, permissions, and/or operations that can modify the corresponding data.
In some embodiments, the computer system 100 selectively grants direct read access to data stored in NVRAM 104 and DRAM-copy write access to copies in DRAM 102 of the data stored in NVRAM 104. For example, when one or more processes request read access to data stored in NVRAM 104, the computer system 100 may grant direct read access to one or more of the processes, thereby allowing the processes to directly read data from the extent/s in NVRAM 104. When one or more processes request write access to data stored in NVRAM 104, the computer system 100 may grant DRAM-copy write access to one or more of the processes, thereby allowing the processes to access and modify a DRAM copy of the requested data in DRAM 102. Direct read access and DRAM-copy write access shall be described in greater detail hereinafter.
In some embodiments, the computer system 100 uses the direct mapping and DRAM-copy mapping mechanism described herein to implement other permissions and/or additional permissions to control access. The computer system 100 may also detect circumstances where permissions other than direct read access and DRAM-copy write access are appropriate. Examples of such circumstances shall be described in greater detail hereafter.
Direct Read Access
An example is provided of direct access implementation of data stored in the file system 106 in NVRAM 104. In response to a request from process P1 to access one or more blocks of extent E1, the computer system 100 populates buffer header H1 with a direct mapping 150. The direct mapping 150 includes a reference to the corresponding data, such as a reference to extent E1, a corresponding memory address in NVRAM 104, and/or other metadata that identifies the corresponding data in NVRAM 104. The computer system 100 then associates the buffer header H1 with process P1. Once process P1 is associated with buffer header H1, process P1 can directly access data stored in one or more blocks of extent E1 in NVRAM 104 using the direct mapping 150 stored in buffer header H1.
Dram-Copy Write Access
An example is provided for DRAM-copy access implementation of data stored the file system 106 in NVRAM. Buffer header H2 includes a DRAM-copy mapping 152 usable to access a DRAM-copy of one or more data blocks of extent E4. The DRAM-copy of extent E4, COPY(E4), is stored in a data buffer 114 in the buffer cache 110. In response to a request from process P2 to access one or more blocks of extent E4, the computer system 100 determines whether the copy COPY(E4) is already maintained in a data buffer 114 of the buffer cache 110. If no copy of extent E4 is maintained in the buffer cache 110, then the computer system 100 populates a data buffer 114 with a copy of extent E4. Then, the computer system 100 populates a buffer header H2 with a DRAM-copy mapping 152. The DRAM-copy mapping 152 includes a reference to the copy of the corresponding data, such as a reference to the data buffer 114 containing COPY(E4). The computer system 100 then associates the buffer header H2 with process P2. Once process P2 is associated with buffer header H2, process P2 can access the copy COPY(E4) of extent E4 stored in data buffers 114 in DRAM 102 using the DRAM-copy mapping 152 stored in buffer header H1.
Changed data, such as data blocks and/or extents, are written back to NVRAM 104 using standard I/O procedures. In some embodiments, the I/O layer of the computer system 100 performs the write from the data buffer 114 to NVRAM 104. The I/O layer may be configured to avoid fractured blocks, torn blocks, and/or other errors.
Reference Count
In some embodiments, the computer system 100 maintains a reference count for one or more extents and/or data blocks for which direct access is currently granted. The reference count may be maintained in DRAM 102, such as in the buffer cache 110. Although the given examples involve reference counts maintained for extents, reference counts may be maintained for a different unit of data stored in NVRAM.
For a particular extent, the reference count indicates the number of references to the extent in NVRAM are currently active. That is, the reference count for a particular extent indicates the number of direct mappings to the extent in the buffer headers 112. The computer system 100 increments the reference count for the particular extent. In some embodiments, the reference count is incremented and decremented by the process to which direct access is granted. For example, process P1 may increment the reference count for extent E1 in the buffer cache 110 in response to being granted direct access to data in extent E1. When process P1 finishes accessing the data, the process decrements the reference count for extent E1.
Reference Count Cleanup
If a process crashes after obtaining direct access to an extent in NVRAM 104 but before releasing access and decrementing the reference count for the extent, another process may be configured to decrement reference counts on behalf of the dead process. In some embodiments, the reference count is decremented by a second process configured to perform other clean up functions after a first process crashes. For example, when a process obtains direct access to an extent via a particular buffer header, the process may store, in a shared memory area of DRAM 102, an indication that it is using the particular buffer header. If the process crashes, a cleanup process checks the shared memory area in DRAM 102 to determine whether the crashed process had direct access to any extents, and decrements the corresponding reference counts in DRAM 102.
The reference count is usable to prevent concurrency issues. In an embodiment, while some processes have been granted direct read access to a NVRAM block, no other process can write a DRAM copy data block buffer to the same NVRAM block. This is because otherwise the processes granted direct read access may read corrupt data. In one embodiment, a database may delay issuing such a write until the reference count for the NVRAM block becomes zero.
Database Implementation
The techniques described herein may be used to implement a database system for a data base stored entirely in non-volatile memory.
Generally, a server, such as a database server, is a combination of integrated software components and an allocation of computational resources, such as memory and processes on a computer for executing the integrated software components. The combination of the software and computational resources are dedicated to providing a particular type of functionality to clients of the server.
A database server 204 runs on the database system 200. The database server 204 includes one or more processes that perform database operations, such as P1 and P2. The processes P1 and P2 are executed by the database system 200 and DRAM 202. The database server 204 manages client access of the database, including data stored in data files 208. For example, client applications may interact with the database server 204 by submitting database commands to the database server 204 that cause the database server 204 to perform database operations on data stored in a database, which is persistently stored in data files 208 in NVRAM.
Database Access Control
The database system 200 controls access to the data files 208 in NVRAM 206. Access control may be performed by the database server 204, an operating system of the database system 200, another file system application executing on the database system 200, other software executing on the database system 200, or any combination thereof. In some embodiments, the database system 200 may control access by granting permissions, such as open, read, write, delete, modify, execute, special permissions, and/or customized permissions. The database system 200 may also control the type of access, such as direct access and DRAM-copy access.
To grant a process direct access to data files 208 stored in NVRAM 206, the database system 200 stores a direct mapping 250 in a particular buffer header of the buffer headers 212 in buffer cache 210, and the particular buffer header is associated with the process to which direct access is granted. The direct mapping 250 includes a reference to the portion of NVRAM 206 to be accessed, such as one or more extents, one or more data blocks, or a portion thereof. For example, buffer header DH1 includes a direct mapping 250 associated with database server process DP1. In response to a request from database server process DP1 to access one or more blocks of extent DE1, the database system 200 populates buffer header DH1 with a direct mapping 250. The direct mapping 250 includes a reference to the corresponding data, such as a reference to extent DE1, a corresponding memory address in NVRAM 206, and/or other metadata that identifies the corresponding data in NVRAM 206. The database system 200 then associates the buffer header DH1 with process DP1. Once process DP1 is associated with buffer header DH1, process DP1 can directly access data stored in one or more blocks of extent DE1 in NVRAM 206 using the direct mapping 250.
To grant a process DRAM-copy access to data in the data files 208 stored in NVRAM 206, the database system 200 stores a DRAM-copy mapping 252 in a particular buffer header of the buffer cache 210, and the particular buffer header is associated with the process to which DRAM-copy access is granted. The DRAM-copy mapping 252 includes a reference to a portion of DRAM 202 that stores a copy of the data to be accessed. In some embodiments, the DRAM-copy of the corresponding data is maintained in the buffer cache 210, such as in one or more data buffers 214. For example, buffer header DH2 includes a DRAM-copy mapping 252 associated with database server process DP2. In response to a request from database server process DP2 to access one or more blocks of extent DE4, the database system 200 determines whether a copy COPY(DE4) of the one or more blocks of extent DE4 is already maintained in buffer cache 210. If no copy is maintained in the buffer cache 210, the database system 200 populates a data buffer 214 with a copy of at least the one or more blocks of extent DE4. Then, the database system 200 populates a buffer header DH2 with a DRAM-copy mapping 252. The DRAM-copy mapping 252 includes a reference to the copy of the corresponding data (e.g. the data buffer 214 containing COPY(DE4)). The database system 200 then associates the buffer header DH2 with the database server process DP2. Once database server process DP2 is associated with buffer header DH2, database server process DP2 can access the copy COPY(E4) of the one or more blocks of extent DE4 stored in DRAM 202 using the DRAM-copy mapping 252.
In some embodiments, the database system 200 selectively grants direct read access to the data files 208 stored in NVRAM 206 and DRAM-copy write access to copies in DRAM 202 of the data stored in NVRAM 206. When one or more database server processes request read access to the data files 208, the database system 200 may grant direct read access to one or more of the processes, thereby allowing the processes to directly read from the extent/s in NVRAM 206. When one or more database server processes request write access to the data files 208 stored in NVRAM 206, the database system 200 may grant DRAM-copy write access to one or more of the processes, thereby allowing the processes to access and modify a DRAM copy of the requested data in DRAM 202.
In some embodiments, the database system 200 uses the direct mapping mechanism and DRAM-copy mapping mechanism described herein to implement other permissions and/or additional permissions to control access. The database system 200 may also detect circumstances where permissions other than direct read access and DRAM-copy write access are appropriate.
Database Change Records
In some embodiments, the database system 200 persistently stores change records in NVRAM 206, such as in change logs 220. Change records can be used to undo changes made to the database. For example, if a change to a database needs to be undone, such as when a transaction is not committed, one or more change records may be processed to determine the necessary steps to undo the change described in the change record. Likewise, a change record may be used to reconstruct changes made to the database. For example, if a data file needs to be restored, a backup of the data file can be loaded, and one or more change records may be processed to redo changes made to the database since the backup. Change records are stored in NVRAM 206 to ensure their availability to the database server 204, since the database server 204 uses the change records to ensure the ACID properties in the database.
As changes are made to the database, the change records are generated and persistently stored in change logs 220 in NVRAM 206. For example, when a process of database server 204 makes a change to the database, a change record is stored in a change log 220. The change record may specify one or more data block(s) of the database being modified and respective change vectors that describe the changes made to the data block. In some embodiments, change records are generated as soon as a change is made to a DRAM copy, whether or not the changed data has been copied to NVRAM 206. The change records may include a logical timestamp. As used herein, the term “logical timestamp” includes any data usable to uniquely identify an order between any two logical timestamps. A logical timestamp may be based on an actual time, an order, or any other data usable to indicate an order.
In some embodiments, the database server 204 uses the change logs 220 to construct a “consistent read” copy of one or more data blocks. A consistent read copy of a data block reflects the state of the data block at a particular point, such as at a particular logical timestamp. For example, a consistent read copy of a data block at a specific timestamp is often used when responding to a query with the specific timestamp. In these cases, the data block is copied into DRAM 202, such as into a data buffer 214. The change records are then read from the change logs 220 and applied to the DRAM copy of the data block in the data buffer 214.
Distributed Volume Implementation
The techniques described herein may be used to implement a distributed file system, and/or a distributed volume of data stored in non-volatile memory of a plurality of computing devices.
Each node 302-304 of the distributed system 300 is a computing system that includes its own resources, including DRAM 306-308 and NVRAM 330-332. A distributed file system 350 storing a distributed volume of data is stored across the NVRAM 330-332 of the plurality of nodes 302-304. Each node 302-304 stores a portion of the distributed volume of data, of the distributed file system 350, as a data set 334-336 in its respective NVRAM 330-332.
Access of the distributed file system 350 may be managed by one or more operating systems of one or more of the nodes 302-304, one or more file system applications executing on one or more of the nodes 302-304, other software executing on one or more of the nodes 302-304, one or more hardware components of one or more of the nodes 302-304 interacting with such software, and/or any combination thereof. When the distributed system 300 grants access, such as direct access and/or DRAM copy access of data stored in the distributed file system 350, the mechanism to grant direct access may be implemented in software and/or hardware, which may include an operating system, a file system, other software and/or hardware operating in one or more of the nodes 302-304.
For each node 302-304, access control may be performed by the operating system of the node 302-304, another file system application executing on the node 302-304, other software executing on the node 302-304, or any combination thereof. In some embodiments, access control is performed by a distributed volume manager 310-312 that executes on the nodes 302-304. In some embodiments, a node 302-304 stores access control data for local processes, including access of data that is local and access that is remote.
Local and Remote Access Control
The distributed system 300 may selectively grant permissions, such as direct access and DRAM-copy access, to processes based on the type of access requested and based on whether the requesting process is requesting access to data stored in NVRAM 330-332 of the same node 302-304 or NVRAM 330-332 of another node 302-304. As used herein, the term “remote” refers to a node other than a current node, such as a current node on which a requesting process runs, and/or the resources of such a node. As used herein, the term “local” refers to resources the current node, such the node on which a requesting process runs, and/or the resources of such a node.
In some embodiments, the distributed system 300 is configured to grant direct read access to processes that request read access to data in the distributed file system 350 that is stored in one or more extents that are local to the requesting process. For example, process A.P1 of other processes 314 is granted direct read access to data stored in extent A.E1. In this case, the requested data is in data set 334, which is local to process A.P1. That is, process A.P1 executes on node A 302, and the extent that stores the requested data, A.E1, is an extent in the NVRAM 330 of node A 302, which is the same node where process A.P1 executes. Node A 302 grants direct read access to process A.P1 by associating a buffer header A.H1, of buffer cache headers 322 in buffer cache 318, with process A.P1 and storing a direct mapping 360 in the buffer header A.H1. The direct mapping 360 includes a reference to extent A.E1 in NVRAM 330. Process A.P1 directly accesses extent A.E1 in NVRAM 330 of node A 302 by using the direct mapping 360 stored in the buffer header A.H1 associated with it.
In some embodiments, the distributed system 300 is configured to grant DRAM-copy write access to processes that request write access to data in the distributed file system 350 that is stored in one or more extents that are local to the requesting process. For example, process A.P2 is granted DRAM-copy write access to data stored in extent A.E4. In this case, the requested data is in a local data set 334 with respect to process A.P2. However, even though extent A.E4 is local to process A.P2, process A.P2 is granted DRAM-copy access because the requested access is write access. Node A 302 grants DRAM-copy write access to process A.P2 by associating a buffer header A.H2 with process A.P2 and storing a DRAM-copy mapping 362 in the buffer header A.H2. The DRAM-copy mapping 362 includes a reference to a data buffer 326 stored in the local DRAM 306 that stores a COPY(A.E4) of the extent A.E4. Process A.P2 accesses a DRAM copy of extent A.E4 in DRAM 306 of node A 302 by using the DRAM-copy mapping 362 stored in the buffer header A.H2 associated with the process A.P2.
In some embodiments, the distributed system 300 is configured to grant DRAM-copy write access and DRAM-copy read access to processes that request access to data in the distributed file system 350 that is stored in one or more extents that are remote to the requesting process. For example, when a process N.P1 of other processes 316 executing on node N 304 requests access to data stored in a remote extent A.E5 in NVRAM 330 of node A, the distributed system 300 grants DRAM-copy access to process N.P1 by associating a buffer header N.H, of buffer cache headers 324 in buffer cache 320, with process N.P1 and storing a DRAM-copy mapping 364 in the buffer header N.H1. The DRAM-copy mapping 364 includes a reference to a data buffer 328 in the local DRAM 308 that stores a COPY(A.E5) of the extent A.E5 in NVRAM 330. The copy COPY(A.E5) may be generated by communicating with node A 302. Process N.P1 accesses the DRAM copy COPY(A.E5) in DRAM 308 of node N 304 by using the DRAM-copy mapping 364 stored in the buffer header N.H1 associated with the process N.P1.
In some embodiments, the distributed volume manager 310-312 may grant a process remote direct read access of NVRAM extents in a node that is remote to the process, such as via remote direct memory access (RDMA). RDMA allows direct access of remote memory without involving the operating system of either the local node or the remote node.
Any component of the distributed system 300 may make the determination of whether data requested by a particular process is local or remote with respect to the node on which the process executes. For example, when a process executing on a particular node requests particular data, a determination that the requested data is local or remote (i.e. resides in a local extent on the particular node or a remote extent of another remote) can be made by one or more components of the distributed system, such as the distributed volume manager 310-312, another component of the particular node, another component of another node, the particular process, another process, or any other component of the distributed system 300.
Distributed Volume Rebalancing
In some embodiments, the distributed volume manager 310-312 performs load rebalancing functions for the distributed file system 350. As used herein, the term “load balancing” refers to distributing and/or redistributing data between the nodes of a distributed system.
Distributed volume manager 310-312 may be configured to perform rebalancing by moving data stored in one or more extents from the NVRAM of one node to one or more extents the NVRAM of another node. Rebalancing may be performed when one or more nodes are added or removed from the distributed system 300, when one or more nodes fail, or to redistribute workload that accesses particular data on particular nodes, or in other situations where rebalancing of data is appropriate in a distributed file system.
In some embodiments, an extent is only moved if no processes currently have permission to directly access the extent. For example, when the distributed volume manager 310-312 determines that data stored in a particular extent of a local node should be moved to a remote node, the distributed volume manager 310-312 checks that no local processes still have permission to directly access the extent at the local node. In some embodiments, the distributed volume manager 310-312 waits until the reference count maintained for the extent at the local node is decremented to zero, indicating that no more local processes will potentially access the extent at the local node.
In some embodiments, when it is determined that data stored in a particular extent should be moved to a remote node, the distributed system 300 prevents additional direct access permissions from being granted for the particular extent. Instead, the access must be performed via other I/O techniques, such as via DRAM-copy access, even if the permission requested is read access to the particular extent from a local process. This prevents further incrementing of the reference count for the particular node.
Distributed Database Implementation
The techniques described herein may be used to implement a distributed database system for a database stored entirely in non-volatile memory of a plurality of computing devices. For example, the distributed system 300 may be a distributed database system, and the nodes 302-304 of the distributed system 300 may be database nodes 302-304. In one embodiment, the nodes 302-304 of the distributed system 300 are database instances, such as Oracle's shared-disk Real Application Clusters (RAC) instances.
The distributed file system 350 of the distributed system 300 can be used to implement shared storage to simulate a shared-disk database system. For example, the distributed volume manager 310-312 can simulate shared storage by forwarding database block access requests and/or by allowing direct remote block accesses via RDMA. The distributed volume manager 310-312 may also rebalance data files among nodes and may implement data file mirroring for the distributed database system. Mirroring may be used to increase the number of nodes that locally store a particular set of data in local NVRAM, and may thereby increase the availability of direct access to the particular set of data.
The data files of the distributed database system are stored across the NVRAM 330-332 of the nodes 302-304. In some embodiments, a database process executing on a particular node may be granted direct read access to read directly from the NVRAM of the particular node. The database process may also be granted DRAM-copy write access to access, at the particular node, a DRAM copy of data stored in the NVRAM of the particular node. The database process may also be granted DRAM-copy write access and/or DRAM copy read access to access a DRAM copy, at the particular node, of data stored in the NVRAM of a different node.
In some embodiments, change log files of the database are stored across the NVRAM 330-332 of the nodes 302-304 of the distributed system 300. Change log files may be shared between one or more database nodes. Alternatively and/or in addition, a database node may maintain one or more dedicated change log files corresponding to local changes.
At block 402, a set of data is maintained in NVRAM of one or more computer systems. The set of data is stored persistently in NVRAM.
At block 404, one or more buffer headers are allocated in DRAM. In some embodiments, a pool of buffer headers are pre-allocated.
At block 406, an access request to access one or more extents is received from a requesting process. In some embodiments, process 400 is performed by a computing device on which the requesting process is executing.
At decision block 408, it is determined whether the one or more extents are local. If it is determined that the one or more extents are local, processing proceeds to block 410. If it is determined that the one or more extents not local, processing proceeds to block 414. When the process is performed in a single-computer system, then the one or more extents are always local with respect to the process.
At decision block 410, it is determined whether the request for local data is a read request. If it is determined that the request for local data is a read request, processing proceeds to block 412. If it is determined that the request for local data is not a read request, processing proceeds to block 414.
At block 412, direct read access to the one or more extents is granted to the process. For example, one or more buffer headers may be associated with the process and populated with one or more references to the one or more extents.
At block 414, DRAM-copy access to the one or more extents is granted to the process. For example, one or more buffer headers may be associated with the process and populated with one or more references to data buffers in DRAM that store a DRAM copy of the data stored in the one or more extents in NVRAM.
At block 416, process 400 returns and/or terminates. For example, processing may continue to processing another access request, passing control to a calling process, generating any appropriate record or notification, returning after a method or function invocation, or terminating. In some embodiments, when access to multiple extents is requested, the process is performed for each extent. Alternatively and/or in addition, multiple access requests are processed.
At block 502, the system determines to grant a process direct read access to an extent. At block 504, the system stores a reference to the extent in a buffer header. In some embodiments, the system stores the reference in an unused buffer header from a pool of allocated buffer headers that are no longer associated with any process.
At block 506, the system associates the buffer header with the process. The process may now use the association to access the buffer header, read the reference, and locate the extent in NVRAM to directly read the extent in NVRAM.
At block 508, the system increments a reference count maintained for the extent. A non-zero reference count indicates that one or more processes currently have permissions to access the extent. In some embodiments, each process that receives permissions to access the extent increments the reference count for the extent.
At block 510, the system decrements a reference count maintained for the extent when access terminates. For example, when the process finishes accessing the extent, the process may decrement the reference count. In some embodiments, another process decrements the reference count when the first process crashes or otherwise fails.
At block 512, process 500 returns and/or terminates. For example, processing may continue to processing another access request, passing control to a calling process, generating any appropriate record or notification, returning after a method or function invocation, or terminating.
At block 602, a computer system determines that a process should be granted DRAM copy access to an extent. For example, the computer system may determine, in response to a request from the process, that DRAM copy access should be granted.
At block 604, the computer system generates a DRAM copy of one or more data blocks of the extent. For example, the computer system may copy the one or more data blocks from NVRAM to a data buffer in DRAM. When the extent resides on a remote node in remote NVRAM, the computer system retrieves the data blocks from the remote node.
At block 606, the computer system stores a reference to the DRAM copy in the buffer header. For example, the computer system may store a reference to the data buffer in DRAM.
At block 608, the computer system associates the buffer header with the process that requested access. The process may use the association to read the reference stored in the buffer header, and then use the reference to access the DRAM copy of the data.
At block 610, process 600 returns and/or terminates. For example, processing may continue to processing another access request, passing control to a calling process, generating any appropriate record or notification, returning after a method or function invocation, or terminating.
Example Database System
Since an embodiment of the present invention is implemented within the context of a database management system (DBMS), a description of a database management system is included herein. A DBMS manages a database. A DBMS may comprise one or more database servers. A database comprises database data and a database dictionary that are stored on a persistent memory mechanism, such as a set of hard disks. Database data may be stored in one or more data containers, each containing one or more records. The data within each record is organized into one or more fields. In relational DBMSs, the data containers are referred to as tables, the records are referred to as rows, and the fields are referred to as columns. In object-oriented databases, the data containers are referred to as object classes, the records are referred to as objects, also referred to herein as object records, and the fields are referred to as attributes. Other database architectures may use other terminology.
Users interact with a database server of a DBMS by submitting to the database server commands that cause the database server to perform operations on data stored in a database. A user may be one or more applications running on a client that interact with a database server.
A database command may be in the form of a database statement that conforms to a syntax of a database language. One example language for expressing database commands is the Structured Query Language (SQL). SQL data definition language (“DDL”) instructions are issued to a DBMS to define database structures such as tables, views, or complex data types. For instance, CREATE, ALTER, DROP, and RENAME, are common examples of DDL instructions found in some SQL implementations. SQL data manipulation language (“DML”) instructions are issued to a DBMS to manage data stored within a database structure. For instance, SELECT, INSERT, UPDATE, and DELETE are common examples of DML instructions found in some SQL implementations. SQL/XML is a common extension of SQL used when manipulating XML data in an object-relational database.
Performing operations within a database server often entails invoking multiple layers software. A layer is set of software modules that perform a functionality that has been dedicated, to an extent, within a database server to the set of software modules. Executing an operation typically involves calling multiple layers of software, with one layer making a call to another layer, which during the execution of the first call, calls another layer. For example, to execute an SQL statement, an SQL layer is invoked. Typically, a client accesses a database server through an interface, such as an SQL interface to the SQL layer. The SQL layer analyzes and parses and executes the statement. During execution of the statement, the SQL layer calls modules of a lower layer to retrieve a particular row from a table and to update a particular in a table. A client, such as a replication client, typically accesses the database via a database command to the database server, such as in the form of a SQL statement.
Although the examples described above are based on Oracle's SQL, the techniques provided herein are not limited to Oracle's SQL, to any proprietary form of SQL, to any standardized version or form of SQL (ANSI standard), or to any particular form of database command or database language. Furthermore, for the purpose of simplifying the explanations contained herein, database commands or other forms of computer instructions may be described as performing an action, such as creating tables, modifying data, and setting session parameters. However, it should be understood that the database command itself performs no actions, but rather the DBMS, upon executing the database command, performs the corresponding actions. Typically, database commands are executed over a synchronous connection to the database.
Example Implementation System
According to some embodiments, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
For example,
Computer system 700 also includes a main memory 706, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 702 for storing information and instructions to be executed by processor 704. Main memory 706 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 704. Such instructions, when stored in non-transitory storage media accessible to processor 704, render computer system 700 into a special-purpose machine that is customized to perform the operations specified in the instructions.
Computer system 700 further includes a read only memory (ROM) 708 or other static storage device coupled to bus 702 for storing static information and instructions for processor 704. A storage device 710, such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to bus 702 for storing information and instructions.
Computer system 700 may be coupled via bus 702 to a display 712, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 714, including alphanumeric and other keys, is coupled to bus 702 for communicating information and command selections to processor 704. Another type of user input device is cursor control 716, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 704 and for controlling cursor movement on display 712. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
Computer system 700 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 700 to be a special-purpose machine. According to some embodiments, the techniques herein are performed by computer system 700 in response to processor 704 executing one or more sequences of one or more instructions contained in main memory 706. Such instructions may be read into main memory 706 from another storage medium, such as storage device 710. Execution of the sequences of instructions contained in main memory 706 causes processor 704 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. As used herein, “non-volatile” refers to a characteristic of a memory that retains data in the absence of any form of electrical power, including external or battery backup. Examples of non-volatile memory include, for example, e-prom memory, flash memory, optical disks, magnetic disks, or solid-state drives, such as storage device 710. Non-volatile memory does not include volatile memory for which power is retained by a battery backup in the absence of another external power source. For example, volatile memory coupled to a board with an embedded battery-backup is not non-volatile memory because without the power provided by a battery, the volatile memory does not retain data.
Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 702. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 704 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 700 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 702. Bus 702 carries the data to main memory 706, from which processor 704 retrieves and executes the instructions. The instructions received by main memory 706 may optionally be stored on storage device 710 either before or after execution by processor 704.
Computer system 700 also includes a communication interface 718 coupled to bus 702. Communication interface 718 provides a two-way data communication coupling to a network link 720 that is connected to a local network 722. For example, communication interface 718 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 718 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 718 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 720 typically provides data communication through one or more networks to other data devices. For example, network link 720 may provide a connection through local network 722 to a host computer 724 or to data equipment operated by an Internet Service Provider (ISP) 726. ISP 726 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 728. Local network 722 and Internet 728 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 720 and through communication interface 718, which carry the digital data to and from computer system 700, are example forms of transmission media.
Computer system 700 can send messages and receive data, including program code, through the network(s), network link 720 and communication interface 718. In the Internet example, a server 730 might transmit a requested code for an application program through Internet 728, ISP 726, local network 722 and communication interface 718.
The received code may be executed by processor 704 as it is received, and/or stored in storage device 710, or other non-volatile storage for later execution.
Software Overview
Software system 800 is provided for directing the operation of computer system 700. Software system 800, which may be stored in system memory (RAM) 706 and on fixed storage (e.g., hard disk or flash memory) 710, includes a kernel or operating system (OS) 810.
The OS 810 manages low-level aspects of computer operation, including managing execution of processes, memory allocation, file input and output (I/O), and device I/O. One or more application programs, represented as 802A, 802B, 802C . . . 802N, may be “loaded” (e.g., transferred from fixed storage 710 into memory 706) for execution by the system 800. The applications or other software intended for use on computer system 700 may also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server, an app store, or other online service).
Software system 800 includes a graphical user interface (GUI) 815, for receiving user commands and data in a graphical (e.g., “point-and-click” or “touch gesture”) fashion. These inputs, in turn, may be acted upon by the system 800 in accordance with instructions from operating system 810 and/or application(s) 802. The GUI 815 also serves to display the results of operation from the OS 810 and application(s) 802, whereupon the user may supply additional inputs or terminate the session (e.g., log off).
OS 810 can execute directly on the bare hardware 820 (e.g., processor(s) 704) of computer system 700. Alternatively, a hypervisor or virtual machine monitor (VMM) 830 may be interposed between the bare hardware 820 and the OS 810. In this configuration, VMM 830 acts as a software “cushion” or virtualization layer between the OS 810 and the bare hardware 820 of the computer system 700.
VMM 830 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine comprises a “guest” operating system, such as OS 810, and one or more applications, such as application(s) 802, designed to execute on the guest operating system. The VMM 830 presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems.
In some instances, the VMM 830 may allow a guest operating system to run as if it is running on the bare hardware 820 of computer system 700 directly. In these instances, the same version of the guest operating system configured to execute on the bare hardware 820 directly may also execute on VMM 830 without modification or reconfiguration. In other words, VMM 830 may provide full hardware and CPU virtualization to a guest operating system in some instances.
In other instances, a guest operating system may be specially designed or configured to execute on VMM 830 for efficiency. In these instances, the guest operating system is “aware” that it executes on a virtual machine monitor. In other words, VMM 830 may provide para-virtualization to a guest operating system in some instances.
A computer system process comprises an allotment of hardware processor time, and an allotment of memory (physical and/or virtual), the allotment of memory being for storing instructions executed by the hardware processor, for storing data generated by the hardware processor executing the instructions, and/or for storing the hardware processor state (e.g. content of registers) between allotments of the hardware processor time when the computer system process is not running. Computer system processes run under the control of an operating system, and may run under the control of other programs being executed on the computer system.
Multiple threads may run within a process. Each thread also comprises an allotment of hardware processing time but share access to the memory allotted to the process. The memory is used to store content of processors between the allotments when the thread is not running. The term thread may also be used to refer to a computer system process in multiple threads are not running.
In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.
This application is a continuation of U.S. patent application Ser. No. 15/693,273, entitled “DIRECTLY MAPPED BUFFER CACHE ON NON-VOLATILE MEMORY”, filed on Aug. 31, 2017 by Juan R. Loaiza, et al., the entire contents of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4425615 | Swenson et al. | Jan 1984 | A |
| 4881166 | Thompson | Nov 1989 | A |
| 5095421 | Freund | Mar 1992 | A |
| 5241675 | Sheth et al. | Aug 1993 | A |
| 5263156 | Bowen et al. | Nov 1993 | A |
| 5287496 | Chen et al. | Feb 1994 | A |
| 5332265 | Orimo et al. | Jul 1994 | A |
| 5333316 | Champagne et al. | Jul 1994 | A |
| 5355477 | Strickland et al. | Oct 1994 | A |
| 5369757 | Spiro et al. | Nov 1994 | A |
| 5388196 | Pajak et al. | Feb 1995 | A |
| 5423037 | Hvasshovd | Jun 1995 | A |
| 5454102 | Tang et al. | Sep 1995 | A |
| 5553279 | Goldring | Sep 1996 | A |
| 5555404 | Torbjoslashedrnsen et al. | Sep 1996 | A |
| 5559991 | Kanfi | Sep 1996 | A |
| 5566315 | Milillo et al. | Oct 1996 | A |
| 5574906 | Morris | Nov 1996 | A |
| 5581753 | Terry et al. | Dec 1996 | A |
| 5603024 | Goldring | Feb 1997 | A |
| 5613113 | Goldring | Mar 1997 | A |
| 5649156 | Vishlizky et al. | Jul 1997 | A |
| 5717890 | Ichida et al. | Feb 1998 | A |
| 5717893 | Mattson | Feb 1998 | A |
| 5774643 | Lubbers | Jun 1998 | A |
| 5806076 | Ngai et al. | Sep 1998 | A |
| 5870758 | Bamford et al. | Feb 1999 | A |
| 5870761 | Demers et al. | Feb 1999 | A |
| 5893086 | Schmuck | Apr 1999 | A |
| 5924096 | Draper et al. | Jul 1999 | A |
| 5943689 | Tamer | Aug 1999 | A |
| 5951695 | Kolovson | Sep 1999 | A |
| 5953719 | Kleewein | Sep 1999 | A |
| 5956731 | Bamford et al. | Sep 1999 | A |
| 5974427 | Reiter | Oct 1999 | A |
| 5983277 | Heile et al. | Nov 1999 | A |
| 5987506 | Carter | Nov 1999 | A |
| 5991771 | Falls et al. | Nov 1999 | A |
| 6014669 | Slaughter et al. | Jan 2000 | A |
| 6122630 | Strickler et al. | Sep 2000 | A |
| 6192377 | Ganesh et al. | Feb 2001 | B1 |
| 6243838 | Liu | Jun 2001 | B1 |
| 6298319 | Heile et al. | Oct 2001 | B1 |
| 6353835 | Lieuwen | Mar 2002 | B1 |
| 6370622 | Chiou et al. | Apr 2002 | B1 |
| 6393485 | Chao et al. | May 2002 | B1 |
| 6457105 | Spencer et al. | Sep 2002 | B1 |
| 6516327 | Zondervan et al. | Feb 2003 | B1 |
| 6526483 | Cho et al. | Feb 2003 | B1 |
| 6574717 | Ngai et al. | Jun 2003 | B1 |
| 6611898 | Slattery et al. | Aug 2003 | B1 |
| 6691139 | Ganesh et al. | Feb 2004 | B2 |
| 6728823 | Walker et al. | Apr 2004 | B1 |
| 6839751 | Dietz et al. | Jan 2005 | B1 |
| 6922754 | Liu et al. | Jul 2005 | B2 |
| 7024656 | Ahad | Apr 2006 | B1 |
| 7069324 | Tiwana et al. | Jun 2006 | B1 |
| 7076508 | Brourbonnais et al. | Jul 2006 | B2 |
| 7159076 | Madter | Jan 2007 | B2 |
| 7165144 | Choubal et al. | Jan 2007 | B2 |
| 7222136 | Brown et al. | May 2007 | B1 |
| 7287034 | Wong et al. | Oct 2007 | B2 |
| 7290017 | Wang et al. | Oct 2007 | B1 |
| 7290090 | Madter | Oct 2007 | B2 |
| 7334064 | Davies | Feb 2008 | B2 |
| 7415723 | Pandya | Aug 2008 | B2 |
| 7461147 | Mowat et al. | Dec 2008 | B1 |
| 7464113 | Girkar et al. | Dec 2008 | B1 |
| 7506103 | Madter | Mar 2009 | B2 |
| 7548898 | Tarenskeen et al. | Jun 2009 | B1 |
| 7558290 | Nucci | Jul 2009 | B1 |
| 7570451 | Bedillion et al. | Aug 2009 | B2 |
| 7627612 | Ahal et al. | Dec 2009 | B2 |
| 7636814 | Karr et al. | Dec 2009 | B1 |
| 7644084 | Rapp | Jan 2010 | B2 |
| 7647443 | Chatterjee | Jan 2010 | B1 |
| 7660945 | Lee | Feb 2010 | B1 |
| 7693886 | Novick | Apr 2010 | B1 |
| 7725559 | Landis | May 2010 | B2 |
| 7769802 | Smith | Aug 2010 | B2 |
| 7774568 | Sudhakar | Aug 2010 | B2 |
| 7836262 | Gunna et al. | Nov 2010 | B2 |
| 7885953 | Chen | Feb 2011 | B2 |
| 7904562 | Takase et al. | Mar 2011 | B2 |
| 7912051 | Rowlands et al. | Mar 2011 | B1 |
| 7917539 | Srinivasan | Mar 2011 | B1 |
| 7921686 | Bagepalli | Apr 2011 | B2 |
| 7962458 | Holenstein | Jun 2011 | B2 |
| 7966293 | Owara et al. | Jun 2011 | B1 |
| 8145838 | Miller et al. | Mar 2012 | B1 |
| 8204892 | Balebail et al. | Jun 2012 | B2 |
| 8244984 | Glasco et al. | Aug 2012 | B1 |
| 8266472 | Bose | Sep 2012 | B2 |
| 8327080 | Der | Dec 2012 | B1 |
| 8359429 | Sharma et al. | Jan 2013 | B1 |
| 8370452 | Harvell et al. | Feb 2013 | B2 |
| 8521923 | Lee et al. | Aug 2013 | B2 |
| 8566297 | Dowers | Oct 2013 | B1 |
| 8627136 | Shankar | Jan 2014 | B2 |
| 8683139 | Gaither | Mar 2014 | B2 |
| 8706687 | Fineberg | Apr 2014 | B2 |
| 8868831 | Goyal et al. | Oct 2014 | B2 |
| 8930647 | Smith | Jan 2015 | B1 |
| 8977801 | Grube | Mar 2015 | B2 |
| 9003159 | Deshkar | Apr 2015 | B2 |
| 9075710 | Talagala | Jul 2015 | B2 |
| 9128965 | Yanacek | Sep 2015 | B1 |
| 9164702 | Nesbit et al. | Oct 2015 | B1 |
| 9256542 | Flower | Feb 2016 | B1 |
| 9263102 | Flynn | Feb 2016 | B2 |
| 9292564 | Kamp et al. | Mar 2016 | B2 |
| 9306947 | Kolbly | Apr 2016 | B2 |
| 9361232 | Umamageswaran et al. | Jun 2016 | B2 |
| 9405694 | Goyal et al. | Aug 2016 | B2 |
| 9405748 | Glover | Aug 2016 | B2 |
| 9514187 | Ziauddin | Dec 2016 | B2 |
| 9703706 | Bagal et al. | Jul 2017 | B2 |
| 10133667 | Mukherjee et al. | Nov 2018 | B2 |
| 10148548 | Griffin | Dec 2018 | B1 |
| 10223326 | Frank et al. | Mar 2019 | B2 |
| 10311154 | Lahiri et al. | Jun 2019 | B2 |
| 10592416 | Baddepudi et al. | Mar 2020 | B2 |
| 20020038384 | Khan | Mar 2002 | A1 |
| 20020059287 | Karasudani | May 2002 | A1 |
| 20020133508 | Larue et al. | Sep 2002 | A1 |
| 20020165724 | Bartus | Nov 2002 | A1 |
| 20030005223 | Coulson | Jan 2003 | A1 |
| 20030046298 | Weedon | Mar 2003 | A1 |
| 20030115324 | Blumenau | Jun 2003 | A1 |
| 20030217236 | Rowlands | Nov 2003 | A1 |
| 20040054860 | Dixit | Mar 2004 | A1 |
| 20040073754 | Cypher | Apr 2004 | A1 |
| 20040117441 | Liu et al. | Jun 2004 | A1 |
| 20040122910 | Douglass et al. | Jun 2004 | A1 |
| 20040148486 | Burton | Jul 2004 | A1 |
| 20040193574 | Suzuki | Sep 2004 | A1 |
| 20040199552 | Ward et al. | Oct 2004 | A1 |
| 20040225719 | Kisley et al. | Nov 2004 | A1 |
| 20040225720 | Pinkerton | Nov 2004 | A1 |
| 20040225845 | Kruckemyer et al. | Nov 2004 | A1 |
| 20040230753 | Amri | Nov 2004 | A1 |
| 20040254943 | Malcom | Dec 2004 | A1 |
| 20050132017 | Biran et al. | Jun 2005 | A1 |
| 20050160224 | Cuomo et al. | Jul 2005 | A1 |
| 20050193160 | Bhatte et al. | Sep 2005 | A1 |
| 20050198062 | Shapiro | Sep 2005 | A1 |
| 20050210202 | Choubal et al. | Sep 2005 | A1 |
| 20060004691 | Sifry | Jan 2006 | A1 |
| 20060010130 | Left et al. | Jan 2006 | A1 |
| 20060064441 | Yamamoto | Mar 2006 | A1 |
| 20060106890 | Paul et al. | May 2006 | A1 |
| 20060146814 | Shah et al. | Jul 2006 | A1 |
| 20060149786 | Nishiyama | Jul 2006 | A1 |
| 20060209444 | Song | Sep 2006 | A1 |
| 20060212481 | Stacey et al. | Sep 2006 | A1 |
| 20060218123 | Chowdhuri et al. | Sep 2006 | A1 |
| 20060271605 | Petruzzo | Nov 2006 | A1 |
| 20060271740 | Mark | Nov 2006 | A1 |
| 20070038689 | Shinkai | Feb 2007 | A1 |
| 20070067575 | Morris et al. | Mar 2007 | A1 |
| 20070078914 | Correl | Apr 2007 | A1 |
| 20070078940 | Fineberg et al. | Apr 2007 | A1 |
| 20070083505 | Ferrari et al. | Apr 2007 | A1 |
| 20070226277 | Holenstein et al. | Sep 2007 | A1 |
| 20070239791 | Cattell | Oct 2007 | A1 |
| 20070260819 | Gao et al. | Nov 2007 | A1 |
| 20080016283 | Madter | Jan 2008 | A1 |
| 20080046736 | Arimilli et al. | Feb 2008 | A1 |
| 20080098044 | Todd | Apr 2008 | A1 |
| 20080104329 | Gaither et al. | May 2008 | A1 |
| 20080155303 | Toeroe | Jun 2008 | A1 |
| 20080177803 | Fineberg et al. | Jul 2008 | A1 |
| 20080201234 | Castro | Aug 2008 | A1 |
| 20080209009 | Katwala et al. | Aug 2008 | A1 |
| 20080215580 | Altinel et al. | Sep 2008 | A1 |
| 20080219575 | Wittenstein | Sep 2008 | A1 |
| 20080222136 | Yates | Sep 2008 | A1 |
| 20080222159 | Aranha et al. | Sep 2008 | A1 |
| 20080235479 | Scales | Sep 2008 | A1 |
| 20080282055 | Yang | Nov 2008 | A1 |
| 20080222111 | Hoang et al. | Dec 2008 | A1 |
| 20090030911 | Guo | Jan 2009 | A1 |
| 20090138944 | Rajasekaran | May 2009 | A1 |
| 20090164536 | Nasre et al. | Jun 2009 | A1 |
| 20090171679 | Salgado et al. | Jul 2009 | A1 |
| 20090193189 | Carswell et al. | Jul 2009 | A1 |
| 20090235230 | Lucas | Sep 2009 | A1 |
| 20090240664 | Dinker et al. | Sep 2009 | A1 |
| 20090248871 | Takase et al. | Oct 2009 | A1 |
| 20090254521 | Raman | Oct 2009 | A1 |
| 20090276479 | Lucas | Nov 2009 | A1 |
| 20090287737 | Hammerly | Nov 2009 | A1 |
| 20090292861 | Kanevsky | Nov 2009 | A1 |
| 20090313311 | Hoffmann | Dec 2009 | A1 |
| 20100017556 | Chin et al. | Jan 2010 | A1 |
| 20100036843 | MacNaughton et al. | Feb 2010 | A1 |
| 20100042587 | Johnson | Feb 2010 | A1 |
| 20100070448 | Omoigui | Mar 2010 | A1 |
| 20100095059 | Kisley et al. | Apr 2010 | A1 |
| 20100122026 | Umamageswaran et al. | May 2010 | A1 |
| 20100145909 | Ngo | Jun 2010 | A1 |
| 20100158486 | Moon | Jun 2010 | A1 |
| 20100274962 | Moesk | Oct 2010 | A1 |
| 20100278446 | Ganesh | Nov 2010 | A1 |
| 20100306234 | Wang et al. | Dec 2010 | A1 |
| 20100332654 | Bose | Dec 2010 | A1 |
| 20110022801 | Flynn | Jan 2011 | A1 |
| 20110029569 | Ganesh | Feb 2011 | A1 |
| 20110040861 | Van Der Merwe | Feb 2011 | A1 |
| 20110041006 | Fowler | Feb 2011 | A1 |
| 20110047330 | Potapov | Feb 2011 | A1 |
| 20110071981 | Ghosh | Mar 2011 | A1 |
| 20110072217 | Hoang | Mar 2011 | A1 |
| 20110087637 | Sundaram et al. | Apr 2011 | A1 |
| 20110153719 | Santoro | Jun 2011 | A1 |
| 20110173325 | Cherian | Jul 2011 | A1 |
| 20110191522 | Condict | Aug 2011 | A1 |
| 20110191543 | Craske et al. | Aug 2011 | A1 |
| 20110238899 | Yano | Sep 2011 | A1 |
| 20110258376 | Young | Oct 2011 | A1 |
| 20110320804 | Chan et al. | Dec 2011 | A1 |
| 20120013758 | Frederiksen | Jan 2012 | A1 |
| 20120017037 | Riddle | Jan 2012 | A1 |
| 20120054225 | Marwah | Mar 2012 | A1 |
| 20120054533 | Shi et al. | Mar 2012 | A1 |
| 20120063533 | Fonseka | Mar 2012 | A1 |
| 20120158650 | Andre | Jun 2012 | A1 |
| 20120158729 | Mital | Jun 2012 | A1 |
| 20120166729 | Donley | Jun 2012 | A1 |
| 20120166886 | Shankar et al. | Jun 2012 | A1 |
| 20120173844 | Punde et al. | Jul 2012 | A1 |
| 20120209873 | He | Aug 2012 | A1 |
| 20120221788 | Raghunathan | Aug 2012 | A1 |
| 20120246202 | Surtani | Sep 2012 | A1 |
| 20120265743 | Ivanova | Oct 2012 | A1 |
| 20120290786 | Mesnier | Nov 2012 | A1 |
| 20120296883 | Ganesh | Nov 2012 | A1 |
| 20120323849 | Garin et al. | Dec 2012 | A1 |
| 20120323970 | Larson | Dec 2012 | A1 |
| 20130007180 | Talpey et al. | Jan 2013 | A1 |
| 20130019000 | Markus | Jan 2013 | A1 |
| 20130024433 | Amit | Jan 2013 | A1 |
| 20130066949 | Colrain | Mar 2013 | A1 |
| 20130086330 | Baddepudi | Apr 2013 | A1 |
| 20130132684 | Ostrovsky | May 2013 | A1 |
| 20130132705 | Ishii | May 2013 | A1 |
| 20130166534 | Yoon | Jun 2013 | A1 |
| 20130166553 | Yoon | Jun 2013 | A1 |
| 20130198312 | Tamir et al. | Aug 2013 | A1 |
| 20130212332 | Umamageswaran | Aug 2013 | A1 |
| 20130275391 | Batwara | Oct 2013 | A1 |
| 20130290462 | Lim | Oct 2013 | A1 |
| 20130290598 | Fiske | Oct 2013 | A1 |
| 20130326152 | Loaiza et al. | Dec 2013 | A1 |
| 20130339572 | Fanning | Dec 2013 | A1 |
| 20140012814 | Bercovici | Jan 2014 | A1 |
| 20140012867 | Moss | Jan 2014 | A1 |
| 20140047263 | Coathney | Feb 2014 | A1 |
| 20140089565 | Lee | Mar 2014 | A1 |
| 20140010842 | Isaacson | Apr 2014 | A1 |
| 20140108421 | Isaacson et al. | Apr 2014 | A1 |
| 20140108751 | Brown et al. | Apr 2014 | A1 |
| 20140143364 | Guerin | May 2014 | A1 |
| 20140149638 | Jain | May 2014 | A1 |
| 20140200166 | Van Rooyen | Jul 2014 | A1 |
| 20140208062 | Cohen | Jul 2014 | A1 |
| 20140281167 | Danilak | Sep 2014 | A1 |
| 20140325115 | Ramsundar | Oct 2014 | A1 |
| 20140337593 | Holbrook | Nov 2014 | A1 |
| 20140372486 | Bose | Dec 2014 | A1 |
| 20140372489 | Jaiswal | Dec 2014 | A1 |
| 20140372702 | Subramanyam | Dec 2014 | A1 |
| 20150006482 | Hardy | Jan 2015 | A1 |
| 20150006813 | Goyal et al. | Jan 2015 | A1 |
| 20150012690 | Bruce | Jan 2015 | A1 |
| 20150012735 | Tamir et al. | Jan 2015 | A1 |
| 20150019834 | Loh | Jan 2015 | A1 |
| 20150039712 | Frank | Feb 2015 | A1 |
| 20150067087 | Guerin | Mar 2015 | A1 |
| 20150088811 | Hase | Mar 2015 | A1 |
| 20150088813 | Lahiri et al. | Mar 2015 | A1 |
| 20150088822 | Raja | Mar 2015 | A1 |
| 20150088824 | Kamp et al. | Mar 2015 | A1 |
| 20150088830 | Kamp | Mar 2015 | A1 |
| 20150088926 | Chavan | Mar 2015 | A1 |
| 20150089121 | Coudhury et al. | Mar 2015 | A1 |
| 20150089125 | Mukherjee et al. | Mar 2015 | A1 |
| 20150089134 | Mukherjee | Mar 2015 | A1 |
| 20150089140 | Sridharan et al. | Mar 2015 | A1 |
| 20150150017 | Hu | May 2015 | A1 |
| 20150187430 | Suzuki | Jul 2015 | A1 |
| 20150317349 | Chao | Nov 2015 | A1 |
| 20150324404 | Glover | Nov 2015 | A1 |
| 20150324410 | Glover | Nov 2015 | A1 |
| 20160026399 | Purkayastha | Jan 2016 | A1 |
| 20160026579 | Samanta | Jan 2016 | A1 |
| 20160103767 | Banerjee et al. | Apr 2016 | A1 |
| 20160110434 | Kakaraddi | Apr 2016 | A1 |
| 20160132411 | Jolad et al. | May 2016 | A1 |
| 20160132432 | Shen | May 2016 | A1 |
| 20160188414 | Jayakumar | Jun 2016 | A1 |
| 20160306574 | Friedman | Oct 2016 | A1 |
| 20160306923 | Van Rooyen | Oct 2016 | A1 |
| 20160308968 | Friedman | Oct 2016 | A1 |
| 20160328301 | Parakh et al. | Nov 2016 | A1 |
| 20160335310 | Lahiri et al. | Nov 2016 | A1 |
| 20160371347 | Aronovich | Dec 2016 | A1 |
| 20170109317 | Hack | Apr 2017 | A1 |
| 20170169133 | Kim | Jun 2017 | A1 |
| 20170177488 | Leung | Jun 2017 | A1 |
| 20170269837 | Stevens | Sep 2017 | A1 |
| 20170300592 | Breslow | Oct 2017 | A1 |
| 20180011893 | Kimura | Jan 2018 | A1 |
| 20180203897 | Van Rest | Jul 2018 | A1 |
| 20180321846 | Zhang | Nov 2018 | A1 |
| 20180341596 | Teotia | Nov 2018 | A1 |
| 20190042410 | Gould | Feb 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 102999519 | Mar 2013 | CN |
| 103177055 | Jun 2013 | CN |
| 0 501 180 | Sep 1992 | EP |
| 2409 301 | Jun 2005 | GB |
| WO 9318461 | Sep 1993 | WO |
| WO2007045839 | Apr 2007 | WO |
| WO2013109640 | Jul 2013 | WO |
| WO 2013109640 | Jul 2013 | WO |
| WO 2015094179 | Jun 2015 | WO |
| Entry |
|---|
| Mitchell et al., “Using One-Sides RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store” dated 2013, 12 pages. |
| Anonymous: “Transaction Handling”, dated Jan. 1, 2002, https://docs.oracle.com/cd/A87860_01/doc/java.817/a83725/trans1.htm, 12 pages. |
| Bober, Paul M., et al., “On Mixing Queries and Transactions via Multiversion Locking”, Computer Sciences Department, University of Wisconsin, 1992, pp. 535-545. |
| Bornhovd et al., “Adaptive Database Caching with DBCache”, IEEE 2004, pp. 11-18. |
| Culley P. et al., “An RDMA Protocol Specification” Internet Draft, dated Sep. 16, 2002, 58 pages. |
| DataStax, “A Solution to the Supernode Problem”, https://www.datastax.com/dev/blog/asolutiontothesupernodeproblem, dated Oct. 25, 2012, 6 pages. |
| Dragojević, et al., “FaRM: Fast Remote Memory”, https://www.usenix.org/conference/nsdi14/technical-sessions/dragojević, dated Apr. 2014, 15 pages. |
| Fan et al., “MemC3: Compact and Concurrent MemCache With Dumber Caching and Smarter Hashing”, NSDI'13, dated Apr. 2013, 14 pages. |
| Feng et al., “Accelerating Relational Databases by Leveraging Remote Memory and RDMA”, Proceedings of the 2016 International Conference on Management of Data, SIGMOD, Jan. 1, 2016, pp. 355-370. |
| Anonymous: “Chapter 6 Handling” Transactions with Enterprise Beans, dated Jan. 1, 2004, https://docs.oracle.com/cd/E19229-01/819-1644/detrans.html, 16 pages. |
| Kalia et al., “Using RDMA Efficiently for Key-Value Services”, SIGCOMM'14, Aug. 17-22, 2014, Chicago, IL, USA, 15 pages. |
| Wikipedia, the free encyclopedia, “Cuckoo Hasing”, https://en.wikipedia.org/wiki/Cuckoo_hashing, last viewed on Jul. 31, 2017, 7 pages. |
| Mohan, C., et al., “Efficient and Flexible Methods for Transient Versioning of Records to Avoid Locking by Read-Only Transactions”,Almaden Research Center, date Feb. 6, 1992, pp. 124-133. |
| Oracle® Clusterware, Administration and Deployment Guide, 11g Release 1(11.1), B28255-06, Oct. 2008. http://download.oracle.com/docs/cd/B28359_01/rac.111/b28255.pdf, 186 pages. |
| Pavlo, Andy, “15-721 Database Systems”, Lecture #23 Non-Volatile Memory, dated Spring 2016, 70 pages. |
| Quinn, Nick, “Making Sense of the Graph Revolution”, ww.JAXenter.com | Sep. 2013, 12 pages. |
| Szepesi, Tyler, et al. “Designing a low-latency cuckoo hash table for write-intensive workloads using RDMA.” First International Workshop on Rack-scale Computing. 2014, 6 pages. |
| The Times Ten Team, Mid-Tier Caching: The Times Ten Approach, Jun. 2002. ACM SIGMOD, 6 pages. |
| Tyler Szepesi, et al. “Nessie: A Decoupled, Client-Driven, Key-Value Store using RDMA”, Copyright 2015 the authors CS-2015-09, 13 pages. |
| Wang et al., “C-Hint: An Effective and Reliable Cache Management for RDMAAccelerated Key-Value Stores”, dated 2014, 2 pages. |
| Hilland et al., “RDMA Protocol Verbs Specification” Version 1.0), dated Apr. 25, 2003, 243 pages. |
| Tao, U.S. Appl. No. 15/720,972, filed Sep. 29, 2017, Notice of Allowance dated Nov. 4, 2020. |
| Meiyyappan, U.S. Appl. No. 15/721,328, filed Sep. 29, 2017, Office Action dated Nov. 13, 2020. |
| Meiyyappan, U.S. Appl. No. 15/721,328, filed Sep. 29, 2017, Notice of Allowance dated Jun. 15, 2020. |
| Lahiri, U.S. Appl. No. 14/097,575, filed Dec. 5, 2013, Notice of Allowance dated Nov. 9, 2018. |
| Loaiza, U.S. Appl. No. 15/693,273, filed Aug. 31, 2017, Notice of Allowance dated Jan. 27, 2020. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Office Action dated May 26, 2020. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Office Action dated Apr. 22, 2019. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Office Action dated Oct. 18, 2017. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Interview Summary dated Jul. 3, 2019. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Final Office Action dated Nov. 29, 2019. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Final Office Action dated Jul. 12, 2018. |
| Baddepudi, U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Advisory Action dated Dec. 29, 2017. |
| Lahiri, U.S. Appl. No. 14/097,575, filed Dec. 5, 2013, Office Action dated Feb. 20, 2018. |
| Loaiza, U.S. Appl. No. 15/693,273, filed Aug. 31, 2017, Final Office Action dated Apr. 3, 2020. |
| Frank, U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Notice of Allowance dated Oct. 10, 2018. |
| Frank, U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Notice of Allowance dated Mar. 26, 2018. |
| Frank, U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Final Office Action dated Apr. 5, 2017. |
| Frank, U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Advisory Action dated Aug. 29, 2017. |
| Frank, U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Supplemental Notice of Allowability dated Jun. 13, 2018. |
| Choudhury, U.S. Appl. No. 15/720,959, filed Sep. 29, 2017, Office Action dated Oct. 4, 2019. |
| Choudhury, U.S. Appl. No. 15/720,959, filed Sep. 29, 2017 Notice of Allowance dated Apr. 15, 2020. |
| Baddepudi, U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Office Action dated Dec. 30, 2016. |
| Baddepudi, U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Notice of Allowance dated Nov. 15, 2019. |
| Baddepudi, U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Final Office Action dated Sep. 8, 2017. |
| Lahiri, U.S. Appl. No. 14/097,575, filed Dec. 5, 2013, Office Action dated Apr. 6, 2017. |
| Tao, U.S. Appl. No. 15/720,972, filed Sep. 29, 2017, Final Office Action dated Jan. 6, 2020. |
| U.S. Appl. No. 14/097,575, filed Dec. 5, 2013, Final Office Action dated Jul. 14, 2016. |
| U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Office Action dated Sep. 9, 2016. |
| U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Office Action dated May 7, 2015. |
| U.S. Appl. No. 13/956,096, filed Jul. 31, 2013, Final Office Action dated Jan. 15, 2016. |
| U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Final Office Action dated Jan. 5, 2015. |
| U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Office Action dated Jul. 1, 2015. |
| U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Office Action dated Mar. 31, 2014. |
| U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Interview Summary dated Oct. 25, 2016. |
| U.S. Appl. No. 13/288,785, filed Nov. 3, 2011, Final Office Action dated May 6, 2016. |
| Loaiza, U.S. Appl. No. 15/693,273, filed Aug. 31, 2017, Office Action dated Aug. 6, 2019. |
| Tao, U.S. Appl. No. 15/720,972, filed Sep. 29, 2017, Office Action dated Sep. 13, 2018. |
| Loaiza, U.S. Appl. No. 15/693,273, filed Aug. 31, 2017, Office Action dated Oct. 2, 2018. |
| Tao, U.S. Appl. No. 15/720,972, filed Sep. 29, 2017, Final Office Action dated Jan. 24, 2019. |
| Shi, U.S. Appl. No. 15/720,949, filed Sep. 29, 2017, Office Action dated Oct. 4, 2019. |
| Shi, U.S. Appl. No. 15/720,949, filed 09/269/2017, Notice of Allowance dated Mar. 25, 2020. |
| Rest, U.S. Appl. No. 15/409,091, filed Jan. 18, 2017, Office Action dated Oct. 18, 2018. |
| Rest, U.S. Appl. No. 15/409,091, filed Jan. 18, 2017, Notice of Allowance dated May 14, 2019. |
| Rest, U.S. Appl. No. 15/409,091, filed Jan. 18, 2017, Interview Summary dated Dec. 18, 2018. |
| Muhkherjee, U.S. Appl. No. 15/257,754, filed Sep. 6, 2016, Corrected Notice of Allowance dated Aug. 28, 2018. |
| Muhkherjee et al., U.S. Appl. No. 15/257,754, filed Sep. 6, 2016, Office Action dated Nov. 16, 2017. |
| Meiyyappan, U.S. Appl. No. 15/721,328, filed Sep. 29, 2017, Office Action dated Nov. 29, 2019. |
| U.S. Appl. No. 14/097,575, filed Dec. 5, 2013, Office Action dated Nov. 4, 2015. |
| Tao, U.S. Appl. No. 15/720,972, filed Sep. 29, 2017, Office Action dated Aug. 14, 2019. |
| Meiyyappan et al., U.S. Appl. No. 15/721,328, filed Sep. 29, 2017, Corrected Notice of Allowability dated Jul. 9, 2021. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Pre-Brief Appeal Conference decision Jun. 29, 2021. |
| Lahiri, U.S. Appl. No. 14/709,018, filed May 11, 2015, Examiner's Answer to Appeal brief, dated Dec. 21, 2021. |
| Number | Date | Country | |
|---|---|---|---|
| 20200320010 A1 | Oct 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15693273 | Aug 2017 | US |
| Child | 16907703 | US |
