Patent Grant
7020746
The present invention is related generally to computer memory storage techniques, and, more particularly, to cache memories.
Some data needed by computer applications are expensive to create or to access. The expenses can include computational resources to calculate the data and transportation costs (including bandwidth and time) to access the data over a network. Often, a computing device, after once expending resources to create or access these data, will store the data in a “cache” memory. Then, if the computing device again needs the data, they can be accessed inexpensively from the cache.
The cache can be local to the original application or to the original computing device, or it can be shared among several applications and devices. The latter type of cache is often called a “central” cache. In some environments, each application supports a local cache for its own use while sharing a central cache with other applications. The central cache is optimized for storing data useful to more than one application, while the local caches are available to provide the benefits of caching for those data specific to each application.
Managing data in a central cache is rarely straightforward. Multiple applications attempting to read data from the central cache rarely cause difficulties, but the same cannot be said when at least one application wishes to add data to the cache. If other applications are allowed to read from the central cache at the same time that one application is writing to the cache, then the readers can get out-of-date, or even garbled, data. This access coordination problem is exacerbated when more than one application wishes to add data to the cache.
A common approach to ameliorating this access coordination problem is called “cache locking.” Whenever one application wishes to change the contents of the cache, by adding, deleting, or modifying its contents, it seeks sole access to a “lock” data structure. While it has the lock, the writer application can modify the cache, and other applications are prevented from accessing the cache as long as a writer has the lock. Thus, readers are prevented from getting out-of-date or garbled data. If two applications both wish to modify the cache, then one of them must wait until the other relinquishes the lock.
Locks can be quite useful in coordinating access to a central cache. However, it is apparent that they delay access for all applications whenever one application wishes to modify the cache. For some central caches, applications readily tolerate this slowdown. For other caches, however, it can be a real nuisance. For example, consider a font-glyph cache. Characters displayed on a computer screen are made up of individual elements called “glyphs.” As some of these glyphs contain a significant amount of data, and as some of the glyphs consume significant computational resources in their generation, they are ideal subjects for a central cache. However, locking the font-glyph cache while a new glyph is added to it can cause a noticeable delay in an application writing to the computer's screen.
When the memory resources available to a central cache are limited, another cache management issue arises. Multiple applications wishing to add data to the cache operate independently of one another. Thus, none of these applications has a “global” view as to which data should be added to the central cache in order to improve the operating environment generally. The same issue arises when the central cache grows too large and is reformulated in a smaller size in order to allow for further additions. No one application can decide which data should be retained in the central cache and which data are best removed in order to free up memory for future cache growth.
In view of the foregoing, the present invention provides a central cache that can be updated without the delay overhead of locking and that has a global view of the importance of the data within the cache. “Atomic” updates provide the benefits of access coordination without incurring the delay overhead of locks. Cache updates are “atomic” in that they are so designed that they cannot be interrupted part way through. They result in a cache that is always up-to-date and consistent when accessed by an application.
Applications are always free to read data in the central cache, accessing the data through a reference table. However, the applications do not directly update the cache, instead, they send update requests to a service routine. To update the cache, the cache service routine proceeds in two phases. In the first phase, the cache service routine prepares the new data and adds them to the cache, without updating the reference table. This first phase may take some time, but the cache remains fully accessible to applications as the cache is not locked. During the first phase, an application accessing the cache cannot “see” the new data because the reference table has not yet been updated. Only after the cache data are fully prepared and loaded into the cache does the cache service routine perform the second phase of the update process: updating the reference table. This update, consisting of changing only one pointer, is performed atomically without locking the cache. Thus, the two-phase update process does not require that the cache ever be locked and leaves the cache, at all times, in a valid state for accessing applications. Because all updates are performed by one cache service routine, there is no need for locks to coordinate among multiple cache writers.
The cache service routine collects statistics on how data in the cache are used. When the cache grows too large, the cache service routine uses these statistics to decide which data should be copied into a new cache. The new cache is created atomically, again in two phases. During the first phase, the cache service routine creates the new cache and populates it with selected data from the old cache. Applications have, as yet, no knowledge of the new cache. When the new cache is ready for use, the cache service routine adds a reference to it in a header of the old cache. Then, in the second phase and using another atomic operation, the cache service routine marks the old cache “obsolete.” On noticing that the old cache is marked obsolete, an application follows the reference to the new cache and starts to use only the new cache. As in updates within a cache, this mechanism for replacing the entire cache is performed in such a manner that applications always see a consistent cache.
Applications can continue to use an obsolete cache until they notice the obsolete flag and switch over to the new cache. Once all applications have switched, the obsolete cache is automatically deleted.
In some embodiments, the reference table within a cache consists of offsets that specify the location of data stored in the cache relative to another location within the cache. This has the advantage that the cache may be stored as a file and immediately re-used after the computing device hosting the cache reboots.
The central cache can be hosted by one computing device and used by applications on that and on other computing devices. Each application can also have its own local cache to use in conjunction with the central cache. If the local cache has the same data structure as the central cache, then the same cache-access code can be used for both caches.
The cache service routine applies a heuristic to the statistics on cache usage that it gathers in order to decide which data to keep when replacing the cache. In some embodiments, a user interface is provided to allow the heuristic to be changed and to allow operation of the cache to be monitored.
While the appended claims set forth the features of the present invention with particularity, the invention, together with its objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
a is a block diagram showing three computing devices sharing cache data via a local area network (LAN);
b is a block diagram showing a central cache memory shared among the computing devices of
a through 3d together form a flowchart illustrating an exemplary method for an application program to attempt to access data from a central cache according to the present invention;
a is a schematic diagram showing an exemplary data structure for a central cache according to the present invention;
b is a schematic diagram showing an exemplary data structure for a data element in a central cache;
a through 5d together form a flowchart of an exemplary method for a routine that maintains a central cache according to the present invention; and
Turning to the drawings, wherein like reference numerals refer to like elements, the present invention is illustrated as being implemented in a suitable computing environment. The following description is based on embodiments of the invention and should not be taken as limiting the invention with regard to alternative embodiments that are not explicitly described herein.
In the description that follows, the present invention is described with reference to acts and symbolic representations of operations that are performed by one or more computing devices, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processing unit of the computing device of electrical signals representing data in a structured form. This manipulation transforms the data or maintains them at locations in the memory system of the computing device, which reconfigures or otherwise alters the operation of the device in a manner well understood by those skilled in the art. The data structures where data are maintained are physical locations of the memory that have particular properties defined by the format of the data. However, while the invention is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that various of the acts and operations described hereinafter may also be implemented in hardware.
The present invention provides an atomically updated, central cache memory. The central cache can be used exclusively by applications running on the computing device that hosts the cache, or, as in
A connection to the Internet 110 is shown indicating that even remote computing devices can join the cached-data-sharing environment 100. In reality, the increased communications time needed for such a remote device to access the central cache runs counter to the cache's purpose of providing fast access to data. Most central cache scenarios will involve only one, or at most a few closely located, computing devices.
b presents structural details of an exemplary embodiment of the cached-data-sharing environment 100 of
If the central cache search routine 114 finds the requested data in either the local cache 116 or in the central cache 118, it returns the data to application A 112. If not, the data are sought elsewhere or are created. Once the data are found or created, the cache search routine 114 requests that the data be added to the local cache 116 by calling a local cache update routine 120. A request is also sent to the central cache service routine 124 to add the data to the central cache 118.
Both local and central caches grow when data are added to them. When they become too big, a new cache is created and is populated with some of the data of the previous cache. To select which data are carried over to the new cache, any number of methods are applied. For example, the most recently used data are selected, or the most often used data.
When creating a new central cache, care is taken to prevent the disruption of applications using the central cache. When the new central cache 118 is ready, the older central cache 126 is marked “obsolete” to warn applications that a newer cache is available. However, those applications need not immediately switch to the newer cache 118, but can choose to continue accessing the older central cache 126 for a while. Once no more applications access the older central cache 126, that cache is deleted.
Application B 128 runs on another computing device in the cached-data-sharing environment 100. This application is shown without a local cache, but it does have a central cache search routine 130. That routine still accesses the older central cache 126, having not yet noticed that this cache is marked obsolete. The cache search routine 130 can request that data be added to the central cache, but the central cache service routine 124 will add that data to the current central cache 118, rather than to the obsolete cache 126.
The computing devices 102, 104, and 106 of
a through 3d present an exemplary method for the local and central cache search routine 114 of
Before the flowchart begins at step 300 of
If a central cache is accessible, then the cache search routine 114 checks, in step 302, to see if the central cache is marked “obsolete.” Note that this step, and the following step 304, are optional: The cache search routine 114 can continue to use an obsolete cache. However, in order to remain current, the cache search routine 114 should periodically check the status of the central cache that it is accessing and release the old cache once it notices that the old cache is obsolete.
Before a central cache is marked obsolete, the central cache service routine 124 creates a new central cache (see steps 512 through 518 of
In step 306, the cache search routine 114 accesses the central cache to see if it contains the data of interest. The specifics of step 306 can vary widely depending upon the specifics of the structure of the central cache. Step 306 as shown in
a illustrates a structure of a central cache 400 that is usable in conjunction with the procedure illustrated in step 306. A fixed size header 402 contains the obsolete flag and, if the flag is set, a reference to a more current central cache 400. Next is a fixed size table of references 404. Each entry in the table of references 404 is either a reference to a data element (see 410 of
The actual cache data are stored as individual data elements 410. The space allocated to these data elements 406 grows as needed into an area of uncommitted space 408. When the uncommitted space 408 is used up, it is time for the cache service routine 124 to create a new central cache 400.
The particular implementation illustrated in step 306 begins in step 308 by hashing the data of interest. The hash is then used, in step 310, as an index into the table of references 404 in the central cache 400. The selected entry in the table of references 404 is examined in step 312 of
If, on the other hand, the selected entry in the table of references 404 does refer to a data element, then that data element may or may not contain the data of interest. The reason for this is that different data values may hash to the same value and lead to selection of the same entry in the table of references 404.
Refer to the exemplary data element structure 410 of
With this exemplary structure of the data element 410 in mind, return to step 316 of
If, on the other hand, the comparison in step 316 reveals that this data element 410 does not contain the data of interest, then the cache search routine 114 proceeds to step 320 of
In step 320, if the reference field 412 of the data element 410 refers to a further data element, then that further data element is examined to see if it contains the data of interest. Step 324 captures this process: The chain of data elements 410 is followed by repeating steps 316 through 322 until either the data of interest are found and retrieved (step 318) or until the end of the chain is reached without finding the data of interest (step 322).
If the central cache 400 does not contain the data of interest, then the cache search routine 114, in step 326 of
a through 3d illustrate how the cache search routine 114 retrieves data of interest, either from a central cache 400 or from elsewhere.
The cache service routine 124 begins in step 500 of
In step 502, some embodiments of the cache service routine 124 collect statistics on the received request and on other requests. In step 504, the cache service routine 124 decides whether to comply with this request. Some embodiments simply comply with all requests, while others consult collected statistics before deciding if the data of interest are “worthy” of being added to the central cache 400. Some embodiments deny the request if the size of the data of interest is too great compared with the uncommitted space 408 in the cache 400. The data of interest are compared against data in the cache 400 (possibly by performing step 306 of
If a decision is made to comply with the request, then in step 510 the cache service routine 124 decides whether a new central cache 400 is needed. For example, if the data of interest will not fit into the uncommitted space 408 in the cache 400, then a new cache 400 is created in steps 512 through 520 of
If a new central cache 400 is needed, an empty shell is created in step 512. Caches are generally stored in RAM so that they can provide fast access to their data. Some operating systems allow an area of RAM to be mapped by the file system. This provides certain advantages to be described later.
In step 514 of
In the header 402 of the old cache, a reference to the new cache 400 is written in step 516, and a flag is set marking the old cache “obsolete” in step 518. The new cache 400 is now ready for use. A cache search routine 114 on seeing the obsolete flag in the old cache follows the reference to the new cache 400 (see steps 302 and 304 of
The cache service routine 124, in step 520, requests that the operating system automatically delete the obsolete cache as soon as there are no applications referring to it, that is, as soon as all of the applications have release their references to the obsolete cache.
Regardless of whether a new cache 400 was just created, the data of interest are added to the current cache 400 in step 522. Step 522 of
The specific embodiment of step 522 begins with hashing the data of interest in step 524 (mirroring step 308 of
Returning to step 528, if the selected entry in the table of references 404 already refers to a data element 410, then the reference field 412 of the data element 410 is examined in step 530. In step 532, the chain of references is followed until it ends. (This mirrors the search down the same chain of references in steps 316 through 324 of
To sum up, whether adding a new data element 410 to an existing central cache 400, or creating a new central cache 400, the cache service routine 124 proceeds in two phases. First, the data are prepared, and all of the time-consuming work is done. During this phase, the changes to the central cache 400 are not visible to cache search routines 114. There is, therefore, no need to lock the central cache 400 to preserve its internal consistency. Also, there is no need for the cache service routine 124 to run at a high priority. Phase two consists of writing a single pointer to the new data element 410 or to the new central cache 400. Writing a single pointer is an inherently non-interruptible procedure so, again, there is no need to lock the central cache 400 during this phase. Once the second phase is complete, the new data or the new cache are accessible to cache search routines 114.
In view of the many possible embodiments to which the principles of the present invention may be applied, it should be recognized that the embodiments described herein with respect to the drawing figures are meant to be illustrative only and should not be taken as limiting the scope of the invention. For example, those of skill in the art will recognize that the illustrated embodiments, especially the data structures and procedures based on them, can be modified in arrangement and detail without departing from the spirit of the invention. Although the invention is described in terms of software modules or components, those skilled in the art will recognize that such may be equivalently replaced by hardware components. Therefore, the invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5276835 | Mohan et al. | Jan 1994 | A |
| 5287473 | Mohan et al. | Feb 1994 | A |
| 5706462 | Matousek | Jan 1998 | A |
| 5809543 | Byers et al. | Sep 1998 | A |
| 5950201 | Van Huben et al. | Sep 1999 | A |
| 6009428 | Kleewein et al. | Dec 1999 | A |
| 6216212 | Challenger et al. | Apr 2001 | B1 |
| 6889301 | Wilson et al. | May 2005 | B1 |
| Number | Date | Country | |
|---|---|---|---|
| 20040148463 A1 | Jul 2004 | US |
