The invention relates to a system and method for processing directory service operations using non-persistent storage.
Directory services are commonly used to manage identities, policies, security information, and network resources. A directory service typically organizes electronic content and runs on a directory server. The directory service is the interface to a directory which holds information about objects that are to be managed by the directory service.
Directories differ from databases in several ways. A primary difference is that information in a directory is generally read more often than it is written. Still, information has to at times be written to directories. Additionally, data in a directory is accessed using one or more directory service protocols. A client requesting from a directory need not know which physical device stores the data. There is an ongoing need to increase directory read and write performance without sacrificing consistency and reliability.
Past attempts to increase read and write performance have resulted in increased complexity of operations and increased cost. Previous attempts at increasing read performance have included the use of various caching techniques. However, in a mixed read/write environment, read performance still suffers in order to maintain consistent information. Caching as well as turning off transaction logging has been used in attempts to increase write performance. However, these techniques do not ensure reliability because information may be lost in the event of a power failure or other service interruption.
The inventors have determined that it would be desirable to provide a system and method to increase both read and write performance of existing systems without sacrificing consistency and reliability.
A system and method are provided for increasing the speed at which data is written in a directory system. The system of the present invention provides a directory server that uses non-persistent storage in order to expedite the process of handling directory service operations. The directory server may include a communications interface, a non-persistent storage mechanism, an evaluation module, and a directory service knowledge module.
The communications interface may be configured to receive one or more directory service operations from one or more client devices and to provide responses to the client device. The evaluation module may be configured to determine whether the directory server is equipped to perform a received directory service operation and to perform the operation.
If the directory is capable of performing the directory service operation, and if the operation is an update operation (e.g., add, modify, or delete), the update operation is broken up into two types, “permanent” and “temporary,” based upon predetermined criteria. This determination is based on the practical reality that some types of “temporary” data do not need to survive a machine failure because they can be either regenerated or are not important (e.g., session information, online status, or any other similar transient data). It could also be the case that the directory system needs to ignore applications that are attempting to update particular attribute types (e.g., the “created by” attribute is not necessary if there is only one application updating a particular part of the directory).
The differentiation between “permanent” and “temporary” may be made according to any predefined criteria. Examples of different criteria which may be utilized in this determination include:
All of the “permanent” data is written through to a persistent storage mechanism (e.g., a hard drive, a file system, etc.). Before, during or after the data has been written to the persistent storage mechanism, an additional determination may also be made as to whether the “permanent” data should be written to a non-persistent storage mechanism.
“Temporary” data is written through to a non-persistent storage mechanism. The entire contents of the non-persistent storage mechanism, or selected portions, may be immediately or periodically written to the persistent storage mechanism.
The non-persistent storage mechanism may be configured to store data that may be accessed by the client devices.
According to some embodiments, the directory server may be part of a directory service network. As such, the directory service knowledge module may be configured have knowledge of the directory service network and the capabilities of each directory server in the directory service network.
In operation, a directory server may receive a directory service operation from a client device. The directory server may then determine whether it is equipped to evaluate and perform the operation. If it is an update operation, and the directory server is capable of handling the operation, the update operation is broken up into “temporary” and “permanent” data. The permanent data is written to a persistent storage mechanism while the temporary data is written to a non-persistent storage mechanism. An additional determination may also be made as to whether the temporary data should also be written to a persistent storage mechanism.
In accordance with various disclosed embodiments, a system and method are provided for efficient directory service performance. Persistent writes to disk or other permanent storage need not be a mandatory part of a directory server. For example, transient data such as a session object created when a user logs into a system, need not be maintained once the user logs out. In another example, tasks created as a part of a workflow application need not be maintained once the task is complete.
In both of the above examples, the data is created as required and it is not necessary that the data pre-exist before being used. Furthermore, the data may be removed once its usage is complete, and the data need not survive a machine failure because the data may be regenerated as needed. Leveraging these concepts, the present invention uses non-persistent information storage to expedite the data read and write processes.
Client 110 may be a person, an application, a directory, and/or any network accessible electronic device. For example, client device may be a desktop computer, a laptop computer, a portable digital assistant, a wireless telephone, and/or other electronic devices. Client 110 may include a directory services communication interface 115 enabling communication with directory service network 120.
Directory service protocols 112 may include any known or proprietary directory service protocols such as, for example, lightweight directory access protocol (LDAP), X.500, directory service markup language (DSML), and/or other directory service protocols. Directory service protocols 112 enable a user at client device 110 to access one or more directory entries stored on one or more of directory servers 122.
Directory services 112 enable clients to locate directory entries and attributes associated with the entries. Each directory server 122 may store directory entries in a hierarchical, tree-like structure. Each directory entry may comprise a collection of attributes having a distinguished name. Additionally, each entry's attributes may have a type and one or more values. The distinguished name may be constructed by taking the name of the entry, known as the relative distinguished name, and concatenating the names of its ancestors in the hierarchy.
Directory service protocols 112 define a plurality of operations for accessing data stored with the directory service network. For example, the protocol may define operations for searching the directory, adding an entry, updating an entry, deleting an entry, and/or other operations. When client device 110 issues a directory service operation, the client need not be aware of which directory server 122 is responsible for evaluating the operation.
According to an exemplary embodiment of the invention, one or more directory servers 122 may include non-persistent storage mechanism 123. In a typical client/server directory service arrangement, a persistent storage mechanism, such as a database, file system, and/or other permanent storage device may be used for evaluating directory service operations. As used herein, persistent storage refers to permanent, static, long-term storage mechanisms. Persistent data is used herein to refer to data that is read from and/or written to a persistent storage mechanism. Each time a read or write directory operation is performed, the responding directory server consults the persistent storage device to either retrieve an object or write an object to the storage device. However, as described above, many directory service operation do not require permanent storage.
Referring now to
Communications interface 210 may be configured to receive directory service operations from one or more client devices. Additionally, communications interface 210 may be configured to forward directory service operations to other directory servers or to receive forwarded directory service operations when connected as part of a network.
Directory server 200 may include non-persistent storage mechanism 220. Non-persistent storage mechanism 220 may be a portion of memory associated with server 200, according to some embodiments of the invention. While depicted in
According to some embodiments of the invention, non-persistent storage mechanism 220 may be an alternate evaluator. The alternate evaluator may be configured to cache an entire directory store. The directory store may be a persistent storage mechanism. Both read and write operations may be evaluated in relation to the alternate evaluator. Data may be read from the alternate evaluator and operations that add or modify data may be written to the alternate evaluator.
Evaluation module 230 may be configured to determine whether directory server 200 is equipped to perform a received directory service operation. For example, directory server 200 may not be equipped to perform a read operation if its accessible storage mechanism(s) is empty. According to some embodiments of the invention, evaluation module 230 may consider only the contents of non-persistent storage mechanism 220 in determining whether directory server 200 may perform a received directory service operation. In other embodiments, evaluation module 230 may consider non-persistent storage mechanism 220 in addition to other storage mechanisms. Additionally, evaluation module 230 may be configured to perform the requested operation.
If directory server 200 is capable of performing the directory service operation, and if the operation is an update operation (e.g., add, modify, or delete), evaluation module 230 breaks up the update operation into two parts, permanent data and temporary data. If the data type is permanent, evaluation module 230 will write the data to a permanent storage mechanism. In some instances, the permanent data may additionally be written to a persistent storage mechanism. If the data type is temporary, evaluation module will write the data to non-persistent storage-mechanism 220.
Directory server knowledge module 240 may be configured to query other directory servers within a directory service network to resolve an operation that evaluation module 230 has determined may not be performed by directory server 200.
The directory server may then determine whether the received operation depends on existing information, as depicted at 304. For example, to perform a search operation, the server must have access to an information storage mechanism that already includes data. For an add operation, the server need not have access to a storage mechanism that contains data.
If the operation does depend on existing information, the directory server may then determine whether it is able to process the operation, as depicted at 306. As described above, a directory server may be configured to evaluate a directory service operation solely with respect to its associated non-persistent storage mechanism or to additionally evaluate an operation with respect to other storage mechanisms. Thus, the directory server may determine whether it has access to the stored data. According to some embodiments of the invention, a non-persistent storage mechanism may be loaded or pre-configured with information from persistent or non-persistent storage mechanism, or other directory servers after which the directory server may commence evaluation of operations. In other embodiments, data may be loaded into the non-persistent storage mechanism via add operations, after which more complex operations such as search, compare, read, list, modify, rename, delete, and/or other operations, may be performed.
If the directory server is unable to process the directory service operation, it may, according to some embodiments, query another server to process the request, as depicted at 308. Alternatively, the server may respond with a message indicating it is unable to complete the request, as depicted at 310.
As depicted at 312, the directory server may perform the directory service operation. According to some embodiments of the invention, the server may store the result of the directory service operation to its non-persistent storage mechanism. A directory server, according to some embodiments of the invention, may include or interface to a persistent storage mechanism in addition to a non-persistent storage mechanism. As such, certain predefined categories of data may be written to the persistent storage mechanism. For example, configuration data may be stored in the persistent storage mechanism and may be retrieved and/or updated as required. Additionally, certain types of data may be stored and/or pre-loaded in the persistent storage mechanism and may be used during evaluation of a particular operation.
In some embodiments of the invention, the directory server may additionally evaluate an operation 322 using process 320 depicted in
A test 324 checks for any attributes in operation 322 which have not yet been processed. If there are attributes to be tested, path 326 is followed and the attribute may be tested repeatedly, for example at test 328 and test 330. The number of order of tests may vary as required, dependent on the particular implementation.
In this embodiment, test 328 checks to determine whether the attribute is temporary. If the attribute is temporary, it is forwarded along path 332 to be added to a non-persistent store buffer in step 334. It is then forwarded along path 336 back to test 324.
If the attribute is permanent, it is forwarded along path 338 to be added to a persistent store buffer in step 340. It is then forwarded along path 342 to test 330 which checks to determine if the attribute needs to also be stored in a non-persistent storage mechanism. If the attribute does need to be stored in non-persistent storage, path 344 is followed and the attribute is added to a non-persistent store buffer in step 334. Otherwise, path 346 is followed back to test 324.
If there are no more attributes associated with operation 322 to be processed, then path 348 is followed and the contents of the respective buffers are applied as necessary and in any order. The attributes from the non-persistent store buffer (from step 334) are applied to a non-persistent storage mechanism in step 350. Attributes in the persistent store buffer (from step 340) are applied to a persistent storage mechanism in step 352.
The application of the attributes is consistent with the type of operation. For example, an add-entry would add attributes, a remove-entry would delete attributes, etc. Furthermore, the application of the attributes can be applied at any time, not necessarily waiting full completion of the various tests noted above. Additionally, the update steps 350 and 352 can occur in any order or in parallel. The update concludes at 354 after all update operations have been applied.
As depicted in
As described above, each directory server may include an evaluation module and a directory server knowledge module.
Server knowledge maintenance module 510 may be used to maintain a list of all servers in directory service network 400. The list may provide information such as, for example, a name associated with the directory server, a network address associated with the directory server, a location of the server, the server status, the type of data maintained by the server, and/or other information. Information maintenance module 510 may be configured to update information a directory server in the directory service network and to add new servers to the server list. According to some embodiments of the invention, each server maintains a server list. In other embodiments of the invention, a dedicated server may be configured to maintain the server list.
Mutual updating module 530 may be configured to enable one or more servers in a directory service network to share received directory service updates with one or more other servers in the network.
The receiving server may determine whether there are other directory servers within the directory service network that should receive the update whether or not the operation was performed locally, as depicted at 604. As described above, a directory server may include a directory server knowledge module which may maintain a knowledge list. The knowledge list may include information about other servers in the network. Determining whether another directory server should receive the update may include, for example, determining whether the other servers store the same type of information, whether the other servers interface to a persistent or non-persistent storage device, and/or other criteria.
If there are no additional servers in the receiving server's knowledge list which should receive the update, the receiving server may simply wait to receive the next operation, as depicted at 610. If there are additional servers in the network directory server list, the receiving server may forward the update to the additional server, as depicted at 608. According to some embodiments of the invention, replication to other servers in the directory server network occurs after confirming the operation with the requesting client device. In other embodiments, replication may occur before confirming the update with the requesting client device. For example, replication to servers which interface to a persistent storage mechanism may be performed after confirming the operation with the client device while replication to servers which interface to a non-persistent storage mechanism may be performed before confirming the operation.
Mutually updating the additional servers may be performed using various synchronization or replication techniques. Table 1 illustrates an exemplary list of techniques that may be used to perform the mutual updating process. Other techniques may be used.
Initialization sub-module 520 may be configured to enable a directory server to receive information stored by a second directory server in the network, prior to bringing the server online. In a replicated directory server network, a new directory server may be initialized by any other directory server in the directory service network. In a distributed directory service network, a new directory server may serve as a replica for one or more other servers in the network. As such, the newly added server may be initialized by the one or more server in the network for which it is a replica.
According to various embodiments of the invention, a routing function may be used to increase the performance of a directory service network.
Router 710 may include a server knowledge list 712. Server knowledge list 712 may provide a list of servers in the network, and an indication of each server's status and/or other information. Status information may include, for example, an indication of the server's availability, a time associated with the last operation performed by the server, and/or other status information. A server may be unavailable, for example, if the server is offline, recovering from another operation, busy, or otherwise marked unavailable.
Server knowledge list 712 may also include information identifying the type or category of information managed by each server. In a replicated environment, multiple servers may serve the same information, whereas in a distributed environment, each server may serve a specific subset of information. In either environment, a client making a request need not be aware of the server allocation.
According to some embodiments of the invention, router 710 may be a directory server adapted to perform one or more routing functions. For example the router may be adapted to perform functions such as load sharing, fail-over, fail-back, and/or other routing functions. If router 710 is configured for fail-over and directs an operation to a server that fails, the operation can be redirected to another server without interrupting the client application. If router 710 is configured for fail-back and a new server is brought online, the new server is configured and used transparently without the need for client input.
As depicted in
Load-sharing between the one or more directory servers 720 may be based on availability and other pre-determined criteria. For example, router 710 may be configured to route directory service operations to one of directory servers 720 in a round robin fashion wherein each server in turn receives a request. In some embodiments, the directory server having the least number of queued requests may be chosen by router 710. Other methods of choosing a directory server may be used such as, for example, choosing randomly, defining directory server precedence, operation complexity, operation type, security classification, and/or other methods.
Router 710 may be configured to respond to a plurality of events. As depicted in
A router receiving a directory operation request may be adapted to forward the request to a directory service that can handle the request. This process, is known as navigation or chaining, and may be based on a unique attribute provided in the directory operation request. For example, deciding where to forward the request may be based on the distinguished name in the request. As described above, the distinguished name is a collection of attributes that make up a directory entry. Using the distinguished name, the router may locate a directory service that manages a namespace that includes the directory name of the target entry.
The directory operation request may include, among other things, a distinguished name associated with the operation. As such, the router may determine which one or more servers manage the provided distinguished name. Other criteria may be used, as would be apparent.
As described above, one or more servers may be capable of responding to a directory operation request. As depicted at 906, a determination is made as to whether more than one server may service the request. The router may consult its knowledge list to determine whether more than one server manages the information identified in the directory operation request. If only one server is identified in the knowledge list, the request is forwarded to the identified server, as depicted at 908. If more than one server may respond to the request, the router may choose a server, as depicted at 910.
The router may determine which of the two or more directory servers to route a directory operation request to based on availability. A server may be unavailable if it is offline, recovering from another operation, busy, or otherwise marked unavailable. In addition to availability, a variety of methods to determine whether a directory operation request should be routed to a server. As described above, this evaluation may be based on one or more of round robin, least busy, multicast, random, preferred server rankings, operation type, operation complexity, security, or any other suitable determination method. For example, if available servers are evaluated using least busy and preferred, the router first determines which server is least busy and then of those which is highest in the preference list. Once a server have been chosen, the directory operation request may be forwarded to the chosen server.
A router in a directory service network may additionally be configured to receive server disconnect indications. According to some embodiments of the invention, a router may receive a disconnect indication due to a server or network failure. Upon receipt of a disconnect indication, the router may mark the server as unavailable in its server availability list. The router may additionally store the time the disconnect indication was received. As such, the router may then send a connection request to the failed server after a pre-defined time-out period has elapsed if the server does not automatically reconnect.
The router may also receive connection indications from one or more directory servers joining the directory service network. When a new server comes online, it may be added to the router's available server list. Furthermore, the router may receive a connection indication when a previously failed server comes back online. The router may then unmark its failure in the availability list.
While the invention has been described with reference to the certain illustrated embodiments, the words that have been used herein are words of description, rather than words of limitation. Changes may be made, within the purview of the associated claims, without departing from the scope and spirit of the invention in its aspects. Although the invention has been described herein with reference to particular structures, acts, and materials, the invention is not to be limited to the particulars disclosed, but rather can be embodied in a wide variety of forms, some of which may be quite different from those of the disclosed embodiments, and extends to all equivalent structures, acts, and, materials, such as are within the scope of the associated claims.
This application is being filed concurrently with the following applications, which are incorporated herein by reference: “Method and System for Configuring a Supplemental Directory,” having a Ser. No. ______, and an attorney docket number of 063170.8171; “Method and System for Providing Enhanced Read Performance for a Supplemental Directory,” having a Ser. No. ______, and an attorney docket number of 063170.8172; “Method and System for Improving Write Performance in a Supplemental Directory,” having a Ser. No. ______, and an attorney docket number of 063170.8173; “Method and System for Providing a Directory Overlay,” having a Ser. No. ______, and an attorney docket number of 063170.8174; “Method and System for Automatic Registration of Attribute Types,” having a Ser. No. ______, and an attorney docket number of 063170.8203; “System and Method for Routing Directory Service Operations in a Directory Service Network,” having a Ser. No. ______, and an attorney docket number of 019232-0316613; “System and Method for Efficient Directory Performance Using Non-Persistent Storage,” having a Ser. No. ______, and an attorney docket number of 019232-0316610; and “System and Method for Providing a Directory Service Network,” having a Ser. No. ______, and an attorney docket number of 019232-0316611. This application is also related to the following applications, which were filed on May 20, 2005, and are incorporated herein by reference: “Method and Apparatus for Enhancing Directory Performance,” U.S. patent application Ser. No. 11/134,047; “Method and Apparatus of Optimizing Directory Performance,” U.S. patent application Ser. No. 11/134,143; “Method and Apparatus for Handling Directory Operations,” U.S. patent application Ser. No. 11/134,251; “Method and Apparatus for Loading Data into an Alternate Evaluator for Directory Operations,” U.S. patent application Ser. No. 11/134,043; “Structure of an Alternate Evaluator for Directory Operations,” U.S. patent application Ser. No. 11/134,237; and “Method of Selecting a Processor for Query Evaluation,” U.S. patent application Ser. No. 11/134,070.