Patent Grant
9069879
The present technology pertains to Globally Unique Identifiers (GUIDs) for content items, and more specifically pertains to GUIDs for individual content items in a multi-user network-based content management environment.
A multi-user network-based content management environment (storage environment) allows users to upload and store content items in a data storage medium associated with an account, which the users may then access from virtually any network-enabled computing device. However, as the number of users and content items increases in such a storage environment, the complexity and quantity of data quickly scale and become difficult to manage. One way to manage these content items is by storing them in a particular directory structure that reflects content item relationships and access permissions. However, this approach can introduce problems or limitations when implementing sharing between user accounts or when attempting to increase storage efficiency.
Particular features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out hereinafter. These and other features of the disclosure will become more fully apparent from the following description and accompanying drawings, or can be learned by the practice of the principles set forth herein.
The approaches set forth herein can be used to assign every content item in a multi-user network-based content management environment (storage environment) a Globally Unique Identifier (GUID), instead of relying solely on content item paths and/or content item names in a hierarchical directory structure. The storage environment can include one or more storage devices, one or more servers, network infrastructure, other computing devices, databases, and so forth.
Systems, methods, and computer-readable storage media for implementing globally unique identifiers (GUIDs) in a multi-user network-based content management environment (storage environment) are further disclosed herein, from the perspective of a server creating a GUID for a content item, and serving requests based on the GUID. The example content management system identifies a content item at a location within the content management system. The content management system can be a multi-user network-based content management environment. Each user of the content management system can be associated with a respective user account having an amount of storage space within the content management system. The content management system can model a user account as a collection of content item objects and globally unique identifiers that are independent of a directory-based hierarchy of content items.
The principles of the present disclosure will become more explicitly understood from the particular description of the principles disclosed with reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only exemplary embodiments of the disclosure and are not therefore to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying drawings in which:
Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the art will recognize that other components and configurations may be used without departing from the spirit and scope of the disclosure.
The present disclosure provides additional flexibility and functionality for multi-user network-based content management environments by the use of GUIDs. A GUID is a globally unique identifier for a content item within a particular storage space, storage domain, content item object space, or other space for storing content items. The storage space can include the entire available storage in a storage environment or a portion thereof. The storage environment can provide accounts for users which can store and access their own content items in the storage environment. For example, a user can upload content items to the storage environment via a web interface or a native storage environment client application on a computer or other computing device. After the content items are uploaded to the account, the user can access those content items via their account from virtually any other network-enabled computing device. The user can also share content items or folders in their account with other users. Thus, the storage environment can maintain different user accounts for different users, each of which can contain numerous content items, folders, and metadata. Each item in the storage environment can be associated with a GUID.
Exemplary system configuration 100 is shown in
In system 100, a user can interact with content management system 106 through client devices 1021, 1022, . . . , 102n (collectively “102”) connected to network 104 by direct and/or indirect communication. Content management system 106 can support connections from a variety of different client devices, such as desktop computers; mobile computers; mobile communications devices, e.g. mobile phones, smart phones, tablets; smart televisions; set-top boxes; and/or any other network enabled computing devices. Client devices 102 can be of varying type, capabilities, operating systems, etc. Furthermore, content management system 106 can concurrently accept connections from and interact with multiple client devices 102.
A user can interact with content management system 106 via a client-side application installed on client device 102i. In some embodiments, the client-side application can include a content management system specific component. For example, the component can be a stand-alone application, one or more application plug-ins, and/or a browser extension. However, the user can also interact with content management system 106 via a third-party application, such as a web browser, that resides on client device 102i and is configured to communicate with content management system 106. In either case, the client-side application can present a user interface (UI) for the user to interact with content management system 106. For example, the user can interact with the content management system 106 via a client-side application integrated with the content item system or via a webpage displayed using a web browser application.
Content management system 106 can make it possible for a user to store content, as well as perform a variety of content management tasks, such as retrieve, modify, browse, and/or share the content. Furthermore, content management system 106 can make it possible for a user to access the content from multiple client devices 102. For example, client device 102i can upload content to content management system 106 via network 104. The content can later be retrieved from content management system 106 using the same client device 102i or some other client device 102j.
To facilitate the various content management services, a user can create an account with content management system 106. The account information can be maintained in user account database 150. User account database 150 can store profile information for registered users. In some cases, the only personal information in the user profile can be a username and/or email address. However, content management system 106 can also be configured to accept additional user information.
User account database 150 can also include account management information, such as account type, e.g. free or paid; usage information, e.g. file edit history; maximum storage space authorized; storage space used; content storage locations; security settings; personal configuration settings; content sharing data; etc. Account management module 124 can be configured to update and/or obtain user account details in user account database 150. The account management module 124 can be configured to interact with any number of other modules in content management system 106.
An account can be used to store content, such as documents, text files, audio files, video files, etc., from one or more client devices 102 authorized on the account. The content can also include folders of various types with different behaviors, or other mechanisms of grouping content items together. For example, an account can include a public folder that is accessible to any user. The public folder can be assigned a web-accessible address. A link to the web-accessible address can be used to access the contents of the public folder. In another example, an account can include a photos folder that is intended for photos and that provides specific attributes and actions tailored for photos; an audio folder that provides the ability to play back audio files and perform other audio related actions; or other special purpose folders. An account can also include shared folders or group folders that are linked with and available to multiple user accounts. The permissions for multiple users may be different for a shared folder.
The content can be stored in content storage 160. Content storage 160 can be a storage device, multiple storage devices, or a server administered by content storage 160. Alternatively, content storage 160 can be a cloud storage provider or network storage accessible via one or more communications networks. Content management system 106 can hide the complexity and details from client devices 102 so that client devices 102 do not need to know exactly where the content items are being stored by content management system 106. In one variation, content management system 106 can store the content items in the same folder hierarchy as they appear on client device 102i. However, content management system 106 can store the content items in its own order, arrangement, or hierarchy. Content management system 106 can store the content items in a network accessible storage (SAN) device, in a redundant array of independent disks (RAID), etc. Content storage 160 can store content items using one or more partition types, such as FAT, FAT32, NTFS, EXT2, EXT3, EXT4, ReiserFS, BTRFS, and so forth.
Content storage 160 can also store metadata describing content items, content item types, and the relationship of content items to various accounts, folders, or groups. The metadata for a content item can be stored as part of the content item or can be stored separately. In one variation, each content item stored in content storage 160 can be assigned a system-wide unique identifier.
Content storage 160 can decrease the amount of storage space required by identifying duplicate content items or duplicate segments of content items. Instead of storing multiple copies, content storage 160 can store a single copy and then use a pointer or other mechanism to link the duplicates to the single copy. Similarly, content storage 160 can store content items more efficiently, as well as provide the ability to undo operations, by using a content item version control that tracks changes to content items, different versions of content items (including diverging version trees), and a change history. The change history can include a set of changes that, when applied to the original content item version, produce the changed content item version.
Content management system 106 can be configured to support automatic synchronization of content from one or more client devices 102. The synchronization can be platform agnostic. That is, the content can be synchronized across multiple client devices 102 of varying type, capabilities, operating systems, etc. For example, client device 102i can include client software, which synchronizes, via a synchronization module 132 at content management system 106, content in client device 102i's content item system with the content in an associated user account. In some cases, the client software can synchronize any changes to content in a designated folder and its sub-folders, such as new, deleted, modified, copied, or moved content items or folders. The client software can be a separate software application, can integrate with an existing content management application in the operating system, or some combination thereof. In one example of client software that integrates with an existing content management application, a user can manipulate content directly in a local folder, while a background process monitors the local folder for changes and synchronizes those changes to content management system 106. Conversely, the background process can identify content that has been updated at content management system 106 and synchronize those changes to the local folder. The client software can provide notifications of synchronization operations, and can provide indications of content statuses directly within the content management application. Sometimes client device 102i may not have a network connection. In this scenario, the client software can monitor the linked folder for content item changes and queue those changes for later synchronization to content management system 106 when a network connection is available. Similarly, a user can manually stop or pause synchronization with content management system 106.
A user can also view or manipulate content via a web interface generated and served by user interface module 122. For example, the user can navigate in a web browser to a web address provided by content management system 106. Changes or updates to content in the content storage 160 made through the web interface, such as uploading a new version of a content item, can be propagated back to other client devices 102 associated with the user's account. For example, multiple client devices 102i each with their own client software, can be associated with a single account and content items in the account can be synchronized between each of the multiple client devices 102.
Content management system 106 can include a communications interface 120 for interfacing with various client devices 102, and can interact with other content and/or service providers 1091, 1092, . . . , 109n (collectively “109”) via an Application Programming Interface (API). Certain software applications can access content storage 160 via an API on behalf of a user. For example, a software package, such as an app on a smartphone or tablet computing device, can programmatically make calls directly to content management system 106, when a user provides credentials, to read, write, create, delete, share, or otherwise manipulate content. Similarly, the API can allow users to access all or part of content storage 160 through a web site.
Content management system 106 can also include authenticator module 126, which can verify user credentials, security tokens, API calls, specific client devices, and so forth, to ensure only authorized clients and users can access content items. Further, content management system 106 can include analytics module 134 that can track and report on aggregate content item operations, user actions, network usage, total storage space used, as well as other technology, usage, or business metrics. A privacy and/or security policy can prevent unauthorized access to user data stored with content management system 106.
Content management system 106 can include sharing module 130 for managing sharing content publicly or privately. Sharing content publicly can include making the content item accessible from any computing device in network communication with content management system 106. Sharing content privately can include linking a content item in content storage 160 with two or more user accounts so that each user account has access to the content item. The sharing can be performed in a platform agnostic manner. That is, the content can be shared across multiple client devices 102 of varying type, capabilities, operating systems, etc. The content can also be shared across varying types of user accounts.
In some embodiments, content management system 106 can include a content item management module 128 for maintaining a content directory identifying the location of each content item in content storage 106. The content directory can include a unique content entry for each content item stored in the content storage.
A content entry can include a content path that can be used to identify the content item. For example, the content path can include the name of the content item and a folder hierarchy associated with the content item. For example, the content path can include a folder or path of folders in which the content item is placed as well as the name of the content item. Content management system 106 can use the content path to present the content items in the appropriate folder hierarchy.
A content entry can also include a content pointer that identifies the location of the content item in content storage 160. For example, the content pointer can include the exact storage address of the content item in memory. In some embodiments, the content pointer can point to multiple locations, each of which contains a portion of the content item.
In addition to a content path and content pointer, a content entry can also include a user account identifier that identifies the user account that has access to the content item. In some embodiments, multiple user account identifiers can be associated with a single content entry indicating that the content item has shared access by the multiple user accounts.
To share a content item privately, sharing module 130 can be configured to add a user account identifier to the content entry associated with the content item, thus granting the added user account access to the content item. Sharing module 130 can also be configured to remove user account identifiers from a content entry to restrict a user account's access to the content item.
To share content publicly, sharing module 130 can be configured to generate a custom network address, such as a uniform resource locator (URL), which allows any web browser to access the content in content management system 106 without any authentication. To accomplish this, sharing module 130 can be configured to include content identification data in the generated URL, which can later be used to properly identify and return the requested content item. For example, sharing module 130 can be configured to include the user account identifier and the content path in the generated URL. Upon selection of the URL, the content identification data included in the URL can be transmitted to content management system 106 which can use the received content identification data to identify the appropriate content entry and return the content item associated with the content entry.
In addition to generating the URL, sharing module 130 can also be configured to record that a URL to the content item has been created. In some embodiments, the content entry associated with a content item can include a URL flag indicating whether a URL to the content item has been created. For example, the URL flag can be a Boolean value initially set to 0 or false to indicate that a URL to the content item has not been created. Sharing module 130 can be configured to change the value of the flag to 1 or true after generating a URL to the content item.
In some embodiments, sharing module 130 can also be configured to deactivate a generated URL. For example, each content entry can also include a URL active flag indicating whether the content should be returned in response to a request from the generated URL. For example, sharing module 130 can be configured to only return a content item requested by a generated link if the URL active flag is set to 1 or true. Thus, access to a content item for which a URL has been generated can be easily restricted by changing the value of the URL active flag. This allows a user to restrict access to the shared content item without having to move the content item or delete the generated URL. Likewise, sharing module 130 can reactivate the URL by again changing the value of the URL active flag to 1 or true. A user can thus easily restore access to the content item without the need to generate a new URL.
While content management system 106 is presented with specific components, it should be understood by one skilled in the art, that the architectural configuration of system 106 is simply one possible configuration and that other configurations with more or less components are also possible. For example, content management system 106 can implement GUIDs for stored content items, such as files. Client devices 102 can include a client application that communicates, via network 104, with content management system 106 to synchronize content items stored in data storage 160 and GUIDs associated with the content items. Client device 102 can include an operating system that maintains unique identifiers for the content items according to an operating system schema that may be different from the GUIDs maintained for use with content management system 106.
Content management system 106 can include, such as part of communications interface 120 and/or content item management module 128, a synchronous client interface and an asynchronous client interface for client devices 102 to access content items through content management system 106. Content management system 106 can also include a GUID generator, and a GUID processor for resolving GUID conflicts for GUIDs generated by the various client devices 102 in an asynchronous access model. Further, content management system 106 can include an encryption module that assists in securing GUID transmissions at communication boundaries.
In one example, content management system 106 can represent GUIDs as 128-bit integers, each of which identifies a specific content item object in content management system 106, but GUIDs can be represented as shorter or longer integers or as other representations that are not strictly numeric, such as an alphanumeric string. Some example randomly generated 128-bit GUIDs are provided below, in hexadecimal form:
GUIDs allow content management system 106 to model each namespace as a set of content item objects with unique identifiers instead of as a directory-based hierarchy of content items. GUIDs can include one of three potential relationships. The first relationship is one GUID to many content item revisions, modeled as (rev_id, ns_id, sj_id). Content management system 106 can track mappings of a GUID to a current path of the content item in content management system 106, modeled as (ns_id, path). This mapping may not exist or may be null if the content item object is not currently located anywhere in content management system 106, such as a GUID for a content item that previously existed but has since been deleted. The first relationship maps a GUID to a sequence of content item revisions for a content item object designated by that GUID. Revisions and content item objects can span multiple storage namespaces. In one embodiment, rev_id is a monotonically increasing integer that indicates the order of the revisions for that GUID.
The second relationship is one GUID to one or zero content item paths, and the third relationship is one GUID to one or zero branch points, which are similar. These relationships can be modeled as (guid, rev_id). The second and third relationships can map a GUID to another GUID at the point that the corresponding content item object was copied. This original GUID can be called a “parent GUID.” GUIDs that lack a parent GUID can represent content item objects that were not derived from other content items. Content management system 106 can store these GUID relationships in one or more GUID databases, or can store these GUID relationships as metadata associated with the GUIDs or with the content item objects.
Content management system 106 may adjust or modify GUID assignments when performing logical operations on content items in the file system structure. Several examples are provided below in terms of a generic application programming interface (API) for accessing data stored in content management system 106. These example content item operations are described in terms of user-level file system operations. In this context, user-level means a high-level operation indicated by the user. Applications that access data via such an API may need to translate user-level operations into the operations described below to interact correctly with a GUID-enabled content management system 106. Each of the example operations described below can operate on an account with content management system 106 in the initial state shown below. Note that the rev parameter in the API can correspond to a server journal identifier (SJID) on a backend of content management system 106.
Content management system 106 can accomplish a simple add operation according to an API call with the following input parameters:
Based on these input parameters, content management system 106 can create a new content item located at “/folder/baz.txt”. Content management system 106 can give this content item a new GUID and that GUID will not have any parent GUID association.
Content management system 106 can accomplish a simple edit operation according to an API call with the following input parameters:
Content management system 106 can retain the same GUID for the content item at /foo.txt, and add a new revision as indicated by the input parameters. Content management system 106 can perform some consistency checking to ensure that the revision indicated in the input parameters is valid, such as checking if a parent revision exists before linking a current revision to the parent revision.
Content management system 106 can accomplish a conflicted edit operation according to an API call with the following input parameters:
Because the parent_rev parameter does not match the current parent_rev of “foo.txt”, content management system 106 can create a new content item, such as “foo (conflicted copy).txt”. This new content item can have a new GUID pointing to the parent GUID 0, at rev_id 0.
Content management system 106 can accomplish a simple copy operation according to an API call with the following input parameters:
Content management system 106 can create a new content item at “/folder/baz.txt” and assign the new content item a new GUID. The GUID of the new content item will have a parent GUID 0, at rev_id 0.
Content management system 106 can accomplish a simple move operation according to an API call that provides the following input parameters:
Content management system 106 can move the content item “/foo.txt” to “/folder/baz.txt”. The content item at “/folder/baz.txt” can keep the GUID 0 that was previously pointing to “/foo.txt”. The GUID 0 also now has another revision, (1, 0, 6), and the SJID becomes 6 because it is the next consecutive SJID for the namespace 0.
Content management system 106 can accomplish a simple restore operation according to an API call with the following input parameters:
Content management system 106 can restore the deleted content item “/qux.txt”, and can maintain the GUID 2 that the deleted content item originally had. Further, content management system 106 can associate GUID 2 with an additional revision (2, 0, 6).
Content management system 106 can accomplish a conflicted restore operation according to an API call with the following input parameters:
Content management system 106 can restore the content item “/baz.txt”, but the GUID 2 that the content item “/baz.txt” originally had was last seen at a different path (“/qux.txt”). Content management system 106 can assign the content item “/baz.txt” a new GUID that points to the parent GUID 2 at the last rev_id for GUID 2 that was at “/baz.txt”, which in this case is rev_id 0.
Having discussed some specific examples of content item operations, API calls, and corresponding actions performed by content management system 106, the disclosure turns to several example scenarios illustrating platform-specific details for implementing GUIDs in content management system 106. Given the preceding description of how GUID assignments change as the file system is modified, content management system 106 could propagate GUIDs correctly if the content items stored in content management system 106 were only accessible via a single, synchronous interface. However, content management system 106 can also provide an asynchronous interface that allows users to make asynchronous changes to content items without mediation. The asynchronous approach can make content item manipulation very responsive, but it can also make GUID propagation difficult.
In one example implementation of the asynchronous approach, a user can make changes to a content item stored at a local storage location, and a client application can synchronize those changes to a copy of the content item that is stored at content management system 106. However, changes to the local file system can occur by third-party applications or by the operating system itself without the client application acting as a mediator. Thus, the client application may not have a definitive way of knowing which content items are copies of other content items and which content items exist at specific locations because they were moved from other locations. The only information available to the client application may be the presence and absence of content items at specific path. The client application may not have move information. To propagate GUIDs properly in this environment, the content management system and the client application may need some way to obtain or derive move information between content items.
Due to the lack of move information, both the client application and content management system 106 synchronization logic has typically been based solely on presence and absence information, leading to an “eventually consistent system” that does not guarantee consistency at every point in time between the local storage and content management system 106. Instead the client application and content management system 106 guarantee that at some point in the future, after all activity has quiesced, the local storage will be consistent with what is stored in content management system 106.
The client application and content management system 106 can infer move and copy relationships between content items without directly mediating all move and copy operations if a unique identifier can be derived for each content item on the local file system. The semantics of such an ideal unique identifier could be the same as GUIDs at content management system 106 for at least some of the content item operations set forth above. The ideal unique identifier in the local file system could follow user-level moves and edits of a content item, but copies would generate new unique identifiers, for example. Then as the client application synchronizes changes with content management system 106, the client application can map the local file system identifier back to the last known GUID with which it was associated.
Microsoft® Windows® provides an Object ID that can serve as such an idealized unique identifier. Object IDs are preserved across edits as long as applications implement their edit behavior using ReplaceFile( ) Mac® OS X® provides a similar concept called ATTR_CMN_OBJPERMANENTID that relies on applications using exchangedata( ) to implement editing. Unfortunately, such an ideal unique identifier is not universally available, such as in Linux™-based operating systems and rare file systems on Mac® OS X® that don't support exchangedata( ). When no such equivalent for the idealized unique identifier exists, the client application or content management system 106 can approximate these special identifiers using a set of heuristics and a combination of index node (inode) numbers and extended file attributes (xattrs). However, inodes may or may not follow user-level edits, and xattrs, which are a modern file system interface that support application-defined file metadata that generally follow edits, moves, and copies, may be prone to error.
Both the client application and content management system 106 can transmit and assign GUIDs to one another, and these GUIDS are synchronized between the client application and content management system 106. An example implementation of a GUID-based architecture can be built on existing, non-GUID-based content item synchronization infrastructure. In this implementation, content management system 106 can hold the canonical state of each account. Client application(s) can attempt to modify the content management system state in response to local changes and what the client application considers to be the current state of content management system 106. With respect to GUIDs, content management system 106 and the client application can be responsible for different functionality. Content management system 106 can be responsible for deciding the canonical GUID assignments and the client application can be responsible for tracking the location of a previously assigned GUID and updating content management system 106 with the new location when content items are moved or edited. Content management system 106 can treat the GUID information received from the client application as a strong hint. However, as the central coordinator for making canonical GUID assignments, content management system 106 can still be free to give any GUID to any content item. Client devices that access content items stored in the account at content management system 106 via an API or via a web interface may not play a part in the GUID assigning system outside of transparently propagating GUIDs across moves and edits using the higher-level APIs described above because they are synchronous access schemes and not asynchronous.
When migrating from an existing synchronization infrastructure that was built around path-based lookup, content management system 106 can associate GUIDs with paths as extra metadata or extended attributes. Extended attributes can associate arbitrary key-value pairs, called xattrs, with content items. Content management system 106 and the client application can transmit GUIDs between each other as xattrs. Transmitting GUIDs in this way can allow for transparent support of non-GUID-aware clients, such as clients on older systems or clients who have not yet upgraded to a GUID-aware version of the client application. The form and semantics of extended attributes for GUID tracking on older clients may depend heavily on the specific combination of application version, file system type, and operating system version being used at the clients. In most cases, xattrs are carried across all edits, moves, and copies. Content management system 106 can implement logic to handle multiple copies with the same GUID by assigning each of the multiple copies a new GUID.
On GUID-aware clients, extended attributes can associate a GUID with a content item in the internal sync logic and can guide transmission of GUIDs between content management system 106 and the client application. When syncing changes to the local file system, in addition to persisting the GUID with the content item using the native xattr system, the client application can further associate the local file system unique identifier with content management system GUID in a separate local database. When syncing local changes back up to content management system 106, the client application can read in the local identifier and lookup content management system GUID in the local database.
Two example approaches for synchronizing content items and GUIDs between content management system 106 and the client application can be commit_batch( ) and list( ). commit_batch( ) can be the ultimate endpoint for mutating content items in an account. list( ) can be used to retrieve all changes to an account that have happened after some point. All GUID bookkeeping and assignment can occur in the commit_batch( ), and the client application can retrieve canonical GUID assignments from content management system 106 via list( ). For the purposes of describing the implementation of GUIDs, an explanation of the interfaces and the extensions made to commit_batch( ) and list( ) in order to support GUIDs is provided below.
The interface to commit_batch( ) can be simple, even though commit_batch( ) internally can be very complex. Commit_batch( ) can accept a list of dictionaries of content item metadata to be added to the account, keyed by namespace ID and path. Commit_batch( ) can perform best when called to modify only a single namespace because namespaces are partitioned across many databases. Modifying a single namespace can limit the amount of databases the server has to interact with. An example usage of commit_batch( ) in Python is provided below:
This example demonstrates that the GUID xattr for the content item can be stored in the guid_xattr variable and sent up in the attrs key of the metadata dictionary used as input to commit_batch( ). The GUID sent up here only serves as a hint, as set forth above. If it conflicts with another GUID already assigned by content management system 106, this content item might be assigned a new GUID.
In addition to the path to GUID mappings provided by the extended attributes system, content management system 106 can track several related mappings, such as the one GUID to many revisions and one GUID to one branch point mapping discussed above. Modifications to commit_batch( ) can accommodate updating these mappings when changes to a content items in an account occur. Performing the GUID bookkeeping synchronously with commit_batch( ) can ensure that at every point in time the GUID state is always consistent with the state of every account.
In one example, the modified assignment logic can filter all of the new rows to the server_file_journal (SFJ) table generated by commit_batch( ) through another layer of logic that can responsible for updating the relevant tables that store the GUID mappings. However, the modified assignment logic can alternatively modify other tables or can be incorporated as part of an existing table. If content management system 106 could trust the path/GUID associations sent by the client application, content management system 106 could update the GUID mappings relatively easily just from the new “inserts” to SFJ generated by commit_batch( ). Unfortunately, some client applications may not have strong GUID tracking mechanisms, such as the older client application versions or Linux™ clients discussed above. Content management system 106 can verify that the content items are not being committed with conflicting GUIDs. Further, commit_batch( ) can include or invoke some additional heuristics to decide what GUID to give a content item. This may be especially important when a client tries to commit a GUID that conflicts with another content item in order to correct the conflict and assign proper GUIDs for one or both of the conflicting content items. Table 1 below shows some example heuristics, a short description of the situation in which the example heuristics may apply, and a corresponding result of applying the heuristic to a GUID conflict.
An additional modification to commit_batch( ) can change the return value to return the final GUID assignment for each content item in its result dictionary. This modification can be similar to how commit_batch( ) returns the list of assigned SJIDs in the order that content items were submitted in the input list. But instead, the modified commit_batch( ) can return a list of assigned GUIDs in the same order. This modification can allow the client application to eagerly associate local content item IDs, such as the native GUID or inode, with the canonical GUID maintained by content management system 106.
The interface to list( ) can be modified to accommodate GUIDs as well. Unlike commit_batch( ), list( ) can be simple in both implementation and interface. The interface for list( ) can accept a dictionary that represents a mapping from namespace ID to SJID and returns all latest content item metadata. For example, the content item metadata can be stored in a format similar to the input to commit_batch( ). The dictionary can include mappings for each path for each namespace that has an SJID larger than the input SJID for the corresponding namespace, for example. Essentially list( ) can perform a query similar to the example database query provided below:
SELECT * FROM server_file_journal WHERE ns_id=% (ns_id)s AND latest=1 AND id>% (13sjid);
The interface to list( ) can be modified to communicate GUIDs back to the client device in the space where xattrs are sent down. Further, list( ) can transform GUIDs from their native form before being sent down to the client.
Either content management system 106 or the client application can generate new GUIDs. GUIDs can be generated according to some algorithm or for a particular range. For example, content management system 106 can assign a client application a specific range within which to generate new GUIDs either serially, randomly, or according to some algorithm. In one embodiment, either content management system 106 or the client application can generate new GUIDs randomly according to the UUID4 algorithm in Python in order to avoid GUID collisions.
After a GUID is generated, content management system 106 can serialize and transmit the GUID to the client application, or vice versa. When serialized into an xattr for the storage backend, content management system 106 can store the UUID.bytes attribute of the GUID. To make targeted GUID spoofing more difficult content management system 106 can also encrypt each GUID at the communication boundary with each external client, such as client applications and clients that access content management system 106 via API calls or a web interface. The encryption key can be derived from the concatenation of a client-specific key and a secret version-specific encryption key, for example. For client applications or a web interface, the client-specific key can be the user ID of the client accessing the data. For API applications, the client-specific key can be a combination of the application ID and the user ID of the user on whose behalf the API application is acting. Content management system 106 can implement AES encryption, for example, to encrypt the GUID with a key derived using bcrypt on the relevant encryption key. Content management system 106 can sign this encrypted key by appending an HMAC using a bcrypt derived key from the concatenation of a secret version-specific “sign key” and the client-specific key. Pseudo code for this process is provided below:
This encryption process can provide for each user, and applications acting on behalf of users, a distinct view of the GUID space. This prevents application developers from sharing GUIDs across applications and users. This process also prevents GUIDs from syncing across user installations of the client application.
Turning now to the storage backend, or where content management system 106 actually stores the content items, GUIDs, and other metadata, the GUID backend can use three major mappings: one GUID to many revisions, one GUID to one or zero content item paths, and one GUID to one or zero branch points. In one implementation, the storage backend maintains these mappings in three database tables: guid_revision, guid_path, and guid_parent. If no row exists in the guid_path table for a specific GUID or the associated columns are 0 (“ ” for the filename column), the storage backend can assume that no path is currently associated with that GUID. These tables may be sharded across multiple physical computing devices using a prefix of the GUID as a key into a global table that maps from this prefix to the corresponding computing device containing the shard. The length of the prefix used for the shard key depends on the number of shards available. The prefix of newly generated GUIDs can be uniformly distributed so that GUID storage is evenly spread out across the shards.
Content management system 106 can get the latest revision row for a GUID, by executing a query such as the following:
Content management system 106 can carry out this query, as well as maintain the guid_latest table, instead of performing a “group by” query on the maximum rev_id in the guid_revision table, to limit the amount of disk pages kept in a page cache of a database server. Even though the “group by” query would be efficient from the perspective of purely doing disk IO, because of the layout of the indexes on disk, the amount of latest rows per page in cache is likely to be much lower compared to the amount of latest rows per page from the guid_latest table. A row in the guid_latest table where the parent_hash and the filename columns are NULL indicates that the GUID is not currently mapped to any location in content management system 106 file system. It is an error for the parent_hash column to be NULL and filename column to not be NULL or vice-versa.
When restoring a content item, content management system 106 can determine the latest rev_id at which the GUID was seen at that path. Content management system 106 can perform a “group by” query on the maximum rev_id for that GUID at that path in the guid_revision table. An example query is provided below:
SELECT guid, max(rev_id) FROM guid_revision WHERE guid=% (guid) AND parent_hash=% (parent_hash) AND filename=% (filename) GROUP BY guid;
Content management system 106 can further ensure concurrency, consistency, and locking of the GUIDs while performing GUID operations. To ensure consistency between the GUID tables, content management system 106 can permit only a single writer to modify the state of any single GUID at any given point in time. Content management system 106 can impose a strong consistency requirement for GUIDs with the state of the server_file_journal table. A GUID cannot be removed from or added to a namespace without ensuring that the consistent operation happens atomically with server_file_journal and vice-versa. To ensure this consistency, content management system 106 can require that all modifications to the state of any GUID occur within a doubly-nested critical region where a mutex is first acquired for the namespace to be modified and then another mutex is acquired for the GUID to be modified.
As part of the current scheme for ensuring consistency within a specific namespace, commit_batch( ) can acquire a mutex for each namespace that it modifies. To further ensure consistency for GUIDs, commit_batch( ) can acquire a mutex for every GUID that will be modified during the request. Content management system 106 can implement application-level locks for GUIDs, but because application-level locks are limited to a single application-level lock on a connection at a time, content management system 106 can optionally use row-level locks to implement the GUID mutex. For example, content management system 106 can obtain a row-level lock on each row of the guid_path table, and before obtaining the lock content management system 106 can ensure the row exists in the table, as shown below:
For a commit_batch( ), content management system 106 can acquire a GUID lock for every GUID that the client application is attempting to assign, as well as a lock for every GUID that is becoming abandoned as a result of paths being deleted or converted into directories or otherwise being given a new GUID. The set of GUIDs that content management system 106 is trying to assign to is easily derivable from the input to commit_batch( ). The set of GUIDs that are being abandoned can be derived from the previous latest SFJ entry for each of the paths. Since content management system 106 acquires the namespace locks first, this information is queryable before acquiring the GUID locks. Thus, the state of server_file_journal and the xattr table should be consistent with the GUID table. Since the GUID tables and server_file_journal are located on different machines, content management system 106 can query the guid_path table and ensure that latest paths are consistent with what was in the server_file_journal. If not, then content management system 106 can assume that path had no previous GUID, otherwise that GUID is available for reassignment in commit_batch( ).
The client application can make changes to the local file system by creating a new content item in a temporary content item stored in a cache location or cache directory. Then, when the temporary content item is ready, the client application moves the original content item to the temporary space and then moves the temporary content item to the location of the original. This approach does not preserve inode data on Mac® OS X® or the Object ID on Windows®, which are the native GUID mechanisms on these operating systems. To limit the potential impact the client application can make modify or intercept the appropriate OS-native calls that implement GUID mechanisms to preserve these identifiers. For example, on Mac® OS X®, the client application can modify or intercept the inode-preserving file update system call exchangedata( ). On Windows®, the client application can modify or intercept the system call ReplaceFile( ).
While GUIDs can provide a convenient way to uniquely reference a specific content item and its revision history, metadata, and other information, regardless of a specific path and content item name, this approach may expose new avenues for exploitation and cracking. For example, an attacker or malicious user may attempt unauthorized access to content item data, GUID spoofing, or ancillary information leakage. An attacker may attempt to use GUIDs to gain access to content item data for which the attacker is not authorized. Indirect leakage of otherwise private information, such as information on whether or not a specific user has had or currently has access to a specific content item object, may also be protected.
With respect to unauthorized access to content item data, the information an attacker can access if they have access to a GUID may include all the revisions across all namespaces that the GUID has traversed, the current content item location of the GUID, and the parent GUID. Content management system 106 can prohibit access to content item data to unauthenticated users unless a user who had access to that data has explicitly made it public, such as via a “Public folder” or other sharing mechanism. For an authenticated user, content management system 106 can grant access to all revisions of all content items in all namespaces to which the authenticated user has access. The namespaces to which a user has access can include the root namespace for that user and all currently subscribed shared folders. To maintain this level of security, content management system 106 may only grant users access to revision data of any GUID for the namespaces that a user currently has access to, as well as anything derivable from that. This is a worst-case security precaution because even if a user has acquired access to every GUID in content management system 106, the user is still only allowed access to content items for which they have already be granted access, and nothing more.
With respect to GUID spoofing, because GUID propagation is influenced in some cases by data that is located in a per-content item local file system specific area (clients which do not have native GUID-like mechanisms already provided by the local file system) and that data is freely available to any application running on that client, an attacker may theoretically artificially attach a GUID to a content item. This is the only vector through which an attacker is able to provoke the system to assign an arbitrary GUID to a content item within their control. To limit the impact of this attack, even if an attacker were to successfully spoof a GUID in this way, the attacker would not actually be able to get access to content item data they would not normally already have access to. Content management system 106 can prevent GUID spoofing by transmitting GUIDs to clients encrypted and signed based on the user ID of the user that they are acting on behalf of, this makes it difficult to spoof an arbitrary GUID retrieved from another source (e.g. a computer running an account linked to a different user, a GUID retrieved via the API, etc.). Content management system 106 can further prevent GUID spoofing by ensuring that that commit_batch( ) will not allow the propagation of a GUID that is not in a namespace the user actually has access to. The only spoofing easily allowed given these two precautions is copying an obfuscated GUID wholesale from one content item to which the user already has access to another content item to which the user has access. This attack does not grant the attacker any more access than they already had and instead just makes the GUID propagation incorrect.
The threat of ancillary information leakage is that an attacker may acquire information about an arbitrary GUID to which they do not already have access. Ancillary information leakage is discussed here in terms of the external interfaces. Users of the API or desktop client will encounter difficulty constructing arbitrary GUIDs that they have not already seen. Instead of querying for metadata based on content item path, the API can allow a user to query for the most recent metadata based on GUID. For a GUID that is not currently at a content item path that is accessible to the user that the API application is acting on behalf of, this call can return error messages that mask whether a particular content item exists, such as returning error 404 (not found) instead of error 403 (forbidden), to prevent accidental leakage about the existence of the GUID in the global system. The other way an attacker may gain information about an arbitrary GUID is via a web interface to content management system 106. To protect against this potential vulnerability, the web interface does not show the user any content item revisions or other data derived from the GUID to which that user does not currently have access.
Having disclosed some system components and GUID concepts, the disclosure now turns to the example method embodiments shown in
To enable user interaction with the computing device 1300, an input device 1345 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device 1335 can also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input to communicate with the computing device 1300. The communications interface 1340 can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.
Storage device 1330 is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) 1325, read only memory (ROM) 1320, and hybrids thereof.
The storage device 1330 can include software modules 1332, 1334, 1336 for controlling the processor 1310. Other hardware or software modules are contemplated. The storage device 1330 can be connected to the system bus 1305. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as the processor 1310, bus 1305, display 1335, and so forth, to carry out the function.
Chipset 1360 can also interface with one or more communication interfaces 1390 that can have different physical interfaces. Such communication interfaces can include interfaces for wired and wireless local area networks, for broadband wireless networks, as well as personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein can include receiving ordered datasets over the physical interface or be generated by the machine itself by processor 1355 analyzing data stored in storage 1370 or 1375. Further, the machine can receive inputs from a user via user interface components 1385 and execute appropriate functions, such as browsing functions by interpreting these inputs using processor 1355.
It can be appreciated that exemplary systems 1300 and 1350 can have more than one processor 1310 or be part of a group or cluster of computing devices networked together to provide greater processing capability.
For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.
In some embodiments the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.
Methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer readable media. Such instructions can comprise, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.
Devices implementing methods according to these disclosures can comprise hardware, firmware and/or software, and can take any of a variety of form factors. Typical examples of such form factors include laptops, smart phones, small form factor personal computers, personal digital assistants, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.
The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.
Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and/or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.
This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 61/746,502, filed on Dec. 27, 2012, which is incorporated herein by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20100037161 | Stading et al. | Feb 2010 | A1 |
| 20100274784 | Acharya | Oct 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 02059724 | Aug 2002 | WO |
| 2011133917 | Oct 2011 | WO |
| Entry |
|---|
| International Search Report dated Jul. 7, 2014, issued in corresponding International Application No. PCT/US2013/077859. |
| Number | Date | Country | |
|---|---|---|---|
| 20140188790 A1 | Jul 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61746502 | Dec 2012 | US |
