Patent Grant
9747333
In a computer system, a kernel is a program that comprises a central core of the computer's operating system. A shell (e.g., bash, csh, ksh, etc., in Unix-based operating systems) is a program and a component of the operating system that interacts with user commands. The shell therefore acts as an interface and a command interpreter for executing user commands. The commands can be programs or tools for performing simple tasks such as reporting a snapshot of the current processes (via the “ps” command) and ascertaining memory usage per process. Users can also combine a set of the shell commands to perform more complex tasks, create shell scripts and save the shell scripts for future use.
Operating Systems (e.g., Linux) expose a large amount of system information that are accessible through various system calls, files, interfaces, libraries, and/or the like. The system information can include monitoring and diagnostic data and/or other data that provide an indication of the operating system state. For example, the “/proc” file system contains system information such as system memory, device mounted, hardware configuration, and so on. Many commands/utilities simply make calls to files in the /proc file system directory to get data. Querying such data encounters two hurdles. Firstly, there is a different way to access and parse different types of data. Secondly, combining this data across machines or hosts requires yet more scripting. The “sysSQL” technology disclosed herein can bridge both of those hurdles by exposing a consistent, declarative interface, based on Structured Query Language (SQL) like language, to a wide range of system information, and by running queries, not just on a single machine, but across a cluster of machines. Via the sysSQL interface, users can ask questions like “what processes are using the most CPU across machines” or “how many network connections are there per machine” using SQL that they are familiar with, without having to write shell scripts.
The following description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding of the disclosure. However, in certain instances, well-known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure can be, but not necessarily are, references to the same embodiment; and, such references mean at least one of the embodiments.
Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments but not other embodiments.
The terms used in this specification generally have their ordinary meanings in the art, within the context of the disclosure, and in the specific context where each term is used. Certain terms that are used to describe the disclosure are discussed below, or elsewhere in the specification, to provide additional guidance to the practitioner regarding the description of the disclosure. For convenience, certain terms may be higighlighted, for example using italics and/or quotation marks. The use of highlighting has no influence on the scope and meaning of a term; the scope and meaning of a term is the same, in the same context, whether or not it is highighlighted. It will be appreciated that same thing can be said in more than one way.
Consequently, alternative language and synonyms may be used for any one or more of the terms discussed herein, nor is any special significance to be placed upon whether or not a term is elaborated or discussed herein. Synonyms for certain terms are provided. A recital of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any terms discussed herein is illustrative only, and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term. Likewise, the disclosure is not limited to various embodiments given in this specification.
Without intent to further limit the scope of the disclosure, examples of instruments, apparatus, methods and their related results according to the embodiments of the present disclosure are given below. Note that titles or subtitles may be used in the examples for convenience of a reader, which in no way should limit the scope of the disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the case of conflict, the present document, including definitions will control.
In some embodiments, the example environment includes a cluster manager server 115 that manages a cluster 135 of host machines (“hosts”) 135A-N and services. A cluster 135, as used herein, is a logical entity that includes a set of hosts (e.g., hosts 135A-N) that runs instances of services. As used herein a service is a category of functionality. Examples of services include the Hadoop Distributed File System (HDFS), MapReduce, YARN, and/or the like. A host (e.g., hosts 135A-N), as used herein, includes a physical or virtual machine that runs an instance of a role. As used herein a role is a category of functionality within a service. For example, the HDFS service can have the following example roles: a namenode, a secondary namenode, datanode and balancer. In a distributed system, one service runs on multiple hosts. In some embodiments, the cluster manager server 115 can be in communication with a repository 110 that can store data such as software, updates, policies etc., for deployment.
In some embodiments, the cluster manager server 115 can be in communication with a management service 125 that consists of a set of roles, for example, an activity monitor, a host monitor, a service monitor, reports manager, and/or the like. Configuration information (e.g., cluster configuration information), monitoring information and/or other information associated with a cluster can be stored in a database 130. The database 130 can include multiple logical databases running across one or more database servers. In some embodiments, the client 105 is an interface for interacting with the cluster manager server 115. The client 105 can include an admin console which is a web-based user interface (UI) or graphical user interface (GUI) that an administrator (“admin”) can use to interact with the cluster manager server 115 and manage one or more clusters (e.g., cluster 135). The client 105, in some embodiments, can also include one or more application programming interfaces (APIs) used by other applications to manage the cluster.
In some embodiments, a sysSQL module 120 can provide a sysSQL interface on the admin console using which admin users can run queries on multiple machines declaratively, using SQL or SQL-like language. In some other embodiments, the sysSQL module 120 can be hosted on the cluster manager server 115, while in other embodiments, the sysSQL module 120 can be hosted on another machine (e.g., a dedicated machine).
As described above, the sysSQL is a query language for ascertaining a current state of a distributed system having multiple hosts. The sysSQL operates on the same data sets that other system commands such as lsof, ps, mount, df, top, netstat, ifconfig, and so on operate on. The sysSQL however aggregates those data sets from multiple hosts and converts the aggregated data sets into a form that is queriable. For example, sysSQL can convert data returned by the /proc file system, which is a virtual file system, into a processes or threads table or dataset. An example excerpt of information returned when a “list processes” (e.g., ps aux) command is issued on a shell in a Linux machine is provided below.
The information for the above can in turn be looked up in /proc file system. Each of the above entries represent a process on that machine. Each process has content such as sub-directories containing data. For example, a process with a process id 10122 can have the following example content:
Data from each of these sub-directories provide system information that can be converted into a queryable form using the sysSQL technology.
In some embodiments, the sysSQL technology can use a schema in converting the “raw” system information (e.g., from the /proc file system) into a queryable form. Some examples of schemas that the sysSQL technology can use are provided below. For example, an example schema for creating a “processes” table can include the following columns:
An example schema for creating a “threads” table can have the following columns:
An example schema for creating a “connections” table can having the following columns:
An example schema for creating a “hosts” table can having the following columns:
An example schema for creating a “mounts” table can having the following columns:
An example schema for creating a “network interfaces” table can having the following columns:
In some embodiments, the user interface module 22( ) provides a user interface in the admin console for an admin user to input a query. An example of a query that can be submitted to the sysSQL module 120 via the user interface of the admin console is provided below.
The above query requests the number of connections per state per host. A query coordinator 210 can receive the query and determine that a “connections” table (or dataset) is needed to execute the query. Unlike a traditional database system, the “connections” table on which the query is to performed does not exist when the query is received, but is materialized, at runtime, in response to the query. The query coordinator 210 can request all the hosts 135A-N (or the hosts specifically identified in the query in some embodiments) in the cluster 135 to send their “connections” tables.
In some embodiments, each host has an agent (e.g., agent 250-A, 250-B, . . . or 250-N) installed on it or residing therein. The agent is responsible for providing data that is requested by the query coordinator 210. For example, an agent of each host receiving the request from the query coordinator 210 can create a “connections table.” In some embodiments, information necessary to create the “connections” table can be obtained by iterating through data provided by the /proc file system or similar interface or API. The /proc file system provides an interface to the internal data structure of the kernel and can return system-level information. In some embodiments, the agent can include adapters to format the system-level information into tables according to table schemas. For example, an adapter can use the schema for the “connections” table to format the system-level information and generate a “connections” table that has a uniform tabular format with the columns defined in the schema. In some embodiments, each agent can also be responsible for other functions such as starting and stopping processes, monitoring, or the like. In some embodiments, remote shells can be used to aggregate datasets from the hosts. In such embodiments, the hosts can include remote shell daemons to gather the datasets to return to the coordinator 210.
Each agent can return its “connections” table to the coordinator 210. The coordinator 210 can create a temporary database using the “connections” tables aggregated from the hosts so that the query execution engine 215 can execute the query on the temporary database. As soon as the query execution engine 215 receives results of the query, the query coordinator 210 can discard the temporary database. In some embodiments, the sysSQL module 120 can include a charting module that can generate graphical diagrams or visualizations using query results.
In some embodiments, the sysSQL module 120 can also handle timeouts associated with fetching datasets from the hosts and/or executing a query via a timeout module 218. In some embodiments, the timeout module 218 can be configured with timeout settings (e.g., fetch timeout, query timeout) specified by the user or by default. Having a timeout on how long the sysSQL module 120 should wait to get the underlying dataset means that the query engine 205 need not wait until it has the entire dataset to execute the query. Instead, after the timeout, the query engine 205 can execute the query on a partial dataset and provide results of the query to the user. This ensures a balance between interactivity (e.g., giving users a result quickly) and correctness (e.g., getting all the data). For example, consider a scenario where there are 100 hosts in a cluster and 3 of them are turned off or too busy to respond to the request from the query engine 205. The query engine 205 then waits until the request times out (e.g., based on an indication from the timeout module 218) and executes the query on the partial dataset (97% of the dataset in this example). In some embodiments, the sysSQL module 120 can also provide feedback to the user if part of the dataset is missing. Similarly, when the query times out (e.g., when the query takes longer than a 10 second query time timeout), the query engine 205 can provide a partial set of query results obtained so far.
A web-based interface 305 (e.g., an admin console) includes a text box 310 that receives a sysSQL query 312 as input. The sysSQL query 312 can be written using the SQL syntax in some embodiments. In this example, the sysSQL query 312 asks the question “how many processes are running in a cluster per user?” When the execute button 315 is selected, the sysSQL module 120 receives and executes the sysSQL query 312 to generate the results table 320, displayed on the web-based user interface 305 shown in
This same query can be run from the text-based shell interface of Unix/Linux-based systems and will provide similar results. For example, shell interface 405 of
In this case, the sysSQL query 512 requests the number of processes per host. Once again, the sysSQL query 512 is written in SQL using common SQL statements such as SELECT, FROM, GROUP BY and ORDER BY. However, there is no “processes” table to select from. This “processes” table encompassing all hosts in the cluster is created on the fly when the execute button is selected in order to execute the sysSQL query 512. The results from the execution of the sysSQL query 512 is displayed in the results table 520. The hostname column lists all the hosts and the count column lists the number of processes per host. In this example, there are 8 hosts and the first host is running the most number of processes.
In this example, the example sysSQL query 612 requests memory usage information for each host. The sysSQL query 612 is directed to the “processes” table with a column “mem_rss” for memory held by each process. When the execute button is selected, the sysSQL module 120 obtains “processes” table from the hosts and uses a temporary database to execute the sysSQL query 612 to select from the “processes” table, for each host, a sum of the memory held by all processes running on the host.
In this case, the sysSQL query 712 acts on a “mounts” table, the schema for which indicates that it has data fields that include hostname, device, mountpoint, fstyp, fstype, total, used and free. The sysSQL query 712 requests a list of hosts having less than 50 GB of disk space. When the execute button is selected, the sysSQL module obtains from each host its own “mounts” table and creates a temporary database including the “mounts” table from the hosts to execute the sysSQL query 712. The results from query can be displayed on a results table 720 shown in
The sysSQL query 812, as shown, is for determining which process is using the most CPU. The sysSQL query 812 includes instructions to take a first snapshot of the “processes” table, and after a pre-defined delay, takes a second snapshot of the same table. The sysSQL query 812 also includes instructions to compute the difference between the first and second snapshots to determine processes that use the most CPU. By having that delay between a first trip to fetch data from the underlying hosts and a second trip to fetch data from the underlying hosts, the sysSQL module 120 can compare data at different times to obtain data such as CPU usage over time. When the execute button is selected, the sysSQL module 120 executes the sysSQL query 812 on the data resulting from the computation of the difference between the first and second snapshots (e.g., CPU usage over time). Thus, unlike traditional databases where users query on data that exists, the sysSQL module 120 enables users to query on data that is ephemeral or transient (e.g., created for the purpose of executing the query). The sysSQL module 120 then outputs the results in a results table 820 shown in
The sysSQL query 912 includes instructions to determine per process connections and determine remote and local hosts that are paired, join the two sets of data and dedupe them to remove duplicates. The results of the sysSQL query 912 are displayed in the results table 920 of
In some embodiments, the web-based interface can include a visualize button or option that can take the results from the query and generate a graphical diagram visualizing the results.
The example method 1100 includes receiving query (e.g., sysSQL query) submitted by a user via a user interface (e.g., the web-based user interface or admin console, a text-based user interface) by the sysSQL module 120 (e.g., via the user interface module 220) at block 1105. The sysSQL module 120 parses the query 1110. At block 1115, the sysSQL module 120 can identify one or more datasets (e.g., tables) needed for the query and request the one or more datasets from each of the multiple hosts. In some embodiments, the requested dataset can be a processes table, a threads table, a connections table, a hosts table, a mounts table and/or a network interfaces table. At block 1120, the sysSQL module 120 can receive the requested dataset from each of the multiple hosts. An agent in each host generates the dataset in response to the request at block 1120a. For example, if a connections table was requested, then a connections table can be returned by an agent of each host. At block 1125, the sysSQL module 120 (e.g., via the query coordinator 210) can create a temporary database to temporarily store the received datasets. The sysSQL module 120 (e.g., via the query execution engine 215) can then execute the received query on the temporary database at block 1130. At block 1140, the sysSQL module 120 can receives results from the execution of the query. In some embodiments, the sysSQL module 120 can display the results on user interface at block 1145. In some embodiments, the sysSQL module 120 can discard the temporary database at block 1150 once the results have been received and/or displayed. In some embodiments, the results can be displayed in a tabular format (e.g., as shown in
In the example of
The processor may be, for example, a conventional microprocessor such as an Intel Pentium microprocessor or Motorola power PC microprocessor. One of skill in the relevant art will recognize that the terms “machine-readable (storage) medium” or “computer-readable (storage) medium” include any type of device that is accessible by the processor.
The memory is coupled to the processor by, for example, a bus. The memory can include, by way of example but not limitation, random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). The memory can be local, remote, or distributed.
The bus also couples the processor to the non-volatile memory and drive unit. The non-volatile memory is often a magnetic floppy or hard disk, a magnetic-optical disk, an optical disk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Some of this data is often written, by a direct memory access process, into memory during execution of software in the computer 1200. The non-volatile storage can be local, remote, or distributed. The non-volatile memory is optional because systems can be created with all applicable data available in memory. A typical computer system will usually include at least a processor, memory, and a device (e.g., a bus) coupling the memory to the processor.
Software is typically stored in the non-volatile memory and/or the drive unit. Indeed, for large programs, it may not even be possible to store the entire program in the memory. Nevertheless, it should be understood that for software to run, if necessary, it is moved to a computer readable location appropriate for processing, and for illustrative purposes, that location is referred to as the memory in this paper. Even when software is moved to the memory for execution, the processor will typically make use of hardware registers to store values associated with the software, and local cache. Ideally, this serves to speed up execution. As used herein, a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as “implemented in a computer-readable medium.” A processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.
The bus also couples the processor to the network interface device. The interface can include one or more of a modem or network interface. It will be appreciated that a modem or network interface can be considered to be part of the computer system. The interface can include an analog modem, isdn modem, cable modem, token ring interface, satellite transmission interface (e.g., “direct PC”), or other interfaces for coupling a computer system to other computer systems. The interface can include one or more input and/or output devices. The I/O devices can include, by way of example but not limitation, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other input and/or output devices, including a display device. The display device can include, by way of example but not limitation, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device. For simplicity, it is assumed that controllers of any devices not depicted in the example of
In operation, the computer system 1200 can be controlled by operating system software that includes a file management system, such as a disk operating system. One example of operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash., and their associated file management systems. Another example of operating system software with its associated file management system software is the Linux operating system and its associated file management system. The file management system is typically stored in the non-volatile memory and/or drive unit and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.
Some portions of the detailed description may be presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.
The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the methods of some embodiments. The required structure for a variety of these systems will appear from the description below. In addition, the techniques are not described with reference to any particular programming language, and various embodiments may thus be implemented using a variety of programming languages.
In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.
The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
While the machine-readable medium or machine-readable storage medium is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” and “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the presently disclosed technique and innovation.
In general, the routines executed to implement the embodiments of the disclosure, may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as “computer programs,” The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute elements involving the various aspects of the disclosure.
Moreover, while embodiments have been described in the context of fully functioning computers and computer systems, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms, and that the disclosure applies equally regardless of the particular type of machine or computer-readable media used to actually effect the distribution.
Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others, and transmission type media such as digital and analog communication links.
Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.
The above detailed description of embodiments of the disclosure is not intended to be exhaustive or to limit the teachings to the precise form disclosed above. While specific embodiments of, and examples for, the disclosure are described above for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.
The teachings of the disclosure provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various embodiments described above can be combined to provide further embodiments.
Any patents and applications and other references noted above, including any that may be listed in accompanying filing papers, are incorporated herein by reference. Aspects of the disclosure can be modified, if necessary, to employ the systems, functions, and concepts of the various references described above to provide yet further embodiments of the disclosure.
These and other changes can be made to the disclosure in light of the above Detailed Description. While the above description describes certain embodiments of the disclosure, and describes the best mode contemplated, no matter how detailed the above appears in text, the teachings can be practiced in many ways. Details of the system may vary considerably in its implementation details, while still being encompassed by the subject matter disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the disclosure should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the disclosure with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the disclosure to the specific embodiments disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of the disclosure encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the disclosure under the claims.
From the foregoing, it will be appreciated that specific embodiments of the sysSQL technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the spirit and scope of the embodiments. Accordingly, the embodiments are not limited except as by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5325522 | Vaughn | Jun 1994 | A |
| 5825877 | Dan et al. | Oct 1998 | A |
| 6542930 | Auvenshine | Apr 2003 | B1 |
| 6553476 | Ayaki et al. | Apr 2003 | B1 |
| 6598058 | Bird | Jul 2003 | B2 |
| 6651242 | Hebbagodi et al. | Nov 2003 | B1 |
| 6678828 | Pham et al. | Jan 2004 | B1 |
| 6687847 | Aguilera et al. | Feb 2004 | B1 |
| 6931530 | Pham et al. | Aug 2005 | B2 |
| 7031981 | DeLuca et al. | Apr 2006 | B1 |
| 7069497 | Desai | Jun 2006 | B1 |
| 7107323 | Hara et al. | Sep 2006 | B2 |
| 7143288 | Pham et al. | Nov 2006 | B2 |
| 7325041 | Hara et al. | Jan 2008 | B2 |
| 7392421 | Bloomstein et al. | Jun 2008 | B1 |
| 7487228 | Preslan et al. | Feb 2009 | B1 |
| 7496829 | Rubin et al. | Feb 2009 | B2 |
| 7620698 | Hara et al. | Nov 2009 | B2 |
| 7631034 | Haustein et al. | Dec 2009 | B1 |
| 7640512 | Appling | Dec 2009 | B1 |
| 7653668 | Shelat et al. | Jan 2010 | B1 |
| 7685109 | Ransil et al. | Mar 2010 | B1 |
| 7698321 | Hackworth | Apr 2010 | B2 |
| 7734961 | Almoustafa et al. | Jun 2010 | B2 |
| 7818313 | Tsimelzon et al. | Oct 2010 | B1 |
| 7831991 | Kiraly | Nov 2010 | B1 |
| 7937482 | Vermeulen et al. | May 2011 | B1 |
| 7970861 | Simitci et al. | Jun 2011 | B2 |
| 7984043 | Waas | Jul 2011 | B1 |
| 8024560 | Alten | Sep 2011 | B1 |
| 8069267 | Powers-Boyle et al. | Nov 2011 | B2 |
| 8108338 | Castro et al. | Jan 2012 | B2 |
| 8108771 | Chijiiwa et al. | Jan 2012 | B2 |
| 8306919 | Sakamura et al. | Nov 2012 | B2 |
| 8311980 | Saito et al. | Nov 2012 | B2 |
| 8468244 | Redlich et al. | Jun 2013 | B2 |
| 8484716 | Hodgson et al. | Jul 2013 | B1 |
| 8631403 | Soundararajan et al. | Jan 2014 | B2 |
| 8655939 | Redlich et al. | Feb 2014 | B2 |
| 8667267 | Garcia et al. | Mar 2014 | B1 |
| 8732674 | Agha | May 2014 | B1 |
| 8788815 | Garcia et al. | Jul 2014 | B1 |
| 8821602 | McAlister | Sep 2014 | B2 |
| 9047019 | Florendo | Jun 2015 | B2 |
| 9183284 | Barsness | Nov 2015 | B2 |
| 9235618 | Sharma | Jan 2016 | B2 |
| 9626411 | Chang | Apr 2017 | B1 |
| 20020055989 | Stringer-Calvert et al. | May 2002 | A1 |
| 20020073322 | Park et al. | Jun 2002 | A1 |
| 20020138762 | Horne | Sep 2002 | A1 |
| 20020174194 | Mooney et al. | Nov 2002 | A1 |
| 20030051036 | Wang et al. | Mar 2003 | A1 |
| 20030055868 | Fletcher et al. | Mar 2003 | A1 |
| 20030093633 | Thiesfeld et al. | May 2003 | A1 |
| 20040003322 | Collins et al. | Jan 2004 | A1 |
| 20040059728 | Miller et al. | Mar 2004 | A1 |
| 20040103166 | Bae et al. | May 2004 | A1 |
| 20040172421 | Saito et al. | Sep 2004 | A1 |
| 20040186832 | Jardin | Sep 2004 | A1 |
| 20050044311 | Lahiri et al. | Feb 2005 | A1 |
| 20050071708 | Bartfai et al. | Mar 2005 | A1 |
| 20050091244 | Marcotte | Apr 2005 | A1 |
| 20050138111 | Aton et al. | Jun 2005 | A1 |
| 20050171983 | Deo et al. | Aug 2005 | A1 |
| 20050182749 | Matsui | Aug 2005 | A1 |
| 20060020854 | Cardona et al. | Jan 2006 | A1 |
| 20060050877 | Nakamura | Mar 2006 | A1 |
| 20060143453 | Imamoto et al. | Jun 2006 | A1 |
| 20060156018 | Lauer et al. | Jul 2006 | A1 |
| 20060224784 | Nishimoto et al. | Oct 2006 | A1 |
| 20060247897 | Lin | Nov 2006 | A1 |
| 20070100913 | Sumner et al. | May 2007 | A1 |
| 20070113188 | Bales et al. | May 2007 | A1 |
| 20070136442 | Palma et al. | Jun 2007 | A1 |
| 20070177737 | Jung et al. | Aug 2007 | A1 |
| 20070180255 | Hanada et al. | Aug 2007 | A1 |
| 20070186112 | Perlin et al. | Aug 2007 | A1 |
| 20070226488 | Lin et al. | Sep 2007 | A1 |
| 20070234115 | Saika | Oct 2007 | A1 |
| 20070255943 | Kern et al. | Nov 2007 | A1 |
| 20070282988 | Bornhoevd et al. | Dec 2007 | A1 |
| 20080104579 | Hartmann | May 2008 | A1 |
| 20080140630 | Sato et al. | Jun 2008 | A1 |
| 20080168135 | Redlich et al. | Jul 2008 | A1 |
| 20080244307 | Dasari et al. | Oct 2008 | A1 |
| 20080256486 | Hagiwara | Oct 2008 | A1 |
| 20080263006 | Wolber et al. | Oct 2008 | A1 |
| 20080276130 | Almoustafa et al. | Nov 2008 | A1 |
| 20080307181 | Kuszmaul et al. | Dec 2008 | A1 |
| 20090013029 | Childress et al. | Jan 2009 | A1 |
| 20090177697 | Gao et al. | Jul 2009 | A1 |
| 20090259838 | Lin | Oct 2009 | A1 |
| 20090307783 | Maeda et al. | Dec 2009 | A1 |
| 20100008509 | Matsushita et al. | Jan 2010 | A1 |
| 20100010968 | Redlich et al. | Jan 2010 | A1 |
| 20100070769 | Shima et al. | Mar 2010 | A1 |
| 20100131817 | Kong et al. | May 2010 | A1 |
| 20100179855 | Chen et al. | Jul 2010 | A1 |
| 20100198972 | Umbehocker | Aug 2010 | A1 |
| 20100296652 | Nakayama et al. | Nov 2010 | A1 |
| 20100306286 | Chiu et al. | Dec 2010 | A1 |
| 20100325713 | Kurita et al. | Dec 2010 | A1 |
| 20100332373 | Crabtree et al. | Dec 2010 | A1 |
| 20110004620 | Butler | Jan 2011 | A1 |
| 20110055578 | Resch | Mar 2011 | A1 |
| 20110078549 | Thueringer et al. | Mar 2011 | A1 |
| 20110119328 | Simitci et al. | May 2011 | A1 |
| 20110179160 | Liu et al. | Jul 2011 | A1 |
| 20110228668 | Pillai et al. | Sep 2011 | A1 |
| 20110236873 | Bowers | Sep 2011 | A1 |
| 20110246816 | Hsieh et al. | Oct 2011 | A1 |
| 20110246826 | Hsieh et al. | Oct 2011 | A1 |
| 20110276396 | Rathod | Nov 2011 | A1 |
| 20110276495 | Varadarajan et al. | Nov 2011 | A1 |
| 20110302417 | Whillock et al. | Dec 2011 | A1 |
| 20110307534 | Peng et al. | Dec 2011 | A1 |
| 20120036146 | Annapragada | Feb 2012 | A1 |
| 20120036357 | Struik | Feb 2012 | A1 |
| 20120102072 | Jia et al. | Apr 2012 | A1 |
| 20120130874 | Mane et al. | May 2012 | A1 |
| 20120131341 | Mane et al. | May 2012 | A1 |
| 20130031240 | Byzek | Jan 2013 | A1 |
| 20130041872 | Aizman et al. | Feb 2013 | A1 |
| 20130054976 | Brown et al. | Feb 2013 | A1 |
| 20130218840 | Smith et al. | Aug 2013 | A1 |
| 20130304761 | Redlich et al. | Nov 2013 | A1 |
| 20140032528 | Mandre | Jan 2014 | A1 |
| 20140164353 | Shankar | Jun 2014 | A1 |
| 20140172898 | Aguilera | Jun 2014 | A1 |
| 20140188841 | Sun et al. | Jul 2014 | A1 |
| 20140195558 | Murthy et al. | Jul 2014 | A1 |
| 20140372533 | Fu | Dec 2014 | A1 |
| Entry |
|---|
| Babaoglu, O., and K. Marzullo, “Consistent global states of distributed systems: Fundamental concepts and mechanisms,” Distributed Systems, 53 pages, Jan. 1993. |
| Canada Exam Report for application No. 2843459, dated Aug. 5, 2014, 3 pages. |
| Chapter 25: Distributed Databases ED—Ramez Elmasri; Shamkant B Navathe (eds), Jan. 1, 2011 (Jan. 1, 2011), Fundamentals of Database Systems (Sixth Edition), Addison-Wesley, pp. 877-927. |
| Cheng, Security Attack Safe Modle and Cloud-Based One-Time Password Tokens Using Rubbing Encryption Algorithm, ACM, Jun. 2011, pp. 304-336. |
| Corbett et al., “Spanner: Google's Globally Distributed Database,” Transactions on Computer Systems (TOCS), vol. 31, No. 3, 14 pages, Aug. 2013. |
| Du et al., “A Pairwise Key Predistribution Scheme for Wireless Sensor Networks,” ACM, May 2005, vol. 8 Issue 2, pp. 228-258. |
| European Search Report for European Application No. 14157984.7, dated Jun. 6, 2014, 10 pages. |
| Exam Report for GB1403929.1, Applicant: Cloudera, Inc, dated May 2, 2014, 6 pages. |
| Kim et al., Design and Implementation of a Private and Public Key Crypto Processor and Its Application to a Security System, IEEE, Feb. 2004, vol. %0, Issue 1, pp. 214-224. |
| Koe et al., “A Study of Encryption Algorithm for RFID tag (SEED: 8 Rounds X 64 bit block),” IEEE, s008, pp. 672-677. |
| Kossmann, Donald, “The State of The Art in Distributed Query Processing,” ACM Computing Surveys, vol. 32, No. 4, Dec. 1, 2000, pp. 422-469. |
| Lamport, L., “Time, clocks, and the ordering of events in a distributed system,” Communications of the ACM, vol. 21, No. 7, pp. 558-565, Jul. 1978. |
| Non-Final Office Action for U.S. Appl. No. 13/362,695, dated Apr. 29, 2013, 23 pgs. |
| Stoller,S.D., “Detecting global predicates in distributed systems with clocks,” Distributed Computing, vol. 13, No. 2, pp. 85-98, Feb. 2000. |
| Tamer Dzsu et al: “Principles of Distributed Database Systems”, Principles of Distributed Database Systems, XX, XX, Jan. 1, 1991 (Jan. 1, 1991), pp. 74-93. |
| Beomseok Nam et al: “Spatial indexing of distributed multidimensional datasets”, Cluster Computing and the Grid, 2005. CCGRID 2005. IEEE International Symposium on Cardiff˜Wales, UK May 9-12, 2005, Piscataway, NJ, USA, IEEE, Piscataway, NJ, USA, vol. 2, May 9, 2005, pp. 743-750. |
| Number | Date | Country | |
|---|---|---|---|
| 20160103874 A1 | Apr 2016 | US |
