Patent Grant
12150129
A managed network (e.g., an enterprise network) often includes a large number of machines and devices configured to perform a wide variety of functions. The amount of computing assets and the amount of data generated and used by these computing assets scale rapidly with the size of the network. System and resource management on a network, such as collecting real-time information regarding systems and resources in the network and dynamically modifying and reallocating resources and data in the network, requires a substantial amount of computation and communication resources.
In a centrally managed network, a central management server is responsible for issuing requests (e.g., requests for status updates, system management operations, and network management operations, etc.) to the targeted destination machines in the network. These requests often take a long time to propagate through the network to the appropriate destination machines. These latencies make real-time management of the machines in the network difficult. For example, it typically takes more time to collect information about the status of machines coupled to the network than it takes for each machine's status to change. Frequently, by the time the requested status information is received by an administrator, such information has already become outdated. In addition, in a centralized managed network, the central server can quickly become overwhelmed by the communication load and becomes a management bottleneck. Furthermore, a centralized management scheme is expensive to implement and maintain.
Some conventional systems attempt to ameliorate the problems of a centralized management scheme by performing some degree of aggregation or processing of data at intermediate control levels, resulting in a hierarchical management structure between the network administrator and the endpoint machines (sometimes called end nodes or endpoint nodes). These systems also do not scale well. For example, for a network with 100,000 machines, it may still take several hours or more to report the status of those individual machines, or even of an aggregate thereof. In that timeframe, many machines would likely have changed their status, making the status report obsolete. In addition, these hierarchical management structures themselves are difficult and complex to create and maintain, and are prone to problems and failures.
Other conventional systems a mass information about network devices into one or more relatively large databases, so that network operators can query those databases for information about devices in the network. These systems also do not scale well. A relatively large network would produce enough data to swamp the operations of a database. One likely consequence is that only a small number of database queries can be made within resource limits of the database or its servers. Another problem with these systems is that their data tend, by the time answers are aggregated, not to reflect the true state of the devices in the network, and, because data is collected over time, the data no longer represent a consistent, snapshot view of those devices.
In one aspect, a method of managing a non-static collection of systems or machines in a network includes establishing one or more linear communication orbits (LCOs) in the network to communicate system, security, and network management requests and data. Machines coupled to the network self-organize themselves into the linear communication orbit(s), where each linear communication orbit is a non-static collection of machines running a respective communication protocol. Minimal intervention or instructions from a central server or an administrator is required for a machine to switch from one linear communication orbit to another (e.g., for a machine to change communication protocols or otherwise leave one LCO and integrate into another LCO). Individual machines are responsible the formation and continued maintenance of the linear communication orbit(s) when machines join and/or exit the network.
More specifically, in some embodiments, a method of managing a non-static collection of machines comprises executing one or more applications at a first client machine in the non-static collection of machines, where the first client machine runs a first communication protocol, and the non-static collection of machines includes a first linear communication orbit, the first linear communication orbit comprising a sequence of machines that run the first communication protocol, and a second linear communication orbit, the second linear communication orbit comprising a sequence of machines that run a second communication protocol distinct from the first communication protocol. The method comprises, at the first client machine: receiving an instruction from a server to install the second communication protocol; installing the second communication protocol; subsequent to installing the second communication protocol, submitting a registration request to the server; obtaining, from the server, contact information of a list of potential neighbors of the first client machine, where each potential neighbor comprises a machine in the non-static collection of machines that runs the second communication protocol; and in response to obtaining the contact information, proactively constructing and maintaining a respective local segment of the second linear communication orbit, the respective local segment connecting the first client machine to a live succeeding machine and a live preceding machine in the list of potential neighbors.
In some embodiments, installing the second communication protocol includes removing the first communication protocol from the first client machine.
In some embodiments, proactively constructing and maintaining a respective local segment of the second linear communication orbit further comprises: proactively establishing, in accordance with the second communication protocol, a respective propagation channel from the first client machine to the live succeeding machine upon detecting that said respective propagation channel to the live succeeding machine does not already exist; and proactively establishing, in accordance with the second communication protocol, a respective reporting channel from the first client machine to the live preceding machine upon detecting that said respective reporting channel to the live preceding machine does not already exist.
In some embodiments, the server maintains two or more lists of potential neighbors, where each potential neighbor in a respective list runs a same respective communication protocol (e.g., the potential neighbors in a first list all run the first communication protocol, and the potential neighbors in the second list all run the second communication protocol).
In some embodiments, installing the second communication protocol at the first client machine includes the first client machine decoupling from the first linear communication orbit.
In some embodiments, the first linear communication orbit includes one or more machines directly connected to a respective server, and the second linear communication orbit includes one or more machines directly connected to the same respective server.
In some embodiments (e.g., where, after constructing the local segment of the second linear communication orbit, the first client machine is the head machine in the second linear communication orbit), the live preceding machine of the first client machine is the server. In some embodiments (e.g., where, after constructing the local segment of the second linear communication orbit, the first client machine is the tail machine in the second linear communication orbit), the live succeeding machine of the first client machine is the server.
In some embodiments, the respective local segment includes the first client machine and four distinct communication channels each originating from or terminating at the first client machine, including respective receiving and reporting communication channels between the first client machine and the live preceding machine, and respective collection and propagation communication channels between the first client machine and the live succeeding machine.
Various embodiments may provide one or more of the following advantages:
In some embodiments, status information can be collected in substantially real-time. Assuming a network of 100,000 machines, an inquiry response time to a server from all the machines within an LCO can be in the order of seconds (e.g., 15 seconds) rather than hours or days, as in the case of conventional network management schemes.
In some embodiments, only a single server is used in the entire network (e.g., to coordinate communication with and between one or more LCOs), and system management infrastructure cost can be greatly reduced as compared to conventional management schemes.
Conventional technologies require massive amounts of data to flow over a wide area network, which is typically a bottleneck, and is typically slow and expensive, and not always available. In some embodiments of the present invention, most of the management communications can be accomplished with local area networks, which are fast, cheap, and always available.
In some embodiments, only a small number of connections need to be maintained. For example, in a network of 100,000 machines, according to conventional management techniques, a single management query may involve opening at least 200,000 connections—100,000 for the query to be sent from the server to every machine, and 100,000 for the answers back to the server. In contrast, in some embodiments of the present invention, only about two hundred connections may be needed, e.g., one hundred connections for the query to be sent from the server to the head machines on one hundred contiguous linear communication orbits, and one hundred connections for the answers back from the tail machines on the contiguous linear communication orbits.
Other embodiments and advantages are apparent to those skilled in the art in light of the descriptions and drawings in this specification.
Like reference numerals refer to corresponding parts throughout the drawings.
Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact, unless the context clearly indicates otherwise.
The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As used herein, the term “if” is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining,” or “in accordance with a determination that . . . ,” or “in response to determining,” or “upon detecting [the stated condition or event],” or “in response to detecting [the stated condition or event],” depending on the context.
In some embodiments, each linear communication orbit includes one or more machines running a respective communication protocol. In some embodiments, managed network 100 also includes server 108 that facilitates the creation and maintenance of the one or more contiguous segments 106. In some embodiments, each contiguous segment 106 includes one head machine (e.g., head machine 102b), one tail machine (e.g., tail machine 102f), and a sequence of zero or more intermediate client machines (e.g., intermediate machine(s) 102c-e) in between the head machine and the tail machine. In some embodiments, the head machine and tail machine of each contiguous segment 106 are connected to server 108, as shown in
In some embodiments, the methods described herein, or portions of those methods, are performed at a remote server (e.g., remote server 110) that is not part of managed network 100 and is optionally separated from managed network 100 by a firewall 104, see
In some embodiments, each machine 102 within managed network 100 is optionally configured to communicate with another machine within or outside of managed network 100 through various connection-oriented and/or connectionless protocols during their normal operations (e.g., user-level operations, such as emailing, Internet browsing, VOIP, database accessing, etc.) The present specification focuses on communication orbits that are established and maintained to facilitate system, security and/or network management operations ascribed to manual and programmed administrations of network 100. Examples of system, security and network management operations include: (1) collection of status information (e.g., bandwidth, load, availability, resource inventory, application status, machine type, date of last update, security breach, errors, etc.) from individual machines of the managed network; (2) issuance of system, security and network management commands (e.g., commands related to shut down, restart, failover, release of resources, change access authorizations, backup, deployment, quarantine, load balancing, etc.) for individual resources and/or machines on the managed network; (3) remotely controlled software installations and updates (e.g., enterprise software installations and updates, patch installations, anti-virus updates, database updates, file sharing, etc.); (4) detecting presence of particular malicious programs, code, script (e.g., virus, malware, security holes, etc.) on individual machines on the managed network; (5) removal of or disabling particular malicious programs, code, script (e.g., virus, malware, security holes, etc.) on individual machines on the managed network; (6) disabling or suspending suspicious or high-risk operations and activities (e.g., Internet or operating system activities of suspected virus programs, malware, etc.) on particular machines on the managed network; (7) detecting unmanaged machines connected to the managed network; (8) detecting data leakage (e.g., transmission of confidential information) from machines on the managed network to locations or machines outside of the managed network; (9) detecting connection or data transfer to/from removable data storage devices (e.g., memory stick, or other removable storage devices) from/to particular ports (e.g., a USB drive) of particular machines on the managed network. Other system, security and network management operations are possible and are apparent to those of ordinary skills in the art.
In some embodiments, as shown in
In some embodiments, a single dedicated server (e.g., server 108) is provided in the entire network (e.g., network 100). In some embodiments, server 108 is elected automatically from among all or a subset of machines 102 according to various predetermined election rules implemented on machines 102. In some embodiments, no server is needed to facilitate the formation and maintenance of the linear communication orbit(s) in network 100, as long as machines 102 have other ways of obtaining their respective ordinal positions (or a respective list of their potential neighbors) in the sequence of all machines currently coupled to network 100. For example, in some embodiments, each machine may store a static list of its potential neighbors rather than relying on a server to provide such a list, provided that only a substantially fixed set of machines can be coupled to the network. In other embodiments, a server (e.g., server 108 or 110) stores one or more lists of potential neighbors, where each list comprises machines that run a respective communication protocol. In such embodiments, the server periodically updates the one or more lists of potential neighbors (e.g., when one or more machines switches from one communication protocol to another, and/or when one or more machines is added to network 100 or removed from network 100).
As set forth in the background, some conventional techniques for system, security and network management rely on a hierarchical system management infrastructure. The hierarchical system management infrastructure includes one or more intermediate servers for funneling management inquiries and requests toward subsidiary intermediate servers and leaf machines of the hierarchical infrastructure. The one or more intermediate servers also aggregate responses and reports from subsidiary intermediate servers and the leaf machines. The hierarchical system management structure is rigid and difficult to modify in real-time. Failure of an intermediate server can cause large scale unavailability of many machines even if those machines do not suffer from any defects themselves. In addition, the hierarchical system management infrastructure has many potential bottlenecks at the intermediate servers, making management communications up and down the hierarchy inefficient, time consuming, and often untimely. Furthermore, the hierarchical system management infrastructure is costly to setup and to maintain both in terms of equipment and personnel.
In contrast, communication orbit(s) 106 shown in
In addition, as discussed in more detail later, each machine in the communication orbit(s) 106 shown in
In some embodiments, system, security and network management communications are propagated along the linear communication orbit(s) in one or both directions to collect system and network information, invoke system, security and network management actions, and/or push out system and network updates. In some embodiments, each machine 102 receiving a communication message from its upstream neighbor machine (or downstream neighbor machine) acts upon the message by providing an update to the message based on its local state or information, performing some aggregation of the information in the message, and/or forwarding the message to its downstream neighbor machine (or upstream neighbor machine) along the linear communication orbit. Essentially, each machine expends a small amount of resources to take on a small part of the duties of an intermediate management server without being overly burdened. More details on how the system, security and network management messages are propagated to and collected from machines 102 in network 100 through linear communication orbit(s) 106 are provided in the Applicants' prior application, U.S. patent application Ser. No. 12/412,623, filed Mar. 27, 2009, now U.S. Pat. No. 8,086,729, entitled “Distributed Statistical Detection of Network Problems and Causes,” which is hereby incorporated by reference in its entirety.
An important feature of the linear communication orbit(s) 106 is that they are automatically formed without global, continuous, and/or active intervention by any network administrative program or personnel. Each machine 102 joining network 100 is equipped with (or provided with) a set of predetermined organization rules. According to the set of predetermined organization rules, each machine 102 finds its neighboring machines and coordinates with these neighboring machines to self-organize into a local segment of the linear communication orbit. The local segments of adjacent machines overlap and fuse into a contiguous segment of the linear communication orbit.
In some embodiments, each machine 102 obtains the set of predetermined organization rules from server 108 as the machine first joins network 100. In some embodiments, each machine 102 queries server 108 to obtain pertinent information (e.g., a list of potential neighbors) that helps the machine to determine its ordinal position in the linear communication orbit. Each machine 102 then proactively contacts its potential predecessor and succeeding machines to identify its upstream and downstream neighbor machines and to establish a local segment of the linear communication orbit centered about said machine 102.
In some embodiments, the linear communication orbit organically grows or contracts as machines join and leave network 100, without global, continuous, and/or active intervention by any network administrative programs or personnel. Although all machines 102 implement the same set of predetermined organization rules and each machine only directly interacts with its immediate neighborhood machines, the predetermined organization rules are designed in a way that causes global coordination of the machines' independent local actions. The global coordination results in self-organization and automatic repair and maintenance of the linear communication orbit(s) 106.
Each linear communication orbit, 204, includes at least one propagation channel 220 traveling downstream from tail machine 208, through each and every intermediate machine in the respective linear communication orbit (e.g., 210-218) to server 108 (e.g., through head machine 206). Each linear communication orbit 204 also includes at least one propagation channel 222 traveling upstream from head machine (e.g., 206), through each and every intermediate machine in the respective linear communication orbit (e.g., 210-218) to server 108 (e.g., through tail machine 208). These propagation channels serve to transmit messages, instructions, data requests, etc. through each linear communication orbit.
As shown in
While intermediate machines 210A-218A and 210B-218B illustrate the configuration of linear communication orbits 204A and 204B, respectively, in some embodiments, linear communication orbit 204A or 204B includes only one intermediate machine, while in other embodiments, it includes dozens, hundreds, or even thousands of intermediate machines.
In some embodiments, input/output interface 306 includes a display and input devices such as a keyboard, a mouse, or a track-pad. However, in some embodiments, endpoint machine 102 does not include an input/output interface 306. In some embodiments, communication buses 310 include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. In some embodiments, non-persistent memory 304 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices. In some embodiments, persistent memory 303 includes CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In some embodiments, persistent memory 303 optionally includes one or more storage devices remotely located from the one or more processors 302. In some embodiments, persistent memory 303 and/or the non-volatile memory device(s) within the non-persistent memory 304, comprises non-transitory computer readable storage medium.
In some embodiments, memory 304 or alternatively the non-transitory computer readable storage medium stores the following programs, modules and data structures, instructions, or a subset thereof:
In some embodiments, input/output interface 406 includes a display and input devices such as a keyboard, a mouse, or a track-pad. However, in some embodiments, server system 108 does not include an input/output interface 406. In some embodiments, communication buses 410 include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. In some embodiments, non-persistent memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices. In some embodiments, persistent memory 403 includes CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In some embodiments, persistent memory 403 optionally includes one or more storage devices remotely located from the one or more processors 402. In some embodiments, persistent memory 403 and/or the non-volatile memory device(s) within the non-persistent memory 404, comprises non-transitory computer readable storage medium.
In some embodiments, memory 404 or alternatively the non-transitory computer readable storage medium stores the following programs, modules, data structures, instructions, or a subset thereof:
In some embodiments, input/output interface 506 includes a display and input devices such as a keyboard, a mouse, or a track-pad. In some embodiments, communication buses 510 include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. In some embodiments, non-persistent memory 504 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices. In some embodiments, persistent memory 503 includes CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. In some embodiments, persistent memory 503 optionally includes one or more storage devices remotely located from the one or more processors 502. In some embodiments, persistent memory 503 and/or the non-volatile memory device(s) within the non-persistent memory 504, comprises a non-transitory computer readable storage medium.
In some embodiments, memory 504 or alternatively the non-transitory computer readable storage medium of memory 504, stores the following programs, modules, data structures, instructions, or a subset thereof:
In some embodiments, the second communication protocol differs from the first communication protocol in one or more of the following respects. In some embodiments, while the first communication protocol has a fixed or predefined set of message types and all messages sent using the first communication protocol must be one of those message types, in the second communication protocol, messages are sent using protocol buffers to define the structure of each message, and the set of message types that can be sent is extensible, meaning that new message types can be added without causing devices or nodes that do not have instructions for processing messages having the new message types to fail or otherwise improperly process messages having the new message types.
In some embodiments, when a machine using the first communication protocol receives a message having an unknown message type (e.g., a message type that is not one of the fixed or predefined set of message types defined by library or set 321 (
In some embodiments, when a respective machine using the second communication protocol receives a message, and the received message is of a known type, but one or more sub-messages of the received message does not match any of the message type definitions 321 (
In some embodiments, messages sent using the first communication protocol include one or more message integrity verification values (so that the message content of such messages can be validated), but are either not encrypted, or are not encrypted in a way that prevents machines or nodes on which the first protocol is installed from being able to read the content of messages sent between other nodes in the network. As a result, messages sent using the first communication protocol can be read by machines or nodes not intended to be the recipients of such messages. However, in some embodiments, each respective message sent using the second communication protocol is encrypted with a session key that is unique to or specific to the sender-receiver pair of machines or nodes between which the respective message is sent. For example, the session keys for each pair of machines that exchange messages may be determined using a mutual authentication method, such as TLS 1.2.
In some embodiments, messages sent using the first communication protocol include one or more message integrity verification values determined using a predefined or fixed public-private key pair, without any mechanism for changing the public-private key pair used for generating and verifying the message integrity verification values in messages. On the other hand, the second communication protocol includes the ability to “rotate among” or otherwise change the keys used to generate and validate message integrity verification values in messages sent using the second communication protocol. In particular, messages sent using the second communication protocol include one or more message integrity verification values as well as one or more indicators of a validation key or validation public-private key-pair used to generate and validate the one or more message integrity verification values. In some embodiments, machines using the second communication protocol change, at predefined times, such as monthly or at other time intervals, the key (e.g., a public or private digital signature generation key) used to generate the message integrity verification values included in sent messages, and use a corresponding key (e.g., a private or public validation key corresponding to the digital signature generation key) to validate the message integrity verification values included in received messages.
In some embodiments, initially, first client machine 102 is connected to (e.g., is a head, tail, or intermediate machine in) the first linear communication orbit 204A and runs communication protocol 602 for communications with server 108 and neighboring machines in the first linear communication orbit 204A. Furthermore, prior to performing the method shown in
In the flow diagram, the first client machine receives (606) instruction from a server (e.g., server 108) to install the second communication protocol 604. Upon receipt of the instruction, machine 102 installs (608) the second communication protocol 604.
In some embodiments, subsequent to receiving the aforementioned instruction from the server, machine 102 removes (610) the first communication protocol and decouples (612) from the first linear communication orbit 204A. In some embodiments, installing (608) the second communication protocol 604 includes removing (610) the first communication protocol 604 from machine 102 (the first client machine). In some embodiments, installing (608) the second communication protocol 604 includes installing a library of message type definitions 321 (
In some embodiments, machine 102 automatically decouples 612 from the first linear communication orbit 204A upon removing the first communication protocol, since removing the first communication protocol results in machine 102 being unable to communicate over the first linear communication orbit with its (former) neighbors. In some embodiments, installing 608 the second communication protocol 604 at the first client machine (e.g., endpoint machine 102) includes the first client machine decoupling from the first linear communication orbit 204A. One of skill in the art will realize that the operation ordering described here is merely for illustration purposes and that other orders of operation are possible for machine 102 regarding the installation of the second communication protocol 604, removal of the first communication protocol 602, decoupling from the first linear communication orbit 204A, registering machine 102 with the server as a machine running the second communication protocol, joining a second linear communication orbit (e.g., by constructing and maintaining segments of the second linear communication orbit that connect machine 102 to upstream and downstream neighbors), etc.
Subsequent to installing the second communication protocol, the first client machine 102 submits 614 a registration request to the server 108. In some embodiments, the registration message includes the first client machine's own unique identifier (e.g., it's IP address or other machine identifier). In some embodiments, the registration message represents a request for information regarding the registering machine's potential neighbors. In some embodiments, machine 102 sends the registration message to server 108 using non-LCO communications module 324 (
In some embodiments, in response to the registration request, server 108 registers the first client machine 102 as a machine using the second communication protocol, and furthermore the server 108 sends, and first client machine 102 receives (or obtains) from server 108 contact information of a list 616 of potential neighbors of the first client machine (e.g., in response to machine 102 sending the registration request to the sever, server 108 then provides the list of potential neighbors to machine 102). The list provides one or more potential upstream neighbors (e.g., live preceding machines) and one or more potential downstream neighbors (e.g., live succeeding machines) that have been identified by the server for the new machine. Each potential neighbor machine in the list comprises a machine 102 in the non-static collection of machines that runs the second communication protocol 604 (e.g., each of the potential neighbors is a machine that is already part of the sequence of machines in the second linear communication orbit 204B). In some embodiments, the server identifies the potential neighbors and sorts them according to respective unique identifiers of all machines currently known to be present in the network and their sequential order relative to the respective identifier of the new machine. In some embodiments, server 108 provides this sorted list to machine 102 in response to receiving a registration request from the first client machine 102.
In some embodiments, the server 108 maintains two or more lists of potential neighbors, one for each distinct protocol, where each potential neighbor machine in a respective list runs the same respective communication protocol (e.g., each list of potential neighbors is a list of live machines in a particularly linear communication orbit, where all the machines in the list are machines running the same communication protocol). In some embodiments, a first list maintained by server 108 identifies machines that run the first communication protocol 602, and a second list maintained by server 108 identifies machines that run the second communication protocol 604.
In response to obtaining the contact information, the first client machine proactively constructs and maintains 618 a respective segment of the second linear communication orbit 204B. The respective local segment connects the first client machine 102 to a live succeeding machine and a live preceding machine in the list of potential neighbors.
In some embodiments, the proactively constructing and maintaining 618 include the first client machine 102 proactively establishing (620), in accordance with the second communication protocol, a respective propagation channel from the first client machine to the live succeeding machine upon detecting that said respective propagation channel to the live succeeding machine does not already exit. The first client machine 102 also proactively establishes (622), in accordance with the second communication protocol, a respective reporting channel to the live preceding machine upon detecting that said respective reporting channel to the live preceding machine does not already exist.
Subsequent to sending to the first client machine the contact information, server 108 sends instructions to and receives responses from the first client machine via the second linear communication orbit, using the second communication protocol.
Additional details of proactively constructing a respective segment of the second linear communication orbit are provided below. Upon receipt of the list of potential neighbors from server 108, the first client machine 102 proceeds to search for its nearest live succeeding machine (i.e., its closest downstream neighbor in the second linear communication orbit 204B) based on the potential neighbor contact information obtained from server 108. In some embodiments, the first client machine runs down the list of potential downstream neighbors one by one, until a live succeeding machine (e.g., a machine that is currently part of the second linear communication orbit and is located downstream of a proposed insertion location in the second linear communication orbit for the first client machine 102) is found. In some embodiments, the list of potential downstream neighbors is sorted according to increasing values of their respective identifiers (e.g., IP addresses). Sometimes, the new machine may encounter one or more inactive machines in the list of potential downstream neighbors before finding the nearest live succeeding machine.
Once the first client machine has identified its downstream neighbor the first client machine proactively establishes (620), in accordance with the second communication protocol, a respective propagation channel from the first client to the live succeeding machine (e.g., such as propagation channels 222 shown in
In some embodiments, after (or, alternatively, before) the first client machine 102 establishes the respective propagation channel to its live succeeding machine, the first client machine searches for its nearest live preceding machine. In some embodiments, the first client machine 102 starts the search for its nearest live preceding machine independently of whether the first client machine has already established the respective propagation channel to its live succeeding machine. In some embodiments, the first client machine 102 runs down the list of potential upstream neighbors one by one, until a live upstream machine is found. In some embodiments, the list of potential upstream neighbors is sorted according to decreasing values of their respective identifiers (e.g., IP addresses). Sometimes, the first client machine 102 may encounter one or more inactive machines in the list of potential upstream neighbors before finding the upstream neighbor. The nearest live preceding machine responds to the contact from the first client machine 102.
Once the first client machine 102 has identified its upstream neighbor, the first client machine proactively establishes a respective propagation channel to the upstream neighbor. The predecessor machine accepts the respective propagation channel request from the first client machine, and the respective propagation channel is thus established. The respective propagation channel serves as a reporting channel for the first client machine (e.g., for reporting results generated in response to received commands or queries), and a collection channel for the upstream neighbor of the first client machine (e.g., for receiving results generated by the first client machine, and possibly other machines downstream of the first client machine, in response to received commands or queries).
The respective local segments connect the first client machine 102 to a live succeeding machine, located downstream of the first client machine in a respective linear communication orbit, and a live preceding machine in the list of potential neighbors, located upstream of the first client machine in the respective linear communication orbit.
In some embodiments, the respective local segment includes the first client machine and four distinct communication channels each originating from or terminating at the first client machine 102. The four distinct communication channels include respective receiving and reporting communication channels between the first client machine and the live preceding machine and also respective collection and propagation communication channels between the first client machine and the live succeeding machine.
In some embodiments, the live preceding machine of the first client machine 102 is the server 108 (e.g., the first client machine is or becomes the head machine of a linear communication orbit). In some embodiments, the live succeeding machine of the first client machine 102 is the server 108 (e.g., the first client machine is or becomes the tail machine of a linear communication orbit). In some embodiments, the live preceding machine and the live succeeding machine of the first client machine 102 are both the server 108 (e.g., the first client machine is a singleton machine connected to server).
From the descriptions above, it can be seen that the server's duties with respect to the formation and maintenance of linear communication orbits are light-weight. Essentially, the server accepts connection requests of new machines, and each new machine establishes a respective propagation channel and a backward connection to the server when the new machine first joins the network. The server also keeps a record of all the machines and their communication protocols that have registered with the server. If a machine has not registered with the server, or alternatively has not affirmed its continued participation in communications with its neighboring machines, for a sufficient amount of time, the server removes that machine from the list of known live machines in the network. The server sorts the list of known live machines according to respective unique identifiers of the machines, such that for each given unique identifier, the server can identify a list of one or more potential neighbor machines in the upstream and the downstream direction (e.g., decreasing value and increasing value of the identifiers), respectively. The server also accepts requests for opening new forward and backward connections and terminating existing forward and backward connections from machines (e.g., head machines, tail machines, new machines, and singleton machines) in the network. In some embodiments, the server also maintains an up-to-date list of head machines so that when a new request is issued (for system and security management operations, etc.) it sends the request to all the head machines in the network, which they download and propagate within their respective segments of the network. As a result, the head machines of the linear communication orbits in the network download and propagate the request immediately. Furthermore, because in a typical network the machine-to-machine communications channels are wide open, overall operations latencies of this system (e.g., from transmission of a request by a server to end point machines throughout the network to receipt by the server of acknowledgments or answers from the endpoint machines) are measured in seconds, not hours or days.
The foregoing description has been provided with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to be limiting to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles disclosed and their practical applications, to thereby enable others to best utilize the disclosure and various embodiments with various modifications as are suited to the particular use contemplated.
This application is a continuation of U.S. application Ser. No. 17/521,686, filed Nov. 8, 2021, which is a continuation of U.S. application Ser. No. 16/854,844, filed Apr. 21, 2020, now U.S. Pat. No. 11,172,470, which claims priority to U.S. Provisional Application Ser. No. 62/873,796, filed Jul. 12, 2019, and is a continuation-in-part of U.S. application Ser. No. 16/194,240, filed Nov. 16, 2018, now U.S. Pat. No. 10,674,486, which is a continuation of U.S. application Ser. No. 15/004,757, filed Jan. 22, 2016, now U.S. Pat. No. 10,136,415, which is a continuation of U.S. application Ser. No. 13/797,946, filed Mar. 12, 2013, now U.S. Pat. No. 9,246,977, which claims under 35 U.S.C. 119(e) the benefit of U.S. Provisional Application Ser. No. 61/774,106, filed Mar. 7, 2013, and U.S. Provisional Application Ser. No. 61/745,236, filed Dec. 21, 2012, all of which are incorporated by reference herein in their entireties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5220596 | Patel | Jun 1993 | A |
| 5842202 | Kon | Nov 1998 | A |
| 5949755 | Upadhya et al. | Sep 1999 | A |
| 6049828 | Dev et al. | Apr 2000 | A |
| 6226493 | Leopold et al. | May 2001 | B1 |
| 6615213 | Johnson | Sep 2003 | B1 |
| 6879979 | Hindawi et al. | Apr 2005 | B2 |
| 6885644 | Knop et al. | Apr 2005 | B1 |
| 6959000 | Lee | Oct 2005 | B1 |
| 7043550 | Knop et al. | May 2006 | B2 |
| 7096503 | Magdych et al. | Aug 2006 | B1 |
| 7120693 | Chang et al. | Oct 2006 | B2 |
| 7225243 | Wilson | May 2007 | B1 |
| 7240044 | Chaudhuri et al. | Jul 2007 | B2 |
| 7299047 | Dolan et al. | Nov 2007 | B2 |
| 7483430 | Yuan et al. | Jan 2009 | B1 |
| 7555545 | McCasland | Jun 2009 | B2 |
| 7600018 | Maekawa et al. | Oct 2009 | B2 |
| 7698453 | Samuels et al. | Apr 2010 | B2 |
| 7720641 | Alagappan et al. | May 2010 | B2 |
| 7761557 | Fellenstein et al. | Jul 2010 | B2 |
| 7769848 | Choy et al. | Aug 2010 | B2 |
| 7844687 | Gelvin et al. | Nov 2010 | B1 |
| 8078668 | Moreau | Dec 2011 | B2 |
| 8086729 | Hindawi et al. | Dec 2011 | B1 |
| 8139508 | Roskind | Mar 2012 | B1 |
| 8185612 | Arolovitch et al. | May 2012 | B1 |
| 8185615 | McDysan et al. | May 2012 | B1 |
| 8271522 | Mehul et al. | Sep 2012 | B2 |
| 8392530 | Manapragada et al. | Mar 2013 | B1 |
| 8477660 | Lee et al. | Jul 2013 | B2 |
| 8504879 | Poletto et al. | Aug 2013 | B2 |
| 8510562 | Ramakrishnan et al. | Aug 2013 | B2 |
| 8650160 | Beatty et al. | Feb 2014 | B1 |
| 8677448 | Kauffman et al. | Mar 2014 | B1 |
| 8813228 | Magee et al. | Aug 2014 | B2 |
| 8819769 | Van et al. | Aug 2014 | B1 |
| 8885521 | Wang et al. | Nov 2014 | B2 |
| 8903973 | Hindawi et al. | Dec 2014 | B1 |
| 8904039 | Hindawi et al. | Dec 2014 | B1 |
| 8972566 | Hindawi et al. | Mar 2015 | B1 |
| 9009827 | Albertson et al. | Apr 2015 | B1 |
| 9059961 | Hindawi et al. | Jun 2015 | B2 |
| 9104794 | Zakonov et al. | Aug 2015 | B2 |
| 9246977 | Hindawi et al. | Jan 2016 | B2 |
| 9576131 | Tuvell et al. | Feb 2017 | B2 |
| 9609007 | Rivlin et al. | Mar 2017 | B1 |
| 9667738 | Hindawi et al. | May 2017 | B2 |
| 9716649 | Bent et al. | Jul 2017 | B2 |
| 9729429 | Hindawi et al. | Aug 2017 | B2 |
| 9769037 | Hindawi et al. | Sep 2017 | B2 |
| 9769275 | Hindawi et al. | Sep 2017 | B2 |
| 9800603 | Sidagni | Oct 2017 | B1 |
| 9910752 | Lippincott et al. | Mar 2018 | B2 |
| 9973525 | Roturier et al. | May 2018 | B1 |
| 9985982 | Bartos et al. | May 2018 | B1 |
| 9998955 | MacCarthaigh | Jun 2018 | B1 |
| 10015185 | Kolman et al. | Jul 2018 | B1 |
| 10095864 | Hunt et al. | Oct 2018 | B2 |
| 10111208 | Hindawi et al. | Oct 2018 | B2 |
| 10136415 | Hindawi et al. | Nov 2018 | B2 |
| 10148536 | Hindawi et al. | Dec 2018 | B2 |
| 10261770 | Devagupthapu et al. | Apr 2019 | B2 |
| 10372904 | Hunt et al. | Aug 2019 | B2 |
| 10412188 | Hindawi et al. | Sep 2019 | B2 |
| 10482242 | Hunt et al. | Nov 2019 | B2 |
| 10484429 | Fawcett et al. | Nov 2019 | B1 |
| 10498744 | Hunt et al. | Dec 2019 | B2 |
| 10649870 | Lippincott et al. | May 2020 | B1 |
| 10674486 | Hindawi et al. | Jun 2020 | B2 |
| 10708116 | Hindawi et al. | Jul 2020 | B2 |
| 10795906 | Teubner | Oct 2020 | B1 |
| 10824729 | Hoscheit et al. | Nov 2020 | B2 |
| 10841365 | White et al. | Nov 2020 | B2 |
| 10873645 | Freilich et al. | Dec 2020 | B2 |
| 10929345 | Stoddard et al. | Feb 2021 | B2 |
| 11032298 | Robbins et al. | Jun 2021 | B1 |
| 11100199 | Subramaniam | Aug 2021 | B2 |
| 11151246 | Davis et al. | Oct 2021 | B2 |
| 11153383 | Richards et al. | Oct 2021 | B2 |
| 11172470 | Guieu et al. | Nov 2021 | B1 |
| 11258654 | Hindawi et al. | Feb 2022 | B1 |
| 11277489 | Freilich et al. | Mar 2022 | B2 |
| 11301568 | Dargude et al. | Apr 2022 | B1 |
| 11343355 | Goela et al. | May 2022 | B1 |
| 11372938 | Stoddard et al. | Jun 2022 | B1 |
| 11461208 | Lippincott et al. | Oct 2022 | B1 |
| 11563764 | Hoscheit et al. | Jan 2023 | B1 |
| 11609835 | Varga et al. | Mar 2023 | B1 |
| 11700303 | Richards et al. | Jul 2023 | B1 |
| 11711810 | Guieu et al. | Jul 2023 | B1 |
| 11777981 | Hoscheit et al. | Oct 2023 | B1 |
| 11809294 | Lippincott et al. | Nov 2023 | B1 |
| 11831670 | Molls et al. | Nov 2023 | B1 |
| 11886229 | Goela et al. | Jan 2024 | B1 |
| 11914495 | Varga et al. | Feb 2024 | B1 |
| 11956335 | Goela et al. | Apr 2024 | B1 |
| 20010056461 | Kampe et al. | Dec 2001 | A1 |
| 20020007404 | Vange et al. | Jan 2002 | A1 |
| 20020042693 | Kampe et al. | Apr 2002 | A1 |
| 20020073086 | Thompson et al. | Jun 2002 | A1 |
| 20020099952 | Lambert et al. | Jul 2002 | A1 |
| 20020198867 | Lohman et al. | Dec 2002 | A1 |
| 20030101253 | Saito et al. | May 2003 | A1 |
| 20030120603 | Kojima et al. | Jun 2003 | A1 |
| 20030131044 | Nagendra et al. | Jul 2003 | A1 |
| 20030212676 | Bruce et al. | Nov 2003 | A1 |
| 20030212821 | Gillies et al. | Nov 2003 | A1 |
| 20040037374 | Gonikberg | Feb 2004 | A1 |
| 20040044727 | Abdelaziz et al. | Mar 2004 | A1 |
| 20040044790 | Loach et al. | Mar 2004 | A1 |
| 20040054723 | Dayal et al. | Mar 2004 | A1 |
| 20040054889 | Pitsos | Mar 2004 | A1 |
| 20040064522 | Zhang et al. | Apr 2004 | A1 |
| 20040076164 | Vanderveen et al. | Apr 2004 | A1 |
| 20040190085 | Silverbrook et al. | Sep 2004 | A1 |
| 20050004907 | Bruno et al. | Jan 2005 | A1 |
| 20050053000 | Oliver et al. | Mar 2005 | A1 |
| 20050108356 | Rosu et al. | May 2005 | A1 |
| 20050108389 | Kempin et al. | May 2005 | A1 |
| 20050195755 | Senta et al. | Sep 2005 | A1 |
| 20060039371 | Castro et al. | Feb 2006 | A1 |
| 20060128406 | MacArtney | Jun 2006 | A1 |
| 20060282505 | Hasha et al. | Dec 2006 | A1 |
| 20070005738 | Alexion-Tiernan et al. | Jan 2007 | A1 |
| 20070171844 | Loyd et al. | Jul 2007 | A1 |
| 20070211651 | Ahmed et al. | Sep 2007 | A1 |
| 20070230482 | Shim et al. | Oct 2007 | A1 |
| 20070261051 | Porter et al. | Nov 2007 | A1 |
| 20080082628 | Rowstron et al. | Apr 2008 | A1 |
| 20080133582 | Andersch et al. | Jun 2008 | A1 |
| 20080258880 | Smith et al. | Oct 2008 | A1 |
| 20080263031 | George et al. | Oct 2008 | A1 |
| 20080288646 | Hasha et al. | Nov 2008 | A1 |
| 20090125639 | Dam et al. | May 2009 | A1 |
| 20090271360 | Bestgen et al. | Oct 2009 | A1 |
| 20090285204 | Gallant et al. | Nov 2009 | A1 |
| 20090319503 | Mehul et al. | Dec 2009 | A1 |
| 20090328115 | Malik | Dec 2009 | A1 |
| 20100011060 | Hilterbrand et al. | Jan 2010 | A1 |
| 20100070570 | Lepeska | Mar 2010 | A1 |
| 20100085948 | Yu et al. | Apr 2010 | A1 |
| 20100094862 | Bent et al. | Apr 2010 | A1 |
| 20100154026 | Chatterjee et al. | Jun 2010 | A1 |
| 20100296416 | Lee et al. | Nov 2010 | A1 |
| 20100306252 | Jarvis et al. | Dec 2010 | A1 |
| 20110099562 | Nandy et al. | Apr 2011 | A1 |
| 20110231431 | Kamiwada et al. | Sep 2011 | A1 |
| 20110271319 | Venable, Sr. | Nov 2011 | A1 |
| 20110299455 | Ordentlich et al. | Dec 2011 | A1 |
| 20120053957 | Atkins et al. | Mar 2012 | A1 |
| 20120110183 | Miranda et al. | May 2012 | A1 |
| 20120221692 | Steiner et al. | Aug 2012 | A1 |
| 20120269096 | Roskind | Oct 2012 | A1 |
| 20120330700 | Garg et al. | Dec 2012 | A1 |
| 20130110931 | Kim et al. | May 2013 | A1 |
| 20130170336 | Chen et al. | Jul 2013 | A1 |
| 20130212296 | Goel et al. | Aug 2013 | A1 |
| 20130276053 | Hugard et al. | Oct 2013 | A1 |
| 20130326494 | Nunez | Dec 2013 | A1 |
| 20140075505 | Subramanian | Mar 2014 | A1 |
| 20140101133 | Carston et al. | Apr 2014 | A1 |
| 20140149557 | Lohmar et al. | May 2014 | A1 |
| 20140164290 | Salter | Jun 2014 | A1 |
| 20140164552 | Kim et al. | Jun 2014 | A1 |
| 20140181247 | Hindawi et al. | Jun 2014 | A1 |
| 20140181295 | Hindawi et al. | Jun 2014 | A1 |
| 20140244727 | Kang et al. | Aug 2014 | A1 |
| 20140279044 | Summers et al. | Sep 2014 | A1 |
| 20140280280 | Singh | Sep 2014 | A1 |
| 20140282586 | Shear et al. | Sep 2014 | A1 |
| 20140372533 | Fu et al. | Dec 2014 | A1 |
| 20140375528 | Ling | Dec 2014 | A1 |
| 20150080039 | Ling et al. | Mar 2015 | A1 |
| 20150149624 | Hindawi et al. | May 2015 | A1 |
| 20150163121 | Mahaffey et al. | Jun 2015 | A1 |
| 20150172228 | Zalepa et al. | Jun 2015 | A1 |
| 20150199511 | Faile et al. | Jul 2015 | A1 |
| 20150199629 | Faile et al. | Jul 2015 | A1 |
| 20150256575 | Scott | Sep 2015 | A1 |
| 20150302458 | Dides et al. | Oct 2015 | A1 |
| 20150312335 | Ying et al. | Oct 2015 | A1 |
| 20150372911 | Yabusaki et al. | Dec 2015 | A1 |
| 20150373043 | Wang et al. | Dec 2015 | A1 |
| 20150378743 | Zellermayer et al. | Dec 2015 | A1 |
| 20160034692 | Singler | Feb 2016 | A1 |
| 20160080408 | Coleman et al. | Mar 2016 | A1 |
| 20160119251 | Solis et al. | Apr 2016 | A1 |
| 20160255142 | Hunt et al. | Sep 2016 | A1 |
| 20160255143 | Hunt et al. | Sep 2016 | A1 |
| 20160269434 | Divalentin et al. | Sep 2016 | A1 |
| 20160286540 | Hindawi et al. | Sep 2016 | A1 |
| 20160352588 | Subbarayan et al. | Dec 2016 | A1 |
| 20160360006 | Hopkins et al. | Dec 2016 | A1 |
| 20160378450 | Fu et al. | Dec 2016 | A1 |
| 20170093915 | Ellis et al. | Mar 2017 | A1 |
| 20170118074 | Feinstein et al. | Apr 2017 | A1 |
| 20170133843 | McNeill-McCallum et al. | May 2017 | A1 |
| 20170257432 | Fu et al. | Sep 2017 | A1 |
| 20170286690 | Chari et al. | Oct 2017 | A1 |
| 20170346824 | Mahabir et al. | Nov 2017 | A1 |
| 20180013768 | Hunt et al. | Jan 2018 | A1 |
| 20180039486 | Kulkarni et al. | Feb 2018 | A1 |
| 20180074483 | Cruz | Mar 2018 | A1 |
| 20180074796 | Alabes et al. | Mar 2018 | A1 |
| 20180191747 | Nachenberg et al. | Jul 2018 | A1 |
| 20180191766 | Holeman et al. | Jul 2018 | A1 |
| 20180267794 | Atchison et al. | Sep 2018 | A1 |
| 20180351792 | Hunter et al. | Dec 2018 | A1 |
| 20180351793 | Hunter et al. | Dec 2018 | A1 |
| 20180375892 | Ganor | Dec 2018 | A1 |
| 20190081981 | Bansal | Mar 2019 | A1 |
| 20190096217 | Pourmohammad et al. | Mar 2019 | A1 |
| 20190138512 | Pourmohammad et al. | May 2019 | A1 |
| 20190260638 | Yocam et al. | Aug 2019 | A1 |
| 20190280867 | Kurian | Sep 2019 | A1 |
| 20190319987 | Levy et al. | Oct 2019 | A1 |
| 20190361843 | Stoddard et al. | Nov 2019 | A1 |
| 20200028890 | White et al. | Jan 2020 | A1 |
| 20200053072 | Glozman et al. | Feb 2020 | A1 |
| 20200195693 | Price et al. | Jun 2020 | A1 |
| 20200198867 | Nakamichi | Jun 2020 | A1 |
| 20200202007 | Nagaraja et al. | Jun 2020 | A1 |
| 20200304536 | Mahabir et al. | Sep 2020 | A1 |
| 20210027401 | Hovhannisyan et al. | Jan 2021 | A1 |
| 20210218711 | Biran et al. | Jul 2021 | A1 |
| 20230036694 | Steven | Feb 2023 | A1 |
| 20230360040 | Childe et al. | Nov 2023 | A1 |
| Number | Date | Country |
|---|---|---|
| 1553747 | Jul 2005 | EP |
| 2493118 | Aug 2012 | EP |
| Entry |
|---|
| Non-Final Office Action, U.S. Appl. No. 15/215,474, Sep. 10, 2018, 10 pages. |
| Non-Final Office Action, U.S. Appl. No. 15/668,665, Aug. 7, 2019, 11 pages. |
| Non-Final Office Action, U.S. Appl. No. 15/702,617, Jun. 1, 2018, 37 pages. |
| Non-Final Office Action, U.S. Appl. No. 16/443,720, Sep. 4, 2020, 13 pages. |
| Non-Final Office Action, U.S. Appl. No. 16/870,742, Oct. 28, 2021, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 16/917,800, Jul. 1, 2021, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 16/943,291, Jul. 16, 2021, 19 pages. |
| Non-Final Office Action, U.S. Appl. No. 16/943,307, Apr. 27, 2022, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 16/952,009, Aug. 1, 2022, 8 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/001,586, Jun. 9, 2022, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/129,638, Jul. 23, 2021, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/182,083, Apr. 27, 2023, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/503,023, Nov. 25, 2022, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/521,686, Oct. 4, 2022, 38 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/732,402, May 21, 2024, 20 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/751,504, Jun. 9, 2023, 31 pages. |
| Non-Final Office Action, U.S. Appl. No. 17/856,787, Apr. 11, 2024, 21 pages. |
| Non-Final Office Action, U.S. Appl. No. 18/123,930, Jul. 14, 2023, 7 pages. |
| Non-Final Office Action, U.S. Appl. No. 18/196,980, Sep. 8, 2023, 17 pages. |
| Non-Final Office Action, U.S. Appl. No. 18/204,351, Jan. 5, 2024, 8 pages. |
| Notice of Allowability, U.S. Appl. No. 17/751,504, Dec. 18, 2023, 11 pages. |
| Notice of Allowance, U.S. Appl. No. 12/412,623, Oct. 5, 2011, 5 pages. |
| Notice of Allowance, U.S. Appl. No. 13/084,923, Jul. 30, 2014, 7 pages. |
| Notice of Allowance, U.S. Appl. No. 13/107,625, Apr. 23, 2014, 7 pages. |
| Notice of Allowance, U.S. Appl. No. 13/107,625, Oct. 22, 2014, 7 pages. |
| Notice of Allowance, U.S. Appl. No. 13/301,250, Jan. 21, 2014, 10 pages. |
| Notice of Allowance, U.S. Appl. No. 13/301,250, Oct. 24, 2014, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 13/797,946, Sep. 11, 2015, 18 pages. |
| Notice of Allowance, U.S. Appl. No. 13/797,962, Feb. 17, 2015, 10 pages. |
| Notice of Allowance, U.S. Appl. No. 14/530,601, Apr. 5, 2017, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 14/553,769, May 19, 2017, 6 pages. |
| Notice of Allowance, U.S. Appl. No. 14/554,711, Jan. 27, 2017, 22 pages. |
| Notice of Allowance, U.S. Appl. No. 14/554,739, May 9, 2017, 20 pages. |
| Notice of Allowance, U.S. Appl. No. 15/004,757, Jul. 16, 2018, 7 pages. |
| Notice of Allowance, U.S. Appl. No. 15/136,790, Nov. 20, 2017, 11 pages. |
| Notice of Allowance, U.S. Appl. No. 15/174,850, Jun. 20, 2018, 39 pages. |
| Notice of Allowance, U.S. Appl. No. 15/215,468, Apr. 1, 2019, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 15/215,468, Jan. 24, 2019, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 15/215,474, Jul. 11, 2019, 9 pages. |
| Notice of Allowance, U.S. Appl. No. 15/215,483, Jun. 7, 2018, 9 pages. |
| Notice of Allowance, U.S. Appl. No. 15/668,665, Mar. 2, 2020, 7 pages. |
| Notice of Allowance, U.S. Appl. No. 15/686,054, Jul. 18, 2018, 6 pages. |
| Notice of Allowance, U.S. Appl. No. 15/702,617, Apr. 23, 2019, 24 pages. |
| Notice of Allowance, U.S. Appl. No. 15/713,518, Apr. 10, 2019, 14 pages. |
| Notice of Allowance, U.S. Appl. No. 15/713,518, Jul. 29, 2019, 13 pages. |
| Notice of Allowance, U.S. Appl. No. 15/878,286, Apr. 25, 2019, 11 pages. |
| Notice of Allowance, U.S. Appl. No. 15/878,286, Jan. 10, 2020, 6 pages. |
| Notice of Allowance, U.S. Appl. No. 15/878,286, Jul. 31, 2019, 5 pages. |
| Notice of Allowance, U.S. Appl. No. 15/930,342, Mar. 24, 2022, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 15/930,342, May 25, 2022, 9 pages. |
| Notice of Allowance, U.S. Appl. No. 16/033,131, Jun. 30, 2020, 27 pages. |
| Notice of Allowance, U.S. Appl. No. 16/194,240, Aug. 14, 2019, 10 pages. |
| Notice of Allowance, U.S. Appl. No. 16/194,240, Mar. 2, 2020, 9 pages. |
| Notice of Allowance, U.S. Appl. No. 16/194,240, Nov. 7, 2019, 10 pages. |
| Notice of Allowance, U.S. Appl. No. 16/430,336, Aug. 7, 2020, 28 pages. |
| Notice of Allowance, U.S. Appl. No. 16/430,336, Sep. 3, 2020, 5 pages. |
| Notice of Allowance, U.S. Appl. No. 16/443,720, Feb. 9, 2021, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 16/443,720, Jun. 15, 2021, 7 pages. |
| Notice of Allowance, U.S. Appl. No. 16/532,391, Oct. 21, 2020, 10 pages. |
| Notice of Allowance, U.S. Appl. No. 16/565,247, Aug. 17, 2020, 19 pages. |
| Notice of Allowance, U.S. Appl. No. 16/854,844, Jul. 6, 2021, 16 pages. |
| Notice of Allowance, U.S. Appl. No. 16/870,742, Mar. 7, 2022, 10 pages. |
| Notice of Allowance, U.S. Appl. No. 16/917,800, Oct. 15, 2021, 7 pages. |
| Notice of Allowance, U.S. Appl. No. 16/943,291, Jan. 27, 2022, 12 pages. |
| Notice of Allowance, U.S. Appl. No. 16/943,291, Oct. 1, 2021, 11 pages. |
| Notice of Allowance, U.S. Appl. No. 16/943,307, Nov. 8, 2022, 11 pages. |
| Notice of Allowance, U.S. Appl. No. 16/943,307, Oct. 6, 2022, 12 pages. |
| Notice of Allowance, U.S. Appl. No. 16/952,009, Jul. 25, 2023, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 16/952,009, Mar. 28, 2023, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 17/001,586, Sep. 8, 2022, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 17/129,638, Nov. 4, 2021, 9 pages. |
| Notice of Allowance, U.S. Appl. No. 17/182,083, Sep. 20, 2023, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 17/503,023, Feb. 24, 2023, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 17/521,686, Mar. 3, 2023, 8 pages. |
| Notice of Allowance, U.S. Appl. No. 17/751,504, Nov. 21, 2023, 12 pages. |
| Notice of Allowance, U.S. Appl. No. 17/959,177, Jun. 21, 2023, 9 pages. |
| Notice of Allowance, U.S. Appl. No. 17/959,177, Mar. 28, 2023, 9 pages. |
| Notice of Allowance, U.S. Appl. No. 18/099,854, May 26, 2023, 12 pages. |
| Notice of Allowance, U.S. Appl. No. 18/123,930, Nov. 7, 2023, 11 pages. |
| Peter Kairouz et al., “Advances and Open Problems in Federated Learning,” 2021, 121 pages, arXIV:1912.04977v3. |
| Ping Wang et al., “Peer-to-Peer Botnets: The Next Generation of Botnet Attacks”, Jan. 2010, pp. 1-25 (Year: 2010). |
| Requirement for Restriction/Election, U.S. Appl. No. 12/412,623, Nov. 22, 2010, 5 pages. |
| Requirement for Restriction/Election, U.S. Appl. No. 13/107,625, Oct. 11, 2013, 6 pages. |
| Sean Rhea et al., “Handling Churn in a DHT”, 2004, pp. 1-14 (Year: 2004). |
| Stoica et al., “Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications”, 2001, pp. 1-12 (Year: 2002). |
| Supplemental Notice of Allowability, U.S. Appl. No. 16/443,720, Aug. 4, 2021, 2 pages. |
| Supplemental Notice of Allowability, U.S. Appl. No. 16/870,742, Apr. 11, 2022, 2 pages. |
| Trevor Hastie et al., “The Elements of Statistical Learning, Data Mining, Inference, and Prediction,” 2001, 545 pages, Springer. |
| Trevor Hastie et al., “The Elements of Statistical Learning, Data Mining, Inference, and Prediction,” 2008, 764 pages, Second Edition, Springer. |
| Weixiong Rao et al., “Optimal Resource Placement in Structured Peer-to-Peer Networks,” IEEE Transactions on Parallel and Distributed Systems, vol. 21, No. 7, Jul. 2010, 16 pgs. |
| “Total Carbon Accounting: A Framework to Deliver Locational Carbon Intensity Data”, White Paper, Nov. 2021, 29 pages. |
| Abdalkarim Awad et al., “Virtual Cord Protocol (VCP): A Flexible DHT-like Routing Service for Sensor Networks”, In Proceedings of the 5th IEEE International Conference on Mobile Ad Hoc and Sensor Systems, 2008, 10 DQ:S. 133-142. |
| Corrected Notice of Allowability, U.S. Appl. No. 15/004,757, Aug. 24, 2018, 4 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 15/174,850, Jul. 25, 2018, 37 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/194,240, Mar. 31, 2020, 6 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/430,336, Oct. 15, 2020, 2 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/565,247, Oct. 15, 2020, 10 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/917,800, Dec. 16, 2021, 2 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/917,800, Nov. 18, 2021, 2 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/917,800, Oct. 25, 2021, 2 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/943,291, Apr. 11, 2022, 2 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/943,291, Feb. 25, 2022, 2 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 16/943,291, Oct. 18, 2021, 5 pages. |
| Corrected Notice of Allowability, U.S. Appl. No. 18/099,854, Jun. 5, 2023, 7 pages. |
| Final Office Action, U.S. Appl. No. 13/084,923, Jul. 1, 2013, 10 pages. |
| Final Office Action, U.S. Appl. No. 15/004,757, Dec. 29, 2017, 27 pages. |
| Final Office Action, U.S. Appl. No. 15/215,474, Apr. 1, 2019, 7 pages. |
| Final Office Action, U.S. Appl. No. 15/668,665, Dec. 10, 2019, 13 pages. |
| Final Office Action, U.S. Appl. No. 15/702,617, Dec. 27, 2018, 54 pages. |
| Final Office Action, U.S. Appl. No. 16/952,009, Dec. 13, 2022, 9 pages. |
| Final Office Action, U.S. Appl. No. 18/196,980, Mar. 11, 2024, 22 pages. |
| H. Brendan McMahan et al., “Communication-Efficient Learning of Deep Networks from Decentralized Data,” 2017, 10 pages, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, Florida, USA. |
| Hood, Cynthhia S., Proactive Network-Fault Detection, Sep. 1997, IEEE Transactions on Reliability, vol. 46, No. 3, pp. 333-341. |
| Ian Goodfellow et al., “Deep Learning,” 2016, 798 pages, MIT Press. |
| International Preliminary Report on Patentability, PCT App. No. PCT/US2013/076971, Jul. 2, 2015, 15 pages. |
| International Preliminary Report on Patentability, PCT App. No. PCT/US2014/067607, Jun. 9, 2016, 11 pages. |
| International Preliminary Report on Patentability, PCT App. No. PCT/US2015/020780, Oct. 6, 2016, 10 pages. |
| International Search Report and Written Opinion, PCT App. No. PCT/US2013/076971, Apr. 4, 2014, 17 pages. |
| International Search Report and Written Opinion, PCT App. No. PCT/US2014/067607, Feb. 18, 2015, 13 pages. |
| International Search Report and Written Opinion, PCT App. No. PCT/US2015/020780, Jul. 2, 2015, 13 pages. |
| IT Services, “Environmental impact of IT: desktops, laptops and screens”, How we are reducing IT waste, and steps you can take to reduce your carbon footprint, available online at <https://www.it.ox.ac.uk/article/environment-and-it>, Apr. 13, 2022, 5 pages. |
| Jae Woo Lee et al., 0 to 10k in 20 seconds: Bootstrapping Large-Scale DHT networks, 2011 IEEE International Conference on Communications, Jun. 9, 2011, pp. 1-6. |
| Justin Sutton-Parker, “Can analytics software measure end user computing electricity consumption?”, Springer, May 5, 2022, 19 pages. |
| Justin Sutton-Parker, “Determining commuting greenhouse gas emissions abatement achieved by information technology enabled remote working”, The 11th International Conference on Sustainable Energy Information Technology (SEIT), Aug. 9-12, 2021, 9 pages. |
| Justin Sutton-Parker, Determining end user computing device Scope 2 GHG emissions with accurate use phase energy consumption measurement, The 10th International Conference on Sustainable Energy Information Technology (SEIT), Aug. 9-12, 2020, pp. 484-491. |
| Justin Sutton-Parker, “Quantifying greenhouse gas abatement delivered by alternative computer operating system displacement strategies”, The 12th International Conference on Sustainable Energy Information Technology, Aug. 9-11, 2022, pp. 1-10. |
| Mongeau et al., “Ensuring integrity of network inventory and configuration data”, Telecommunications Network Strategy and Planning Symposium, Networks 2004, 11th International Vienna, Austria, Jun. 13-16, 2004, 6 pgs. |
| Non-Final Office Action, U.S. Appl. No. 12/412,623, Mar. 7, 2011, 10 pages. |
| Non-Final Office Action, U.S. Appl. No. 13/084,923, Dec. 9, 2013, 13 pages. |
| Non-Final Office Action, U.S. Appl. No. 13/084,923, Feb. 14, 2013, 8 pages. |
| Non-Final Office Action, U.S. Appl. No. 13/107,625, Jan. 14, 2014, 9 pages. |
| Non-Final Office Action, U.S. Appl. No. 13/301,250, Jun. 26, 2013, 11 pages. |
| Non-Final Office Action, U.S. Appl. No. 13/797,946, Feb. 27, 2015, 18 pages. |
| Non-Final Office Action, U.S. Appl. No. 14/530,601, Nov. 10, 2016, 8 pages. |
| Non-Final Office Action, U.S. Appl. No. 14/553,769, Feb. 9, 2017, 16 pages. |
| Non-Final Office Action, U.S. Appl. No. 14/554,711, Jul. 29, 2016, 23 pages. |
| Non-Final Office Action, U.S. Appl. No. 14/554,739, Aug. 26, 2016, 30 pages. |
| Non-Final Office Action, U.S. Appl. No. 15/004,757, Jun. 21, 2017, 23 pages. |
| Non-Final Office Action, U.S. Appl. No. 15/004,757, Mar. 9, 2018, 57 pages. |
| Non-Final Office Action, U.S. Appl. No. 15/215,468, Oct. 4, 2018, 13 pages. |
| Non-Final Office Action, U.S. Appl. No. 18/196,980, Sep. 19, 2024, 24 pages. |
| Non-Final Office Action, U.S. Appl. No. 18/374,621, Aug. 16, 2024, 9 pages. |
| Non-Final Office Action, U.S. Appl. No. 18/440,922, Sep. 5, 2024, 14 pages. |
| Non-Final Office Action, U.S. Appl. No. 18/516,882, Aug. 28, 2024, 11 pages. |
| Notice of Allowance, U.S. Appl. No. 17/683,213, Sep. 16, 2024, 18 pages. |
| Notice of Allowance, U.S. Appl. No. 18/204,351, Jun. 21, 2024, 10 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 62873796 | Jul 2019 | US | |
| 61774106 | Mar 2013 | US | |
| 61745236 | Dec 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17521686 | Nov 2021 | US |
| Child | 18225620 | US | |
| Parent | 16854844 | Apr 2020 | US |
| Child | 17521686 | US | |
| Parent | 15004757 | Jan 2016 | US |
| Child | 16194240 | US | |
| Parent | 13797946 | Mar 2013 | US |
| Child | 15004757 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16194240 | Nov 2018 | US |
| Child | 16854844 | US |
