Correlated risk in cybersecurity

Information

  • Patent Grant
  • 12273367
  • Patent Number
    12,273,367
  • Date Filed
    Friday, August 4, 2023
    a year ago
  • Date Issued
    Tuesday, April 8, 2025
    5 days ago
Abstract
Computer-implemented methods are provided herein for quantifying correlated risk in a network of a plurality of assets having at least one dependency, where each asset belongs to at least one entity. The method includes generating a dependency graph based on relationships between the assets, at least one dependency, and at least one entity, and executing a plurality of Monte Carlo simulations over the dependency graph. Executing a plurality of Monte Carlo simulations includes generating a seed event in the dependency graph, where the seed event has a probability distribution, and propagating disruption through the dependency graph based on the seed event. The method further includes assessing loss for each of the assets, and aggregating losses for two or more assets to determine correlated risk in the network.
Description
TECHNICAL FIELD

The following disclosure is directed to methods and systems for quantifying correlated risk in networks and, more specifically, methods and systems for quantifying correlated risk in networks due to cybersecurity incidents.


BACKGROUND

Network-based software and services (including websites, electronic communications, software-as-a-service (SaaS) offerings, and others) rely on an increasingly large and complex set of dependencies to operate. A failure or breach of any of these dependencies can cause service disruptions, outages, and other negative outcomes for the services that depend on them (directly or indirectly), resulting in loss of business continuity or other financial harm to the organizations that operate them.


There are many possible kinds of dependencies. One major category is service providers. These include (but are not limited to) hosting providers, domain name systems (DNS), content delivery networks (CDN), cloud infrastructure, managed Web servers, email services, payment processors, certificate authorities, and analytics and monitoring.


A second category includes components used to build and operate products and services. These include (but are not limited to) operating systems, application servers, code libraries, databases, networking systems, and hardware. A systematic fault in one of these components can simultaneously affect large numbers of services that use the component. For example, a bug in the Linux kernel related to leap seconds caused widespread disruption in 2012.


A third—and less obvious—category consists of software defects and malicious software. Although these are not intentional dependencies, they, too, can pose significant aggregate risk. For example, a wide-scale ransomware attack has the potential to disrupt large numbers of software services and businesses.


Dependency relationships are often not immediately apparent. For example, if Web site A is hosted on hosting provider B, and hosting provider B uses a domain-name service (DNS) provider C, a failure of C can lead to a failure of A, even though no direct business or technical relationship—only a transitive one—exists between A and C.


Furthermore, large numbers of services (and businesses) may rely on a single dependency (direct or transitive). A failure of that dependency can thus cause surprisingly widespread disruptions. These dependencies therefore create aggregate risk (also known as correlated risk) from the point of view of a business operating multiple services or service instances, or from the point of view of an organization with a financial interest in a portfolio of businesses (e.g., insurance or investments).


Many methods of reliability and risk analysis assume that failures are uncorrelated and independent, because this greatly simplifies the analysis. However, because of the above points, this assumption often leads to inaccuracies and/or understated risks in networked environments. There is a current need for methods of identifying dependencies and other risk factors which pose high levels of aggregate risk, and of quantifying this risk.


SUMMARY

Methods and systems are described herein to identify such dependencies that may be the sources of risk and accurately quantify that risk. The methods include capturing the relationships among entities (such as organizations or companies), their assets, and the dependencies that pose risks to the operation of those assets. These relationships can be identified via a dependency graph. The dependency graph may also include portfolio level nodes to aid risk managers in identifying risk in a collection across multiple entities—even if those entities are not directly related. Note that the assets may be weighted by their importance to their owners, based on traffic data and other metrics. Risk is evaluated and quantified by carrying out repeated sampling and probabilistic simulations (e.g., Monte Carlo trials) over the dependency graph. In each simulation, a seed event is generated representing a disruption caused by the failure of a dependency (or the occurrence of a botnet attack, software defect, or other systemic effect). Next, the disruption caused by the seed event is propagated probabilistically though the graph. Once propagation is complete, the loss is assessed for each asset, and the aggregate losses are accumulated at the entity and portfolio levels. Statistics are gathered from the collection of individual runs.


In accordance with an embodiment of the disclosure, a computer-implemented method is provided for quantifying correlated risk in a network of assets having one or more dependencies, where each asset belongs to at least one entity. The method includes generating a dependency graph based on relationships between the assets, at least one dependency, and at least one entity, and executing Monte Carlo simulations over the dependency graph, including generating a seed event in the dependency graph, where the seed event has a probability distribution, and propagating disruption through the dependency graph based on the seed event. An estimated loss is assessed for each of the assets, which may be aggregated across multiple assets to determine correlated risk in the network.


In a related embodiment, the assets are selected from a group consisting of: Internet Protocol (IP) address, domain name, and server system; and each entity may be a company or an organization. The dependencies may result from using a hosting provider and/or a software version. Optionally, the method includes receiving information indicative of the relationships between the plurality of assets, at least one dependency, and at least one entity.


In another related embodiment, the method includes storing information indicative of the relationships among the assets, the dependencies, and at the entities in a database, wherein the information is at least one of the group consisting of: domain name system (DNS) record, server banner, traffic data, malware infection, and software version. Optionally, the method includes observing traffic to and from a particular asset in the network to identify at least one of (i) an entity and (ii) a dependency related to the particular asset.


In a further related embodiment, each of the assets may be weighted according to its importance to an entity owning, controlling or using the asset. Optionally, the dependency graph includes (i) edges representing relationships among the assets, at least one dependency, and at least one entity and (ii) nodes representing the assets, at least one dependency, and at least one entity, wherein each edge has a conditional probability that the asset on a receiving node of a particular edge is compromised given that the providing node is compromised.


In yet another related embodiment, propagating disruption through the dependency graph may continue through the dependency graph until a threshold number of nodes is affected, such as a threshold amount of loss aggregated among assets. Optionally or alternatively, the seed event may be a breach or failure of the at least one dependency. In a related embodiment, the probability distribution is a probability that the asset will become unavailable if the dependency fails. In a further related embodiment, the method may include storing information related to the aggregated losses in a database.


In another related embodiment, the entity includes at least two entities, and a first asset belongs to a first entity and a second asset belongs to a second entity. In a related embodiment, at least one of the assets belongs to another entity of the entities. Optionally, the method includes aggregating losses for two or more entities to determine correlated risk in the network. In a related embodiment, the method includes storing information related to the aggregated losses for the two or more entities in a database. Optionally, each of the at least two entities is assigned to at least one portfolio, and wherein the method further includes aggregating losses for two or more portfolios to determine correlated risk in the network. Optionally or alternatively, the method includes storing information related to the aggregated losses for the two or more portfolios in a database.


In a related embodiment, the aggregating losses for two or more assets to determine correlated risk in the network further includes aggregating losses in a nonlinear sum for the two or more assets. In another related embodiment, a number of the plurality of Monte Carlo simulations is selected to reduce a statistical variance of the plurality of Monte Carlo simulations. Optionally, the execution of Monte Carlo simulations over the dependency graph further includes determining if a statistical variance is equal to or less than a threshold, and terminating the Monte Carlo simulations if the statistical variance is equal to or less than the threshold.





BRIEF DESCRIPTION OF THE DRAWINGS


FIGS. 1A-1B are flowcharts of exemplary computer-implemented methods for quantifying correlated risk in a network.



FIG. 2 is a schematic diagram of an exemplary network of plurality of assets, at least one dependency, at least one entity, and at least one portfolio.



FIG. 3 is a graph of the loss exceedance curve as a function of loss in U.S. Dollars produced from the simulation results.



FIG. 4 is a diagram of an exemplary system 400 configured to quantify correlated risk in network of assets, one or more dependencies, one or more entities, and/or one or more portfolio.





DETAILED DESCRIPTION

Described herein are methods and systems that enable the identification and quantification of aggregate risk posed to entities and portfolios of entities by dependencies and other risk factors. In the following, FIGS. 1-4 are described together for clarity purposes.


Identifying Relationships


FIG. 1A is a flowchart of an exemplary computer-implemented method 100 for quantifying correlated risk in a network. An exemplary network includes a plurality of assets, at least one dependency, and at least one entity. In process 102, a processor (such as processor 408, 410, and/or 412 illustrated in FIG. 4) can generate a dependency graph based on relationships among the plurality of assets, the at least one dependency, and the at least one entity, as detailed below. Optionally, the dependency graph can include at least one portfolio containing the at least one entity, as detailed below. In other embodiments, the processor stores or defines information related to the relationships among the different constituents (assets, dependency or dependencies, entity or entities, portfolio(s)) in a table, relational database, or flat file instead of a graph.



FIG. 2 is a schematic diagram of an exemplary dependency graph 200 of plurality of assets (A(n+1)), at least one dependency (Dn), at least one entity (En), and at least one portfolio (Pn), wherein n is any natural number {1, 2, 3, . . . }. Dependency level 202 includes at least one dependency D1, D2 (e.g., hosting providers or software version). Asset level 204 includes the plurality of assets A1-A4 (e.g., domain names, servers, Internet Protocol (IP) addresses, hosts, etc.) belonging to one or more entities E1-E3. The edges 210 between the dependency and asset levels 202, 204 indicate a dependency relationship involving the asset and the dependency. For example, A1 and A2 are dependent on same dependency D1 while A2 is also dependent on D2. Thus, each asset An may have one or more dependency Dn. The assets are each weighted by their importance to their owners (entities), based on traffic data and other metrics. Importance may be assessed in terms of financial impact (e.g. percentage of revenue), fraction of network traffic, or custom metrics defined by the user. Entity level 206 includes entities E1, E2, E3 (companies or organizations), which are linked to their assets. For example, entity E1 is linked to both assets A1 and A2 while E1 is linked to asset A2. Portfolio level 208 includes at least one portfolio. Each portfolio Pn may include one or more entities En. The relationships between the constituents represent links critical to the operation of, for instance, the assets.


Various constituents of the network can be identified in one of several ways, which include observations of network traffic and published records. For more detail on the identification of assets and their relationships to entities, refer to commonly owned U.S. Publication No. 2017/0236077, published on Aug. 17, 2017 and titled “Relationships among technology assets and services and the entities responsible for them” the entire disclosure of which is included herein by reference. The data that are used to generate the graph 200 are collected via multiple processes, including (but not limited to) network maps, observations of network traffic, domain name system (DNS) records, server responses, malware infections, and/or published information. In some embodiments, inter-business payment data (e.g., data provided by Dun & Bradstreet, Inc., New Jersey) to discover business relationships, and therefore dependencies.


In some embodiments, network observations can be collected and stored in a database coupled to the processor. For more detail, refer to commonly owned U.S. Pat. No. 9,705,932, issued on Jul. 11, 2017 and titled “Methods and systems for creating, de-duplicating, and accessing data using an object storage system” the entire disclosure of which is included herein by reference. These network observations include DNS records (to identify DNS providers, email providers, and other relationships), server banners (to identify software and operating system dependencies and risk factors), references to resources used (e.g. JavaScript libraries), traffic data, malware (e.g. botnet) infections, software versions used within an organization, and/or records of insecure systems. In some examples, observations can be assigned to entities via their network maps. A network map can include Internet Protocol (IP, Version 4 or Version 6) address ranges, autonomous system (AS) numbers, domain names, and hostnames owned by a particular entity. Each of these entries can be associated with a date range for which it is effective. The data can be partitioned and ordered by timestamp to facilitate efficiency of processing. Some or all of these data may also be used in the hazard model that is used to determine the probability of seed events, as discussed in more detail below. For example, a service provider with a poor security posture suffers a greater average rate of disruptions and failures due to cybercrime attacks. In some examples, some or all of these data also may be used to assess the relative importance or criticality of assets. For example, those with higher volumes network traffic may have greater value, and may generate larger losses when compromised.


Disruption Simulation

Referring back to FIG. 1A, in process 104, the processor executes a plurality of Monte Carlo simulations over the dependency graph 200. To do so, the processor generates a ‘seed event’ according to a hazard model. A hazard model is a probability distribution over possible events (e.g. an outage of a particular network component). This probability distribution can be derived from empirical observations, from theoretical models, or from some combination of empirical observations and theoretical models. A seed event can be a failure or disruption (such as an occurrence of a botnet attack, software defect, or other systematic effect) at one or more of the constituents (assets, dependency or dependencies, entity or entities, portfolio(s)). In some examples, the seed event is generated at the dependency level 202. The seed events are drawn from a probability distribution that may be based on empirical observations of similar events (e.g., the incidence of malware infections or DNS attacks), from theoretical models, or from other models derived from empirical data. In some examples, the seed events may optionally be chosen by an end user of the system to investigate a particular scenario.


In process 106, disruptions are propagated probabilistically through the graph 200, using the conditional probability distributions on each edge 210. For example, an edge 210 may have the probability that a given asset will become unavailable, given that the dependency in question failed. The probability may be defined as the probability that the asset on the receiving end of the edge is compromised (or affected in some other specified way), given that the node on the provider end of the edge is compromised. The conditional probabilities may be determined by empirical observations of actual events, by theoretical models, or some combination of the two. In some examples, the propagation proceeds for either a fixed number of steps or until a threshold for the affected number of nodes, the magnitude of the loss, or other criterion is reached. This propagation technique can take complex dependency relationships, including transitive and circular dependent relationships, into account.


Assessing and Aggregating Loss(es)

Referring back to FIG. 1A, in process 110, potential losses assessed for some or all of the plurality of assets. In process 112, the losses are then aggregated for each asset An. In some embodiments, losses at the assets An are aggregated at the portfolio level 208, at the entity level 206, or both. In some embodiments, the losses can be aggregated per portfolio Pn or per entity En. The dependency nodes Dn, which are the sources of risk, propagate risk to assets An that rely on the dependencies. In some embodiments, other nodes, such as assets, can be made into sources of risk by creating additional dependency nodes. In some embodiments, the aggregated losses may be simple sums, or they may involve more complex loss models including thresholds, limits, and other conditions. For example, certain policies may limit losses to a certain level or certain constituent (such as per entity or per asset).


Each simulation produces an output of projected loss data. Multiple simulations are run and the outputs from the runs are stored in a database. Having repeated the simulations a plurality of times, statistics can be gathered or derived from the simulation results. These statistics can include mean expected loss or loss exceedance curves. Referring to the method 114 of FIG. 1B, in some embodiments, once the losses are assessed in process 110, the processor can determine if the statistical variance is equal to or less than a threshold in process 116. If so, the losses are aggregated in process 112. Otherwise, control returns to process 106 for one or more runs of the Monte Carlo simulation. Note that, in some embodiments, process 116 can occur after process 112 in that the variance can be measured after aggregation. In this embodiment, the post-aggregation measured variance can be optionally used as a convergence criterion. Thus, if the statistics have not converged, an additional number of simulations are executed until they do converge. The process terminates when the variance of the desired statistics reaches an acceptable level (i.e. the statistics have converged). In some embodiments, the number of simulation iterations may be chosen a priori. In other embodiments, iterations may be continued until the variance of the statistics is reduced to the desired level. For instance, the expected variance for the financial loss can be limited to 1% of the square of the mean.


The simulation results can be used to identify dependencies that pose large aggregate risks, and to quantify the level of aggregate risk exposure for each portfolio or entity of interest. These results, and the statistics derived therefrom, are used to generate charts, tables, and/or metrics that can be used by a risk manager to assess and quantify risk(s) associated with entities and/or portfolios. FIG. 3 is a graph 300 of the loss exceedance curve 302 as a function of loss in U.S. Dollars produced from the simulation results. The relationship between loss and the exceedance rate (expressed in 1/year) is indirectly proportional and generally non-linear. The loss exceedance curve 302 enables a risk analyst to determine, for example, whether the rate of losses exceeding a threshold (such as a predetermined quantity of $1 million) falls within acceptable bounds of the exceedance rate (such as an incidence of once per 100 years).


In some embodiments, the results are stored in a manner that allows losses at the portfolio level to be determined dynamically (for portfolios that were not yet defined when the simulations were run), without having to repeat the computationally expensive simulation steps. This enables a user interacting with a user interface of the system to quickly evaluate how changes in portfolio composition or dependencies affect the aggregate risk. In some embodiments, the simulation results can be used to produce visualizations, tables, and other displays that may be used to interactively identify sources of aggregate risk, and to evaluate the level of aggregate risk posed by dependencies, at the portfolio and individual entity level.


System(s) for Quantifying Correlated Risk


FIG. 4 is a diagram of an exemplary system 400 configured to quantify correlated risk in network of assets, one or more dependencies, one or more entities, and/or one or more portfolio. The one or more processes of methods described herein (see FIGS. 1A-1B) may be executed by a processor of a server system 402, a computer 404, and/or a mobile device 406. In some examples, a processor 408 of computer 404 can execute a first subset of the processes (such as process 102 of FIGS. 1A-1B) while relying on a processor 410 of server system 402 to execute a second subset of the processes (such as processes 104-108 of FIGS. 1A-1B). In some embodiments, the first and second subsets of processes may overlap. Server 402, computer 404, and mobile device 406 may each have their own storage (414, 416, and 418, respectively) or be coupled to an external data storage system 420. Computer 404 and mobile device 406 may each have their own display (422 and 424, respectively) or be coupled to an external display device 426.


The term “system” 400 may encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. A processing system can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.


A computer program (also known as a program, software, software application, script, executable logic, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.


Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile or volatile memory, media, and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks or magnetic tapes; magneto optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Sometimes a server (e.g., forming a portion of the server system 402) is a general-purpose computer, and sometimes it is a custom-tailored special purpose electronic device, and sometimes it is a combination of these things.


Implementations can include a back-end component, e.g., a data server, or a middleware component, e.g., an application server, or a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.


Server 402, computer 404, mobile device 406, display device 426, and data storage server 420 may communicate with each other (as well as other devices and data sources) via a network 428. Network communication may take place via any media such as standard and/or cellular telephone lines, LAN or WAN links (e.g., T1, T3, 56 kb, X.25), broadband connections (ISDN, Frame Relay, ATM), wireless links, and so on. Preferably, the network 125 can carry TCP/IP protocol communications, and HTTP/HTTPS requests made by the mobile device and the connection between the mobile device 406 and the server 402 can be communicated over such networks. In some implementations, the network includes various cellular data networks such as 2G, 3G, 4G, and others. The type of network is not limited, however, and any suitable network may be used. Typical examples of networks that can serve as the communications network 428 include a wireless or wired Ethernet-based intranet, a local or wide-area network (LAN or WAN), and/or the global communications network known as the Internet, which may accommodate many different communications media and protocols.


Certain features that are described above in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, features that are described in the context of a single implementation can be implemented in multiple implementations separately or in any sub-combinations.


The order in which operations are performed as described above can be altered. In certain circumstances, multitasking and parallel processing may be advantageous. The separation of system components in the implementations described above should not be understood as requiring such separation.


The terms and expressions employed herein are used as terms and expressions of description and not of limitation and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof. In addition, having described certain embodiments of the invention, it will be apparent to those of ordinary skill in the art that other embodiments incorporating the concepts disclosed herein may be used without departing from the spirit and scope of the invention. The structural features and functions of the various embodiments may be arranged in various combinations and permutations, and all are considered to be within the scope of the disclosed invention. Unless otherwise necessitated, recited steps in the various methods may be performed in any order and certain steps may be performed substantially simultaneously. Accordingly, the described embodiments are to be considered in all respects as only illustrative and not restrictive. Furthermore, the configurations described herein are intended as illustrative and in no way limiting. Similarly, although physical explanations have been provided for explanatory purposes, there is no intent to be bound by any particular theory or mechanism, or to limit the claims in accordance therewith.

Claims
  • 1. A computer-implemented method for quantifying correlated risk in a network of a plurality of assets having at least one dependency, each asset belonging to at least one entity, the method comprising: executing a plurality of Monte Carlo simulations over a dependency graph that is based on relationships between the plurality of assets, the at least one dependency, and the at least one entity, wherein each of the plurality of Monte Carlo simulations executes by: generating a seed event in the dependency graph, the seed event having a probability distribution; andpropagating disruption through the dependency graph based on the seed event; andterminating the respective Monte Carlo simulation when a threshold amount of loss is aggregated among two or more assets of the plurality of assets affected by the disruption is exceeded;assessing, based on the plurality of Monte Carlo simulations, a loss for each asset of the plurality of assets; andaggregating the losses for the two or more assets of the plurality of assets to determine correlated risk in the network,wherein the dependency graph comprises (i) a plurality of edges representing the relationships between the plurality of assets, the at least one dependency, and the at least one entity and (ii) a plurality of nodes representing the plurality of assets, the at least one dependency, and the at least one entity, andwherein each edge has a conditional probability that the asset on a receiving node of a particular edge, of the plurality of edges, is compromised given that a providing node, of the plurality of nodes, is compromised.
  • 2. The method of claim 1, wherein: each of the plurality of assets is selected from the group consisting of: Internet Protocol (IP) address, domain name, and server system;each of the at least one entity is selected from the group consisting of: a company and an organization; andeach of the at least one dependency is selected from the group consisting of: hosting provider and software version.
  • 3. The method of claim 1, further comprising: receiving information indicative of the relationships between the plurality of assets, the at least one dependency, and the at least one entity.
  • 4. The method of claim 3, further comprising: storing the information indicative of the relationships between the plurality of assets, the at least one dependency, and the at least one entity in a database, wherein the information is at least one of the group consisting of: domain name system (DNS) record, server banner, traffic data, malware infection, and software version.
  • 5. The method of claim 3, further comprising: observing traffic to and from a particular one of the plurality of assets in the network to identify at least one of (i) an entity and (ii) a dependency related to the particular asset.
  • 6. The method of claim 3, wherein the information indicative of the relationships includes inter-business payment data.
  • 7. The method of claim 1, wherein each of the plurality of assets is weighted according to its importance to the at least one entity having the asset.
  • 8. The method of claim 1, further comprising: generating the dependency graph.
  • 9. The method of claim 1, wherein the propagating disruption through the dependency graph based on the seed event further comprises: propagating disruption through the dependency graph until a threshold number of nodes is affected.
  • 10. The method of claim 1, wherein the seed event is a breach or failure of the at least one dependency.
  • 11. The method of claim 1, wherein the probability distribution is a probability that the asset will become unavailable if the at least one dependency fails.
  • 12. The method of claim 1, further comprising: storing information related to the aggregated losses for the two or more assets of the plurality of assets in a database.
  • 13. The method of claim 1, wherein the at least one entity comprises at least two entities, and a first asset of the two or more assets belongs to a first entity of the at least two entities and a second entity of the two or more assets belongs to a second asset of the at least two entities.
  • 14. The method of claim 13, wherein at least one of the first and second assets belongs to another entity of the at least two entities.
  • 15. The method of claim 13, further comprising: aggregating losses for two or more entities of the at least two entities to determine correlated risk in the network.
  • 16. The method of claim 15, further comprising: storing information related to the aggregated losses for the two or more entities of the at least two entities in a database.
  • 17. The method of claim 13, wherein each of the at least two entities is assigned to at least one portfolio, and wherein the method further comprises: aggregating losses for two or more portfolios of the at least one portfolio to determine correlated risk in the network.
  • 18. The method of claim 17, further comprising: storing information related to the aggregated losses for the two or more portfolios of the at least one portfolio in a database.
  • 19. The method of claim 1, wherein the aggregating losses for the two or more assets of the plurality of assets to determine correlated risk in the network further comprises: aggregating losses in a nonlinear sum for the two or more assets of the plurality of assets.
  • 20. The method of claim 1, wherein a number of the plurality of Monte Carlo simulations is selected to reduce a statistical variance of the plurality of Monte Carlo simulations.
  • 21. The method of claim 1, wherein the executing a plurality of Monte Carlo simulations over the dependency graph further comprises: comparing a statistical variance to a threshold; andterminating the plurality of Monte Carlo simulations when the statistical variance is equal to or less than the threshold.
  • 22. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to: execute, for a network of a plurality of assets having at least one dependency, each asset belonging to at least one entity, a plurality of Monte Carlo simulations over a dependency graph that is based on relationships between the plurality of assets, the at least one dependency, and the at least one entity, wherein each of the plurality of Monte Carlo simulations executes by: generating a seed event in the dependency graph, the seed event having a probability distribution; andpropagating disruption through the dependency graph based on the seed event; andterminating the respective Monte Carlo simulation when a threshold amount of loss is aggregated among two or more assets of the plurality of assets affected by the disruption is exceeded;assess, based on the plurality of Monte Carlo simulations, a loss for each asset of the plurality of assets; andaggregate the losses for the two or more assets of the plurality of assets to determine correlated risk in the network,wherein the dependency graph comprises (i) a plurality of edges representing the relationships between the plurality of assets, the at least one dependency, and the at least one entity and (ii) a plurality of nodes representing the plurality of assets, the at least one dependency, and the at least one entity, andwherein each edge has a conditional probability that the asset on a receiving node of a particular edge, of the plurality of edges, is compromised given that a providing node, of the plurality of nodes, is compromised.
  • 23. A computing apparatus comprising: a processor; anda memory storing instructions that, when executed by the processor, configure the apparatus to: execute, for a network of a plurality of assets having at least one dependency, each asset belonging to at least one entity, a plurality of Monte Carlo simulations over a dependency graph that is based on relationships between the plurality of assets, the at least one dependency, and the at least one entity, wherein each of the plurality of Monte Carlo simulations executes by: generating a seed event in the dependency graph, the seed event having a probability distribution; andpropagating disruption through the dependency graph based on the seed event; andterminating the respective Monte Carlo simulation when a threshold amount of loss is aggregated among two or more assets of the plurality of assets affected by the disruption is exceeded;assess, based on the plurality of Monte Carlo simulations, a loss for each asset of the plurality of assets; andaggregate the losses for the two or more assets of the plurality of assets to determine correlated risk in the network,wherein the dependency graph comprises (i) a plurality of edges representing the relationships between the plurality of assets, the at least one dependency, and the at least one entity and (ii) a plurality of nodes representing the plurality of assets, the at least one dependency, and the at least one entity, andwherein each edge has a conditional probability that the asset on a receiving node of a particular edge, of the plurality of edges, is compromised given that a providing node, of the plurality of nodes, is compromised.
CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 17/179,630, titled “Correlated Risk in Cybersecurity” and filed on Feb. 19, 2021, which is a continuation of U.S. patent application Ser. No. 16/795,056, titled “Correlated Risk in Cybersecurity” and filed on Feb. 19, 2020, now U.S. Pat. No. 10,931,705, issued on Feb. 23, 2021, which is a continuation of U.S. patent application Ser. No. 16/292,956, titled “Correlated Risk in Cybersecurity” and filed on Mar. 5, 2019 now U.S. Pat. No. 10,594,723 issued on Mar. 17, 2020, which is a continuation of U.S. patent application Ser. No. 15/918,286, titled “Correlated Risk in Cybersecurity” and filed on Mar. 12, 2018 now U.S. Pat. No. 10,257,219 issued on Apr. 9, 2019, which are hereby incorporated by reference herein in their entireties.

US Referenced Citations (577)
Number Name Date Kind
5867799 Lang et al. Feb 1999 A
6016475 Miller et al. Jan 2000 A
6578066 Logan et al. Jun 2003 B1
6745150 Breiman Jun 2004 B1
6785732 Bates et al. Aug 2004 B1
6792401 Nigro et al. Sep 2004 B1
7062572 Hampton Jun 2006 B1
D525264 Chotai et al. Jul 2006 S
D525629 Chotai et al. Jul 2006 S
7100195 Underwood Aug 2006 B1
7124055 Breiman Oct 2006 B2
7194769 Lippmann et al. Mar 2007 B2
7257630 Cole et al. Aug 2007 B2
7290275 Baudoin et al. Oct 2007 B2
7343626 Gallagher Mar 2008 B1
7389262 Lange Jun 2008 B1
7409357 Schaf Aug 2008 B2
D604740 Matheny et al. Nov 2009 S
7650570 Torrens et al. Jan 2010 B2
7747778 King et al. Jun 2010 B1
7748038 Olivier et al. Jun 2010 B2
7827607 Sobel et al. Nov 2010 B2
D630645 Tokunaga et al. Jan 2011 S
7971252 Lippmann et al. Jun 2011 B2
8000698 Wolman et al. Aug 2011 B2
8042184 Batenin Oct 2011 B1
8056132 Chang et al. Nov 2011 B1
D652048 Joseph Jan 2012 S
8150538 Dubinsky Apr 2012 B2
8239939 Dunagan et al. Aug 2012 B2
D667022 LoBosco et al. Sep 2012 S
8266695 Clay, IV Sep 2012 B1
8321791 Dixon et al. Nov 2012 B2
8359651 Wu et al. Jan 2013 B1
8370933 Buckler Feb 2013 B1
8370938 Daswani et al. Feb 2013 B1
8429630 Nickolov et al. Apr 2013 B2
D682287 Cong et al. May 2013 S
D688260 Pearcy et al. Aug 2013 S
8504556 Rice et al. Aug 2013 B1
8505094 Xuewen et al. Aug 2013 B1
D691164 Lim et al. Oct 2013 S
D694252 Helm Nov 2013 S
D694253 Helm Nov 2013 S
8578496 Krishnappa Nov 2013 B1
8578499 Zhu et al. Nov 2013 B1
8584233 Yang et al. Nov 2013 B1
8601575 Mullarkey et al. Dec 2013 B2
8621621 Burns et al. Dec 2013 B1
8661146 Alex et al. Feb 2014 B2
D700616 Chao Mar 2014 S
8677481 Lee Mar 2014 B1
8683584 Daswani et al. Mar 2014 B1
8752183 Heiderich et al. Jun 2014 B1
8775402 Baskerville et al. Jul 2014 B2
8776240 Wu et al. Jul 2014 B1
8806646 Daswani et al. Aug 2014 B1
8825662 Kingman et al. Sep 2014 B1
8839432 Patil Sep 2014 B1
8850570 Ramzan Sep 2014 B1
8898776 Molnar et al. Nov 2014 B2
8949988 Adams et al. Feb 2015 B2
8949990 Hsieh et al. Feb 2015 B1
8966639 Roytman et al. Feb 2015 B1
D730918 Park et al. Jun 2015 S
9049222 He et al. Jun 2015 B1
9053210 Elnikety et al. Jun 2015 B2
9075990 Yang Jul 2015 B1
D740847 Yampolskiy et al. Oct 2015 S
D740848 Bolts et al. Oct 2015 S
D741351 Kito et al. Oct 2015 S
D746832 Pearcy et al. Jan 2016 S
9241252 Dua et al. Jan 2016 B2
9244899 Greenbaum Jan 2016 B1
9294498 Yampolskiy et al. Mar 2016 B1
D754690 Park et al. Apr 2016 S
D754696 Follett et al. Apr 2016 S
9323930 Satish Apr 2016 B1
D756371 Bertnick et al. May 2016 S
D756372 Bertnick et al. May 2016 S
D756392 Yun et al. May 2016 S
D759084 Yampolskiy et al. Jun 2016 S
D759689 Olson et al. Jun 2016 S
9372994 Yampolskiy et al. Jun 2016 B1
9373144 Ng et al. Jun 2016 B1
D760782 Kendler et al. Jul 2016 S
9384206 Bono et al. Jul 2016 B1
9401926 Dubow Jul 2016 B1
9407658 Kuskov et al. Aug 2016 B1
9413774 Liu et al. Aug 2016 B1
9420049 Talmor et al. Aug 2016 B1
9424333 Bisignani et al. Aug 2016 B1
9432383 Johns et al. Aug 2016 B2
9438615 Gladstone et al. Sep 2016 B2
9479526 Yang Oct 2016 B1
D771103 Eder Nov 2016 S
D771695 Yampolskiy et al. Nov 2016 S
D772276 Yampolskiy et al. Nov 2016 S
9501647 Yampolskiy et al. Nov 2016 B2
D773507 Sagrillo et al. Dec 2016 S
9530016 Pomerantz Dec 2016 B1
D775635 Raji et al. Jan 2017 S
D776136 Chen et al. Jan 2017 S
D776153 Yampolskiy et al. Jan 2017 S
D777177 Chen et al. Jan 2017 S
9548988 Roundy et al. Jan 2017 B1
9560072 Xu Jan 2017 B1
D778927 Bertnick et al. Feb 2017 S
D778928 Bertnick et al. Feb 2017 S
D779512 Kimura et al. Feb 2017 S
D779514 Baris et al. Feb 2017 S
D779531 List et al. Feb 2017 S
9578048 Hunt et al. Feb 2017 B1
D780770 Sum et al. Mar 2017 S
D785009 Lim et al. Apr 2017 S
D785010 Bachman et al. Apr 2017 S
D785016 Berwick et al. Apr 2017 S
9620079 Curtis Apr 2017 B2
D787530 Huang May 2017 S
D788128 Wada May 2017 S
9641547 Yampolskiy et al. May 2017 B2
9646110 Byrne et al. May 2017 B2
D789947 Sun Jun 2017 S
D789957 Wu et al. Jun 2017 S
9680855 Schultz Jun 2017 B2
9680858 Boyer et al. Jun 2017 B1
D791153 Rice et al. Jul 2017 S
D791834 Eze et al. Jul 2017 S
D792427 Weaver et al. Jul 2017 S
D795891 Kohan et al. Aug 2017 S
9736019 Hardison et al. Aug 2017 B2
9742796 Salsamendi Aug 2017 B1
D796523 Bhandari et al. Sep 2017 S
D801989 Iketsuki et al. Nov 2017 S
D803237 Wu et al. Nov 2017 S
9813440 Hoover et al. Nov 2017 B1
9825976 Gomez et al. Nov 2017 B1
9825984 Hoover et al. Nov 2017 B1
D804528 Martin et al. Dec 2017 S
D806735 Olsen et al. Jan 2018 S
D806737 Chung et al. Jan 2018 S
D809523 Lipka et al. Feb 2018 S
D809989 Lee et al. Feb 2018 S
D812633 Saneii Mar 2018 S
D814483 Gavaskar et al. Apr 2018 S
D815119 Chalker et al. Apr 2018 S
D815148 Martin et al. Apr 2018 S
D816105 Rudick et al. Apr 2018 S
D816116 Selassie Apr 2018 S
9954893 Zhao et al. Apr 2018 B1
D817970 Chang et al. May 2018 S
D817977 Kato et al. May 2018 S
D818475 Yepez et al. May 2018 S
9973524 Boyer et al. May 2018 B2
D819687 Yampolskiy et al. Jun 2018 S
10044750 Livshits et al. Aug 2018 B2
10079854 Scott et al. Sep 2018 B1
10084817 Saher et al. Sep 2018 B2
10142364 Baukes et al. Nov 2018 B2
D835631 Yepez et al. Dec 2018 S
10180966 Lang et al. Jan 2019 B1
10185924 McClintock et al. Jan 2019 B1
10210329 Malik et al. Feb 2019 B1
10217071 Mo et al. Feb 2019 B2
10230753 Yampolskiy et al. Mar 2019 B2
10230764 Ng et al. Mar 2019 B2
10235524 Ford Mar 2019 B2
10242180 Haefner et al. Mar 2019 B2
D847169 Sombreireiro et al. Apr 2019 S
10257219 Geil Apr 2019 B1
10305854 Alizadeh-Shabdiz et al. May 2019 B2
10331502 Hart Jun 2019 B1
10339321 Tedeschi Jul 2019 B2
10339484 Pai et al. Jul 2019 B2
10348755 Shavell et al. Jul 2019 B1
10412083 Zou et al. Sep 2019 B2
D863335 Hardy et al. Oct 2019 S
D863345 Hardy et al. Oct 2019 S
10453142 Mun Oct 2019 B2
10469515 Helmsen et al. Nov 2019 B2
10482239 Liu et al. Nov 2019 B1
10491619 Yampolskiy et al. Nov 2019 B2
10491620 Yampolskiy et al. Nov 2019 B2
10521583 Bagulho Monteiro Pereira Dec 2019 B1
D872574 Deylamian et al. Jan 2020 S
10540374 Singh et al. Jan 2020 B2
D874506 Kang et al. Feb 2020 S
10572945 McNair Feb 2020 B1
D880512 Greenwald et al. Apr 2020 S
10757127 Schultz Aug 2020 B2
D894939 Braica Sep 2020 S
10764298 Light et al. Sep 2020 B1
10776483 Bagulho Monteiro Pereira Sep 2020 B2
10796260 Brannon et al. Oct 2020 B2
10805331 Boyer et al. Oct 2020 B2
D903693 Li et al. Dec 2020 S
D905712 Li et al. Dec 2020 S
D908139 Hardy et al. Jan 2021 S
10896394 Brannon et al. Jan 2021 B2
10909488 Hecht et al. Feb 2021 B2
D918955 Madden, Jr. et al. May 2021 S
D920343 Bowland May 2021 S
D920353 Boutros et al. May 2021 S
D921031 Tessier et al. Jun 2021 S
D921662 Giannino et al. Jun 2021 S
D921674 Kmak et al. Jun 2021 S
D921677 Kmak et al. Jun 2021 S
D922397 Modi et al. Jun 2021 S
D924909 Nasu et al. Jul 2021 S
11126723 Bagulho Monteiro Pereira Sep 2021 B2
11222388 Baumgartner Jan 2022 B2
11334832 Dumoulin et al. May 2022 B2
11379773 Vescio Jul 2022 B2
11455322 Yang et al. Sep 2022 B2
11652834 Gladstone et al. May 2023 B2
11727114 Bagulho Monteiro Pereira Aug 2023 B2
20010044798 Nagral et al. Nov 2001 A1
20020083077 Vardi Jun 2002 A1
20020133365 Grey et al. Sep 2002 A1
20020164983 Raviv et al. Nov 2002 A1
20030011601 Itoh et al. Jan 2003 A1
20030050862 Bleicken et al. Mar 2003 A1
20030074248 Braud et al. Apr 2003 A1
20030123424 Jung Jul 2003 A1
20030187967 Walsh Oct 2003 A1
20040003284 Campbell et al. Jan 2004 A1
20040010709 Baudoin et al. Jan 2004 A1
20040024859 Bloch et al. Feb 2004 A1
20040088570 Roberts et al. May 2004 A1
20040098375 DeCarlo May 2004 A1
20040111358 Lange Jun 2004 A1
20040133561 Burke Jul 2004 A1
20040133689 Vasisht Jul 2004 A1
20040193907 Patanella Sep 2004 A1
20040193918 Green et al. Sep 2004 A1
20040199791 Poletto et al. Oct 2004 A1
20040199792 Tan et al. Oct 2004 A1
20040221296 Ogielski et al. Nov 2004 A1
20040250122 Newton Dec 2004 A1
20040250134 Kohler et al. Dec 2004 A1
20050065754 Schaf Mar 2005 A1
20050065807 DeAngelis et al. Mar 2005 A1
20050066195 Jones Mar 2005 A1
20050071450 Allen et al. Mar 2005 A1
20050076245 Graham et al. Apr 2005 A1
20050080720 Betz et al. Apr 2005 A1
20050108415 Turk et al. May 2005 A1
20050131830 Juarez et al. Jun 2005 A1
20050138413 Lippmann et al. Jun 2005 A1
20050160002 Roetter et al. Jul 2005 A1
20050228899 Wendkos et al. Oct 2005 A1
20050234767 Bolzman et al. Oct 2005 A1
20050278726 Cano et al. Dec 2005 A1
20050278786 Tippett et al. Dec 2005 A1
20060036335 Banter et al. Feb 2006 A1
20060075490 Boney et al. Apr 2006 A1
20060075494 Bertman et al. Apr 2006 A1
20060107226 Matthews et al. May 2006 A1
20060173992 Weber et al. Aug 2006 A1
20060212925 Shull et al. Sep 2006 A1
20060230039 Shull et al. Oct 2006 A1
20060253458 Dixon et al. Nov 2006 A1
20060253581 Dixon et al. Nov 2006 A1
20060271564 Meng Muntz et al. Nov 2006 A1
20070016948 Dubrovsky et al. Jan 2007 A1
20070067845 Wiemer Mar 2007 A1
20070113282 Ross May 2007 A1
20070136622 Price et al. Jun 2007 A1
20070143851 Nicodemus et al. Jun 2007 A1
20070174915 Gribble et al. Jul 2007 A1
20070179955 Croft et al. Aug 2007 A1
20070198275 Malden et al. Aug 2007 A1
20070214151 Thomas et al. Sep 2007 A1
20070282730 Carpenter et al. Dec 2007 A1
20080017526 Prescott et al. Jan 2008 A1
20080033775 Dawson et al. Feb 2008 A1
20080047018 Baudoin et al. Feb 2008 A1
20080091834 Norton Apr 2008 A1
20080097980 Sullivan Apr 2008 A1
20080127338 Cho et al. May 2008 A1
20080140495 Bhamidipaty et al. Jun 2008 A1
20080140728 Fraser et al. Jun 2008 A1
20080148408 Kao et al. Jun 2008 A1
20080162931 Lord et al. Jul 2008 A1
20080172382 Prettejohn Jul 2008 A1
20080175266 Alperovitch et al. Jul 2008 A1
20080208995 Takahashi et al. Aug 2008 A1
20080209565 Baudoin et al. Aug 2008 A2
20080222287 Bahl Sep 2008 A1
20080222736 Boodaei et al. Sep 2008 A1
20080262895 Hofmeister et al. Oct 2008 A1
20080270458 Gvelesiani Oct 2008 A1
20090019525 Yu et al. Jan 2009 A1
20090024663 McGovern Jan 2009 A1
20090044272 Jarrett Feb 2009 A1
20090064337 Chien Mar 2009 A1
20090094265 Vlachos et al. Apr 2009 A1
20090094697 Provos et al. Apr 2009 A1
20090125427 Atwood et al. May 2009 A1
20090132861 Costa et al. May 2009 A1
20090150999 Dewey et al. Jun 2009 A1
20090161629 Purkayastha et al. Jun 2009 A1
20090193054 Karimisetty et al. Jul 2009 A1
20090204235 Dubinsky Aug 2009 A1
20090216700 Bouchard et al. Aug 2009 A1
20090228830 Herz et al. Sep 2009 A1
20090265787 Baudoin et al. Oct 2009 A9
20090276835 Jackson et al. Nov 2009 A1
20090293128 Lippmann et al. Nov 2009 A1
20090299802 Brennan Dec 2009 A1
20090300768 Krishnamurthy et al. Dec 2009 A1
20090319420 Sanchez et al. Dec 2009 A1
20090323632 Nix Dec 2009 A1
20090328063 Corvera et al. Dec 2009 A1
20100017880 Masood Jan 2010 A1
20100024033 Kang et al. Jan 2010 A1
20100042605 Cheng et al. Feb 2010 A1
20100057582 Arfin et al. Mar 2010 A1
20100114634 Christiansen et al. May 2010 A1
20100114757 Jeng May 2010 A1
20100180344 Malyshev et al. Jul 2010 A1
20100186088 Banerjee et al. Jul 2010 A1
20100205042 Mun Aug 2010 A1
20100218256 Thomas et al. Aug 2010 A1
20100235910 Ku et al. Sep 2010 A1
20100251000 Lyne et al. Sep 2010 A1
20100251371 Brown Sep 2010 A1
20100262444 Atwal et al. Oct 2010 A1
20100275263 Bennett et al. Oct 2010 A1
20100281124 Westman et al. Nov 2010 A1
20100281151 Ramankutty et al. Nov 2010 A1
20100309206 Xie et al. Dec 2010 A1
20110099620 Stavrou et al. Apr 2011 A1
20110106920 Longo May 2011 A1
20110137704 Mitra et al. Jun 2011 A1
20110145168 Dirnstorfer Jun 2011 A1
20110145576 Bettan Jun 2011 A1
20110148880 De Peuter Jun 2011 A1
20110185403 Dolan et al. Jul 2011 A1
20110185427 Aciicmez et al. Jul 2011 A1
20110213742 Lemmond et al. Sep 2011 A1
20110219455 Bhagwan et al. Sep 2011 A1
20110225085 Takeshita Sep 2011 A1
20110231395 Vadlamani et al. Sep 2011 A1
20110239294 Kim et al. Sep 2011 A1
20110239300 Klein et al. Sep 2011 A1
20110249002 Duplessis et al. Oct 2011 A1
20110276514 Kalagnanam Nov 2011 A1
20110282997 Prince et al. Nov 2011 A1
20110289582 Kejriwal et al. Nov 2011 A1
20110296519 Ide et al. Dec 2011 A1
20120008974 Kawai et al. Jan 2012 A1
20120036263 Madden et al. Feb 2012 A1
20120036580 Gorny et al. Feb 2012 A1
20120059823 Barber et al. Mar 2012 A1
20120079596 Thomas et al. Mar 2012 A1
20120089745 Turakhia Apr 2012 A1
20120158725 Molloy et al. Jun 2012 A1
20120166458 Laudanski et al. Jun 2012 A1
20120174219 Hernandez et al. Jul 2012 A1
20120198558 Liu et al. Aug 2012 A1
20120215892 Wanser et al. Aug 2012 A1
20120221376 Austin Aug 2012 A1
20120254993 Sallam Oct 2012 A1
20120255021 Sallam Oct 2012 A1
20120255027 Kanakapura et al. Oct 2012 A1
20120290498 Jones Nov 2012 A1
20120291129 Shulman et al. Nov 2012 A1
20130014253 Neou et al. Jan 2013 A1
20130055070 Sacks et al. Feb 2013 A1
20130055386 Kim et al. Feb 2013 A1
20130060351 Imming et al. Mar 2013 A1
20130080505 Nielsen et al. Mar 2013 A1
20130086521 Grossele et al. Apr 2013 A1
20130086681 Jaroch Apr 2013 A1
20130086687 Chess et al. Apr 2013 A1
20130091574 Howes Apr 2013 A1
20130124644 Hunt et al. May 2013 A1
20130124653 Vick et al. May 2013 A1
20130142050 Luna Jun 2013 A1
20130145437 Zaitsev Jun 2013 A1
20130173791 Longo Jul 2013 A1
20130212479 Willis et al. Aug 2013 A1
20130227078 Wei et al. Aug 2013 A1
20130227697 Zandani Aug 2013 A1
20130238527 Jones Sep 2013 A1
20130263270 Cote et al. Oct 2013 A1
20130276056 Epstein Oct 2013 A1
20130282406 Snyder et al. Oct 2013 A1
20130291105 Yan Oct 2013 A1
20130298244 Kumar et al. Nov 2013 A1
20130305368 Ford Nov 2013 A1
20130318594 Hoy et al. Nov 2013 A1
20130333038 Chien Dec 2013 A1
20130347116 Flores Dec 2013 A1
20140006129 Heath Jan 2014 A1
20140019196 Wiggins et al. Jan 2014 A1
20140052998 Bloom et al. Feb 2014 A1
20140101006 Pitt Apr 2014 A1
20140108474 David et al. Apr 2014 A1
20140114755 Mezzacca Apr 2014 A1
20140114843 Klein et al. Apr 2014 A1
20140130158 Wang et al. May 2014 A1
20140137254 Ou et al. May 2014 A1
20140137257 Martinez et al. May 2014 A1
20140146370 Banner et al. May 2014 A1
20140173066 Newton et al. Jun 2014 A1
20140173736 Liu Jun 2014 A1
20140189098 MaGill et al. Jul 2014 A1
20140189864 Wang et al. Jul 2014 A1
20140204803 Nguyen et al. Jul 2014 A1
20140237545 Mylavarapu et al. Aug 2014 A1
20140244317 Roberts et al. Aug 2014 A1
20140282261 Ranz et al. Sep 2014 A1
20140283056 Bachwani et al. Sep 2014 A1
20140283067 Call et al. Sep 2014 A1
20140283068 Call et al. Sep 2014 A1
20140283069 Call et al. Sep 2014 A1
20140288996 Rence et al. Sep 2014 A1
20140304816 Klein et al. Oct 2014 A1
20140330616 Lyras Nov 2014 A1
20140334336 Chen et al. Nov 2014 A1
20140337086 Asenjo et al. Nov 2014 A1
20140337633 Yang et al. Nov 2014 A1
20140344332 Giebler Nov 2014 A1
20150033331 Stern et al. Jan 2015 A1
20150033341 Schmidtler et al. Jan 2015 A1
20150052607 Al Hamami Feb 2015 A1
20150074579 Gladstone et al. Mar 2015 A1
20150081860 Kuehnel et al. Mar 2015 A1
20150088783 Mun Mar 2015 A1
20150156084 Kaminsky et al. Jun 2015 A1
20150180883 Aktas et al. Jun 2015 A1
20150195299 Zoldi et al. Jul 2015 A1
20150207776 Morin et al. Jul 2015 A1
20150213259 Du et al. Jul 2015 A1
20150248280 Pillay et al. Sep 2015 A1
20150261955 Huang et al. Sep 2015 A1
20150264061 Ibatullin et al. Sep 2015 A1
20150288706 Marshall Oct 2015 A1
20150288709 Singhal et al. Oct 2015 A1
20150310188 Ford et al. Oct 2015 A1
20150310213 Ronen et al. Oct 2015 A1
20150317672 Espinoza et al. Nov 2015 A1
20150331932 Georges et al. Nov 2015 A1
20150339479 Wang et al. Nov 2015 A1
20150347754 Born Dec 2015 A1
20150347756 Hidayat et al. Dec 2015 A1
20150350229 Mitchell Dec 2015 A1
20150381649 Schultz Dec 2015 A1
20160014081 Don, Jr. et al. Jan 2016 A1
20160023639 Cajiga et al. Jan 2016 A1
20160028746 Tonn Jan 2016 A1
20160036849 Zakian Feb 2016 A1
20160065613 Cho et al. Mar 2016 A1
20160078382 Watkins et al. Mar 2016 A1
20160088015 Sivan et al. Mar 2016 A1
20160104071 Brueckner Apr 2016 A1
20160119373 Fausto Apr 2016 A1
20160140466 Sidebottom et al. May 2016 A1
20160142419 Antipa et al. May 2016 A1
20160142428 Pastore et al. May 2016 A1
20160147992 Zhao et al. May 2016 A1
20160162602 Bradish et al. Jun 2016 A1
20160171415 Yampolskiy et al. Jun 2016 A1
20160173520 Foster et al. Jun 2016 A1
20160173522 Yampolskiy et al. Jun 2016 A1
20160182537 Tatourian et al. Jun 2016 A1
20160189301 Ng et al. Jun 2016 A1
20160191554 Kaminsky Jun 2016 A1
20160205126 Boyer et al. Jul 2016 A1
20160212101 Reshadi et al. Jul 2016 A1
20160239772 Dahlberg Aug 2016 A1
20160241560 Reshadi et al. Aug 2016 A1
20160248797 Yampolskiy et al. Aug 2016 A1
20160253500 Alme et al. Sep 2016 A1
20160259945 Yampolskiy et al. Sep 2016 A1
20160335232 Born et al. Nov 2016 A1
20160337387 Hu et al. Nov 2016 A1
20160344769 Li Nov 2016 A1
20160344801 Akkarawittayapoom Nov 2016 A1
20160359875 Kim et al. Dec 2016 A1
20160364496 Li Dec 2016 A1
20160373485 Kamble Dec 2016 A1
20160378978 Singla et al. Dec 2016 A1
20170048267 Yampolskiy et al. Feb 2017 A1
20170063901 Muddu et al. Mar 2017 A1
20170063923 Yang et al. Mar 2017 A1
20170104783 Vanunu et al. Apr 2017 A1
20170126719 Mason May 2017 A1
20170142148 Buber et al. May 2017 A1
20170161253 Silver Jun 2017 A1
20170161409 Martin Jun 2017 A1
20170161859 Baumgartner Jun 2017 A1
20170213292 Sweeney et al. Jul 2017 A1
20170221072 AthuluruTlrumala et al. Aug 2017 A1
20170223002 Sabin et al. Aug 2017 A1
20170236078 Rasumov Aug 2017 A1
20170237764 Rasumov Aug 2017 A1
20170264623 Ficarra et al. Sep 2017 A1
20170277892 MacDermid Sep 2017 A1
20170279843 Schultz Sep 2017 A1
20170289109 Caragea Oct 2017 A1
20170300911 Alnajem Oct 2017 A1
20170316324 Barrett et al. Nov 2017 A1
20170318045 Johns et al. Nov 2017 A1
20170324555 Wu et al. Nov 2017 A1
20170324766 Gonzalez Granadillo Nov 2017 A1
20170337487 Nock et al. Nov 2017 A1
20180013716 Connell et al. Jan 2018 A1
20180041521 Zhang et al. Feb 2018 A1
20180052999 Mitola, III Feb 2018 A1
20180088968 Myhre et al. Mar 2018 A1
20180103043 Kupreev et al. Apr 2018 A1
20180121659 Sawhney et al. May 2018 A1
20180123934 Gissing et al. May 2018 A1
20180124091 Sweeney et al. May 2018 A1
20180124110 Hunt et al. May 2018 A1
20180139180 Napchi et al. May 2018 A1
20180146004 Belfiore, Jr. et al. May 2018 A1
20180157468 Stachura Jun 2018 A1
20180191768 Broda et al. Jul 2018 A1
20180218157 Price et al. Aug 2018 A1
20180219910 Greenshpan et al. Aug 2018 A1
20180285414 Kondiles et al. Oct 2018 A1
20180322584 Crabtree Nov 2018 A1
20180324201 Lowry et al. Nov 2018 A1
20180332076 Callahan et al. Nov 2018 A1
20180336348 Ng et al. Nov 2018 A1
20180337938 Kneib et al. Nov 2018 A1
20180337941 Kraning et al. Nov 2018 A1
20180349641 Barday et al. Dec 2018 A1
20180365519 Pollard et al. Dec 2018 A1
20180375896 Wang et al. Dec 2018 A1
20180375953 Casassa Mont et al. Dec 2018 A1
20190034845 Mo et al. Jan 2019 A1
20190052650 Hu et al. Feb 2019 A1
20190065545 Hazel et al. Feb 2019 A1
20190065748 Foster et al. Feb 2019 A1
20190079869 Baldi et al. Mar 2019 A1
20190089711 Faulkner Mar 2019 A1
20190098025 Lim Mar 2019 A1
20190124091 Ujiie et al. Apr 2019 A1
20190140925 Pon et al. May 2019 A1
20190141060 Lim May 2019 A1
20190147378 Mo et al. May 2019 A1
20190166152 Steele et al. May 2019 A1
20190166156 King-Wilson May 2019 A1
20190179490 Barday et al. Jun 2019 A1
20190215331 Anakata et al. Jul 2019 A1
20190238439 Pugh et al. Aug 2019 A1
20190297106 Geil Sep 2019 A1
20190303574 Lamay et al. Oct 2019 A1
20190303584 Yang et al. Oct 2019 A1
20190362280 Vescio Nov 2019 A1
20190379632 Dahlberg et al. Dec 2019 A1
20190391707 Ristow et al. Dec 2019 A1
20190392252 Fighel et al. Dec 2019 A1
20200012794 Saldanha et al. Jan 2020 A1
20200053127 Brotherton et al. Feb 2020 A1
20200065213 Poghosyan et al. Feb 2020 A1
20200074084 Dorrans et al. Mar 2020 A1
20200092172 Kumaran et al. Mar 2020 A1
20200097845 Shaikh et al. Mar 2020 A1
20200104488 Li et al. Apr 2020 A1
20200106798 Lin Apr 2020 A1
20200120118 Shu et al. Apr 2020 A1
20200125734 Light et al. Apr 2020 A1
20200134175 Marwah et al. Apr 2020 A1
20200183655 Barday et al. Jun 2020 A1
20200186546 Dichiu et al. Jun 2020 A1
20200272763 Brannon et al. Aug 2020 A1
20200285737 Kraus et al. Sep 2020 A1
20200356689 McEnroe et al. Nov 2020 A1
20200356695 Brannon et al. Nov 2020 A1
20210064746 Koide et al. Mar 2021 A1
20210073377 Coull et al. Mar 2021 A1
Foreign Referenced Citations (2)
Number Date Country
WO-2017142694 Jan 2019 WO
WO-2019023045 Jan 2019 WO
Non-Patent Literature Citations (217)
Entry
U.S. Appl. No. 15/271,655 Published as: US 2018/0083999, Self-Published Security Risk Management, filed Sep. 21, 2016.
U.S. Appl. No. 15/377,574 U.S. Pat. No. 9,705,932, Methods and Systems for Creating, De-Duplicating, and Accessing Data Using an Object Storage System, filed Dec. 13, 2016.
U.S. Appl. No. 14/021,585 U.S. Pat. No. 9,438,615 Published as: US2015/0074579, Security Risk Management, filed Sep. 9, 2013.
U.S. Appl. No. 15/216,955 U.S. Pat. No. 10,326,786 Published as: US 2016/0330231, Methods for Using Organizational Behavior for Risk Ratings, filed Jul. 22, 2016.
U.S. Appl. No. 15/239,063 U.S. Pat. No. 10,341,370 Published as: US2017/0093901, Security Risk Management, filed Aug. 17, 2016.
U.S. Appl. No. 18/453,488, Information Technology Security Assessment System, filed Aug. 22, 2023.
U.S. Appl. No. 18/461,087, Information Technology Security Assessment System, filed Sep. 5, 2023.
U.S. Appl. No. 17/523,166 Published as: US2022/0121753, Systems and Methods for Forecasting Cybersecurity Ratings Based on Event-Rate Scenarios, filed Nov. 10, 2021.
U.S. Appl. No. 18/455,838, Systems and Methods for Forecasting Cybersecurity Ratings Based on Event-Rate Scenarios, filed Aug. 25, 2023.
U.S. Appl. No. 18/454,959, Systems and Methods for Rapidly Generating Security Ratings, filed Aug. 24, 2023.
U.S. Appl. No. 29/815,855, Computer Display With a Graphical User Interface for Cybersecurity Risk Management, filed Nov. 17, 2021.
U.S. Appl. No. 18/359,183, Systems and Methods for Assessing Cybersecurity Efficacy of Entities Against Common Control and Maturity Frameworks Using Externally-Observed Datasets, filed Jul. 26, 2023.
“Agreed Upon Procedures,” Version 4.0, BITS, The Financial Institution Shared Assessments Program, Assessment Guide, Sep. 2008, 56 pages.
“Amazon Mechanical Turk,” accessed on the internet at https://www.mturk.com/, (Nov. 9, 2018), 7 pages.
“An Executive View of IT Governance,” IT Governance Institute, 2009, 32 pages.
“Assessing Risk in Turbulent Times,” A Workshop for Information Security Executives, Glassmeyter/McNamee Center for Digital Strategies, Tuck School of Business at Dartmouth, Institute for Information Infrastructure Protection, 2009, 17 pages.
“Assuring a Trusted and Resilient Information and Communications Infrastructure,” Cyberspace Policy Review, May 2009, 76 pages.
“Computer Network Graph,” http://www.opte.org, accessed on the internet at http://www.opte.org, (Nov. 9, 2018), 1 page.
“Creating Transparency with Palantir,” accessed on the internet at https://www.youtube.com/watch?v=8cbGChfagUA; Jul. 5, 2012; 1 page.
“Master Security Criteria,” Version 3.0, BITS Financial Services Security Laboratory, Oct. 2001, 47 pages.
“Neo4j (neo4j.com),” accessed on the internet at https://web.archive.org/web/20151220150341/http://neo4j.com:80/developer/guide-data-visualization/; Dec. 20, 2015; 1 page.
“Palantir Cyber: Uncovering malicious behavior at petabyte scale,” accessed on the internet at https://www.youtube.com/watch?v= EhYezV06EE; Dec. 21, 2012; 1 page.
“Palantir.com,” accessed on the internet at http://www.palantir.com/; Dec. 2015; 2 pages.
“Plugging the Right Holes,” Lab Notes, MIT Lincoln Library, Posted Jul. 2008, retrieved Sep. 14, 2010 from http://www.ll.miLedufpublicationsflabnotesfpluggingtherightho! . . . , 2 pages.
“Rapid7 Nexpose Vulnerability Scanner,” accessed on the internet at https://web.archive.org/web/20170520082737/https://www.rapid7.com/products/nexpose/; May 20, 2017.
“Report on Controls Placed in Operation and Test of Operating Effectiveness,” EasCorp, Jan. 1 through Dec. 31, 2008, prepared by Crowe Horwath, 58 pages.
“Shared Assessments: Getting Started,” BITS, 2008, 4 pages.
“Tenable Nessus Network Vulnerability Scanner,” accessed on the internet at https://www.tenable.com/products/nessus/nessus-professional, (Nov. 9, 2018), 13 pages.
“Twenty Critical Controls for Effective Cyber Defense: Consensus Audit,” Version 2.3, Nov. 13, 2009, retrieved on Apr. 9, 2010 from http://www.sans.org/critical-security-controls/print.php., 52 pages.
2009 Data Breach Investigations Report, study conducted by Verizon Business RISK Team, 52 pages.
U.S. Appl. No. 13/240,572.
Artz, Michael Lyle, “NetSPA: A Network Security Planning Architecture,” Massachusetts Institute of Technology, May 24, 2002, 97 pages.
Azman, Mohamed et al. Wireless Daisy Chain and Tree Topology Networks for Smart Cities. 2019 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT). https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber= 8869252 (Year: 2019).
Basinya, Evgeny A.; Yushmanov, Anton A. Development of a Comprehensive Security System. 2019 Dynamics of Systems, Mechanisms and Machines (Dynamics). https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8944700 (Year: 2019).
Bhilare et al., “Protecting Intellectual Property and Sensitive Information in Academic Campuses from Trusted Insiders: Leveraging Active Directory”, SIGUCC, Oct. 2009 (5 pages).
Bitsight, “Cyber Security Myths Versus Reality: How Optimism Bias Contributes to Inaccurate Perceptions of Risk”, Jun. 2015, Dimensional Research, pp. 1-9.
Borgatti, et al., “On Social Network Analysis in a Supply Chain Context,” Journal of Supply Chain Management; 45(2): 5-22; Apr. 2009, 18 pages.
Boyer, Stephen, et al., Playing with Blocks: SCAP-Enable Higher-Level Analyses, MIT Lincoln Laboratory, 5th Annual IT Security Automation Conference, Oct. 26-29, 2009, 35 pages.
Buckshaw, Donald L., “Use of Decision Support Techniques for Information System Risk Management,” submitted for publication in Wiley's Encyclopedia of Quantitative Risk Assessment in Jan. 2007, 11 pages.
Buehler, Kevin S., et al., “Running with risk,” The Mckinsey Quarterly, No. 4, 2003, pp. 40-49.
Camelo, “Botnet Cluster Identification,” Sep. 2014, 90 pages.
Camelo, “Condenser: A Graph-based Approach for Detecting Botnets,” AnubisNetworks R&D, Amadora, Portugal and CENTRIA, Universidade NOVA de Lisboa, Portugal, 8 pages, (Oct. 31, 2014).
Carstens, et al., “Modeling Company Risk and Importance in Supply Graphs,” European Semantic Web Conference 2017: The Semantic Web, pp. 18-31, (May 7, 2017).
Chernyshev, M. et al., “On 802.11 Access Point Locatability and Named Entity Recognition in Service Set Identifiers”, IEEE Trans. on Info. and Sec., vol. 11 No. 3 (Mar. 2016).
Chu, Matthew, et al., “Visualizing Attack Graphs, Reachability, and Trust Relationships with NAVIGATOR,” MIT Lincoln Library, VizSEC '10, Ontario, Canada, Sep. 14, 2010, 12 pages.
Chuvakin, “SIEM: Moving beyond compliance”, RSA White Paper (2010) (16 pages).
Computer Network Graph-Bees, http://bioteams.com/2007/04/30/visualizing_complex_networks.html, date accessed Sep. 28, 2016, 2 pages.
Computer Network Graph—Univ. of Michigan, http://people.cst.cmich.edu/liao1q/research.shtml, date accessed Sep. 28, 2016, 5 pages.
Crowther, Kenneth G., et al., “Principles for Better Information Security through More Accurate, Transparent Risk Scoring,” Journal of Homeland Security and Emergency Management, vol. 7, Issue 1, Article 37, 2010, 20 pages.
Davis, Lois M., et al., “The National Computer Security Survey (NCSS) Final Methodology,” Technical report prepared for the Bureau of Justice Statistics, Safety and Justice Program, RAND Infrastructure, Safety and Environment (ISE), 2008, 91 pages.
Dillon-Merrill, PhD., Robin L, et al., “Logic Trees: Fault, Success, Attack, Event, Probability, and Decision Trees,” Wiley Handbook of Science and Technology for Homeland Security, 13 pages, (Mar. 15, 2009).
Dun & Bradstreet Corp. Stock Report, Standard & Poor's, Jun. 6, 2009, 8 pages.
Edmonds, Robert, “ISC Passive DNS Architecture”, Internet Systems Consortium, Inc., Mar. 2012, 18 pages.
Equifax Inc. Stock Report, Standard & Poor's, Jun. 6, 2009, 8 pages.
Gephi (gephi.org), accessed on the internet at https://web.archive.org/web/20151216223216/https://gephi.org/; Dec. 16, 2015; 1 page.
Gilgur, et al., “Percentile-Based Approach to Forecasting Workload Growth” Proceedings of CMG'15 Performance and Capacity International Conference by the Computer Measurement Group. No. 2015 (Year:2015), 16 pages.
Gundert, Levi, “Big Data in Security—Part III: Graph Analytics,” accessed on the Internet at https://blogs.cisco.com/security/big-data-in-security-part-iii-graph-analytics; Cisco Blog, Dec. 2013, 8 pages.
Hachem, Sara; Toninelli, Alessandra; Pathak, Animesh; Issany, Valerie. Policy-Based Access Control in Mobile Social Ecosystems. 2011 IEEE International Symposium on Policies for Distributed Systems and Networks (POLICY). Http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=5976796. 8 pages, (Jun. 6, 2011).
Hacking Exposed 6, S. McClure et al., copyright 2009, 37 pages.
Ingols, Kyle, et al., “Modeling Modern Network Attacks and Countermeasures Using Attack Graphs,” MIT Lincoln Laboratory, 16 pages, (Dec. 7, 2009).
Ingols, Kyle, et al., “Practical Attack Graph Generation for Network Defense,” MIT Lincoln Library, IEEE Computer Society, Proceedings of the 22nd Annual Computer Security Applications Conference (ACSAC'06), 2006, 10 pages.
Ingols, Kyle, et al., “Practical Experiences Using SCAP to Aggregate CND Data,” MIT Lincoln Library, Presentation to NIST SCAP Conference, Sep. 24, 2008, 59 pages.
Jean, “Cyber Security: How to use graphs to do an attack analysis,” accessed on the internet at https://linkurio.us/blog/cyber-security-use-graphs-attack-analysis/; Aug. 2014, 11 pages.
Jin et al., “Identifying and tracking suspicious activities through IP gray space analysis”, MineNet, Jun. 12, 2007 (6 pages).
Johnson, Eric, et al., “Information Risk and the Evolution of the Security Rating Industry,” Mar. 24, 2009, 27 pages.
Joslyn, et al., “Massive Scale Cyber Traffic Analysis: A Driver for Graph Database Research,” Proceedings of the First International Workshop on Graph Data Management Experience and Systems (GRADES 2013), 6 pages.
KC Claffy, “Internet measurement and data analysis: topology, workload, performance and routing statistics,” accessed on the Internet at http://www.caida.org/publications/papers/1999/Nae/Nae.html., NAE '99 workshop, 1999, 22 pages.
Li et al., “Finding the Linchpins of the Dark Web: a Study on Topologically Dedicated Hosts on Malicious Web Infrastructures”, IEEE, 2013 (15 pages).
Lippmann, Richard, et al., “Validating and Restoring Defense in Depth Using Attack Graphs,” MIT Lincoln Laboratory, 10 pages, (Oct. 23, 2006).
Lippmann, RP., et al., “An Annotated Review of Papers on Attack Graphs,” Project Report IA-1, Lincoln Laboratory, Massachusetts Institute of Technology, Mar. 31, 2005, 39 pages.
Lippmann, RP., et al., “Evaluating and Strengthening Enterprise Network Security Using Attack Graphs,” Project Report IA-2, MIT Lincoln Laboratory, Oct. 5, 2005, 96 pages.
Luo, Hui; Henry, Paul. A Secure Public Wireless LAN Access Technique That Supports Walk-Up Users. GLOBECOM '03. IEEE Global Telecommunications Conference. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber= 1258471 (Year: 2003).
Maltego XL, accessed on the Internet at https://www.paterva.com/web7/buy/maltego-clients/maltego-xl.php, 5 pages, (Nov. 7, 2018).
Massimo Candela, “Real-time BGP Visualisation with BGPlay,” accessed on the Internet at https://labs.ripe.net/Members/massimo_candela/real-time-bgp-visualisationwith-bgplay), Sep. 30, 2015, 8 pages.
MaxMind, https://www.maxmind.com/en/about-maxmind, https://www.maxmind.com/en/geoip2-isp-database, date accessed Sep. 28, 2016, 3 pages.
McNab, “Network Security Assessment,” copyright 2004, 13 pages.
McNab, “Network Security Assessment,” copyright 2004, 56 pages.
Method Documentation, CNSS Risk Assessment Tool Version 1.1, Mar. 31, 2009, 24 pages.
Mile 2 CPTE Maltego Demo, accessed on the internet at https://www.youtube.com/watch?v=o2oNKOUzPOU; Jul. 12, 2012; 1 page.
Moradi, et al., “Quantitative Models for Supply Chain Management,” IGI Global, 2012, 29 pages.
Morningstar Direct, dated to Nov. 12, 2020, morningstardirect.com [online]. Retrieved Feb. 26, 2021 from internet URL:https://web.archive.org/web/20201112021943/https://www.morningstar.com/products/direct, (Year: 2020).
Netcraft, www.netcraft.com, date accessed Sep. 28, 2016, 2 pages.
NetScanTools Pro, http://www.netscantools.com/nstpromain.html, date accessed Sep. 28, 2016, 2 pages.
Noel, et al., “Big-Data Architecture for Cyber Attack Graphs, Representing Security Relationships in NoSQL Graph Databases,” The MITRE Corporation, 2014, 6 pages.
Nye, John, “Avoiding Audit Overlap,” Moody's Risk Services, Presentation, Source Boston, Mar. 14, 2008, 19 pages.
U.S. Appl. No. 14/021,585.
U.S. Appl. No. 14/944,484.
U.S. Appl. No. 61/386,156.
Paxson, Vern, “How the Pursuit of Truth Led Me to Selling Viagra,” EECS Department, University of California, International Computer Science Institute, Lawrence Berkeley National Laboratory, Aug. 13, 2009, 68 pages.
Proposal and Award Policies and Procedures Guide, Part I—Proposal Preparation & Submission Guidelines GPG, The National Science Foundation, Feb. 2009, 68 pages.
Provos et al., “The Ghost in the Browser Analysis of Web-based Malware”, 2007 (9 pages).
Rare Events, Oct. 2009, Jason, The MITRE Corporation, Oct. 2009, 104 pages.
Rees, L. P. et al., “Decision support for cybersecurity risk planning.” Decision Support Systems 51.3 (2011): pp. 493-505.
Report to the Congress on Credit Scoring and Its Effects on the Availability and Affordability of Credit, Board of Governors of the Federal Reserve System, Aug. 2007, 304 pages.
RFC 1834, https://tools.ietf.org/html/rfc1834, date accessed Sep. 28, 2016, 7 pages.
RFC 781, https://tools.ietf.org/html/rfc781, date accessed Sep. 28, 2016, 3 pages.
RFC 950, https://tools.ietf.org/html/rfc950, date accessed Sep. 28, 2016, 19 pages.
RFC 954, https://tools.ietf.org/html/rfc954, date accessed Sep. 28, 2016, 5 pages.
SamSpade Network Inquiry Utility, https://www.sans.org/reading-room/whitepapers/tools/sam-spade-934, date accessed Sep. 28, 2016, 19 pages.
Santos, J. R. et al., “A framework for linking cybersecurity metrics to the modeling of macroeconomic interdependencies.” Risk Analysis: An International Journal (2007) 27.5, pp. 1283-1297.
SBIR Phase I: Enterprise Cyber Security Scoring, CyberAnalytix, LLC, http://www.nsf.gov/awardsearch/showAward. do?AwardNumber=1013603, Apr. 28, 2010, 2 pages.
Search Query Report form IP.com (performed Apr. 27, 2020).
Search Query Report from IP.com (performed Jul. 29, 2022).
Security Warrior, Cyrus Peikari, Anton, Chapter 8: Reconnaissance, 6 pages, (Jan. 2004).
Seigneur et al., A Survey of Trust and Risk Metrics for a BYOD Mobile Worker World: Third International Conference on Social Eco-Informatics, 2013, 11 pages.
Seneviratne et al., “SSIDs in the Wild: Extracting Semantic Information from WiFi SSIDs” HAL archives-ouvertes.fr, HAL Id: hal-01181254, Jul. 29, 2015, 5 pages.
Snort Intrusion Monitoring System, http://archive.oreilly.com/pub/h/1393, date accessed Sep. 28, 2016, 3 pages.
Srivastava, Divesh; Velegrakis, Yannis. Using Queries to Associate Metadata with Data. IEEE 23rd International Conference on Data Engineering. Pub. Date: 2007. http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=4221823, 3 pages.
Stone-Gross, Brett, et al., “FIRE: Finding Rogue Networks,” 10 pages, (Dec. 7, 2009).
Taleb, Nassim N., et al., “The Six Mistakes Executives Make in Risk Management,” Harvard Business Review, Oct. 2009, 5 pages.
The CIS Security Metrics vl.0.0, The Center for Internet Security, May 11, 2009, 90 pages.
The Fair Credit Reporting Act (FCRA) of the Federal Trade Commission (FTC), Jul. 30, 2004, 86 pages.
The Financial Institution Shared Assessments Program, Industry Positioning and Mapping Document, BITS, Oct. 2007, 44 pages.
Wagner, et al., “Assessing the vulnerability of supply chains using graph theory,” Int. J. Production Economics 126 (2010) 121-129.
Wikipedia, https://en.wikipedia.org/wiki/Crowdsourcing, date accessed Sep. 28, 2016, 25 pages.
Williams, Leevar, et al., “An Interactive Attack Graph Cascade and Reachability Display,” MIT Lincoln Laboratory, 17 pages, (Jan. 2007).
Williams, Leevar, et al., “GARNET: A Graphical Attack Graph and Reachability Network Evaluation Tool,” MIT Lincoln Library, VizSEC 2009, pp. 44-59, (Sep. 15, 2008).
Winship, C., “Models for sample selection bias”, Annual review of sociology, 18(1) (Aug. 1992), pp. 327-350.
U.S. Appl. No. 16/405,121 U.S. Pat. No. 10,785,245 Published as: US2019/0260791, Methods for Using Organizational Behavior for Risk Ratings, filed May 7, 2019.
U.S. Appl. No. 17/025,930 U.S. Pat. No. 11,652,834 Published as: US2021/0006581, Methods for Using Organizational Behavior for Risk Ratings, filed Sep. 18, 2020.
U.S. Appl. No. 18/297,863 Published as: US2023/0247041, Methods for Using Organizational Behavior for Risk Ratings, filed Apr. 10, 2023.
U.S. Appl. No. 13/240,572 U.S. Pat. No. 10,805,331 Published as: US2016/0205126, Information Technology Security Assessment System, filed Sep. 22, 2011.
U.S. Appl. No. 14/944,484 U.S. Pat. No. 9,973,524 Published as: US2016/0323308, Information Technology Security Assessment System, filed Nov. 18, 2015.
U.S. Appl. No. 17/069,151 U.S. Pat. No. 11,777,976 Published as: US2021/0211454, Information Technology Security Assessment System, filed Oct. 13, 2020.
U.S. Appl. No. 18/453,488 Published as: US2023/0403295, Information Technology Security Assessment System, filed Aug. 22, 2023.
U.S. Appl. No. 18/461,087 Published as: US2023/0421600, Information Technology Security Assessment System, filed Sep. 5, 2023.
U.S. Appl. No. 15/142,677 U.S. Pat. No. 9,830,569 Published as: US2016/0239772, Security Assessment Using Service Provider Digital Asset Information, filed Apr. 29, 2016.
U.S. Appl. No. 18/637,577, Information Technology Security Assessment System, filed Apr. 17, 2024.
U.S. Appl. No. 15/134,845 U.S. Pat. No. 9,680,858, Annotation Platform for a Security Risk System, filed Apr. 21, 2016.
U.S. Appl. No. 15/044,952 U.S. Pat. No. 11,182,720 Published as: US2017/0236077, Relationships Among Technology Assets and Services and the Entities Responsible for Them, filed Feb. 16, 2016.
U.S. Appl. No. 15/089,375 U.S. Pat. No. 10,176,445 Published as: US2017/0236079, Relationships Among Technology Assets and Services and the Entities Responsible for Them, filed Apr. 1, 2016.
U.S. Appl. No. 29/598,298 U.S. Pat. No. D. 835,631, Computer Display Screen With Graphical User Interface, filed Mar. 24, 2017.
U.S. Appl. No. 29/598,299 U.S. Pat. No. D. 818,475, Computer Display With Security Ratings Graphical User Interface, filed Mar. 24, 2017.
U.S. Appl. No. 29/599,622 U.S. Pat. No. D. 847,169, Computer Display With Security Ratings Graphical User Interface, filed Apr. 5, 2017.
U.S. Appl. No. 29/599,620 U.S. Pat. No. D. 846,562, Computer Display With Security Ratings Graphical User Interface, filed Apr. 5, 2017.
U.S. Appl. No. 16/015,686 U.S. Pat. No. 10,425,380 Published as: US2018/0375822, Methods for Mapping IP Addresses and Domains to Organizations Using User Activity Data, filed Jun. 22, 2018.
U.S. Appl. No. 16/543,075 U.S. Pat. No. 10,554,619 Published as: US Published as: US2019/0379632, Methods for Mapping IP Addresses and Domains to Organizations Using User Activity Data, filed Aug. 16, 2019.
U.S. Appl. No. 16/738,825 U.S. Pat. No. 10,893,021 Published as: US2020/0153787, Methods for Mapping IP Addresses and Domains to Organizations Using User Activity Data, filed Jan. 9, 2020.
U.S. Appl. No. 17/146,064 U.S. Pat. No. 11,627,109 Published as: US2021/0218702, Methods for Mapping IP Addresses and Domains to Organizations Using User Activity Data, filed Jan. 11, 2021.
U.S. Appl. No. 15/918,286 U.S. Pat. No. 10,257,219, Correlated Risk in Cybersecurity, filed Mar. 12, 2018.
U.S. Appl. No. 16/292,956 U.S. Pat. No. 10,594,723 Published as: US2019/0297106, Correlated Risk in Cybersecurity, filed Mar. 5, 2019.
U.S. Appl. No. 16/795,056 U.S. Pat. No. 10,931,705 Published as: US2020/0195681, Correlated Risk in Cybersecurity, filed Feb. 19, 2020.
U.S. Appl. No. 17/179,630 U.S. Pat. No. 11,770,401 Published as: US2021/0176269, Correlated Risk in Cybersecurity, filed Feb. 19, 2021.
U.S. Appl. No. 16/170,680 U.S. Pat. No. 10,521,583, Systems and Methods for Remote Detection of Software Through Browser WebInjects, filed Oct. 25, 2018.
U.S. Appl. No. 16/688,647 U.S. Pat. No. 10,776,483 Published as: US2020/0134174, Systems and Methods for Remote Detection of Software Through Browser WebInjects, filed Nov. 19, 2019.
U.S. Appl. No. 17/000,135 U.S. Pat. No. 11,126,723 Published as: US2021/0004457, Systems and Methods for Remote Detection of Software Through Browser WebInjects, filed Aug. 21, 2020.
U.S. Appl. No. 17/401,683 U.S. Pat. No. 11,727,114 Published as: US2021/0374243, Systems and Methods for Remote Detection of Software Through Browser WebInjects, filed Aug. 13, 2021.
U.S. Appl. No. 18/333,768 Published as: US2023/0325502, Systems and Methods for Remote Detection of Software Through Browser WebInjects, filed Jun. 13, 2023.
U.S. Appl. No. 15/954,921 U.S. Pat. No. 10,812,520 Published as: US2019/0319979, Systems and Methods for External Detection of Misconfigured Systems, filed Apr. 17, 2018.
U.S. Appl. No. 17/014,495 U.S. Pat. No. 11,671,441 Published as: US2020/0404017, Systems and Methods for External Detection of Misconfigured Systems, filed Sep. 8, 2020.
U.S. Appl. No. 18/302,925 Published as: US2023/0269267, Systems and Methods for External Detection of Misconfigured Systems, filed Apr. 19, 2023.
U.S. Appl. No. 16/549,764 Published as: US2021/0058421, Systems and Methods for Inferring Entity Relationships via Network Communications of Users or User Devices, filed Aug. 23, 2019.
U.S. Appl. No. 16/787,650 U.S. Pat. No. 10,749,893, Systems and Methods for Inferring Entity Relationships via Network Communications of Users or User Devices, filed Feb. 11, 2020.
U.S. Appl. No. 18/429,539, Systems and Methods for Inferring Entity Relationships via Network Communications of Users or User Devices, filed Feb. 1, 2024.
U.S. Appl. No. 16/583,991 U.S. Pat. No. 10,848,382, Systems and Methods for Network Asset Discovery and Association Thereof With Entities, filed Sep. 26, 2019.
U.S. Appl. No. 17/085,550 U.S. Pat. No. 11,329,878 Published as: US2021/0099347, Systems and Methods for Network Asset Discovery and Association Thereof With Entities, filed Oct. 30, 2020.
U.S. Appl. No. 29/666,942 U.S. Pat. No. D. 892,135, Computer Display With Graphical User Interface, filed Oct. 17, 2018.
U.S. Appl. No. 16/360,641 U.S. Pat. No. 11,200,323 Published as: US2020/0125734, Systems and Methods for Forecasting Cybersecurity Ratings Based on Event-Rate Scenarios, filed Mar. 21, 2019.
U.S. Appl. No. 17/523,166 U.S. Pat. No. 11,783,052 Published as: US2022/0121753, Systems and Methods for Forecasting Cybersecurity Ratings Based on Event-Rate Scenarios, filed Nov. 10, 2021.
U.S. Appl. No. 16/514,771 U.S. Pat. No. 10,726,136, Systems and Methods for Generating Security Improvement Plans for Entities, filed Jul. 17, 2019.
U.S. Appl. No. 16/922,673 U.S. Pat. No. 11,030,325 Published as: US2021/0019424, Systems and Methods for Generating Security Improvement Plans for Entities, filed Jul. 7, 2020.
U.S. Appl. No. 17/307,577 U.S. Pat. No. 11,675,912 Published as: US2021/0211454, Systems and Methods for Generating Security Improvement Plans for Entities, filed May 4, 2021.
U.S. Appl. No. 18/138,803 Published as US2023/0267215, Systems and Methods for Generating Security Improvement Plans for Entities, filed Apr. 25, 2023.
U.S. Appl. No. 29/677,306 U.S. Pat. No. D. 905,702, Computer Display Screen With Corporate Hierarchy Graphical User Interface, filed Jan. 18, 2019.
U.S. Appl. No. 16/775,840 U.S. Pat. No. 10,791,140, Systems and Methods for Assessing Cybersecurity State of Entities Based on Computer Network Characterization, filed Jan. 29, 2020.
U.S. Appl. No. 17/018,587 U.S. Pat. No. 11,050,779, Systems and Methods for Assessing Cybersecurity State of Entities Based on Computer Network Characterization, filed Sep. 11, 2020.
U.S. Appl. No. 16/779,437 U.S. Pat. No. 10,893,067 Published as: US2021/0243221, Systems and Methods for Rapidly Generating Security Ratings, filed Jan. 31, 2020.
U.S. Appl. No. 17/132,512 U.S. Pat. No. 11,595,427 Published as: US2021/0243221, Systems and Methods for Rapidly Generating Security Ratings, filed Dec. 23, 2020.
U.S. Appl. No. 18/158,594 U.S. Pat. No. 11,777,983, Systems and Methods for Rapidly Generating Security Ratings, filed Jan. 24, 2023.
U.S. Appl. No. 18/454,959 Published as: US2024/129332, Systems and Methods for Rapidly Generating Security Ratings, filed Aug. 24, 2023.
U.S. Appl. No. 17/119,822 U.S. Pat. No. 11,122,073, Systems and Methods for Cybersecurity Risk Mitigation and Management, filed Dec. 11, 2020.
U.S. Appl. No. 29/815,855 U.S. Pat. No. D. 1,010,666, Computer Display With a Graphical User Interface for Cybersecurity Risk Management, filed Nov. 17, 2021.
U.S. Appl. No. 17/392,521 U.S. Pat. No. 11,689,555 Published as US 2022/0191232, Systems and Methods for Cybersecurity Risk Mitigation and Management, filed Aug. 3, 2021.
U.S. Appl. No. 18/141,654 Published as: US2023/0269265, Systems and Methods for Cybersecurity Risk Mitigation and Management, filed May 1, 2023.
U.S. Appl. No. 29/916,503, Computer Display With a Graphical User Interface, filed Nov. 13, 2023.
U.S. Appl. No. 29/916,519, Computer Display With a Graphical User Interface, filed Nov. 13, 2023.
U.S. Appl. No. 16/802,232 U.S. Pat. No. 10,764,298, Systems and Methods for Improving a Security Profile of an Entity Based on Peer Security Profiles, filed Feb. 26, 2020.
U.S. Appl. No. 16/942,452 U.S. Pat. No. 11,265,330 Published as: US2021/0266324, Systems and Methods for Improving a Security Profile of an Entity Based on Peer Security Profiles, filed Jul. 29, 2020.
U.S. Appl. No. 29/725,724, Computer Display With Risk Vectors Graphical User Interface, filed Feb. 26, 2020.
U.S. Appl. No. 29/736,641 U.S. Pat. No. D. 937,870, Computer Display With Peer Analytics Graphical User Interface, filed Jun. 2, 2020.
U.S. Appl. No. 17/039,675 U.S. Pat. No. 11,032,244 Published as: US2021/0099428, Systems and Methods for Determining Asset Importance in Security Risk Management, filed Sep. 30, 2020.
U.S. Appl. No. 17/320,997 Published as US 2021/0344647, Systems and Methods for Determining Asset Importance in Security Risk Management, filed May 14, 2021.
U.S. Appl. No. 18/422,470, Systems and Methods for Determining Asset Importance in Security Risk Management, filed Jan. 25, 2024.
U.S. Appl. No. 16/884,607 U.S. Pat. No. 11,023,585, Systems and Methods for Managing Cybersecurity Alerts, filed May 27, 2020.
U.S. Appl. No. 17/236,594 U.S. Pat. No. 11,720,679 Published as: US2021/0374246, Systems and Methods for Managing Cybersecurity Alerts, filed Apr. 21, 2021.
U.S. Appl. No. 18/335,384 Published as: US2023/0325505, Systems and Methods for Managing Cybersecurity Alerts, filed Jun. 15, 2023.
U.S. Appl. No. 17/710,168 Published as: US2022/0318400, Systems and Methods for Assessing Cybersecurity Risk in a Work From Home Environment, filed Mar. 31, 2022.
U.S. Appl. No. 18/770,949, Systems and Methods for Assessing Cybersecurity Risk in a Work From Home Environment, filed Jul. 12, 2024.
U.S. Appl. No. 17/945,337 Published as US2023/0091953, Systems and Methods for Precomputation of Digital Asset Inventories, filed Sep. 15, 2022.
U.S. Appl. No. 18/359,183 Published as: US2024/0045950, Systems and Methods for Assessing Cybersecurity Efficacy of Entities Against Common Control and Maturity Frameworks Using Externally-Observed Datasets, filed Jul. 26, 2023.
U.S. Appl. No. 17/856,217 Published as: US2023/0004655, Systems and Methods for Accelerating Cybersecurity Assessments, filed Jul. 1, 2022.
U.S. Appl. No. 18/162,154 Published as: US2023/0244794, Systems and Methods for Assessment of Cyber Resilience, filed Jan. 31, 2023.
U.S. Appl. No. 18/328,142, Systems and Methods for Modeling Cybersecurity Breach Costs, filed Jun. 2, 2023.
U.S. Appl. No. 18/678,378, Systems and Methods for Predicting Cybersecurity Risk Based on Entity Firmographics, filed May 30, 2024.
834 Patent Claim Chart, BitSight Technologies, Inc. v. NormShield Inc. d/b/a Black Kite Inc., Case No. 1:23-cv-12055-MJJ, D.I. 39-11 (Dec. 11, 2023), 28 pages.
“Maltego User Guide” webpage http://ctas.paterva.com/view/Userguide, 35 pages, Jun. 6, 2012, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20120606172056/http://ctas.paterva.com/view/Userguide on Sep. 6, 2024.
“MW Metadata”, webpage https://mattw.io/youtube-metadata, 7 pages, retrieved on Aug. 21, 2024.
Anderson, H., “Nessus, Part 3: Analysing Reports,” webpage http://www.securityfocus.com/infocus/1759, 5 pages, Oct. 20, 2006, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20061020202310/http://www.securityfocus.com/infocus/1759 on Aug. 16, 2024.
Aug. 29, 2024 Email from Melissa Nezhnik, 3 pages.
Gates, C., “New School Information Gathering,” (2008), available at https://www.carnal0wnage.com/papers/17_Gates.pdf, 84 pages.
Gates, C., “Toorcon X Gates: New School Information Gathering,” 2 pages, Mar. 2, 2009, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20090302045813/vimeo.com//2745624 on Aug. 13, 2024.
Gates, C., “Toorcon X Gates: New School Information Gathering,” available at http://vimeo.com/2745624, 2 pages, retrieved on Aug. 13, 2024.
Khalil, J. et al., “Discovering Malicious Domains through Passive DNS Data Graph Analysis,” Conference Paper, (Jun. 2016), 13 pages.
Knowles, D. et al., “W32.Blaster.Worm: Technical Details” webpage http://www.symantec.com/security_response/writeup.jsp?docid=2003- 081113-0229-99&tabid=2, 3 pages, May 3, 2007, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20070503023514/http://www.symantec.com/security_response/writeup.jsp?docid=2003-081113-0229-99&tabid=2 on Aug. 16, 2024.
Levy, E., “The Making of a Spam Zombie Army,” IEEE Computer & Security (2003), pp. 58-59.
Lippmann, Rich, et al., “NetSPA: a Network Security Planning Architecture”, MIT Lincoln Laboratory, 2002, 11 pages.
Matta Security Limited, “An Introduction to Internet Attack & Penetration,” available at http:/www.trustmatta.com/downloads/pdf/, Matta_Attack_and_Penetration_Introduction.pdf, (2001-2002), 14 pages.
McNab, C., “Network Security Assessment,” O'Reilly Media, Inc., Second Edition, (2008), 506 pages.
Moore & Valsmith, et al., “Tactical Exploitation,” 37 pages, Feb. 8, 2010, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20100208161237/https:/www.blackhat.com/presentations/bh-usa-07/Moore_and_Valsmith/Whitepaper/bh-usa-07-moore_and_valsmith-WP.pdf.
Nessus, “Documentation,” webpage http://www.nessus.org/documentation/, 2 pages, Feb. 19, 2007, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20070219213924/http://www.nessus.org/documentation/ on Aug. 16, 2024.
Nessus, “Plugins: Symantec Anti Virus Corporate Edition Check,” webpage http://www.nessus.org/plugins/index.php?view=single&id=21725, 2 pages, Feb. 22, 2007, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20070222133717/http://www.nessus.org/plugins/index.php?view=single&id=21725 on Aug. 13, 2024.
Nessus, “Plugins: The remote host is infected by a virus”, webpage http://www.nessus.org/plugins/index.php?view=single&id=11329, 2 pages, Feb. 22, 2007, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20070222091638/http://www.nessus.org/plugins/index.php?view=single&id=11329 on Aug. 13, 2024.
Nessus, “Plugins: The remote host is infected by msblast.exe”, webpage http://www.nessus.org/plugins/index.php?view=single&id=11818, 1 page, Sep. 24, 2006, retrieved from Internet Archive Wayback Machine https://web.archive.org/web/20060924205758/http://www.nessus.org/plugins/index.php?view=single&id=11818 on Aug. 13, 2024.
Social-Engineer, LLC, “Social Engineering Using Maltego,” webpage <www.youtube.com/watch?v=qiv4-wy3mxo>, 2 pages, Sep. 14, 2009, retrieved on Aug. 13, 2024.
Social-Engineer, LLC, Screen captures from “Social Engineering Using Maltego,” webpage <www.youtube.com/watch?v=qiv4-wy3mxo>, 43 pages, Sep. 14, 2009.
Stoneburner, G. et al., “Risk Management Guide for Information Technology Systems,” NIST, available at https://www.archives.gov/files/era/recompete/sp800-30.pdf. (Jul. 2002), 55 pages.
Tenable Network Security, Inc., “Nessus 3.0 Client Guide,” available at http://nessus.org/documentation/nessus_3.0_client_guide.pdf, Mar. 6, 2007, 32 pages.
Related Publications (1)
Number Date Country
20230396644 A1 Dec 2023 US
Continuations (4)
Number Date Country
Parent 17179630 Feb 2021 US
Child 18365384 US
Parent 16795056 Feb 2020 US
Child 17179630 US
Parent 16292956 Mar 2019 US
Child 16795056 US
Parent 15918286 Mar 2018 US
Child 16292956 US