Patent Grant
10264014
This application is also related to the following U.S. Patent Applications: application Ser. No. 14/139,628, titled “TAX DATA CLUSTERING,” application Ser. No. 14/139,603, titled “MALWARE DATA CLUSTERING,” and application Ser. No. 14/139,713, titled “USER-AGENT DATA CLUSTERING.” Each of the above identified applications is hereby incorporated by reference herein in its entirety and for all purposes.
Embodiments of the present disclosure generally relate to data analysis and, more specifically, to generating data clusters of related data entities with customizable analysis strategies.
In financial and security investigations an analyst may have to make decisions regarding data entities within a collection of data. For instance, the analyst could have to decide whether an account data entity represents a fraudulent bank account. However, an individual data entity oftentimes includes insufficient information for the analyst to make such decisions.
The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, several non-limiting features will now be discussed briefly.
In the example noted above, the analyst may make better decisions based upon a collection of related data entities. For instance, two financial transactions may be related by an identical account identifier or two accounts belonging to one customer may be related by an identical customer identifier or other attribute (e.g., a shared phone number or address). Some currently available systems assist the analyst by identifying data entities that are directly related to an initial data entity. For example, the analyst could initiate an investigation with a single suspicious data entity or “seed,” such as a fraudulent credit card account. If the analyst examined this data entity by itself, then the analyst would not observe any suspicious characteristics. However, the analyst could request a list of data entities related to the seed by a shared attribute, such as a customer identifier. In doing so, the analyst could discover an additional data entity, such as an additional credit card account, which relates to the original fraudulent account because of a shared customer identifier. The analyst could then mark the additional credit card account as potentially fraudulent, based upon the relationship of the shared customer identifier.
Although these systems can be very helpful in discovering related data entities, they typically require the analyst to manually repeat the same series of searches for many investigations. Repeating the same investigation process consumes time and resources, such that there are oftentimes more investigations than can be performed. Thus, analysts typically prioritize investigations based upon the characteristics of the seeds. However, there may be insignificant differences between the seeds, so the analyst may not be able to determine the correct priority for investigations. For instance, the analyst could have to choose between two potential investigations based upon separate fraudulent credit card accounts. One investigation could reveal more potentially fraudulent credit card accounts than the other, and therefore could be more important to perform. Yet, the characteristics of the two original credit card accounts could be similar, so the analyst would not be able to choose the more important investigation. Without more information, prioritizing investigations is difficult and error prone.
According to various embodiments, a data analysis system is disclosed in which clusters of related data entities may be generated from initial data entities, called “seeds.” A data entity may include any data, information or thing, such as a person, a place, an organization, an account, a computer, an activity, and event, and the like. Seeds may be generated according to seed generation strategies, while clusters of related data entities may be generated based on those seeds and according to cluster generation strategies. The system may further generate a score, multiple scores, and/or metascores for each generated cluster, and may optionally rank or prioritize the generated clusters based on the generated metascores. High priority clusters may be of greater interest to an analyst as they may contain related data entities that meet particular criteria related to the analyst's investigation. In an embodiment, the system may enable an analyst to advantageously start an investigation with a prioritized cluster including many related data entities rather than a single randomly selected data entity.
In one embodiment, a method for generating a cluster of related data from a seed is disclosed. This method may generally include retrieving a seed and adding the seed to a first cluster. This method may further include retrieving a cluster strategy referencing one or more data bindings. Each data binding may specify a search protocol for retrieving data. For each of the one or more data bindings, data parameters input to the search protocol are identified, the search protocol is performed using the identified data parameters, and data returned by the search protocol is evaluated for inclusion in the first cluster.
In a particular embodiment, the search protocol for a first one of the data bindings uses the seed as the data parameters input to the search protocol. Further, the search protocol for a first one of the data bindings returns data used as the identified data parameters input to the search protocol for a second one of the data bindings. This process may iteratively continue until all data bindings in the search strategy have been executed for all available data obtained while growing the first cluster.
In a particular embodiment, this method may further include retrieving a second cluster from a plurality of clusters, comparing data from the first cluster to data from the second cluster, and determining whether to merge the first cluster and the second cluster based on the comparison.
Other embodiments include, without limitation, a computer-readable medium that includes instructions that enable a processing unit to implement one or more aspects of the disclosed methods as well as a system having a processor, memory, and application programs configured to implement one or more aspects of the disclosed methods.
According to another embodiment, a computer system is disclosed that comprises: one or more computer readable storage devices configured to store: one or more software modules including computer executable instructions; a plurality of tax-related data items and properties associated with respective tax-related data items, each of the properties including associated property values, at least some of the tax-related data items including tax-return data items; a plurality of tax-fraud indicators; a clustering strategy; and at least one scoring criterion; and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the one or more software modules in order to cause the computer system to: designate one or more seeds by: accessing, from the one or more computer readable storage devices, the tax-fraud indicators and at least one tax-related data item; comparing the tax-fraud indicators to the at least one tax-related data item and associated properties; and based on the comparison and in response to determining the at least one tax-related data item is related to at least one tax-fraud indicator, designating the at least one tax-related data item as a seed; for each designated seed, generate a cluster by: adding the seed to the cluster; adding one or more tax-return data items associated with the seed to the cluster; accessing, from the one or more computer readable storage devices, the clustering strategy; and adding to the cluster, based on the clustering strategy, one or more tax-related data items associated with the added tax-return data items; and score each generated cluster by: accessing, from the one or more computer readable storage devices, the least one scoring criterion; and generating a cluster score by assessing the generated cluster based on the accessed at least one scoring criterion.
According to yet another embodiment, a computer-implemented is disclosed that comprises: under control of one or more hardware computing devices configured with specific computer executable instructions, enabling communication with one or more computer readable storage devices configured to store: a plurality of tax-related data items and properties associated with respective tax-related data items, each of the properties including associated property values; a plurality of tax-fraud indicators; a clustering strategy; and at least one scoring criterion; designating one or more seeds by: accessing, from the one or more computer readable storage devices, the tax-fraud indicators and at least one tax-related data item; comparing the tax-fraud indicators to the at least one tax-related data item and associated properties; and based on the comparison and in response to determining the at least one tax-related data item is related to at least one tax-fraud indicator, designating the at least one tax-related data item as a seed; for each designated seed, generating a cluster by: adding the seed to the cluster; accessing, from the one or more computer readable storage devices, the clustering strategy; and adding to the cluster, based on the clustering strategy, one or more tax-related data items associated with the seed; and scoring each generated cluster by: accessing, from the one or more computer readable storage devices, the least one scoring criterion; and generating a cluster score by assessing the generated cluster based on the accessed at least one scoring criterion.
According to another embodiment, a non-transitory computer-readable storage medium is disclosed that stores computer-executable instructions that, when executed by a computer system, configure the computer system to perform operations comprising: enabling communication with one or more computer readable storage devices configured to store: a plurality of tax-related data items and properties associated with respective tax-related data items, each of the properties including associated property values; a plurality of tax-fraud indicators; a clustering strategy; and at least one scoring criterion; designating one or more seeds by: accessing, from the one or more computer readable storage devices, the tax-fraud indicators and at least one tax-related data item; comparing the tax-fraud indicators to the at least one tax-related data item and associated properties; and based on the comparison and in response to determining the at least one tax-related data item is related to at least one tax-fraud indicator, designating the at least one tax-related data item as a seed; for each designated seed, generating a cluster by: adding the seed to the cluster; accessing, from the one or more computer readable storage devices, the clustering strategy; and adding to the cluster, based on the clustering strategy, one or more tax-related data items associated with the seed; and scoring each generated cluster by: accessing, from the one or more computer readable storage devices, the least one scoring criterion; and generating a cluster score by assessing the generated cluster based on the accessed at least one scoring criterion.
According to yet another embodiment, a computer system is disclosed that comprises: one or more computer readable storage devices configured to store: one or more software modules including computer executable instructions; and a plurality of beaconing malware-related data items and properties associated with respective malware-related data items, each of the properties including associated property values, the beaconing malware-related data items including at least one of: data items associated with captured communications between an internal network and an external network, users of particular computerized devices, internal Internet Protocol addresses, external Internet Protocol addresses, external domains, internal computerized devices, external computerized devices, data feed items, or host-based events; and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the one or more software modules in order to cause the computer system to: access, from the one or more computer readable storage devices, the plurality of beaconing malware-related data items; generate, based on the accessed beaconing malware-related data items, a plurality of connection pairs, each of the connection pairs indicating communications between a particular internal source within the internal network and a particular external destination that is not within the internal network; identify a plurality of connection pairs having a common internal source and a common external destination; generate a time series of connection pairs based on the identified plurality of connection pairs; filter out noise from the at least one time series; compute a variance in the filtered at least one time series; and based on a determination that the variance satisfies a particular threshold, designate a connection pair associated with the filtered at least one time series as a seed, the designated connection pair including the common internal source and the common external source; generate a data item cluster based on the designated seed; and score the generated data item cluster.
According to another embodiment, a computer-implemented is disclosed that comprises: under control of one or more hardware computing devices configured with specific computer executable instructions, enabling communication with one or more computer readable storage devices configured to store: a plurality of beaconing malware-related data items and properties associated with respective malware-related data items, each of the properties including associated property values, the beaconing malware-related data items including at least one data item associated with captured communications between an internal network and an external network; accessing, from the one or more computer readable storage devices, the plurality of beaconing malware-related data items; generating, based on the accessed beaconing malware-related data items, a plurality of connection pairs, each of the connection pairs indicating communications between a particular internal source within the internal network and a particular external destination that is not within the internal network; identifying a plurality of connection pairs having a common internal source and a common external destination; generating a time series of connection pairs based on the identified plurality, of connection pairs; filtering out noise from the at least one time series; computing a variance in the filtered at least one time series; and based on a determination that the variance satisfies a particular threshold, designating a connection pair associated with the filtered at least one time series as a seed, the designated connection pair including the common internal source and the common external source; generating a data item cluster based on the designated seed; and scoring the generated data item cluster.
According to yet another embodiment, a non-transitory computer-readable storage medium is disclosed that stores computer-executable instructions that, when executed by a computer system, configure the computer system to perform operations comprising: enabling communication with one or more computer readable storage devices configured to store: a plurality of beaconing malware-related data items and properties associated with respective malware-related data items, each of the properties including associated property values, the beaconing malware-related data items including at least one data item associated with captured communications between an internal network and an external network; accessing, from the one or more computer readable storage devices, the plurality of beaconing malware-related data items; generating, based on the accessed beaconing malware-related data items, a plurality of connection pairs, each of the connection pairs indicating communications between a particular internal source within the internal network and a particular external destination that is not within the internal network; identifying a plurality of connection pairs having a common internal source and a common external destination; generating a time series of connection pairs based on the identified plurality of connection pairs; filtering out noise from the at least one time series; computing a variance in the filtered at least one time series; and based on a determination that the variance satisfies a particular threshold, designating a connection pair associated with the filtered at least one time series as a seed, the designated connection pair including the common internal source and the common external source; generating a data item cluster based on the designated seed; and scoring the generated data item cluster.
According to another embodiment, a computer system is disclosed that comprises: one or more computer readable storage devices configured to store: one or more software modules including computer executable instructions; and a plurality of user-agent-related data items and properties associated with respective user-agent-related data items, each of the properties including associated property values, the user-agent-related data items including at least one of: data items associated with captured communications between an internal network and an external network, user-agent strings, users of particular computerized devices, internal Internet Protocol addresses, external Internet Protocol addresses, external domains, internal computerized devices, external computerized devices, data feed items, or host-based events; and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the one or more software modules in order to cause the computer system to: designate one or more seeds by: accessing, from the one or more computer readable storage devices, a plurality of user-agent-related data items associated with the captured communications; filtering the accessed data items to remove data items that are unlikely to be related to malware activity; determining a first set of filtered data items associated with a test time period and a second set of filtered data items associated with a reference time period; and identifying one or more filtered data items in the first set that are not included among the filtered data items in the second set and designating each of the one or more identified filtered data items as a seed; and for each designated seed: generate a data item cluster; and score the generated data item cluster.
According to yet another embodiment, a computer-implemented is disclosed that comprises: under control of one or more hardware computing devices configured with specific computer executable instructions, enabling communication with one or more computer readable storage devices configured to store: a plurality of user-agent-related data items and properties associated with respective user-agent-related data items, each of the properties including associated property values; designating one or more seeds by: accessing, from the one or more computer readable storage devices, a plurality of user-agent-related data items associated with the captured communications; filtering the accessed data items to remove data items that are unlikely to be related to malware activity; determining a first set of filtered data items associated with a test time period and a second set of filtered data items associated with a reference time period; and identifying one or more filtered data item in the first set that are not included among the filtered data items in the second set and designating each of the one or more identified filtered data items as a seed; and for each designated seed; generating a data item cluster; and scoring the generated data item cluster.
According to another embodiment, a non-transitory computer-readable storage medium is disclosed that stores computer-executable instructions that, when executed by a computer system, configure the computer system to perform operations comprising: enabling communication with one or more computer readable storage devices configured to store: a plurality of user-agent-related data items and properties associated with respective user-agent-related data items, each of the properties including associated property values; designating one or more seeds by: accessing, from the one or more computer readable storage devices, a plurality of user-agent-related data items associated with the captured communications; filtering the accessed data items to remove data items that are unlikely to be related to malware activity; determining a first set of filtered data items associated with a test time period and a second set of filtered data items associated with a reference time period; and identifying one or more filtered data item in the first set that are not included among the filtered data items in the second set and designating each of the one or more identified filtered data items as a seed; and for each designated seed: generating a data item cluster; and scoring the generated data item cluster.
According to yet another embodiment, a computer system is disclosed that comprises: one or more computer readable storage devices configured to store: one or more software modules including computer executable instructions; and a plurality of activity trend-related data items and properties associated with respective trend-related data items, each of the properties including associated property values, the activity trend-related data items including at least one of: host-based events, data items associated with captured host-based events, Internet Protocol addresses, external domains, users, or computerized devices, wherein hosts comprise computerized devices in a network; and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the one or more software modules in order to cause the computer system to: designate one or more seeds by: accessing, from the one or more computer readable storage devices, a plurality of activity trend-related data items associated with the host-based events; determining a group of the plurality of activity trend-related data items each indicating a same particular activity type and associated with a particular host; determining, based on the group of activity trend-related data items, a statistical deviation in the particular type of activity on the particular host; and in response to determining that the statistical deviation satisfies a particular threshold, designating an activity trend-related data item from the group as a seed; and for each designated seed: generate a data item cluster; and score the generated data item cluster.
According to another embodiment, a computer-implemented is disclosed that comprises: under control of one or more hardware computing devices configured with specific computer executable instructions, enabling communication with one or more computer readable storage devices configured to store: a plurality of activity trend-related data items and properties associated with respective trend-related data items, each of the properties including associated property values; designating one or more seeds by: accessing, from the one or more computer readable storage devices, a plurality of activity trend-related data items associated with the host-based events; determining a group of the plurality of activity trend-related data items each indicating a same particular activity type and associated with a particular host; determining, based on the group of activity trend-related data items, a statistical deviation in the particular type of activity on the particular host; and in response to determining that the statistical deviation satisfies a particular threshold, designating an activity trend-related data item from the group as a seed; and for each designated seed: generating a data item cluster; and scoring the generated data item cluster.
According to yet another embodiment, a non-transitory computer-readable storage medium is disclosed that stores computer-executable instructions that, when executed by a computer system, configure the computer system to perform operations comprising: enabling communication with one or more computer readable storage devices configured to store: a plurality of activity trend-related data items and properties associated with respective trend-related data items, each of the properties including associated property values; designating one or more seeds by: accessing, from the one or more computer readable storage devices, a plurality of activity trend-related data items associated with the host-based events; determining a group of the plurality of activity trend-related data items each indicating a same particular activity type and associated with a particular host; determining, based on the group of activity trend-related data items, a statistical deviation in the particular type of activity on the particular host; and in response to determining that the statistical deviation satisfies a particular threshold, designating an activity trend-related data item from the group as a seed; and for each designated seed: generating a data item cluster; and scoring the generated data item cluster.
Advantageously, according to various embodiments, the disclosed techniques provide a more effective starting point for an investigation of financial, security, and/or other data entities. An analyst may be able to start an investigation from a cluster of related data entities instead of an individual data entity, which may reduce the amount of time and effort required to perform the investigation. The disclosed techniques may also, according to various embodiments, provide a prioritization of multiple clusters. For example, the analyst may also able to start the investigation from a high priority cluster, which may allow the analyst to focus on the most important investigations.
So that the manner in which the above recited features of the present invention may be understood in detail, a more particular description of various embodiments, briefly summarized above, may be had by reference to the appended drawings and detailed description. It is to be noted, however, that the appended drawings illustrate only typical embodiments of present disclosure and are therefore not to be considered limiting of its scope, for present disclosure may admit to other equally effective embodiments.
Example Application of the Data Analysis System to Tax Fraud Detection
Example Application of the Data Analysis System to Beaconing Malware Detection
Example Application of the Data Analysis System to Malware User-Agent Detection
Example Application of the Data Analysis System to Activity Trend Detection
In order to facilitate an understanding of the systems and methods discussed herein, a number of terms are defined below. The terms defined below, as well as other terms used herein, should be construed to include the provided definitions, the ordinary and customary meaning of the terms, and/or any other implied meaning for the respective terms. Thus, the definitions below do not limit the meaning of these terms, but only provide exemplary definitions.
Ontology: Stored information that provides a data model for storage of data in one or more databases. For example, the stored data may comprise definitions for object types and property types for data in a database, and how objects and properties may be related.
Database: A broad term for any data structure for storing and/or organizing data, including, but not limited to, relational databases (for example, Oracle database, mySQL database, and the like), spreadsheets, XML files, and text file, among others. The various terms “database,” “data store,” and “data source” may be used interchangeably in the present disclosure.
Data Entity (Entity), Data Object (Object), or Data Item (Item): A data container for information representing specific things in the world that have a number of definable properties. For example, a data entity may represent an entity such as a person, a place, an organization, an account, a computer, an activity, a market instrument, or other noun. A data entity may represent an event that happens at a point in time or for a duration. A data entity may represent a document or other unstructured data source such as an e-mail message, a news report, or a written paper or article. Each data entity may be associated with a unique identifier that uniquely identifies the data entity. The data entity's attributes (for example, metadata about the data entity) may be represented in one or more properties. The terms “data entity,” “data object,” and “data item” may be used interchangeably and/or synonymously in the present disclosure.
Entity (or Object or Item) Type: Type of a data entity (for example, Person, Event, or Document). Data entity types may be defined by an ontology and may be modified or updated to include additional data entity types. An data entity definition (for example, in an ontology) may include how the data entity is related to other data entities, such as being a sub-data entity type of another data entity type (for example, an agent may be a sub-data entity of a person data entity type), and the properties the data entity type may have.
Properties: Also referred to as “metadata,” includes attributes of a data entity that represent individual data items. At a minimum, each property of a data entity has a property type and a value or values. Properties/metadata associated with data entities may include any information relevant to that object. For example, properties associated with a person data entity may include a name (for example, John Doe), an address (for example, 123 S. Orange Street), and/or a phone number (for example, 800-0000), among other properties. In another example, metadata associated with a computer data entity may include a list of users (for example, user1, user 2, and the like), and/or an IP (internet protocol) address, among other properties.
Property Type: The type of data a property is, such as a string, an integer, or a double. Property types may include complex property types, such as a series data values associated with timed ticks (for example, a time series), and the like.
Property Value: The value associated with a property, which is of the type indicated in the property type associated with the property. A property may have multiple values.
Link: A connection between two data objects, based on, for example, a relationship, an event, and/or matching properties. Links may be directional, such as one representing a payment from person A to B, or bidirectional.
Link Set: Set of multiple links that are shared between two or more data objects.
Seed: One or more data entities that may be used as a basis, or starting point, for generating a cluster. A seed may be generated, determined, and/or selected from one or more sets of data entities according to a seed generation strategy. For example, seeds may be generated from data entities accessed from various databases and data sources including, for example, databases maintained by financial institutions, government entities, private entities, public entities, and/or publicly available data sources.
Cluster: A group or set of one or more related data entities/objects/items. A cluster may be generated, determined, and/or selected from one or more sets of data entities according to a cluster generation strategy. A cluster may further be generated, determined, and/or selected based on a seed. For example, a seed may comprise an initial data entity of a cluster. Data entities related to the seed may be determined and added to the cluster. Further, additional data entities related to any clustered data entity may also be added to the cluster iteratively as indicated by a cluster generation strategy. Data entities may be related by any common and/or similar properties, metadata, types, relationships, and/or the like.
Seed/Cluster Generation Strategy (or Rule): Seed and cluster generation strategies/rules indicate processes, methods, and/or strategies for generating seeds and generating clusters, respectively. For example, a seed generation strategy may indicate that data entities having a particular property (for example, data entities that are credit card accounts) are to be designated as seeds. In another example, a cluster generation strategy may indicate that data entities having particular properties in common with (or similar to) a seed or other data entity in a cluster are to be added to the cluster. Seed and/or cluster generation strategies may specify particular searches and/or rule matches to perform on one or more sets of data entities. Execution of a seed and/or cluster generation strategy may produce layers of related data entities. Additionally, a seed/cluster generation strategy/rule may include multiple strategies, sub-strategies, rules, and/or sub-rules.
According to various embodiments, a data analysis system is disclosed in which clusters of related data entities may be generated from initial data entities, called “seeds.” Seeds and related data entities may be accessed from various databases and data sources including, for example, databases maintained by financial institutions, government entities, private entities, public entities, and/or publicly available data sources. Such databases and data sources may include a variety of information and data, such as, for example, personal information, financial information, tax-related information, computer network-related data, and/or computer-related activity data, among others. Further, the databases and data sources may include various relationships that link and/or associate data entities with one another. Various data entities and relationships may be stored across different systems controlled by different entities and/or institutions. According to various embodiments, the data analysis system may bring together data from multiple data sources in order to build clusters.
In various embodiments, the data analysis system may enable a user to efficiently perform analysis and investigations of various data entities and clusters of data entities. For example, the system may enable a user (also referred to herein as an “analyst”) to perform various financial and security investigations related to a seed (for example, an initial data entity or data object). In such an investigation, the system may enable an analyst to search and/or investigate several layers of related data entities. For example, a credit card account may be a seed that is linked by the system to various data entities including, for example, customer identifiers and/or phone numbers associated with the credit card account. Further, the system may link, for example, various other credit card accounts related to the customer identifiers, to the seed credit card account. Accordingly, in various embodiments, the system may automatically determine and provide to a user or analyst various layers of data entities related to the seed credit card account. Such an investigation may, in an embodiment, enable the analyst to determine fraudulent activity. For example, if the seed credit card account was suspected to be fraudulent, then the analyst may determine that the additional credit card accounts may also be fraudulent. Further, if the seed credit card account was linked to other known fraudulent credit card accounts, the analyst may determine that the seed credit card account was likely to be fraudulent. As mentioned above, in such an investigation the analyst may discover relationships between the additional credit card accounts and the seed credit card account through several layers of related data entities. Such techniques, enabled by various embodiments of the data analysis system, may be particularly valuable for investigations in which relationships between data entities may include several layers, and in which such relationships may be otherwise very difficult or impossible to manually identify.
In various embodiments, the data analysis system may automatically generate, or determine, seeds based on a seed generation strategy (also referred to as “seed generation rules”). For example, for a particular set of data entities, the data analysis system may automatically generate, based on a seed generation strategy, seeds by designating particular data entities (or groups of data entities) as seeds. Examples of various seed generation strategies are described below.
Further, in various embodiments, the data analysis system may automatically discover data entities related to a seed, and store the resulting relationships and related data entities together in a “cluster.” A cluster generation strategy (also referred to as “cluster generation rules”) may specify particular searches to perform at each step of an investigation, or cluster generation, process. Such searches may produce layers of related data entities to add to the cluster. Further, according to an embodiment, multiple cluster may be merged and/or collapsed into a single cluster when the multiple cluster share one or more common data entities and/or properties. Thus, according to an embodiment, an analyst may start an investigation with the resulting cluster, rather than the seed alone. Starting with the cluster, the analyst may form opinions regarding the related data entities, conduct further analysis of the related data entities, and/or may query for additional related data entities.
According to various embodiments, the data analysis system may further generate various “cluster scores.” Cluster scores may include scores based on various characteristics and/or attributes associated with the cluster and/or the various data entities of the cluster. In various embodiments, the data analysis system may also generate “cluster metascores” which may include, for example, an overall cluster score. Cluster metascores may, for example, be based on a combination of cluster scores of a cluster associated with a seed.
Further, in various embodiments, for a particular set of data entities, multiple clusters may be generated by the data analysis system. For example, the data analysis system may generate multiple seeds according to a seed generation strategy, and then multiple clusters based on those seeds (and based on a cluster generation strategy). In such embodiments, the data analysis system may prioritize the multiple generated clusters based upon cluster scores and/or cluster metascores. In an embodiment, the data analysis system may provide a user interface including a display of summaries of the clusters, including cluster scores, cluster metascores, and/or various other cluster information. Such summaries may be displayed according to a prioritization of clusters. In various embodiments, cluster prioritization may assist an analyst in selecting particular clusters to investigate.
In various embodiments, the data analysis system may be used in various data analysis applications. Such applications may include, for example, financial fraud detection, tax fraud detection, beaconing malware detection, malware user-agent detection, activity trend detection, health insurance fraud detection, financial account fraud detection, detection of activity by networks of individuals, and/or criminal activity detection, among various others. A number of examples of such applications are described in detail below in reference to, for example,
In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments of the present disclosure. However, it will be apparent to one of skill in the art that the systems and methods of the present disclosure may be practiced without one or more of these specific details.
In various embodiments, different types of data entities may have different property types. For example, a “Person” data entity may have an “Eye Color” property type and an “Event” data entity may have a “Date” property type. Each property as represented by data in a database may have a property type defined by an ontology used by the database. Further, data entities may be instantiated in a database in accordance with a corresponding object definition for the particular data entity in the ontology. For example, a specific monetary payment (for example, an entity of type “event”) of US$30.00 (for example, a property of type “currency” having a property value of “US$30.00”) taking place on Mar. 27, 2009 (for example, a property of type “date” having a property value of “Mar. 27, 2009”) may be stored in the database as an event object with associated currency and date properties as defined within the ontology.
Data objects defined in an ontology may support property multiplicity. In particular, a data entity may be allowed to have more than one property of the same property type. For example, a “Person” data object might have multiple “Address” properties or multiple “Name” properties.
A link represents a connection between two data entities and may be through any of a relationship, an event, and/or matching properties. A link may be asymmetrical or symmetrical. For example, “Person” data entity A may, be connected to “Person” data entity B by a “Child Of” relationship (where “Person” data entity B has an asymmetric “Parent Of” relationship to “Person” data entity A), a “Kin Of” symmetric relationship to “Person” data entity C, and an asymmetric “Member Of” relationship to “Organization” data entity X. The type of relationship between two data entities may vary depending on the types of the data entities. For example, “Person” data entity A may have an “Appears In” relationship with “Document” data entity Y or have a “Participate In” relationship with “Event” data entity E. As an example of an event connection, two “Person” data entities may be connected by an “Airline Flight” data entity representing a particular airline flight if they traveled together on that flight, or by a “Meeting” data entity representing a particular meeting if they both attended that meeting. In one embodiment, when two data entities are connected by an event, they are also connected by relationships, in which each data entity has a specific relationship to the event, such as, for example, an “Appears In” relationship.
As an example of a matching properties connection, two “Person” data entities representing a brother and a sister may both have an “Address” property that indicates where they live. If the brother and the sister live in the same home, then their “Address” properties likely contain similar, if not identical property values. In one embodiment, a link between two data entity may be established based on similar or matching properties (for example, property types and/or property values) of the data entity. These are just some examples of the types of connections that may be represented by a link and other types of connections may be represented; embodiments are not limited to any particular types of connections between data entities. For example, a document may contain references to two different entities. For example, a document may contain a reference to a payment (one data entity), and a person (a second data entity). A link between these two data entities may represent a connection between these two entities through their co-occurrence within the same document.
Each data entity may have multiple links with another data entity to form a link set. For example, two “Person” data entities representing a husband and a wife may be linked through a “Spouse Of” relationship, a matching “Address” property, and/or one or more matching “Event” properties (for example, a wedding). Each link, as represented by data in a database, may have a link type defined by the database ontology used by the database.
In various embodiments, the data analysis system may access various data entities and associated properties from various databases and data sources. Such databases and data sources may include a variety of information and data, such as, for example, personal information (for example, names, addresses, phone numbers, personal identifiers, and the like), financial information (for example, financial account information, transaction information, balance information, and the like), tax-related information (for example, tax return data, and the like), computer network-related data (for example, network traffic information, IP (Internet Protocol) addresses, user account information, domain information, network connection information, and the like), and/or computer-related activity data (for example, computer events, user actions, and the like), among others.
Embodiments of the disclosure will now be described with reference to the accompanying Figures, wherein like numerals refer to like elements throughout. The terminology used in the description presented herein is not intended to be interpreted in any limited or restrictive manner, simply because it is being utilized in conjunction with a detailed description of certain specific embodiments of the disclosure. Furthermore, embodiments of the disclosure may include several novel features, no single one of which is solely responsible for its desirable attributes or which is essential to practicing the embodiments of the disclosure herein described.
The application server 115 may include a cluster engine 120 and a workflow engine 125. The cluster engine 120 and a workflow engine 125 may be software modules as described below in reference to
In an embodiment, the cluster engine 120 is configured to score the clusters, according to a defined scoring strategy. The score may indicate the importance of analyzing the cluster. For instance, the cluster engine 120 may execute a scoring strategy that aggregates the account balances of credit card accounts within the cluster. Because, for example, a large aggregated total balance may indicate a large liability for a financial institution, a cluster with such a large total balance may be considered to have a higher score relative to other clusters with lower aggregated total balances (and, therefore, lower scores). Thus, a cluster with a higher score relative to a cluster with a lower score may be considered more important to analyze.
In one embodiment, the cluster engine 120 organizes and presents the clusters according to the assigned scores. The cluster engine 120 may present summaries of the clusters and/or interactive representations of the clusters within the cluster analysis UI. For example, the representations may provide visual indications (e.g., graphs or other visualizations) of the related data entities within the clusters. The cluster engine 120 may generate a cluster analysis user interface (UI), such as a web application and/or a dynamic web page displayed within the client 135. The cluster engine 120 may also allow an analyst to create tasks associated with the clusters. Example operations of the cluster engine 120 are discussed in detail below in conjunction with
Analysts may also assign tasks to themselves or one another via a workflow UI generated by the workflow engine, for example. The workflow engine 125 may consume scores generated by the cluster engine 120. For example, the workflow engine 125 may present an analyst with clusters generated, scored, and ordered by the cluster engine 120.
The client 135 may represent one or more software applications or modules configured to present data and translate input, from the analyst, into requests for data analyses by the application server 115. In one embodiment, the client 135 and the application server 115 may be embodied in the same software module and/or may be included in the same computing system. However, several clients 135 may execute on the client computer 130, and/or several clients 135 on several client computers 130 may interact with the application server 115. In one embodiment, the client 135 may be a browser accessing a web service.
While the client 135 and application server 115 are shown running on distinct computing systems, the client 135 and application server 115 may run on the same computing system. Further, the cluster engine 120 and the workflow engine 125 may run on separate applications servers 115, on separate server computing systems, or some combination thereof. Additionally, a history service may store the results generated by an analyst relative to a given cluster
In one embodiment, the cluster data sources 160 provide data available to the cluster engine to create or generate seeds and/or to create or generate clusters from a seed or a set of seeds. Such data sources may include relational data sources, web services data, XML data, and the like. Further, such data sources may include a variety of information and data, for example, personal information, financial information, tax-related information, computer network-related data, and/or computer-related activity data, among others. For example, the data sources may be related to customer account records stored by a financial institution. In such a case, the data sources may include a credit card account data, bank account data, customer data, and transaction data. The data may include data attributes such as account numbers, account balances, phone numbers, addresses, and transaction amounts, and the like. Of course, cluster data sources 160 is included to be representative of a variety of data available to the server computer system 110 over network 150, as well as locally available data sources.
The database 140 may be a Relational Database Management System (RDBMS) that stores the data as rows in relational tables. The term “database,” as used herein, may refer to an database (e.g., RDBMS or SQL database), or may refer to any other data structure, such as, for example a comma separated values (CSV), extensible markup language (XML), text (TXT) file, flat file, spreadsheet file, and/or any other widely used or proprietary format. While the database 140 is shown as a distinct computing system, the database 140 may operate on the same server computing system 110 as the application server 115.
Seeds 212 (including one, some, or all of seeds 212-1 through 212-S) may be generated by the cluster engine 120 according to various seed generation strategies/rules. Examples of seed generation are described below in reference to various example applications of the data analysis system. According to an embodiment, once generated, seeds 212 may be the starting point for generating a cluster 252. To generate a cluster, the cluster engine 120 may retrieve a given seed 212 from the seed list 210. The seed 212 may be a data entity or group of data entities within the database 140, such as a customer name, a customer social security number, an account number, and/or a customer telephone number.
The cluster engine 120 may generate the cluster 252 from the seed 212. In one embodiment, the cluster engine 120 generates the cluster 252 as a collection of data entities and the relationships between the various data entities. As noted above, the cluster strategy may execute data bindings in order to add each additional layer of data entities to the cluster. For example, the cluster engine 120 may generate the cluster 252-1 from a seed credit card account. The cluster engine 120 may first add the credit card account to the cluster 252-1. The cluster engine 120 may then add customers related to the credit card account to the cluster 252-1. The cluster engine 120 may complete the cluster 252-1 by adding additional credit card accounts related to those customers. As the cluster engine 120 generates the cluster 252-1, the cluster engine 120 may store the cluster 252-1 within the cluster list 250. The cluster 252-1 may be stored as a graph data structure or other appropriate data structure.
The cluster list 250 may be a collection of tables in the database 140. In such a case, there may be a table for the data entities of each cluster 252, such as those of example cluster 252-1 discussed above, a table for the relationships between the various data entities, a table for the attributes of the data entities, and a table for scores of the clusters. The cluster list 250 may include clusters 252 from multiple investigations. Note that the cluster engine 120 may store portions of clusters 252 in the cluster list 250 as the cluster engine 120 generates the clusters 252. Persons skilled in the art will recognize that many technically feasible techniques exist for creating and storing data structures that may be used to implement the systems and methods of the data analysis system.
The cluster strategy store 230 may include cluster strategies 232-1, 232-2 . . . 232-N. Each cluster strategy may include data binding references 235 to one or more data bindings 237. As noted, each data binding may be used to identify data that may grow a cluster (as determined by the given search strategy 232). For example, the cluster engine 120 may execute a cluster strategy 232-1 to generate the cluster 252-1. Specifically, the cluster engine 120 may execute the cluster strategy 232-1 in response to selection of that cluster strategy by an analyst. The analyst may submit a selection of one or more cluster strategies to perform on a seed or group of seeds to the cluster engine 120 through the client 135. Alternatively, the cluster engine 120 may automatically select one or more cluster strategies, such as based on user preferences or rules.
According to an embodiment, each cluster strategy 232 is configured so as to perform an investigation processes for generating a cluster 252. Again, for example, the cluster strategy 232-2 may include data binding references 235 to a collection of data bindings executed to add layer after layer of data to a cluster. The investigation process may include searches to retrieve data entities related to a seed 212 that is selected for clustering using cluster strategy 232-2. For example, the cluster strategy 232-2 may start with a possibly fraudulent credit card account as the seed 212-2. The cluster strategy 232-2 may search for customers related to the credit card account, and then additional credit card accounts related to those customers. A different cluster strategy 232-3 may search for customers related to the credit card account, phone numbers related to the customers, additional customers related to the phone numbers, and additional credit card accounts related to the additional customers, for example.
In an embodiment, cluster strategies 232 include references to at least one data binding 237 (such as data bindings 237-1 through 237-3). The cluster engine 120 may execute a search protocol specified by the data binding 237 to retrieve data, and the data returned by a given data binding may form a layer within the cluster 252. For instance, the data binding 237 (and/or the search protocol of the data binding 237) may retrieve sets of customers related to an account by an account owner attribute. The data binding 237 (and/or the search protocol of the data binding 237) may retrieve the set of related data entities from a data source. For instance, the data binding 237-1 may specify a database query to perform against a database. Likewise, the data binding 237-2 may define a connection and/or query to a remote relational database system and the data binding 237-3 may define a connection and/or query against a third-party web service. Once retrieved, the cluster strategy 232 may evaluate whether the returned data should be added to a cluster being grown from a given seed 212.
Multiple cluster strategies 232 may reference a given data binding 237. The analyst may update the data binding 237, but typically updates the data binding 237 only if the associated data source changes. A cluster strategy 232 may also include a given data binding 237 multiple times. For example, executing a data binding 237 using one seed 212 may generate additional seeds for that data binding 237 (or generate seeds for another data binding 237). More generally, different cluster strategies 232-1, 232-2 . . . 232-N may include different arrangements of various data bindings 237 to generate different types of clusters 252.
The cluster strategies 232 may, specify that the cluster engine 120 use an attribute from the related data entities retrieved with one data binding 237, as input to a subsequent data binding 237. The cluster engine 120 may use the subsequent data binding 237 to retrieve a subsequent layer of related date entities for the cluster 252. For instance, a particular cluster strategy 232 may specify, that the cluster engine 120 retrieve a set of credit card account data entities with a first data binding 237-1. That cluster strategy 232 may also specify that the cluster engine 120 then use the account number attribute from credit card account data entities as input to a subsequent data binding 237-2. The cluster strategy 232 may also specify filters for the cluster engine 120 to apply, to the attributes before performing the subsequent data binding 237. For instance, if the first data binding 237-1 were to retrieve a set of credit card account data entities that included both personal and business credit card accounts, then the cluster engine 120 could filter out the business credit card accounts before performing the subsequent data binding 237-2.
In operation, according to an embodiment, the cluster engine 120 generates a cluster 252-1 from a seed 212-1 by first retrieving a cluster strategy 232. Assuming the analyst selected a cluster strategy 232-2, the cluster engine 120 would retrieve the cluster strategy 232-2 from the cluster strategy store 230. The cluster engine 120 may then retrieve the seed 212-1 as input to the cluster strategy 232-2. The cluster engine 120 may execute the cluster strategy 232-2 by retrieving sets of data by executing data bindings 237 referenced by the cluster strategy 232-2. For example, the cluster strategy 232-2 may execute data bindings 237-1, 237-2, and 237-3. Accordingly, the cluster engine 120 may evaluate data returned by each data binding 237 to determine whether to use that data to grow the cluster 252-1. The cluster engine 120 may then use elements of the returned data as input to the next data binding 237. Of course, a variety of execution paths are possible for the data bindings 237. For example, assume one data binding 237 returned a set of phone numbers. In such a case, another data binding 237 may evaluate each phone number individually. As another example, one data binding 237 may use input parameters obtained by executing multiple, other data bindings 237. More generally, the cluster engine 120 may retrieve data for each data binding referenced by the cluster strategy 232-2. The cluster engine 120 may then store the complete cluster 252-1 in the cluster list 250.
As the cluster engine 120 generates the clusters 252-1, 252-2 . . . , 252-C from seeds 212-1, 212-2 . . . 212-S, the cluster list 250 may include overlapping clusters 252. For example, two clusters 252-1 and 252-C may overlap if both clusters 252-1 and 252-C include a common data entity. In an example, a larger cluster 252 formed by merging two smaller clusters 252-1 and 252-C may be a better investigation starting point than the smaller clusters 252-1 and 252-C individually. The larger cluster 252 may provide additional insight or relationships, which may not be available if the two clusters 252-1 and 252-C remain separate.
In an embodiment, the cluster engine 120 includes a resolver 226 that is configured to detect and merge two or more overlapping clusters 252 together. For example, the resolver 226 may compare the data entities within a cluster 252-1 to the data entities within each one of the other clusters 252-2 through 252-C. If the resolver 226 finds the same data entity within the cluster 252-1 and a second cluster 252-C, then the resolver 226 may merge the two clusters 252-1 and 252-C into a single larger cluster 252. For example, the cluster 252-1 and cluster 252-C may both include the same customer. The resolver 226 may compare the data entities of cluster 252-1 to the data entities of cluster 252-C and detect the same customer in both clusters 252. Upon detecting the same customer in both clusters 252, the resolver 226 may merge the cluster 252-1 with cluster 252-C. The resolver 226 may test each pair of clusters 252 to identify overlapping clusters 252. Although the larger clusters 252 may be better investigation starting points, an analyst may want to understand how the resolver 226 formed the larger clusters 252. Accordingly, the resolver 226, may store a history of each merge.
In an embodiment, cluster merging (for example, by resolver 226) may be optionally disabled for particular types of data entities, and/or particular data entities. For example, when a particular data entity, or type of data entity, is so common that it may be included in many different clusters (for example, an institutional entity such as a bank), merging of cluster based on that common entity (for example, the particular bank) or common type of entity (for example, banks in general) may be disabled. In another embodiment, cluster may be merged only when they share two or more common data entities and/or other properties. In an embodiment, when two clusters are determined to share a data entity that this very common (such that they cluster may not be merged based on that entity) the system may automatically determine whether the two clusters share one or more other data entities and/or properties such that they may be merged. In various embodiments, cluster merging may be disabled based on other criteria. For example, cluster merging between two related clusters may be disabled when one or both of the two clusters reach a particular size (for example, include a particular number of data entities).
After the cluster engine generates a group of clusters from a given collection of seeds (and after merging or resolving the cluster), the cluster engine 120 may score, rank, and/or otherwise order the clusters relative to a scoring strategy 442. In some embodiments, clusters are scored and provided to the analysis without resolving.
In one embodiment, the analysis system 100, and more specifically, the cluster engine 120, receives a request for cluster generation. In response to the request, a list of seeds may be generated, clusters may be generated based on those seeds, and the clusters may be ranked, ordered, and presented to analysts. In an embodiment, the cluster engine 120 may consume seeds generated by other systems. Alternatively, in other embodiments, cluster engine 120 may generate the seeds 212-1, 212-2 . . . 212-S. For instance, the cluster engine 120 may include a seed generation strategy (also referred to as a “lead generation strategy”) that identifies data entities, or groups of data entities, as potential seeds 212. The seed generation (or lead generation) strategy, may apply to a particular business type, such as credit cards, stock trading, or insurance claims, and may be run against a cluster data source 160 or an external source of information.
In an embodiment, the analysis system 100 may not include data bindings as described above. Rather, according to an embodiment, the analysis system 100 may include one or more interfaces and/or connections to various internal and/or external data stores of data entities and/or other information. According to an embodiment, the system may include a generic interface and/or connection to various internal and/or external data stores of data entities and/or other information. For example, the analysis system 100 may include a generic data interface through which the system may search, access, and/or filter various data entity information during seed and/or cluster generation. The generic interface may include various aspects that enable searching, accessing, and/or filtering of data. For example, the generic interface may access various data sources that each have differing data formats. The generic interface may accordingly covert and/or filter the accessed data to a common format. Alternatively, the data sources may include functionality through which stored data may, be searched and/or converted to a standard format automatically. In an embodiment, the generic interface may enable Federated search of multiple data stores of data entity-related information.
Accordingly, in various embodiments, the analysis system 100 may access various data sources for data entity clustering and seed generation.
In the example, assuming the seed 212 is fraudulent credit card account, the cluster engine 120 would add the credit card account to the cluster 252 as the seed entity 302. The cluster engine 120 may then use the account owner attribute of the credit card account as input to a data binding 237. The cluster engine 120 may execute the search protocol of the data binding 237 to retrieve the customer data identifying the owner of the fraudulent credit card account. The cluster engine 120 would then add the customer data to the cluster 252 as the related data entity 305-1. The cluster engine 120 would also add the account owner attribute as the link 303-1 that relates the account number to the customer data of the owner. The cluster engine 120 would execute the next search of the cluster strategy 232 by inputting the customer identifier attribute of the customer data into a data binding 237 to retrieve a phone data. The cluster engine 120 would then add the phone data as the related data entity 305-2 and the customer identifier attribute as the link 303-2 between the customer data and the phone data. At this point in the investigation process, the cluster 252 would include the seed entity 302, two links 303-1 and 303-2, and two related data entities 305-1 and 305-2. That is, the cluster 252 would include the fraudulent credit card account, the customer data of the owner of the credit card, and the phone number of the owner. By carrying the investigation process further, the cluster engine 120 may reveal further related information, for example, additional customers and/or potentially fraudulent credit card accounts.
Turning to
Continuing the example,
In an embodiment, the analyst may identify and determine whether the additional data account entities, stored as related data entities 305-5 and 305-6, represent fraudulent credit card accounts more efficiently than if the analyst started an investigation with only the seed 302. As the foregoing example illustrates, according to various embodiments, the data analysis system may enable an analyst to advantageously start an investigation with a cluster including many related data entities (such as the example cluster 252 with the seed entity 302 and related data entities 305) rather than a single data entity.
In an embodiment, the cluster engine 120 executes a scoring strategy 442 to score a cluster 252. For example, the cluster engine 120 may generate a cluster (for example, via a cluster strategy/data bindings) and attempt to resolve it with existing clusters. Thereafter, the cluster engine 120 may score the resulting cluster with any scoring strategies associated with a given cluster generation strategy. In an embodiment, the multiple scores may be generated for a given cluster. The multiple scores may be based on various aspects, metrics, or data associated with the cluster. In one embodiment, a cluster metascore may be generated based on a combination or aggregation of scores associated with a given cluster. Ordering for a group of clusters, (according to a given scoring strategy) may be performed on demand when requested by a client. Alternatively, the analyst may select a scoring strategy 442 through the client 135 and/or the analyst may include the selection within a script or configuration file. In another alternative, the data analysis system may automatically select a scoring strategy. In other embodiments, the cluster engine 120 may execute several scoring strategies 442 to determine a combined score for the cluster 252.
In an embodiment, a scoring strategy (such as scoring strategy 442) specifies an approach for scoring a cluster (such as cluster 252). A score may indicate a relative importance or significance of a given cluster. For example, the cluster engine 120 may execute a scoring strategy 442-1 to determine a score by counting the number of a particular data entity type that are included within the cluster 252. Assume, for example, a data entity corresponds with a credit account. In such a case, a cluster with a large number of accounts opened by a single individual (possibly within a short time) might correlate with a higher fraud risk. Of course, a cluster score may be related to a high risk of fraud based on the other data in the cluster, as appropriate for a given case. More generally, each scoring strategy 442 may be tailored based on the data in clusters created by a given cluster strategy 230 and a particular type of risk or fraud (or amounts at risk) of interest to an analyst.
According to an embodiment, the cluster engine 120 scores a cluster 252-1 by first retrieving a scoring strategy 442. For example, assume an analyst selects scoring strategy 442-1. In response, the cluster engine 120 may retrieve the scoring strategy 442-1. The cluster engine 120 may also retrieve the cluster 252-1 from the cluster list 250. After determining the score of the cluster 252-1, the cluster engine 120 may store the score with the cluster 252-1 in the cluster list 250.
The cluster engine 120 may score multiple clusters 252-1, 252-2 . . . 252-C in the cluster list 250. The cluster engine 120 may also rank the clusters 252-1, 252-2 . . . 252-C based upon the scores. For instance, the cluster engine 120 may rank the cluster 252-1, 252-2 . . . 252-C from highest score to lowest score. In various embodiment, cluster may be ranked according into multiple scores, combinations of scores, and/or metascores.
In the example UI 500 of
For example, when the analyst selects a particular seed generation strategy, the system may generate a seed list (for example, seed list 210) and then may generate clusters based on seeds of the seed list. The seed list and/or clusters may, in an embodiment, be generated in response to a selection of a particular seed generation strategy. The seed generation strategy may generate a seed list (for example, seed list 210) and/or clusters (for example, clusters 252-1, 252-2, . . . 252-C of the cluster list 250) from the database 140 and/or an external source of information (for example, a cluster data source 160). Alternatively, when the analyst selects a previously generated seed or seed list (for example, seed list 210), the system may retrieve data related to the selected seed list (for example, the seed entities, clusters, and/or related clustered data entities) from, for example, database 140 and/or an external source of information (for example, a cluster data source 160). In an embodiment, clusters may be generated in response to a selection of a previously generated seed list (or seed). Alternatively, cluster may be been previously generated, and may be retrieved in response to selection of a previously generated seed list (or seed). In an embodiment, the analyst may select a particular cluster of interest via the selection box 510.
Further, in the example UI 500 the cluster strategy box 530 displays the cluster strategies 232 that the cluster engine 120 ran against the seed list 210. The cluster engine 120 may execute multiple cluster strategies 232 against the seed list 210, so there may be multiple cluster strategies 232 listed in the cluster strategy box 530. The analyst may click on the name of a given cluster strategy 232 in the cluster strategy box 530 to review the clusters 252 that the cluster strategy 232 generated.
In an embodiment, the workflow engine 125 generates for display in the UI 500 summaries of the clusters 252 in the cluster summary list 525. For example, the summaries may include characteristics of the clusters 252, such as identifiers, scores, and/or analysts assigned to analyze the clusters 252. The workflow engine 125 may select the clusters 252 for display in the cluster summary list 525 according to those or other characteristics. For instance, the workflow engine 125 may display the summaries in the order of the scores of the clusters 252, where a summary of the highest scoring cluster 252 is displayed first.
The workflow engine 125 may control the order and selection of the summaries within the cluster summary list 525 based upon an input from the analyst. The cluster search box 520 may include a search text box coupled to a search button and a pull-down control. The analyst may enter a characteristic of a cluster 252 in the search text box and then instruct the workflow engine 125 to search for and display clusters 252 that include the characteristic by pressing the search button. For example, the analyst may search for clusters with a particular score. The pull-down control may include a list of different characteristics of the clusters 252, such as score, size, assigned analyst, and/or date created. The analyst may select one of the characteristics to instruct the workflow engine 125 to present the summaries of the clusters 252 arranged by that characteristic.
In an embodiment, the workflow engine 125 is also configured to present details of a given cluster 252 within the cluster review window 515. The workflow engine 125 displays the details of the cluster 252, for example, the score, and/or average account balances within a cluster, when the analyst clicks a mouse pointer on the associated summary within the cluster summary list 525. The workflow engine 125 may present details of the cluster 252, such as the name of an analyst assigned to analyze the cluster 252, a score of the cluster 252, and/or statistics or graphs generated from the cluster 252. These details may allow the analyst to determine whether to investigate the cluster 252 further. The cluster review window 515 may also include a button which may be clicked to investigate a cluster 252 within a graph, and an assign button for assigning a cluster to an analyst.
An analyst may click a mouse pointer on an “Investigate in Graph” button representing a cluster to investigate the cluster within an interactive graph. The interactive representation may be a visual graph of the cluster 252, where icons represent the entities of the cluster 252 and lines between the icons represent the links between entities of the cluster 252. For example, the workflow engine 125 may display the interactive graph of the cluster 252 similar to the representation of the cluster 252 in
In an embodiment, an administrative user may click a mouse pointer on an assign button to assign the associated cluster 252 to an analyst. The workflow engine 125 may also allow the administrative user to create tasks associated with the clusters 252, while the administrative user assigns the cluster 252. For example, the administrative user may create a task for searching within the three highest scoring clusters 252 for fraudulent credit card accounts. The workflow engine 125 may display the summaries in the cluster summary list 525 according to the names of the analysts assigned to the clusters 252. Likewise, the workflow engine 125 may only display summaries for the subset of the clusters 252 assigned to an analyst.
The interface shown in
In an alternative embodiment, and as described below in reference to
As shown, example cluster generation method 600 begins at block 605, where the cluster engine 120 retrieves a cluster strategy (e.g., cluster strategy 232-2) and a seed 212. Once a cluster strategy is selected, the cluster engine 120 may identify a list of seeds from which to build clusters using the selected cluster strategy. At block 610, the cluster engine 120 initializes a cluster 252 with one of the seeds in the list. The cluster 252 may be stored as a graph data structure. The cluster engine 120 may initialize the graph data structure and then add the seed 212-1 to the graph data structure as the first data entity.
At block 615, the cluster engine 120 may grow the cluster 252 by executing the search protocol of a data binding 237 from the cluster strategy 232-2. The cluster strategy 232-2 may include a series of data bindings 237 that the cluster engine 120 executes to retrieve related data entities. A given data binding 237 may include queries to execute against a cluster data source 160 using the seed as an input parameter. For example, if the seed 212-1 is an account number, then the data binding 237 may retrieve the data identifying the owner of the account with the account number. After retrieving this information, the cluster engine 120 may add the customer data entity to the cluster as a related data entity and the account owner attribute as the link between the seed 212-1 and the related data entity. After retrieving the related data entities, the cluster engine 120 may add them to the cluster 252.
At block 620, the cluster engine 120 determines if the cluster strategy 232-2 is fully executed. If not the method 600 returns to block 615 to execute additional data bindings for a given seed. Alternatively, as described above, the cluster engine 120 may grow the cluster by searching for, accessing, and/or filtering various data entities through, for example, a generic interface to various internal and/or external data sources. Further, in an embodiment, the cluster engine 120 may determine whether the cluster being generated is to be merged with another cluster, as described above. Once the cluster strategy is executed for that seed, the cluster engine 120 may determine and assign a score (or multiple scores) to that cluster (relative to a specified scoring strategy). After generating clusters for a group of seeds, such clusters may be ordered or ranked based on the relative scores. Doing so may allow an analyst to rapidly identify and evaluate clusters determined to represent, for example, a high risk of fraud.
At block 625, the cluster engine 120 may store the cluster 252 in cluster list 250. As mentioned above, the cluster list 250 may be a collection of tables within a relational database, where a table may include the seed and related data entities of the cluster 252 and another table may include links between the related data entities of the cluster 252.
At block 630, the cluster engine 120 determines if there are more seeds 212 to analyze in the seed list 210. If so, the method 600 returns to block 605 to generate another cluster from the next seed. Otherwise, the method 600 ends. Note, while method 600 describes a single cluster being generated, one of skill in the art will recognize that multiple instances of the cluster generation process illustrated by method 600 may be performed in parallel.
As shown, the example cluster scoring method 700 begins at block 705, where the cluster engine 120 retrieves a scoring strategy 442 and a cluster 252 (for example, a cluster just created using the method 600 of
At block 710, the cluster engine 120 executes the scoring strategy 442 against the cluster 252. The scoring strategy 442 may specify characteristics of the related data entities within the cluster 252 to aggregate. The cluster engine 120 may execute the scoring strategy 442 by aggregating the specified characteristics together to determine a score. For instance, the cluster engine 120 may aggregate account balances of related data entities that are account data entities. In such a case, a total amount of dollars (or average dollars or any other aggregated, averaged, or normal attribute of the cluster) included within the balances of the account data entities of the cluster 252 may be the score of the cluster 252.
At block 715, the cluster engine 120 may store the score with the cluster 252 in the cluster list 250. At step 720, the cluster engine 120 determines if there are more clusters 252 to score. For example, in one embodiment, a set of clusters may be re-scored using an updated scoring strategy. In other cases, the cluster engine may score each cluster when it is created from a seed (based on a given cluster generation and corresponding scoring strategy). If more clusters remain to be scored (or re-scored), the method 700 returns to block 705.
At block 725, the cluster engine 120 may rank the clusters 252 according to the scores of the clusters 252. For example, after re-scoring a set of clusters (or after scoring a group of clusters generated from a set of seeds), the cluster engine 125 may rank the clusters 252 from highest score to lowest score. The ranking may be used to order a display of summaries of the clusters 252 presented to the analyst. The analyst may rely upon the ranking and scores to determine which clusters 252 to analyze first. The ranking and sorting may generally be performed on-demand when an analyst is looking for a cluster to investigate. Thus, the ranking need not happen at the same time as scoring. Further, the clusters may be scored (and later ranked) using different raking strategies.
In various embodiments, multiple scores for each cluster may be determined according to methods similar to the example method 700. Accordingly, clusters may be ranked according to any of multiple scores. Additionally, in various embodiments, multiple scores may be combined and/or aggregated into a metascore that may be used to rank the clusters. Various example score and metascore determinations are described below in reference to
The CPU 860 may retrieve and execute programming instructions stored in memory 820, as well as store and retrieve application data residing in memory 820. The bus 840 may be used to transmit programming instructions and application data between the CPU 860, I/O device interface 870, storage 830, network interface 850, and memory 820. Note that the CPU 860 is included to be representative of, for example, a single CPU, multiple CPUs, a single CPU having multiple processing cores, a CPU with an associate memory management unit, and the like.
The memory 820 is included to be representative of, for example, a random access memory (RAM), cache and/or other dynamic storage devices for storing information and instructions to be executed by CPU 860. Memory 820 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by CPU 860. Such instructions, when stored in storage media accessible to CPU 860, render server computing system 110 into a special-purpose machine that is customized to perform the operations specified in the instructions.
The storage 830 may be a disk drive storage device, a read only memory (ROM), or other static, non-transitory, and/or computer-readable storage device or medium coupled to bus 840 for storing static information and instructions for CPU 860. Although shown as a single unit, the storage 830 may be a combination of fixed and/or removable storage devices, such as fixed disc drives, removable memory cards, and/or optical storage, network attached storage (NAS), and/or a storage area-network (SAN).
Programming instructions, such as the cluster engine 120 and/or the workflow engine 125, may be stored in the memory 820 and/or storage 830 in various software modules. The modules may be stored in a mass storage device (such as storage 830) as executable software codes that are executed by the server computing system 110. These and other modules may include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.
Illustratively, according to an embodiment, the memory 820 stores a seed list 210, a cluster engine 120, a cluster list 250, and a workflow engine 125 (as described with reference to the various figures above). The cluster engine 120 may include a cluster strategy 232-2. The particular cluster strategy 232-2 may include data bindings 237-1, 237-2, and 237-3, with which the cluster engine 120 may access the cluster data source 160. The workflow engine 125 may include a scoring strategy 442-1.
Illustratively, according to an embodiment, the storage 830 includes a cluster strategy store 230, data bindings store 835, and a scoring strategy store 440. As described above, the cluster strategy store 230 may include a collection of different cluster strategies 232, such as cluster strategy 232-2. For example, the cluster strategy store 230 may be a directory that includes the cluster strategies 232-1, 232-2 . . . 232-N as distinct modules. The scoring strategy store 440 may include a collection of different scoring strategies 442, such as scoring strategy 442-2, and may also be a directory of distinct modules. The data binding store 835 may include data bindings 237-1, 237-2 . . . 237-M, which may also be stored as distinct modules within a directory.
Although shown in memory 820, the seed list 210, cluster engine 120, cluster list 250, and workflow engine 125, may be stored in memory 820, storage 830, and/or split between memory 820 and storage 830. Likewise, copies of the cluster strategy 232-2, data binding 237-1, 237-2, and 237-3, and scoring strategy 442-2 may be stored in memory 820, storage 830, and/or split between memory 820 and storage 830.
The network 150 may be any wired network, wireless network, or combination thereof. In addition, the network 150 may be a personal area network, local area network, wide area network, cable network, satellite network, cellular telephone network, or combination thereof. Protocols and components for communicating via the Internet or any of the other aforementioned types of communication networks are well known to those skilled in the art of computer communications and thus, need not be described in more detail herein.
As described above, in various embodiments the system may be accessible by an analyst (or other operator or user) through a web-based viewer, such as a web browser. In this embodiment, the user interface may be generated by the server computing system 110 and transmitted to the web browser of the analyst. Alternatively, data necessary for generating the user interface may be provided by the server computing system 110 to the browser, where the user interface may be generated. The analyst/user may then interact with the user interface through the web-browser. In an embodiment, the user interface of the data analysis system may be accessible through a dedicated software application. In an embodiment, the client computing device 130 may be a mobile computing device, and the user interface of the data analysis system may be accessible through such a mobile computing device (for example, a smartphone and/or tablet). In this embodiment, the server computing system 110 may generate and transmit a user interface to the mobile computing device. Alternatively, the mobile computing device may include modules for generating the user interface, and the server computing system 110 may provide user interaction data to the mobile computing device. In an embodiment, the server computing system 110 comprises a mobile computing device. Additionally, in various embodiments any of the components and/or functionality described above with reference to the server computing system 110 (including, for example, memory, storage, CPU, network interface, I/O device interface, and the like), and/or similar or corresponding components and/or functionality, may be included in the client computing device 130.
According to various embodiments, the data analysis system and other methods and techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, server computer systems, portable computer systems, handheld devices, networking devices or any other device or combination of devices that incorporate hard-wired and/or program logic to implement the techniques.
Computing devices of the data analysis system may generally be controlled and/or coordinated by operating system software, such as iOS, Android, Chrome OS, Windows XP, Windows Vista, Windows 7, Windows 8, Windows Server, Windows CE, Unix, Linux, SunOS, Solaris, iOS, Blackberry OS, VxWorks, or other compatible operating systems. In other embodiments, the computing devices may be controlled by a proprietary operating system. Conventional operating systems control and schedule computer processes for execution, perform memory management, provide file system, networking, I/O services, and provide a user interface functionality, such as a graphical user interface (“GUI”), among other things.
In general, the word “module,” as used herein, refers to a collection of software instructions, possibly having entry and exit points, written in a programming language, such as, for example, Java, Lua, C or C++. A software module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be appreciated that software modules may be callable from other modules or from themselves, and/or may be invoked in response to detected events or interrupts. Software modules configured for execution on computing devices may be provided on a computer readable medium, such as a compact disc, digital video disc, flash drive, magnetic disc, or any other tangible medium, or as a digital download (and may be originally stored in a compressed or installable format that requires installation, decompression or decryption prior to execution). Such software code may be stored, partially or fully, on a memory device of the executing computing device, for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It will be further appreciated that hardware devices (such as processors and CPUs) may be comprised of connected logic units, such as gates and flip-flops, and/or may be comprised of programmable units, such as programmable gate arrays or processors. The modules or computing device functionality described herein are preferably implemented as software modules, but may be represented in hardware devices. Generally, the modules described herein refer to software modules that may be combined with other modules or divided into sub-modules despite their physical organization or storage.
Server computing system 110 may implement various of the techniques and methods described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which, in combination with various software modules, causes the server computing system 110 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by server computing system 110 in response to CPU 860 executing one or more sequences of one or more modules and/or instructions contained in memory 820. Such instructions may be read into memory 820 from another storage medium, such as storage 830. Execution of the sequences of instructions contained in memory 820 may cause CPU 840 to perform the processes and methods described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The term “non-transitory media,” and similar terms, as used herein refers to any media that store data and/or instructions that cause a machine to operate in a specific fashion. Such non-transitory media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage 830. Volatile media includes dynamic memory, such as memory 820. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, and networked versions of the same.
Non-transitory media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between non-transitory media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 840. Transmission media may also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Various forms of media may be involved in carrying one or more sequences of one or more instructions to CPU 860 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions and/or modules into its dynamic memory and send the instructions over a telephone or cable line using a modem. A modem local to server computing system 820 may receive the data on the telephone/cable line and use a converter device including the appropriate circuitry to place the data on bus 840. Bus 840 carries the data to memory 820, from which CPU 860 retrieves and executes the instructions. The instructions received by memory 820 may optionally be stored on storage 830 either before or after execution by CPU 860.
While financial fraud using credit card accounts is used as a primary reference example in the discussion above, the techniques described herein may be adapted for use with a variety of data sets and in various applications. For example, information from data logs of online systems may be evaluated as seeds to improve cyber security. In such a case, a seed may be a suspicious IP address, a compromised user account, and the like. From the seeds, log data, DHCP logs, IP blacklists, packet captures, webapp logs, and other server and database logs may be used to create clusters of activity related to the suspicions seeds. Other examples include data quality analysis used to cluster transactions processed through a computer system (whether financial or otherwise). A number of examples of such applications are described in detail below in reference to, for example,
As described above, and as shown in
Further, as described above, the seeds may include one or multiple data entities, and may be generated based on seed generation strategies and/or rules. Similarly, the clusters may include one or multiple data entities related to a seed, including the seed, and may be generated based on cluster generation strategies and/or rules (including data bindings and/or searching and filtering are performed through, for example, a generic interface to various data sources). Scores and metascores may be determined based on attributes, characteristics, and/or properties associated with data entities that make up a given cluster.
Four example applications of the data analysis system are described below (
According to various embodiments, and as described below, tax-related data entity seeds may be generated by the system as described below in reference to
a. Tax Fraud Detection: Seed Generation
Referring to
At block 1014, various tax fraud indicators may be received and/or accessed by the data analysis system. Tax fraud indicators may include, for example, various data, rules, and/or criteria that may be compared with the tax-related data entities to determine that a tax-related entity is likely associated with fraudulent activity. Examples of tax fraud indicators may include Internet Protocol (IP) addresses that are frequently used, for example, to file multiple tax returns in a short period of time (1014a); financial institution accounts that receive multiple tax refunds (1014b); and/or locations (such as physical addresses) associated with and/or receiving multiple tax refunds (1014c); just to name a few. Other tax fraud indicators may include, for example, various of the cluster scoring criteria described below in reference to
At block 1016, the tax-related data may be analyzed in view of, or compared to, the tax fraud indicators, and particular tax-related data entities that may be associated with tax fraud are determined. For example, a tax return data entity may include a return address that is a known tax fraud indicator. In this example, the tax return data entity may be determined by the system possibly to be related to tax fraud. In another example, a tax return may be filed from a particular IP address known to be a tax fraud indicator, and/or may include a bank account number that is a known tax fraud indicator. In this example, the tax return may again be determined by the system to be possibly related to tax fraud.
At block 1018, the tax-related entities that are determined to be possibly related to tax fraud may be used as seeds by the data analysis system. Accordingly, the method 910a may identify data entity seeds that may be used by the data analysis system in a tax-fraud detection application.
b. Tax Fraud Detection: Cluster Generation
Turning now to
At block 1024, any tax-return data entities that are related to the seed may be clustered. Clustering of data entities may be accomplished as generally described above, in which data bindings are executed and/or searching and filtering are performed (through, for example, a generic interface to various data sources) as part of a clustering strategy. Additionally, as described above, clustered data entities may be related by, for example, sharing the same or similar properties, characteristics, and/or metadata. For example, assume a seed is a bank account. In block 1024, any tax returns that include mention of the bank account may be clustered with the bank account. This example may be seen visually in
Returning to
Returning again to
Returning to
Additionally, in an embodiment a cluster graph similar to the cluster illustration of
c. Tax Fraud Detection: Cluster Scoring
Turning now to
At block 1034, the data analysis system may access and/or receive tax fraud scoring criteria. The tax fraud scoring criteria may include any number of rules or scoring strategies such that multiple scores may be generated for each cluster. Several non-limiting examples of tax fraud scoring criteria may include: a number of tax returns in the cluster known to be fraudulent; a presence of particular types of physical addresses (for example, post office boxes and/or private delivery service mail boxes); a total number of tax returns in the cluster; a similarity between requested tax return amounts among the tax returns in the cluster; a number of tax returns in the cluster associated with persons having a recent change in address and/or bank account; an inclusion in the cluster of persons under a particular age and/or persons known to be dead; tax returns including requests for particular types of credits in the cluster; a number of persons in the cluster that have never before filed a tax return; and/or a number of known fraud accounts in the cluster.
At block 1036, the tax fraud scoring criteria may be applied to the clusters and cluster scores may be generated. In an embodiment, each cluster score may include an absolute value and/or a weighted value. For example, a given cluster may include two known fraudulent tax returns. When a scoring criteria is applied that indicates a known number of fraudulent tax returns, the absolute value of the score generated would be two (2), as the cluster includes two such tax returns. However, the scoring criteria may also include a relative weighting that indicates the importance of the particular score (“a number of known fraudulent tax returns”) to an overall evaluation of the cluster in the context of tax fraud. Such a relative weighting, in one embodiment, may be a number between 0 and 1, where a number closer to one indicates a greater importance. For example, if the “number of known fraudulent tax returns” score is considered relatively important to an evaluation of the cluster, it may be given a relative weight of, for example, “0.7”. On the other hand, a relatively less important score/consideration (for example, a number of first time filers) may be given a relative weight of, for example, “0.15”. Then, when a score is calculated by the data analysis system, the relative weight may be multiplied by the absolute value to arrive at a corresponding relative value. For example, in the “number of fraudulent returns” example, where an absolute value of 2 was determined, a relative value may be determined by multiplying 2 by 0.7, to arrive at a relative value of 1.4. In other embodiments, various other methods may be employed by the data analysis system to determine relative values of cluster scores. For example, as may be recognized by one of skill in the art, the system may normalize the absolute values of each of the scores before applying a relative weighting to arrive at a weighted value. Examples of cluster scores presented to an analyst or other user of the data analysis system are shown and described below in reference to
At block 1042, a metascore may be generated for the clusters. The cluster metascore may be based on a combination or aggregation of the individual scores generated in block 1036. Alternatively, the metascores may be separately determined scores. In an embodiment, a metascore may be calculated by summing, multiplying, and/or otherwise aggregating or averaging the various individual scores together. The metascore may, in an embodiment, capture the relative importance of each of the individual scores by weighting each of the individual scores in a manner similar to that described above with reference to block 1036.
In various embodiments, metascores and/or scores may advantageously enable an analyst to directly compare and/or prioritize various clusters, and/or may advantageously be used by the data analysis system to prioritize a list of clusters and/or scores related to a cluster.
d. Tax Fraud Detection: Example User Interface
In the example user interface of
According to an embodiment, various items of information may be included in the user interface that may be useful to an analyst in evaluating and/or investigating the generated clusters. For example, metascores associated with each of the generated clusters may be shown in the list of clusters 1082, and/or the clusters may be prioritized according to the metascores. In another example, absolute values (1090) and/or weighted values (1088) may be displayed in the list of scores 1084 for each score. In the example shown, a metascore of “0.5” may be calculated for “Tax Fraud Cluster 1” by, for example, averaging the various cluster scores (for example, (1.0+0.8+0.6+0.6+0+0)/6=0.5). In another example, the detailed view 1086 may include a graph that shows additional information related to the selected score. For example, in
In an embodiment, the data analysis system may automatically evaluate the generated clusters to determine a likelihood of tax fraud. For example, the system may determine that a cluster having a metascore below a particular threshold is likely not fraud, while a cluster having a metascore above another particular threshold likely is fraud. In an embodiment, the system may determine that a cluster having a metascore within a particular range of thresholds requires additional analysis by an analyst as the likelihood of fraud is not conclusive. In an embodiment, an analyst may adjust the thresholds, the metadata calculations, and/or the weighting applied to the scores. Further, the analyst may mark various clusters as, for example, fraudulent, likely fraudulent, likely not fraudulent, and/or not fraudulent. Additionally, the analyst may dispatch other analysts to review particular clusters and/or mark particular clusters for further analysis.
According to various embodiments, beaconing malware-related data entity seeds (referred to herein as “beaconing seeds”) may be generated by the system as described below in reference to
In an embodiment, and as described below, the data analysis system may be used in a network environment in which an internal network is in communication with an external network. The system may be used to determine whether any computer systems of the internal network have been infected by beaconing malware that is communicating with computer systems of the external network. Various computerized devices may be included in the internal network that may be capable to capturing and/or logging data traffic between the internal network and the external network including, for example, network routers and/or switches.
a. Beaconing Malware Detection: Seed Generation
Referring to
At block 1113, the system may generate internal-external connection pairs. Each of the internal-external connection pairs may include a particular internal IP address and a particular external IP address and/or domain that was contacted by the internal IP address. At block 1114, time series of the generated internal-external connection pairs may be generated. For example, the system may determine sets of connection pairs that have common internal IP addresses and external IP addresses or domains. Then, for each set, a time series may be generated that represents each point in time that the same or a similar connection is made between a particular internal IP address and external IP address or domains. Each of the time series may span a particular time period. For example, each time series may span a number of days, weeks, months, or years. Thus, a connection pair time-series (or simply “connection pair series” or “connection series”), may indicate multiple connections made between a particular internal and external IP address (or domain or other device identifier) and/or a periodicity or other pattern indicating when the connections were made. The internal-external connection pairs may be plotted along each time series for the particular time period.
At block 1116, the data analysis system may filter out any noise in each time series. For example, the connection pairs in each connection series may be analyzed in order to identify any connection pairs of the particular connection series that should be indicated as noise. Noise in a connection series may include, for example, any internal-external connection pairs that have a low likelihood of being related to beaconing activity and/or to malicious activity. Various filter criteria may be applied to filter out noise. Examples of noise filtering criteria may include, but are not limited to: filter 1116a, which detects frequently established connections, such as the same or similar connection pairs (for example, multiple connection pairs from the same internal IP to the same external IP and/or domain) that occur with short intervals (or deltas) of time between them (for example, intervals on the order of seconds, or intervals that are shorter than are typically employed by beaconing malware); filter 1116b, which detects connection pairs that have only been occurring for a short period of time (for example, for a week or less); filter 1116c, which detects connection pairs with popular or well-known legitimate external domains (for example, a third-party produced list of popular domains may be used by the system); and/or filter 1116d, which detects connection pairs made by legitimate software for, for example, software updates (in an embodiment, this filter criteria may be applied on a per-computer system basis, such that a determination may be made regarding the legitimacy of particular pieces of software on each individual computer system).
Once connection pairs that include noise, or which are not likely related to beaconing malware, are filtered from each connection series, at block 1117 a beaconing score may be computed for each connection pair series. A beaconing score may be computed in any of various ways. One example of computing a beaconing score is shown in block 1117a. In the example of block 1117a, the system may calculate a variance of the particular connection pair series. The variance may, for example, provide an indication of the regularity, or periodicity, of the connection pairs over time. Higher variances may indicate that the connection pair is less likely to be related to malware beaconing activity, as malware beaconing activity may generally occur at very regular intervals. Thus, lower variances may indicate that the connection pair is more likely to be related to malware beaconing activity. Another example of computing a beaconing score is shown in block 1117b. In the example of block 1117b, the system may calculate a mean of the particular connection pair series. The mean may, for example, provide an indication of the average time between each connection pair over time. Particular mean values, for example, a particular number of days, weeks, and/or months, may indicate higher or lower likelihood that the connection series is related to malware beaconing activity. In another example, some combination of a variance and a mean of a connection pair series may be used by the system as a beaconing score (for example, a variance divided or normalized by a mean or a mean squared). In an embodiment, the variance is calculated based on an average of squared differences from the mean time between connections in a time series.
At block 1118, the system may determine which connection pairs have beaconing scores that satisfy a particular threshold. For example, the system may determine that any beaconing pairs having beaconing scores below a particular variance are likely to represent malware beaconing activity. Accordingly, the data analysis system may designate and use those connection pairs as seeds. Thus, the method 910b may be used to generate seeds including a connection pair (e.g., an internal IP address and an external IP address or domain) that may be used by the data analysis system in a beaconing malware detection application.
b. Beaconing Malware Detection: Cluster Generation
Turning now to
At block 1124, any data entities that are related to the seed may be clustered. Clustering of data entities may be accomplished as generally described above, in which data bindings are executed and/or searching and filtering are performed (through, for example, a generic interface to various data sources) as part of a clustering strategy. Additionally, as described above, clustered data entities may be related by, for example, sharing the same or similar properties, characteristics, and/or metadata. Examples of data entities that may be clustered include, but are not limited to: users (for example, persons having accounts on particular computer systems), internal IP addresses, internal IP addresses that connect to external domains, internal computer systems, internal computer systems that connect to external domains, external IP addresses, external domains, external IP addresses associated with external domains, other data feed data entities (for example, data entities drawn from public and/or private whitelists or blacklists, such as data entities representing known bad domains, known good domains, known bad IP addresses, and the like), host-based events (such as, for example, virus scan alerts and/or logged events, intrusion prevention system alerts and/or logged events, and the like), and the like.
Returning again to
In an embodiment, the various clustered data entities may include various properties and characteristics, including information regarding data communications and requests between internal and external computer systems. For example, a given connection pair (or seed) may represent multiple connections over a period of time (as described above in reference to
Additionally, in an embodiment a cluster graph similar to the cluster illustration of
c. Beaconing Malware Detection: Cluster Scoring
Turning now to
At block 1134, the data analysis system may access and/or receive beaconing scoring criteria. The beaconing scoring criteria may include any number of rules or scoring strategies such that multiple scores may be generated for each cluster. Several non-limiting examples of beaconing scoring criteria may include: a number of external domains in the cluster known to be malicious; a number of blacklists on which an external domain in the cluster appears; a trustworthiness (and/or number) of blacklists on which external domains in the cluster appear; a number and/or severity of host-based events in the cluster (such as, for example, virus scan alerts and/or logged events, intrusion prevention system alerts and/or logged events, and the like); a number of requests and/or connections between internal and external network devices associated with the cluster that were blocked by a proxy, router, or other appliance linking the internal network to the external network; and/or an average request size (for example, an amount of data transmitted) between the internal and external devices associated with the cluster (for example, smaller request sizes may indicate a higher likelihood that the activity is related to beaconing activity).
At block 1136, the beaconing scoring criteria may be applied to the clusters and cluster scores may be generated. In an embodiment, each cluster score may include an absolute value and/or a weighted value as described above in reference to
At block 1142, a metascore may be generated for the clusters. The cluster metascore may be based on a combination or aggregation of the individual scores generated in block 1136. Alternatively, the metascores may be separately determined scores. In an embodiment, a metascore may be calculated by summing, multiplying, and/or otherwise aggregating or averaging the various individual scores together. The metascore may, in an embodiment, capture the relative importance of each of the individual scores by weighting each of the individual scores in a manner similar to that described above with reference to
In various embodiments, metascores and/or scores may advantageously enable an analyst to directly compare and/or prioritize various clusters, and/or may advantageously be used by the data analysis system to prioritize a list of clusters and/or scores related to a cluster.
At optional block 1144, analyst (or other user) feedback may optionally be used in future scoring by the data analysis system. For example, if the analyst determines that a particular domain, identified by the system as potentially malicious, is not malicious, this information may be used by the system in future scoring of clusters. For example, the domain determined by the analyst to not be malicious may be whitelisted, or less weight may be applied to scores related to that domain.
d. Beaconing Malware Detection: Example User Interface
In the example user interface of
According to an embodiment, various items of information may be included in the user interface that may be useful to an analyst in evaluating and/or investigating the generated clusters. For example, metascores associated with each of the generated clusters may be shown in the list of clusters 1182, and/or the clusters may be prioritized according to the metascores. In another example, as described above with reference to
According to various embodiments, the data analysis system as applied to beaconing malware detection may advantageously enable an analyst to detect and proactively remove an item of malware from various computer systems. Further, according to various embodiments the data analysis system as applied to beaconing malware detection may advantageously enable an analyst to block particular domains determined to be related to beaconing malware, and/or rake other step to protect and internal network from attack.
In an embodiment, the data analysis system may automatically evaluate the generated clusters to determine a likelihood that a given cluster represents beaconing malware activity. For example, the system may determine that a cluster having a metascore below a particular threshold is likely not related to beaconing malware activity, while a cluster having a metascore above another particular threshold likely is beaconing malware activity. In an embodiment, the system may determine that a cluster having a metascore within a particular range of thresholds requires additional analysis by an analyst as the likelihood of beaconing malware activity is not conclusive. In an embodiment, an analyst may adjust the thresholds, the metadata calculations, and/or the weighting applied to the scores. Further, the analyst may marks various clusters as, for example, beaconing malware, likely beaconing malware, likely not beaconing malware, and/or not beaconing malware. Additionally, the analyst may dispatch other analysts to review particular clusters and/or mark particular clusters for further analysis.
According to various embodiments, user-agent-related data entity seeds may be generated by the system as described below in reference to
In an embodiment, and as described below, the data analysis system may be used in a network environment in which an internal network is in communication with an external network. The system may be used to determine, for example, whether any computer systems of the internal network have been infected by malware that is communicating with computer systems of the external network. Various computerized devices may be included in the internal network that may be capable to capturing and/or logging data traffic between the internal network and the external network including, for example, network routers and/or switches.
a. Malware User-Agent Detection: Seed Generation
Referring to
At block 1213, the system may filter out particular user-agent strings, and associated data entities, based on various criteria so as to remove any communications that are not likely to be malicious. For example, the system may make use of a list of whitelisted domains (1213a) by removing any communications items that involve communications with known non-malicious domains. Similarly, the system may make use of a list of whitelisted user-agent strings by removing any communications items that involve communications with known non-malicious user-agent strings. In another example, the system may determine that particular connections between certain internal computer systems and external computer systems are common or frequent (1213b), and thus not likely malicious. In an embodiment, common connections between many internal computer systems and particular external computer systems may indicate that the connections are not performed by malicious software. Such common connection may be associated with, for example, software updates by legitimate software installed on many computer systems of the internal network. In yet another example, the data analysis system may optionally filter out any communications related to random user-agent strings (1213c).
Once a set of communications associated with the filtered list of user-agent strings is determined, at block 1214 the data analysis system compares the detected, filtered, user-agent strings from a test period to the detected, filtered, user-agent strings from a reference period. In an embodiment, a test period may include a current day, while a reference period may include a three, four, or five week period before the current day. Any other test and/or reference period may be used by the data analysis system. According to an embodiment, comparing user-agent strings between a test period and a reference period enables the data analysis system to detect any new (for example, not previously used during the reference period) user-agent strings. At block 1215, any user-agent strings determined to be new (for example, used during the test period but not during the reference period) may be used (along with related communication data entities) as seeds.
At block 1216, the data analysis system may use an additional or alternative method for determining seeds. This may be accomplished by determining a frequency distribution of detected, filtered, user-agent strings used by hosts (or various computer systems) of the internal network. Such a frequency distribution may be determined for a particular test period of time. Additionally, such a frequency distribution may indicate, for example, the number of hosts that used any particular user-agent string during the test period of time. As indicated by block 1218, any user-agent strings that are determined to have been used rarely (for example, used by less than a particular threshold number or percentage of hosts) may be used (along with related communication data entities) as seeds by the data analysis system. For example, a particular threshold number of hosts may be five. In this example, the system may determine that a particular user-agent was used by two hosts during the test period of time. Accordingly, the system may designate the particular user-agent as a seed. Alternatively, the system may determine that a second particular user-agent was used by 100 hosts during the test period of time. Accordingly, the system may not designate the second particular user-agent as a seed.
b. Malware User-Agent Detection: Cluster Generation
Turning now to
At block 1224, any user-agent-related data entities that are related to the seed may be clustered. Clustering of data entities may be accomplished as generally described above, in which data bindings are executed and/or searching and filtering are performed (through, for example, a generic interface to various data sources) as part of a clustering strategy. Additionally, as described above, clustered data entities may be related by, for example, sharing the same or similar properties, characteristics, and/or metadata. Examples of data entities that may be clustered include, but are not limited to: hosts or source computing systems (for example, computing systems of the internal network); host owners, such as users of hosts or computing systems (for example, persons having accounts on particular computer systems); host-based events (such as, for example, virus scan alerts and/or logged events, intrusion prevention system alerts and/or logged events, and the like); internal IP addresses; internal IP addresses that connect to external domains; internal computer systems; internal computer systems that connect to external domains; external IP addresses; external domains; external IP addresses associated with external domains; and the like.
Returning again to
In an embodiment, the various clustered data entities may include various properties and characteristics, including information regarding data communications and requests between internal and external computer systems and the associated user-agent strings. For example, a given connection between a host and an external domain may represent multiple connections over a period of time. In an embodiment, the data analysis system may cluster various data entities with the seed so as to enable an analyst to quickly determine, for example, any computer systems associated with the user agent, any users of those computer systems, any actions (including communications) taken by those computer systems, any events happening on those computer systems (for example, host-based events, including file changes, software installs, and the like), and the like.
Additionally, in an embodiment a cluster graph similar to the cluster illustration of
c. Malware User-Agent Detection: Cluster Scoring
Turning now to
At block 1234, the data analysis system may access and/or receive user-agent scoring criteria. The user-agent scoring criteria may include any number of rules or scoring strategies such that multiple scores may be generated for each cluster. Several non-limiting examples of user-agent scoring criteria may include: a number of external domains in the cluster known to be malicious; a number of user agents known to be malicious (including, for example, being included on a blacklist or a number of blacklists); a number of blacklists on which an external domain or user agent in the cluster appears; a trustworthiness (or number) of blacklists on which external domains or user agents in the cluster appear; a number and/or severity of host-based events in the cluster (such as, for example, virus scan alerts and/or logged events, intrusion prevention system alerts and/or logged events, and the like); a number of requests and/or connections between internal and external network devices associated with the cluster that were blocked by a proxy, router, or other appliance linking the internal network to the external network; and/or an average request size (for example, an amount of data transmitted) between internal and external devices associated with the cluster (for example, smaller request sizes may indicate a higher likelihood that the activity is related to malware activity).
At block 1236, the user-agent scoring criteria may be applied to the clusters and cluster scores may be generated. In an embodiment, each cluster score may include an absolute value and/or a weighted value as described above in reference to
At optional block 1242, a metascore may optionally be generated for the clusters. The cluster metascore may be based on a combination or aggregation of the individual scores generated in block 1236. Alternatively, the metascores may be separately determined scores. In an embodiment, a metascore may be calculated by summing, multiplying, and/or otherwise aggregating or averaging the various individual scores together. The metascore may, in an embodiment, capture the relative importance of each of the individual scores by weighting each of the individual scores in a manner similar to that described above with reference to
In various embodiments, metascores and/or scores may advantageously enable an analyst to directly compare and/or prioritize various clusters, and/or may advantageously be used by the data analysis system to prioritize a list of clusters and/or scores related to a cluster.
At optional block 1244, an analyst (or other user) may optionally investigate particular hosts included in a cluster and/or associated with a particular user agent. The analyst may further optionally place a user agent determined to be associated with malicious software on a blacklist that may be used in subsequent applications of the data analysis system.
d. Malware User-Agent Detection: Example User interface
In the example user interface of
According to an embodiment, various items of information may be included in the user interface that may be useful to an analyst in evaluating and/or investigating the generated clusters. For example, metascores associated with each of the generated clusters may be shown in the list of clusters 1282, and/or the clusters may be prioritized according to the metascores. In another example, as described above with reference to
According to various embodiments, the data analysis system as applied to malware user-agent detection may advantageously enable an analyst to detect and proactively remove an item of malware from various computer systems. Further, according to various embodiments the data analysis system as applied to malware user-agent detection may advantageously enable an analyst to block particular domains determined to be related to malicious software, and/or take other step to protect and internal network from attack.
In an embodiment, the data analysis system may automatically evaluate the generated clusters to determine a likelihood that a given cluster represents malicious activity. For example, the system may determine that a cluster having a metascore below a particular threshold is likely not related to malware activity, while a cluster having a metascore above another particular threshold likely is malware activity. In an embodiment, the system may, determine that a cluster having a metascore within a particular range of thresholds requires additional analysis by an analyst as the likelihood of malware activity is not conclusive. In an embodiment, an analyst may adjust the thresholds, the metadata calculations, and/or the weighting applied to the scores. Further, the analyst may mark various clusters as, for example, malware, likely malware, likely not malware, and/or not malware. Additionally, the analyst may dispatch other analysts to review particular clusters and/or mark particular clusters for further analysis.
According to various embodiments, activity-related data entity seeds may be generated by the system as described below in reference to
In an embodiment, and as described below, the data analysis system may be used in a network environment in which an internal network is in communication with an external network. The internal network may include multiple computer systems (also referred to as hosts) that may be in communication with one another. Various activities and events occurring on each of the hosts, as well as network communications, may be tracked and/or logged by aspects of the data analysis system such that they may be analyzed as described below. Various computerized devices may be included in the network that may be capable to capturing and/or logging data traffic and communications, for example, network routers and/or switches. Further, various software programs or other aspects may be included in the network and/or on computer systems in the network that may be capable to capturing and/or logging various host-based events.
a. Activity Trend Detection: Seed Generation
Referring to
As various types of events may be captured by the data analysis system, in the flowchart of
At block 1314, for a particular type of captured event on a particular host, any events of the particular type captured on the particular host may be analyzed by the data analysis system and a current Z-score may be calculated with respect to a previous time period. For example, in an embodiment the data analysis system may determine any of a particular type of event that is captured on a particular host. The data analysis system may also determine a current time period and a previous time period into which the events of a particular type may be divided. In one example, a current time period may be a most recent day, week, or month. In an example, a previous time period may be a day, week, or month immediately preceding the current time period. Then, the data analysis system may calculate a probability distribution of events of the particular type on the host during the previous time period. Next, the data analysis system may determine a Z-score of events of the particular type on the host during the current time period along the previous time period distribution. A Z-score may indicate, for example, a number of standard deviations the events during the current time period are above or below the mean of the distribution of the previous time period. In an embodiment, the calculated Z-score may provide an indication of an unusual change in activity of a particular type of event on a particular host.
At block 1315, the data analysis system may compare the calculated Z-score, for each event type on each host, to a threshold. The threshold may be, for example, +/−1, +/−1.5, +/−2, +/−2.5, among other possibilities. At block 1316, any events having Z-scores that satisfy the threshold may be used as seeds. Accordingly, the method 910d may generate seeds that may be used by the data analysis system in an activity trend detection application.
b. Activity Trend Detection: Cluster Generation
Turning now to
At block 1324, any activity-related data entities that are related to the seed may be clustered. Clustering of data entities may be accomplished as generally described above, in which data bindings are executed and/or searching and filtering are performed (through, for example, a generic interface to various data sources) as part of a clustering strategy. Additionally, as described above, clustered data entities may be related by, for example, sharing the same or similar properties, characteristics, and/or metadata. Examples of data entities that may be clustered include, but are not limited to: hosts or source computing systems (for example, computing systems of the network); host owners, such as users of hosts or computing systems (for example, persons having accounts on particular computer systems); host-based events related to the seed event (such as, for example, virus scan alerts and/or logged events, intrusion prevention system alerts and/or logged events, and the like); internal IP addresses; internal IP addresses that connect to external domains; internal computer systems; internal computer systems that connect to external domains; external IP addresses; external domains; external IP addresses associated with external domains; and the like.
Returning again to
In an embodiment, the various clustered data entities may include various properties and characteristics, including information regarding data communications and requests between computer systems in the network. For example, a given connection between as host and an external domain may represent multiple connections over a period of time. In an embodiment, the data analysis system may cluster various data entities with the seed so as to enable an analyst to quickly determine, for example, any computer systems and/or other events associated with a particular detected unusual event, any users of related computer systems, any actions (including communications) taken by related computer systems, any events happening on related computer systems (for example, host-based events, including file changes, software installs, and the like), and the like.
Additionally, in an embodiment a cluster graph similar to the cluster illustration of
c. Activity Trend Detection: Cluster Scoring
Turning now to
At block 1334, the data analysis system may access and/or received the previously determined Z-score of the seed. This Z-score may be considered by the data analysis system as a Z-score of the cluster, as it indicates how the detected event (the seed) deviates from the normal activity on the host. At block 1335, the cluster Z-score may be compared to a calculated “universe,” or entire-network, Z-score. In an embodiment, the universe Z-score may be calculated in a manner similar to the calculation of the seed Z-score, but with reference to the entire network of computers from which the data analysis system has captured events. For example, the universe Z-score may be calculated as follows: The data analysis system may use the same current and previous time periods as were used in calculation of the seed Z-score, however a probability distribution of events (of the same or similar type to the seed event) across the network may be calculated for the previous time period. Then, the data analysis system may determine a universe Z-score of events of the particular type on the network during the current time period along the previous time period distribution. In an embodiment, the calculated universe Z-score may provide an indication of an unusual change in activity of a particular type of event on the network.
In various embodiments, the comparison between the cluster Z-score and the universe Z-score may provide an indication, for example, of whether or not the activity trend detected on the host is unusual in the context of the network. For example, when the activity trend detected on the host is also present on the network as a whole (for example, the cluster Z-score is similar to the universe Z-score), the activity trend may not be particularly interesting to an analyst. Alternatively, when the activity trend detected on the host is different from related activity that is present on the network as a whole (for example, the cluster Z-score is different from the universe Z-score), the activity trend may be interesting to an analyst. Accordingly, at block 1336, a cluster score is generated that indicates how significantly the cluster Z-score and the universe Z-score differ from one another. Additionally, in an embodiment, the cluster score may capture the absolute value of the cluster Z-score such that larger cluster Z-scores (indicating significant activity trends) may result in larger cluster scores as compared to smaller cluster scores, even in view of the difference between the cluster Z-score and the universe Z-score.
d. Activity Trend Detection: Example User Interface
In the example user interface of
According to an embodiment, various items of information may be included in the user interface that may be useful to an analyst in evaluating and/or investigating the generated clusters. For example, the clusters may be prioritized according to the cluster scores. In another example, the detailed view 1386 may include a graph that shows additional information related to the selected score. For example, in
In various embodiments, any other mathematical and/or statistical measures and/or methods may be used in determining and/or detecting unusual and/or interesting activity trends. For example, any other mathematical methods may be used to determine activities that are statistical outliers. Additionally, in various embodiments, the other methods may be used in any of the seed generation, cluster generation, and/or scoring of clusters.
According to various embodiments, the data analysis system as applied to activity trend detection may advantageously enable an analyst to detect and proactively take action when unusual activity is detected. In an embodiment, the data analysis system may automatically evaluate the generated clusters to determine a likelihood that a given cluster represents an activity trend or unusual activity. For example, the system may determine that a cluster having a cluster score below a particular threshold is likely not an activity trend, while a cluster having a cluster score above another particular threshold likely is an activity trend. In an embodiment, the system may determine that a cluster having a cluster score within a particular range of thresholds requires additional analysis by an analyst as the likelihood of an activity trend is not conclusive. In an embodiment, an analyst may adjust the thresholds, the cluster score calculations, and/or the weighting applied to the Z-scores. Further, the analyst may mark various clusters as, for example, activity trend, likely activity trend, likely not activity trend, and/or not activity trend. Additionally, the analyst may dispatch other analysts to review particular clusters and/or mark particular clusters for further analysis.
While the foregoing is directed to various embodiments, other and further embodiments may be devised without departing from the basic scope thereof. For example, aspects of the present disclosure may be implemented in hardware or software or in a combination of hardware and software. An embodiment of the disclosure may be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and may be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. Each of the processes, methods, and algorithms described in the preceding sections may be embodied in, and fully or partially automated by, code modules executed by one or more computer systems or computer processors comprising computer hardware. The processes and algorithms may alternatively be implemented partially or wholly in application-specific circuitry.
The various features and processes described above may be used independently of one another, or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. In addition, certain method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate. For example, described blocks or states may be performed in an order other than that specifically disclosed, or multiple blocks or states may be combined in a single block or state. The example blocks or states may be performed in serial, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The example systems and components described herein may be configured differently than described. For example, elements may be added to, removed from, or rearranged compared to the disclosed example embodiments.
Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment.
The term “comprising” as used herein should be given an inclusive rather than exclusive interpretation. For example, a general purpose computer comprising one or more processors should not be interpreted as excluding other computer components, and may possibly include such components as memory, input/output devices, and/or network interfaces, among others.
Any process descriptions, elements, or blocks in the flow diagrams described herein and/or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those skilled in the art.
It should be emphasized that many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure. The foregoing description details certain embodiments of the invention. It will be appreciated, however, that no matter how detailed the foregoing appears in text, the invention may be practiced in many ways. As is also stated above, it should be noted that the use of particular terminology when describing certain features or aspects of the invention should not be taken to imply that the terminology is being re-defined herein to be restricted to including any specific characteristics of the features or aspects of the invention with which that terminology is associated. The scope of the invention should therefore be construed in accordance with the appended claims and any equivalents thereof.
This application is a continuation of U.S. patent application Ser. No. 14/139,640, filed Dec. 23, 2013, titled “TREND DATA CLUSTERING,” which is a continuation-in-part of U.S. patent application Ser. No. 13/968,265, filed Aug. 15, 2013, titled “GENERATING DATA CLUSTERS WITH CUSTOMIZABLE ANALYSIS STRATEGIES,” and is a continuation-in-part of U.S. patent application Ser. No. 13/968,213, filed Aug. 15, 2013, titled “PRIORITIZING DATA CLUSTERS WITH CUSTOMIZABLE SCORING STRATEGIES,” each of which claims benefit of U.S. Provisional Patent Application No. 61/800,887, filed Mar. 15, 2013. Each of the above identified applications is hereby incorporated by reference herein in its entirety and for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5109399 | Thompson | Apr 1992 | A |
| 5329108 | Lamoure | Jul 1994 | A |
| 5632009 | Rao et al. | May 1997 | A |
| 5670987 | Doi et al. | Sep 1997 | A |
| 5781704 | Rossmo | Jul 1998 | A |
| 5790121 | Sklar et al. | Aug 1998 | A |
| 5798769 | Chiu et al. | Aug 1998 | A |
| 5845300 | Comer | Dec 1998 | A |
| 5978475 | Schneier et al. | Nov 1999 | A |
| 6057757 | Arrowsmith et al. | May 2000 | A |
| 6091956 | Hollenberg | Jul 2000 | A |
| 6141659 | Barker et al. | Oct 2000 | A |
| 6161098 | Wallman | Dec 2000 | A |
| 6219053 | Tachibana et al. | Apr 2001 | B1 |
| 6232971 | Haynes | May 2001 | B1 |
| 6247019 | Davies | Jun 2001 | B1 |
| 6279018 | Kudrolli et al. | Aug 2001 | B1 |
| 6341310 | Leshem et al. | Jan 2002 | B1 |
| 6366933 | Ball et al. | Apr 2002 | B1 |
| 6369835 | Lin | Apr 2002 | B1 |
| 6374251 | Fayyad et al. | Apr 2002 | B1 |
| 6456997 | Shukla | Sep 2002 | B1 |
| 6549944 | Weinberg et al. | Apr 2003 | B1 |
| 6560620 | Ching | May 2003 | B1 |
| 6567936 | Yang et al. | May 2003 | B1 |
| 6581068 | Bensoussan et al. | Jun 2003 | B1 |
| 6594672 | Lampson et al. | Jul 2003 | B1 |
| 6631496 | Li et al. | Oct 2003 | B1 |
| 6642945 | Sharpe | Nov 2003 | B1 |
| 6674434 | Chojnacki et al. | Jan 2004 | B1 |
| 6714936 | Nevin, III | Mar 2004 | B1 |
| 6725240 | Asad et al. | Apr 2004 | B1 |
| 6775675 | Nwabueze et al. | Aug 2004 | B1 |
| 6807569 | Bhimani et al. | Oct 2004 | B1 |
| 6820135 | Dingman | Nov 2004 | B1 |
| 6828920 | Owen et al. | Dec 2004 | B2 |
| 6839745 | Dingari et al. | Jan 2005 | B1 |
| 6877137 | Rivette et al. | Apr 2005 | B1 |
| 6976210 | Silva et al. | Dec 2005 | B1 |
| 6980984 | Huffman et al. | Dec 2005 | B1 |
| 6985950 | Hanson et al. | Jan 2006 | B1 |
| 7017046 | Doyle et al. | Mar 2006 | B2 |
| 7036085 | Barros | Apr 2006 | B2 |
| 7043702 | Chi et al. | May 2006 | B2 |
| 7055110 | Kupka et al. | May 2006 | B2 |
| 7139800 | Bellotti et al. | Nov 2006 | B2 |
| 7158878 | Rasmussen et al. | Jan 2007 | B2 |
| 7162475 | Ackerman | Jan 2007 | B2 |
| 7168039 | Bertram | Jan 2007 | B2 |
| 7171427 | Witowski et al. | Jan 2007 | B2 |
| 7225468 | Waisman et al. | May 2007 | B2 |
| 7269786 | Malloy et al. | Sep 2007 | B1 |
| 7278105 | Kitts | Oct 2007 | B1 |
| 7290698 | Poslinski et al. | Nov 2007 | B2 |
| 7333998 | Heckerman et al. | Feb 2008 | B2 |
| 7370047 | Gorman | May 2008 | B2 |
| 7373669 | Eisen | May 2008 | B2 |
| 7379811 | Rasmussen et al. | May 2008 | B2 |
| 7379903 | Caballero et al. | May 2008 | B2 |
| 7426654 | Adams et al. | Sep 2008 | B2 |
| 7451397 | Weber et al. | Nov 2008 | B2 |
| 7454466 | Bellotti et al. | Nov 2008 | B2 |
| 7467375 | Tondreau et al. | Dec 2008 | B2 |
| 7487139 | Fraleigh et al. | Feb 2009 | B2 |
| 7502786 | Liu et al. | Mar 2009 | B2 |
| 7525422 | Bishop et al. | Apr 2009 | B2 |
| 7529727 | Arning et al. | May 2009 | B2 |
| 7529734 | Dirisala | May 2009 | B2 |
| 7533008 | Mangino et al. | May 2009 | B2 |
| 7546245 | Surpin et al. | Jun 2009 | B2 |
| 7558677 | Jones | Jul 2009 | B2 |
| 7574409 | Patinkin | Aug 2009 | B2 |
| 7574428 | Leiserowitz et al. | Aug 2009 | B2 |
| 7579965 | Bucholz | Aug 2009 | B2 |
| 7596285 | Brown et al. | Sep 2009 | B2 |
| 7614006 | Molander | Nov 2009 | B2 |
| 7617232 | Gabbert et al. | Nov 2009 | B2 |
| 7620628 | Kapur et al. | Nov 2009 | B2 |
| 7627812 | Chamberlain et al. | Dec 2009 | B2 |
| 7634717 | Chamberlain et al. | Dec 2009 | B2 |
| 7640173 | Surpin et al. | Dec 2009 | B2 |
| 7703021 | Flam | Apr 2010 | B1 |
| 7706817 | Bamrah et al. | Apr 2010 | B2 |
| 7712049 | Williams et al. | May 2010 | B2 |
| 7716067 | Surpin et al. | May 2010 | B2 |
| 7716077 | Mikurak | May 2010 | B1 |
| 7725530 | Sah et al. | May 2010 | B2 |
| 7725547 | Albertson et al. | May 2010 | B2 |
| 7730082 | Sah et al. | Jun 2010 | B2 |
| 7730109 | Rohrs et al. | Jun 2010 | B2 |
| 7770032 | Nesta et al. | Aug 2010 | B2 |
| 7770100 | Chamberlain et al. | Aug 2010 | B2 |
| 7783658 | Bayliss | Aug 2010 | B1 |
| 7801871 | Gosnell | Sep 2010 | B2 |
| 7805457 | Viola et al. | Sep 2010 | B1 |
| 7809703 | Balabhadrapatruni et al. | Oct 2010 | B2 |
| 7814102 | Miller et al. | Oct 2010 | B2 |
| 7818291 | Ferguson et al. | Oct 2010 | B2 |
| 7818658 | Chen | Oct 2010 | B2 |
| 7870493 | Pall et al. | Jan 2011 | B2 |
| 7894984 | Rasmussen et al. | Feb 2011 | B2 |
| 7899611 | Downs et al. | Mar 2011 | B2 |
| 7917376 | Bellin et al. | Mar 2011 | B2 |
| 7920963 | Jouline et al. | Apr 2011 | B2 |
| 7933862 | Chamberlain et al. | Apr 2011 | B2 |
| 7962281 | Rasmussen et al. | Jun 2011 | B2 |
| 7962495 | Jain et al. | Jun 2011 | B2 |
| 7962848 | Bertram | Jun 2011 | B2 |
| 7970240 | Chao et al. | Jun 2011 | B1 |
| 7971150 | Raskutti et al. | Jun 2011 | B2 |
| 7984374 | Caro et al. | Jul 2011 | B2 |
| 8001465 | Kudrolli et al. | Aug 2011 | B2 |
| 8001482 | Bhattiprolu et al. | Aug 2011 | B2 |
| 8010545 | Stefik et al. | Aug 2011 | B2 |
| 8010886 | Gusmorino et al. | Aug 2011 | B2 |
| 8015487 | Roy et al. | Sep 2011 | B2 |
| 8019709 | Norton et al. | Sep 2011 | B2 |
| 8024778 | Cash et al. | Sep 2011 | B2 |
| 8036632 | Cona et al. | Oct 2011 | B1 |
| 8042110 | Kawahara et al. | Oct 2011 | B1 |
| 8046362 | Bayliss | Oct 2011 | B2 |
| 8082172 | Chao et al. | Dec 2011 | B2 |
| 8103543 | Zwicky | Jan 2012 | B1 |
| 8134457 | Velipasalar et al. | Mar 2012 | B2 |
| 8135679 | Bayliss | Mar 2012 | B2 |
| 8135719 | Bayliss | Mar 2012 | B2 |
| 8145703 | Frishert et al. | Mar 2012 | B2 |
| 8185819 | Sah et al. | May 2012 | B2 |
| 8190893 | Benson et al. | May 2012 | B2 |
| 8196184 | Amirov et al. | Jun 2012 | B2 |
| 8214361 | Sandler et al. | Jul 2012 | B1 |
| 8214764 | Gemmell et al. | Jul 2012 | B2 |
| 8225201 | Michael | Jul 2012 | B2 |
| 8229947 | Fujinaga | Jul 2012 | B2 |
| 8230333 | Decherd et al. | Jul 2012 | B2 |
| 8239668 | Chen et al. | Aug 2012 | B1 |
| 8266168 | Bayliss | Sep 2012 | B2 |
| 8271461 | Pike et al. | Sep 2012 | B2 |
| 8280880 | Aymeloglu et al. | Oct 2012 | B1 |
| 8290926 | Ozzie et al. | Oct 2012 | B2 |
| 8290942 | Jones et al. | Oct 2012 | B2 |
| 8301464 | Cave et al. | Oct 2012 | B1 |
| 8301904 | Gryaznov | Oct 2012 | B1 |
| 8312367 | Foster | Nov 2012 | B2 |
| 8312546 | Alme | Nov 2012 | B2 |
| 8321943 | Walters et al. | Nov 2012 | B1 |
| 8347398 | Weber | Jan 2013 | B1 |
| 8352881 | Champion et al. | Jan 2013 | B2 |
| 8368695 | Howell et al. | Feb 2013 | B2 |
| 8397171 | Klassen et al. | Mar 2013 | B2 |
| 8411046 | Kruzeniski et al. | Apr 2013 | B2 |
| 8412707 | Mianji | Apr 2013 | B1 |
| 8447674 | Choudhuri et al. | May 2013 | B2 |
| 8447722 | Ahuja et al. | May 2013 | B1 |
| 8452790 | Mianji | May 2013 | B1 |
| 8463036 | Ramesh et al. | Jun 2013 | B1 |
| 8484168 | Bayliss | Jul 2013 | B2 |
| 8489331 | Kopf et al. | Jul 2013 | B2 |
| 8489623 | Jain et al. | Jul 2013 | B2 |
| 8489641 | Seefeld et al. | Jul 2013 | B1 |
| 8495077 | Bayliss | Jul 2013 | B2 |
| 8498969 | Bayliss | Jul 2013 | B2 |
| 8498984 | Hwang et al. | Jul 2013 | B1 |
| 8510743 | Hackborn et al. | Aug 2013 | B2 |
| 8514082 | Cova et al. | Aug 2013 | B2 |
| 8515207 | Chau | Aug 2013 | B2 |
| 8515912 | Garrod et al. | Aug 2013 | B2 |
| 8527461 | Ducott, III et al. | Sep 2013 | B2 |
| 8554579 | Tribble et al. | Oct 2013 | B2 |
| 8554653 | Falkenborg et al. | Oct 2013 | B2 |
| 8554709 | Goodson et al. | Oct 2013 | B2 |
| 8560413 | Quarterman | Oct 2013 | B1 |
| 8577911 | Stepinski et al. | Nov 2013 | B1 |
| 8589273 | Creeden et al. | Nov 2013 | B2 |
| 8595234 | Siripuapu et al. | Nov 2013 | B2 |
| 8600872 | Yan | Dec 2013 | B1 |
| 8620641 | Farnsworth et al. | Dec 2013 | B2 |
| 8639757 | Zang et al. | Jan 2014 | B1 |
| 8646080 | Williamson et al. | Feb 2014 | B2 |
| 8676597 | Buehler et al. | Mar 2014 | B2 |
| 8676857 | Adams et al. | Mar 2014 | B1 |
| 8689108 | Duffield et al. | Apr 2014 | B1 |
| 8707185 | Robinson et al. | Apr 2014 | B2 |
| 8713467 | Goldenberg et al. | Apr 2014 | B1 |
| 8726379 | Stiansen et al. | May 2014 | B1 |
| 8739278 | Varghese | May 2014 | B2 |
| 8742934 | Sarpy et al. | Jun 2014 | B1 |
| 8744890 | Bernier | Jun 2014 | B1 |
| 8745516 | Mason et al. | Jun 2014 | B2 |
| 8756244 | Dassa et al. | Jun 2014 | B2 |
| 8781169 | Jackson et al. | Jul 2014 | B2 |
| 8787939 | Papakipos et al. | Jul 2014 | B2 |
| 8788405 | Sprague et al. | Jul 2014 | B1 |
| 8788407 | Singh et al. | Jul 2014 | B1 |
| 8799799 | Cervelli et al. | Aug 2014 | B1 |
| 8799812 | Parker | Aug 2014 | B2 |
| 8812960 | Sun et al. | Aug 2014 | B1 |
| 8818892 | Sprague et al. | Aug 2014 | B1 |
| 8826438 | Perdisci et al. | Sep 2014 | B2 |
| 8830322 | Nerayoff et al. | Sep 2014 | B2 |
| 8832594 | Thompson et al. | Sep 2014 | B1 |
| 8868486 | Tamayo | Oct 2014 | B2 |
| 8868537 | Colgrove et al. | Oct 2014 | B1 |
| 8917274 | Ma et al. | Dec 2014 | B2 |
| 8924872 | Bogomolov et al. | Dec 2014 | B1 |
| 8937619 | Sharma et al. | Jan 2015 | B2 |
| 8938686 | Erenrich et al. | Jan 2015 | B1 |
| 8972376 | Gailis et al. | Mar 2015 | B1 |
| 9009171 | Grossman et al. | Apr 2015 | B1 |
| 9009827 | Albertson et al. | Apr 2015 | B1 |
| 9021260 | Falk et al. | Apr 2015 | B1 |
| 9021384 | Beard et al. | Apr 2015 | B1 |
| 9043696 | Meiklejohn et al. | May 2015 | B1 |
| 9043894 | Dennison et al. | May 2015 | B1 |
| 9069842 | Melby | Jun 2015 | B2 |
| 9116975 | Shankar et al. | Aug 2015 | B2 |
| 9135658 | Sprague et al. | Sep 2015 | B2 |
| 9165299 | Stowe et al. | Oct 2015 | B1 |
| 9171334 | Visbal et al. | Oct 2015 | B1 |
| 9177344 | Singh et al. | Nov 2015 | B1 |
| 9202249 | Cohen et al. | Dec 2015 | B1 |
| 9230280 | Maag et al. | Jan 2016 | B1 |
| 9256664 | Chakerian et al. | Feb 2016 | B2 |
| 9344447 | Cohen et al. | May 2016 | B2 |
| 9367872 | Visbal et al. | Jun 2016 | B1 |
| 9589299 | Visbal et al. | Mar 2017 | B2 |
| 9674662 | Freeland et al. | Jun 2017 | B2 |
| 9965937 | Cohen et al. | May 2018 | B2 |
| 9998485 | Cohen et al. | Jun 2018 | B2 |
| 20010021936 | Bertram | Sep 2001 | A1 |
| 20020033848 | Sciammarella et al. | Mar 2002 | A1 |
| 20020065708 | Senay et al. | May 2002 | A1 |
| 20020091707 | Keller | Jul 2002 | A1 |
| 20020095658 | Shulman | Jul 2002 | A1 |
| 20020112157 | Doyle et al. | Aug 2002 | A1 |
| 20020116120 | Ruiz et al. | Aug 2002 | A1 |
| 20020130907 | Chi et al. | Sep 2002 | A1 |
| 20020174201 | Ramer et al. | Nov 2002 | A1 |
| 20020194119 | Wright et al. | Dec 2002 | A1 |
| 20030028560 | Kudrolli et al. | Feb 2003 | A1 |
| 20030033228 | Bosworth-Davies et al. | Feb 2003 | A1 |
| 20030036848 | Sheha et al. | Feb 2003 | A1 |
| 20030039948 | Donahue | Feb 2003 | A1 |
| 20030074368 | Schuetze et al. | Apr 2003 | A1 |
| 20030097330 | Hillmer et al. | May 2003 | A1 |
| 20030140106 | Raguseo | Jul 2003 | A1 |
| 20030144868 | MacIntyre et al. | Jul 2003 | A1 |
| 20030163352 | Surpin et al. | Aug 2003 | A1 |
| 20030200217 | Ackerman | Oct 2003 | A1 |
| 20030225755 | Iwayama et al. | Dec 2003 | A1 |
| 20030229848 | Arend et al. | Dec 2003 | A1 |
| 20040032432 | Baynger | Feb 2004 | A1 |
| 20040064256 | Barinek et al. | Apr 2004 | A1 |
| 20040085318 | Hassler et al. | May 2004 | A1 |
| 20040095349 | Bito et al. | May 2004 | A1 |
| 20040111410 | Burgoon et al. | Jun 2004 | A1 |
| 20040126840 | Cheng et al. | Jul 2004 | A1 |
| 20040143602 | Ruiz et al. | Jul 2004 | A1 |
| 20040143796 | Lerner et al. | Jul 2004 | A1 |
| 20040153418 | Hanweck | Aug 2004 | A1 |
| 20040163039 | Gorman | Aug 2004 | A1 |
| 20040181554 | Heckerman et al. | Sep 2004 | A1 |
| 20040193600 | Kaasten et al. | Sep 2004 | A1 |
| 20040205524 | Richter et al. | Oct 2004 | A1 |
| 20040221223 | Yu et al. | Nov 2004 | A1 |
| 20040250124 | Chesla et al. | Dec 2004 | A1 |
| 20040260702 | Cragun et al. | Dec 2004 | A1 |
| 20040267746 | Marcjan et al. | Dec 2004 | A1 |
| 20050027705 | Sadri et al. | Feb 2005 | A1 |
| 20050028094 | Allyn | Feb 2005 | A1 |
| 20050039119 | Parks et al. | Feb 2005 | A1 |
| 20050065811 | Chu et al. | Mar 2005 | A1 |
| 20050078858 | Yao et al. | Apr 2005 | A1 |
| 20050080769 | Gemmell | Apr 2005 | A1 |
| 20050086207 | Heuer et al. | Apr 2005 | A1 |
| 20050108063 | Madill et al. | May 2005 | A1 |
| 20050125715 | Di Franco et al. | Jun 2005 | A1 |
| 20050154628 | Eckart et al. | Jul 2005 | A1 |
| 20050154769 | Eckart et al. | Jul 2005 | A1 |
| 20050162523 | Darrell et al. | Jul 2005 | A1 |
| 20050166144 | Gross | Jul 2005 | A1 |
| 20050180330 | Shapiro | Aug 2005 | A1 |
| 20050182793 | Keenan et al. | Aug 2005 | A1 |
| 20050183005 | Denoue et al. | Aug 2005 | A1 |
| 20050210409 | Jou | Sep 2005 | A1 |
| 20050222928 | Steier et al. | Oct 2005 | A1 |
| 20050229256 | Banzhof | Oct 2005 | A2 |
| 20050246327 | Yeung et al. | Nov 2005 | A1 |
| 20050251786 | Citron et al. | Nov 2005 | A1 |
| 20060026120 | Carolan et al. | Feb 2006 | A1 |
| 20060026170 | Kreitler et al. | Feb 2006 | A1 |
| 20060045470 | Poslinski et al. | Mar 2006 | A1 |
| 20060053096 | Subramanian et al. | Mar 2006 | A1 |
| 20060059139 | Robinson | Mar 2006 | A1 |
| 20060069912 | Zheng et al. | Mar 2006 | A1 |
| 20060074866 | Chamberlain et al. | Apr 2006 | A1 |
| 20060074881 | Vembu et al. | Apr 2006 | A1 |
| 20060080619 | Carlson et al. | Apr 2006 | A1 |
| 20060093222 | Saffer et al. | May 2006 | A1 |
| 20060095521 | Patinkin | May 2006 | A1 |
| 20060129746 | Porter | Jun 2006 | A1 |
| 20060139375 | Rasmussen et al. | Jun 2006 | A1 |
| 20060142949 | Helt | Jun 2006 | A1 |
| 20060143034 | Rothermel | Jun 2006 | A1 |
| 20060149596 | Surpin et al. | Jul 2006 | A1 |
| 20060161558 | Tamma et al. | Jul 2006 | A1 |
| 20060203337 | White | Sep 2006 | A1 |
| 20060218637 | Thomas et al. | Sep 2006 | A1 |
| 20060241974 | Chao et al. | Oct 2006 | A1 |
| 20060242040 | Rader | Oct 2006 | A1 |
| 20060242630 | Koike et al. | Oct 2006 | A1 |
| 20060253578 | Dixon | Nov 2006 | A1 |
| 20060265747 | Judge | Nov 2006 | A1 |
| 20060271277 | Hu et al. | Nov 2006 | A1 |
| 20060279630 | Aggarwal et al. | Dec 2006 | A1 |
| 20070011150 | Frank | Jan 2007 | A1 |
| 20070016363 | Huang et al. | Jan 2007 | A1 |
| 20070038646 | Thota | Feb 2007 | A1 |
| 20070038962 | Fuchs et al. | Feb 2007 | A1 |
| 20070057966 | Ohno et al. | Mar 2007 | A1 |
| 20070078832 | Ott et al. | Apr 2007 | A1 |
| 20070083541 | Fraleigh et al. | Apr 2007 | A1 |
| 20070088596 | Berkelhamer et al. | Apr 2007 | A1 |
| 20070094389 | Nussey et al. | Apr 2007 | A1 |
| 20070106582 | Baker et al. | May 2007 | A1 |
| 20070150369 | Zivin | Jun 2007 | A1 |
| 20070174760 | Chamberlain et al. | Jul 2007 | A1 |
| 20070192265 | Chopin et al. | Aug 2007 | A1 |
| 20070198571 | Ferguson et al. | Aug 2007 | A1 |
| 20070208497 | Downs et al. | Sep 2007 | A1 |
| 20070208498 | Barker et al. | Sep 2007 | A1 |
| 20070208736 | Tanigawa et al. | Sep 2007 | A1 |
| 20070233709 | Abnous | Oct 2007 | A1 |
| 20070240062 | Christena et al. | Oct 2007 | A1 |
| 20070266336 | Nojima et al. | Nov 2007 | A1 |
| 20070294200 | Au | Dec 2007 | A1 |
| 20070294643 | Kyle | Dec 2007 | A1 |
| 20070294766 | Mir et al. | Dec 2007 | A1 |
| 20080016216 | Worley et al. | Jan 2008 | A1 |
| 20080040275 | Paulsen et al. | Feb 2008 | A1 |
| 20080040684 | Crump | Feb 2008 | A1 |
| 20080051989 | Welsh | Feb 2008 | A1 |
| 20080052142 | Bailey et al. | Feb 2008 | A1 |
| 20080077597 | Butler | Mar 2008 | A1 |
| 20080077642 | Carbone et al. | Mar 2008 | A1 |
| 20080082486 | Lermant et al. | Apr 2008 | A1 |
| 20080104019 | Nath | May 2008 | A1 |
| 20080126951 | Sood et al. | May 2008 | A1 |
| 20080133567 | Ames et al. | Jun 2008 | A1 |
| 20080148398 | Mezack et al. | Jun 2008 | A1 |
| 20080155440 | Trevor et al. | Jun 2008 | A1 |
| 20080162616 | Gross et al. | Jul 2008 | A1 |
| 20080195417 | Surpin et al. | Aug 2008 | A1 |
| 20080195608 | Clover | Aug 2008 | A1 |
| 20080201580 | Savitzky et al. | Aug 2008 | A1 |
| 20080222295 | Robinson et al. | Sep 2008 | A1 |
| 20080229422 | Hudis et al. | Sep 2008 | A1 |
| 20080249983 | Meisels et al. | Oct 2008 | A1 |
| 20080255973 | El Wade et al. | Oct 2008 | A1 |
| 20080263468 | Cappione et al. | Oct 2008 | A1 |
| 20080267107 | Rosenberg | Oct 2008 | A1 |
| 20080270328 | Lafferty et al. | Oct 2008 | A1 |
| 20080276167 | Michael | Nov 2008 | A1 |
| 20080278311 | Grange et al. | Nov 2008 | A1 |
| 20080281819 | Tenenbaum et al. | Nov 2008 | A1 |
| 20080288306 | MacIntyre et al. | Nov 2008 | A1 |
| 20080288425 | Posse et al. | Nov 2008 | A1 |
| 20080288475 | Kim et al. | Nov 2008 | A1 |
| 20080301559 | Martinsen et al. | Dec 2008 | A1 |
| 20080301643 | Appleton et al. | Dec 2008 | A1 |
| 20080313281 | Scheidl et al. | Dec 2008 | A1 |
| 20090002492 | Velipasalar et al. | Jan 2009 | A1 |
| 20090007272 | Huang et al. | Jan 2009 | A1 |
| 20090018940 | Wang et al. | Jan 2009 | A1 |
| 20090024505 | Patel et al. | Jan 2009 | A1 |
| 20090027418 | Maru et al. | Jan 2009 | A1 |
| 20090030915 | Winter et al. | Jan 2009 | A1 |
| 20090037912 | Stoitsev et al. | Feb 2009 | A1 |
| 20090044279 | Crawford et al. | Feb 2009 | A1 |
| 20090055251 | Shah et al. | Feb 2009 | A1 |
| 20090070162 | Leonelli et al. | Mar 2009 | A1 |
| 20090076845 | Bellin et al. | Mar 2009 | A1 |
| 20090082997 | Tokman et al. | Mar 2009 | A1 |
| 20090083184 | Eisen | Mar 2009 | A1 |
| 20090088964 | Schaaf et al. | Apr 2009 | A1 |
| 20090103442 | Douville | Apr 2009 | A1 |
| 20090119309 | Gibson et al. | May 2009 | A1 |
| 20090125359 | Knapic | May 2009 | A1 |
| 20090125369 | Kloosstra et al. | May 2009 | A1 |
| 20090125459 | Norton et al. | May 2009 | A1 |
| 20090132921 | Hwangbo et al. | May 2009 | A1 |
| 20090132953 | Reed et al. | May 2009 | A1 |
| 20090143052 | Bates et al. | Jun 2009 | A1 |
| 20090144262 | White et al. | Jun 2009 | A1 |
| 20090144274 | Fraleigh et al. | Jun 2009 | A1 |
| 20090164934 | Bhattiprolu et al. | Jun 2009 | A1 |
| 20090171939 | Athsani et al. | Jul 2009 | A1 |
| 20090172511 | Decherd et al. | Jul 2009 | A1 |
| 20090172821 | Daira et al. | Jul 2009 | A1 |
| 20090177962 | Gusmorino et al. | Jul 2009 | A1 |
| 20090179892 | Tsuda et al. | Jul 2009 | A1 |
| 20090187464 | Bai et al. | Jul 2009 | A1 |
| 20090192957 | Subramanian et al. | Jul 2009 | A1 |
| 20090222400 | Kupershmidt et al. | Sep 2009 | A1 |
| 20090222759 | Drieschner | Sep 2009 | A1 |
| 20090222760 | Halverson et al. | Sep 2009 | A1 |
| 20090228701 | Lin | Sep 2009 | A1 |
| 20090234720 | George et al. | Sep 2009 | A1 |
| 20090249244 | Robinson et al. | Oct 2009 | A1 |
| 20090254970 | Agarwal et al. | Oct 2009 | A1 |
| 20090254971 | Herz | Oct 2009 | A1 |
| 20090271359 | Bayliss | Oct 2009 | A1 |
| 20090281839 | Lynn et al. | Nov 2009 | A1 |
| 20090287470 | Farnsworth et al. | Nov 2009 | A1 |
| 20090292626 | Oxford | Nov 2009 | A1 |
| 20090300589 | Watters et al. | Dec 2009 | A1 |
| 20090318775 | Michelson et al. | Dec 2009 | A1 |
| 20090328222 | Helman et al. | Dec 2009 | A1 |
| 20100004857 | Pereira et al. | Jan 2010 | A1 |
| 20100011282 | Dollard et al. | Jan 2010 | A1 |
| 20100042922 | Bradateanu et al. | Feb 2010 | A1 |
| 20100057716 | Stefik et al. | Mar 2010 | A1 |
| 20100070523 | Delgo et al. | Mar 2010 | A1 |
| 20100070842 | Aymeloglu et al. | Mar 2010 | A1 |
| 20100070845 | Facemire et al. | Mar 2010 | A1 |
| 20100070897 | Aymeloglu et al. | Mar 2010 | A1 |
| 20100077481 | Polyakov et al. | Mar 2010 | A1 |
| 20100077483 | Stolfo et al. | Mar 2010 | A1 |
| 20100100963 | Mahaffey | Apr 2010 | A1 |
| 20100103124 | Kruzeniski et al. | Apr 2010 | A1 |
| 20100106611 | Paulsen et al. | Apr 2010 | A1 |
| 20100106752 | Eckardt et al. | Apr 2010 | A1 |
| 20100114817 | Broeder et al. | May 2010 | A1 |
| 20100114887 | Conway et al. | May 2010 | A1 |
| 20100122152 | Chamberlain et al. | May 2010 | A1 |
| 20100125546 | Barrett et al. | May 2010 | A1 |
| 20100131457 | Heimendinger | May 2010 | A1 |
| 20100162176 | Dunton | Jun 2010 | A1 |
| 20100169237 | Howard et al. | Jul 2010 | A1 |
| 20100185691 | Irmak et al. | Jul 2010 | A1 |
| 20100191563 | Schlaifer et al. | Jul 2010 | A1 |
| 20100198684 | Eraker et al. | Aug 2010 | A1 |
| 20100199225 | Coleman et al. | Aug 2010 | A1 |
| 20100228812 | Uomini | Sep 2010 | A1 |
| 20100235915 | Memon et al. | Sep 2010 | A1 |
| 20100250412 | Wagner | Sep 2010 | A1 |
| 20100262688 | Hussain et al. | Oct 2010 | A1 |
| 20100280857 | Liu et al. | Nov 2010 | A1 |
| 20100293174 | Bennett et al. | Nov 2010 | A1 |
| 20100306029 | Jolley | Dec 2010 | A1 |
| 20100306713 | Geisner et al. | Dec 2010 | A1 |
| 20100313119 | Baldwin et al. | Dec 2010 | A1 |
| 20100318924 | Frankel et al. | Dec 2010 | A1 |
| 20100321399 | Ellren et al. | Dec 2010 | A1 |
| 20100325526 | Ellis et al. | Dec 2010 | A1 |
| 20100325581 | Finkelstein et al. | Dec 2010 | A1 |
| 20100330801 | Rouh | Dec 2010 | A1 |
| 20110004498 | Readshaw | Jan 2011 | A1 |
| 20110029526 | Knight et al. | Feb 2011 | A1 |
| 20110047159 | Baid et al. | Feb 2011 | A1 |
| 20110055140 | Roychowdhury | Mar 2011 | A1 |
| 20110060753 | Shaked et al. | Mar 2011 | A1 |
| 20110060910 | Gormish et al. | Mar 2011 | A1 |
| 20110061013 | Bilicki et al. | Mar 2011 | A1 |
| 20110066933 | Ludwig | Mar 2011 | A1 |
| 20110074811 | Hanson et al. | Mar 2011 | A1 |
| 20110078055 | Faribault et al. | Mar 2011 | A1 |
| 20110078173 | Seligmann et al. | Mar 2011 | A1 |
| 20110087519 | Fordyce, III et al. | Apr 2011 | A1 |
| 20110093327 | Fordyce, III et al. | Apr 2011 | A1 |
| 20110117878 | Barash et al. | May 2011 | A1 |
| 20110119100 | Ruhl et al. | May 2011 | A1 |
| 20110131122 | Griffin et al. | Jun 2011 | A1 |
| 20110137766 | Rasmussen et al. | Jun 2011 | A1 |
| 20110153384 | Horne et al. | Jun 2011 | A1 |
| 20110161096 | Buehler et al. | Jun 2011 | A1 |
| 20110167054 | Bailey et al. | Jul 2011 | A1 |
| 20110167105 | Ramakrishnan et al. | Jul 2011 | A1 |
| 20110167493 | Song et al. | Jul 2011 | A1 |
| 20110170799 | Carrino et al. | Jul 2011 | A1 |
| 20110173032 | Payne et al. | Jul 2011 | A1 |
| 20110173093 | Psota et al. | Jul 2011 | A1 |
| 20110178842 | Rane et al. | Jul 2011 | A1 |
| 20110185316 | Reid et al. | Jul 2011 | A1 |
| 20110208724 | Jones et al. | Aug 2011 | A1 |
| 20110213655 | Henkin | Sep 2011 | A1 |
| 20110218934 | Elser | Sep 2011 | A1 |
| 20110219450 | McDougal et al. | Sep 2011 | A1 |
| 20110225198 | Edwards et al. | Sep 2011 | A1 |
| 20110225650 | Margolies et al. | Sep 2011 | A1 |
| 20110231223 | Winters | Sep 2011 | A1 |
| 20110238495 | Kang | Sep 2011 | A1 |
| 20110238510 | Rowen et al. | Sep 2011 | A1 |
| 20110238553 | Raj et al. | Sep 2011 | A1 |
| 20110238570 | Li et al. | Sep 2011 | A1 |
| 20110246229 | Pacha | Oct 2011 | A1 |
| 20110251951 | Kolkowtiz | Oct 2011 | A1 |
| 20110258158 | Resende et al. | Oct 2011 | A1 |
| 20110270705 | Parker | Nov 2011 | A1 |
| 20110289397 | Eastmond et al. | Nov 2011 | A1 |
| 20110289407 | Naik et al. | Nov 2011 | A1 |
| 20110289420 | Morioka et al. | Nov 2011 | A1 |
| 20110291851 | Whisenant | Dec 2011 | A1 |
| 20110302397 | Mitola, III | Dec 2011 | A1 |
| 20110307382 | Siegel et al. | Dec 2011 | A1 |
| 20110310005 | Chen et al. | Dec 2011 | A1 |
| 20110314007 | Dassa et al. | Dec 2011 | A1 |
| 20120004904 | Shin et al. | Jan 2012 | A1 |
| 20120019559 | Siler et al. | Jan 2012 | A1 |
| 20120036013 | Neuhaus et al. | Feb 2012 | A1 |
| 20120036434 | Oberstein | Feb 2012 | A1 |
| 20120050293 | Carlhian et al. | Mar 2012 | A1 |
| 20120066296 | Appleton et al. | Mar 2012 | A1 |
| 20120072825 | Sherkin et al. | Mar 2012 | A1 |
| 20120079363 | Folting et al. | Mar 2012 | A1 |
| 20120084118 | Bai et al. | Apr 2012 | A1 |
| 20120084135 | Nissan et al. | Apr 2012 | A1 |
| 20120084866 | Stolfo | Apr 2012 | A1 |
| 20120106801 | Jackson | May 2012 | A1 |
| 20120110633 | An et al. | May 2012 | A1 |
| 20120110674 | Belani et al. | May 2012 | A1 |
| 20120117082 | Koperda et al. | May 2012 | A1 |
| 20120131512 | Takeuchi et al. | May 2012 | A1 |
| 20120137235 | Ts et al. | May 2012 | A1 |
| 20120144335 | Abeln et al. | Jun 2012 | A1 |
| 20120159307 | Chung et al. | Jun 2012 | A1 |
| 20120159362 | Brown et al. | Jun 2012 | A1 |
| 20120159399 | Bastide et al. | Jun 2012 | A1 |
| 20120170847 | Tsukidate | Jul 2012 | A1 |
| 20120173985 | Peppel | Jul 2012 | A1 |
| 20120180002 | Campbell et al. | Jul 2012 | A1 |
| 20120196557 | Reich et al. | Aug 2012 | A1 |
| 20120196558 | Reich et al. | Aug 2012 | A1 |
| 20120197651 | Robinson et al. | Aug 2012 | A1 |
| 20120203708 | Psota et al. | Aug 2012 | A1 |
| 20120208636 | Feige | Aug 2012 | A1 |
| 20120215898 | Shah et al. | Aug 2012 | A1 |
| 20120221511 | Gibson et al. | Aug 2012 | A1 |
| 20120221553 | Wittmer et al. | Aug 2012 | A1 |
| 20120221580 | Barney | Aug 2012 | A1 |
| 20120245976 | Kumar et al. | Sep 2012 | A1 |
| 20120246148 | Dror | Sep 2012 | A1 |
| 20120254129 | Wheeler et al. | Oct 2012 | A1 |
| 20120266245 | McDougal et al. | Oct 2012 | A1 |
| 20120284345 | Costenaro et al. | Nov 2012 | A1 |
| 20120290879 | Shibuya et al. | Nov 2012 | A1 |
| 20120296907 | Long et al. | Nov 2012 | A1 |
| 20120304244 | Xie et al. | Nov 2012 | A1 |
| 20120310831 | Harris et al. | Dec 2012 | A1 |
| 20120310838 | Harris et al. | Dec 2012 | A1 |
| 20120311684 | Paulsen et al. | Dec 2012 | A1 |
| 20120323829 | Stokes et al. | Dec 2012 | A1 |
| 20120323888 | Osann, Jr. | Dec 2012 | A1 |
| 20120330801 | McDougal et al. | Dec 2012 | A1 |
| 20120330973 | Ghuneim et al. | Dec 2012 | A1 |
| 20130006426 | Healey et al. | Jan 2013 | A1 |
| 20130006655 | Van Arkel et al. | Jan 2013 | A1 |
| 20130006668 | Van Arkel et al. | Jan 2013 | A1 |
| 20130006725 | Simanek et al. | Jan 2013 | A1 |
| 20130006916 | McBride et al. | Jan 2013 | A1 |
| 20130018796 | Kolhatkar et al. | Jan 2013 | A1 |
| 20130019306 | Lagar-Cavilla et al. | Jan 2013 | A1 |
| 20130024268 | Manickavelu | Jan 2013 | A1 |
| 20130024307 | Fuerstenberg et al. | Jan 2013 | A1 |
| 20130024339 | Choudhuri et al. | Jan 2013 | A1 |
| 20130046635 | Grigg et al. | Feb 2013 | A1 |
| 20130046842 | Muntz et al. | Feb 2013 | A1 |
| 20130060786 | Serrano et al. | Mar 2013 | A1 |
| 20130061169 | Pearcy et al. | Mar 2013 | A1 |
| 20130073377 | Heath | Mar 2013 | A1 |
| 20130073454 | Busch | Mar 2013 | A1 |
| 20130078943 | Biage et al. | Mar 2013 | A1 |
| 20130086482 | Parsons | Apr 2013 | A1 |
| 20130097482 | Marantz et al. | Apr 2013 | A1 |
| 20130101159 | Chao et al. | Apr 2013 | A1 |
| 20130110822 | Ikeda et al. | May 2013 | A1 |
| 20130110877 | Bonham et al. | May 2013 | A1 |
| 20130111320 | Campbell et al. | May 2013 | A1 |
| 20130117651 | Waldman et al. | May 2013 | A1 |
| 20130139268 | An et al. | May 2013 | A1 |
| 20130150004 | Rosen | Jun 2013 | A1 |
| 20130151148 | Parundekar et al. | Jun 2013 | A1 |
| 20130151388 | Falkenborg et al. | Jun 2013 | A1 |
| 20130157234 | Gulli et al. | Jun 2013 | A1 |
| 20130160120 | Malaviya et al. | Jun 2013 | A1 |
| 20130166550 | Buchmann et al. | Jun 2013 | A1 |
| 20130176321 | Mitchell et al. | Jul 2013 | A1 |
| 20130179420 | Park et al. | Jul 2013 | A1 |
| 20130197925 | Blue | Aug 2013 | A1 |
| 20130211985 | Clark et al. | Aug 2013 | A1 |
| 20130224696 | Wolfe et al. | Aug 2013 | A1 |
| 20130225212 | Khan | Aug 2013 | A1 |
| 20130226318 | Procyk | Aug 2013 | A1 |
| 20130226953 | Markovich et al. | Aug 2013 | A1 |
| 20130232045 | Tai et al. | Sep 2013 | A1 |
| 20130238616 | Rose et al. | Sep 2013 | A1 |
| 20130246170 | Gross et al. | Sep 2013 | A1 |
| 20130251233 | Yang et al. | Sep 2013 | A1 |
| 20130262527 | Hunter et al. | Oct 2013 | A1 |
| 20130262528 | Foit | Oct 2013 | A1 |
| 20130263019 | Castellanos et al. | Oct 2013 | A1 |
| 20130267207 | Hao et al. | Oct 2013 | A1 |
| 20130268520 | Fisher et al. | Oct 2013 | A1 |
| 20130276799 | Davidson | Oct 2013 | A1 |
| 20130279757 | Kephart | Oct 2013 | A1 |
| 20130282696 | John et al. | Oct 2013 | A1 |
| 20130288719 | Alonzo | Oct 2013 | A1 |
| 20130290011 | Lynn et al. | Oct 2013 | A1 |
| 20130290825 | Arndt et al. | Oct 2013 | A1 |
| 20130297619 | Chandrasekaran et al. | Nov 2013 | A1 |
| 20130311375 | Priebatsch | Nov 2013 | A1 |
| 20130318594 | Hoy et al. | Nov 2013 | A1 |
| 20130339218 | Subramanian et al. | Dec 2013 | A1 |
| 20140006109 | Callioni et al. | Jan 2014 | A1 |
| 20140013434 | Ranum et al. | Jan 2014 | A1 |
| 20140019936 | Cohanoff | Jan 2014 | A1 |
| 20140032506 | Hoey et al. | Jan 2014 | A1 |
| 20140033010 | Richardt et al. | Jan 2014 | A1 |
| 20140040371 | Gurevich et al. | Feb 2014 | A1 |
| 20140047319 | Eberlein | Feb 2014 | A1 |
| 20140047357 | Alfaro et al. | Feb 2014 | A1 |
| 20140058763 | Zizzamia et al. | Feb 2014 | A1 |
| 20140059038 | McPherson et al. | Feb 2014 | A1 |
| 20140059683 | Ashley | Feb 2014 | A1 |
| 20140067611 | Adachi et al. | Mar 2014 | A1 |
| 20140068487 | Steiger et al. | Mar 2014 | A1 |
| 20140074855 | Zhao et al. | Mar 2014 | A1 |
| 20140081652 | Klindworth | Mar 2014 | A1 |
| 20140081685 | Thacker et al. | Mar 2014 | A1 |
| 20140095273 | Tang et al. | Apr 2014 | A1 |
| 20140095509 | Patton | Apr 2014 | A1 |
| 20140101124 | Scriffignano et al. | Apr 2014 | A1 |
| 20140108068 | Williams | Apr 2014 | A1 |
| 20140108380 | Gotz et al. | Apr 2014 | A1 |
| 20140108985 | Scott et al. | Apr 2014 | A1 |
| 20140123279 | Bishop et al. | May 2014 | A1 |
| 20140129261 | Bothwell et al. | May 2014 | A1 |
| 20140143009 | Brice et al. | May 2014 | A1 |
| 20140149130 | Getchius | May 2014 | A1 |
| 20140149272 | Hirani et al. | May 2014 | A1 |
| 20140149436 | Bahrami et al. | May 2014 | A1 |
| 20140156484 | Chan et al. | Jun 2014 | A1 |
| 20140156527 | Grigg et al. | Jun 2014 | A1 |
| 20140157172 | Peery et al. | Jun 2014 | A1 |
| 20140164502 | Khodorenko et al. | Jun 2014 | A1 |
| 20140189536 | Lange et al. | Jul 2014 | A1 |
| 20140195515 | Baker et al. | Jul 2014 | A1 |
| 20140195887 | Ellis et al. | Jul 2014 | A1 |
| 20140214579 | Shen et al. | Jul 2014 | A1 |
| 20140222521 | Chait | Aug 2014 | A1 |
| 20140244388 | Manouchehri et al. | Aug 2014 | A1 |
| 20140258246 | Lo Faro et al. | Sep 2014 | A1 |
| 20140267294 | Ma | Sep 2014 | A1 |
| 20140267295 | Sharma | Sep 2014 | A1 |
| 20140279824 | Tamayo | Sep 2014 | A1 |
| 20140304582 | Bills et al. | Oct 2014 | A1 |
| 20140310266 | Greenfield | Oct 2014 | A1 |
| 20140310282 | Sprague et al. | Oct 2014 | A1 |
| 20140316911 | Gross | Oct 2014 | A1 |
| 20140333651 | Cervelli et al. | Nov 2014 | A1 |
| 20140337772 | Cervelli et al. | Nov 2014 | A1 |
| 20140344230 | Krause et al. | Nov 2014 | A1 |
| 20140351070 | Christner et al. | Nov 2014 | A1 |
| 20140366132 | Stiansen et al. | Dec 2014 | A1 |
| 20150019394 | Unser et al. | Jan 2015 | A1 |
| 20150046870 | Goldenberg et al. | Feb 2015 | A1 |
| 20150073929 | Psota et al. | Mar 2015 | A1 |
| 20150089424 | Duffield et al. | Mar 2015 | A1 |
| 20150100897 | Sun et al. | Apr 2015 | A1 |
| 20150100907 | Erenrich et al. | Apr 2015 | A1 |
| 20150134633 | Colgrove et al. | May 2015 | A1 |
| 20150134666 | Gattiker et al. | May 2015 | A1 |
| 20150169709 | Kara et al. | Jun 2015 | A1 |
| 20150169726 | Kara et al. | Jun 2015 | A1 |
| 20150170077 | Kara et al. | Jun 2015 | A1 |
| 20150178825 | Huerta | Jun 2015 | A1 |
| 20150178877 | Bogomolov et al. | Jun 2015 | A1 |
| 20150186821 | Wang et al. | Jul 2015 | A1 |
| 20150187036 | Wang et al. | Jul 2015 | A1 |
| 20150227295 | Meiklejohn et al. | Aug 2015 | A1 |
| 20150235334 | Wang et al. | Aug 2015 | A1 |
| 20150242401 | Liu | Aug 2015 | A1 |
| 20150309719 | Ma et al. | Oct 2015 | A1 |
| 20150317342 | Grossman et al. | Nov 2015 | A1 |
| 20150324868 | Kaftan et al. | Nov 2015 | A1 |
| 20160004764 | Chakerian et al. | Jan 2016 | A1 |
| 20160006749 | Cohen et al. | Jan 2016 | A1 |
| 20160034470 | Sprague et al. | Feb 2016 | A1 |
| 20160048937 | Mathura et al. | Feb 2016 | A1 |
| 20160180451 | Visbal et al. | Jun 2016 | A1 |
| 20170132200 | Noland et al. | May 2017 | A1 |
| 20170244735 | Visbal et al. | Aug 2017 | A1 |
| 20170308402 | Bills et al. | Oct 2017 | A1 |
| 20180270264 | Cohen et al. | Sep 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 101729531 | Jun 2010 | CN |
| 103281301 | Sep 2013 | CN |
| 102014103482 | Sep 2014 | DE |
| 102014215621 | Feb 2015 | DE |
| 1191463 | Mar 2002 | EP |
| 1672527 | Jun 2006 | EP |
| 2551799 | Jan 2013 | EP |
| 2555153 | Feb 2013 | EP |
| 2560134 | Feb 2013 | EP |
| 2778977 | Sep 2014 | EP |
| 2778983 | Sep 2014 | EP |
| 2779082 | Sep 2014 | EP |
| 2835745 | Feb 2015 | EP |
| 2835770 | Feb 2015 | EP |
| 2838039 | Feb 2015 | EP |
| 2846241 | Mar 2015 | EP |
| 2851852 | Mar 2015 | EP |
| 2858014 | Apr 2015 | EP |
| 2858018 | Apr 2015 | EP |
| 2863326 | Apr 2015 | EP |
| 2863346 | Apr 2015 | EP |
| 2869211 | May 2015 | EP |
| 2881868 | Jun 2015 | EP |
| 2884439 | Jun 2015 | EP |
| 2884440 | Jun 2015 | EP |
| 2891992 | Jul 2015 | EP |
| 2911078 | Aug 2015 | EP |
| 2911100 | Aug 2015 | EP |
| 2940603 | Nov 2015 | EP |
| 2940609 | Nov 2015 | EP |
| 2963577 | Jan 2016 | EP |
| 2985729 | Feb 2016 | EP |
| 3037991 | Jun 2016 | EP |
| 2513247 | Oct 2014 | GB |
| 2516155 | Jan 2015 | GB |
| 2518745 | Apr 2015 | GB |
| 2012778 | Nov 2014 | NL |
| 2013306 | Feb 2015 | NL |
| 624557 | Dec 2014 | NZ |
| WO 2000009529 | Feb 2000 | WO |
| WO 2002065353 | Aug 2002 | WO |
| WO 2005104736 | Nov 2005 | WO |
| WO 2008011728 | Jan 2008 | WO |
| WO 2008064207 | May 2008 | WO |
| WO 2008113059 | Sep 2008 | WO |
| WO 2009061501 | May 2009 | WO |
| WO 2010000014 | Jan 2010 | WO |
| WO 2010030913 | Mar 2010 | WO |
| WO 2013010157 | Jan 2013 | WO |
| WO 2013102892 | Jul 2013 | WO |
| WO 2013126281 | Aug 2013 | WO |
| WO 2015047803 | Apr 2015 | WO |
| Entry |
|---|
| US 8,712,906 B1, 04/2014, Sprague et al. (withdrawn) |
| TR35: the young innovators with this year's best ideas, Technology Review (Cambridge, Mass.)109.4: 43(28). Technology Review, Inc. (Sep. 2006-Oct. 2006). |
| “A Word About Banks and the Laundering of Drug Money,” Aug. 18, 2012, http://www.golemxiv.co.uk/2012/08/a-word-about-banks-and-the-laundering-of-drug-money/. |
| “Potential Money Laundering Warning Signs,” snapshot taken 2003, https://web.archive/org/web/20030816090055/http:/finsolinc.com/ANTI-MONEY&20LAUNDRING%20TRAINING%20GUIDES.pdf. |
| Alfred, Rayner “Summarizing Relational Data Using Semi-Supervised Genetic Algorithm-Based Clustering Techniques”, Journal of Computer Science, 2010, vol. 6, No. 7, pp. 775-784. |
| Keylines.com, “An Introduction to KeyLines and Network Visualization,” Mar. 2014, <http://keylines.com/wp-content/uploads/2014/03/KeyLines-White-Paper.pdf> downloaded May 12, 2014 in 8 pages. |
| Keylines.com, “KeyLines Datasheet,” Mar. 2014, <http://keylines.com/wp-content/uploads/2014/03/KeyLines-datasheet.pdf> downloaded May 12, 2014 in 2 pages. |
| Keylines.com, “Visualizing Threats: Improved Cyber Security Through Network Visualization,” Apr. 2014, <http://keylines.com/wp-content/uploads/2014/04/Visualizing-Threats1.pdf> downloaded May 12, 2014 in 10 pages. |
| Shah, Chintan, “Periodic Connections to Control Server Offer New Way to Detect Botnets,” Oct. 24, 2013 in 6 pages, <http://www.blogs.mcafee.com/mcafee-labs/periodic-links-to-control-server-offer-new-way-to-detect-botnets>. |
| Wiggerts, T.A., “Using Clustering Algorithms in Legacy Systems Remodularization,” Reverse Engineering, Proceedings of the Fourth Working Conference, Netherlands, Oct. 6-8, 1997, IEEE Computer Soc., pp. 33-43. |
| Notice of Allowance for U.S. Appl. No. 14/139,628 dated Jun. 24, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/139,640 dated Jun. 17, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/139,713 dated Jun. 12, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/264,445 dated May 14, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/473,552 dated Jul. 24, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/473,860 dated Jan. 5, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/486,991 dated May 1, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/616,080 dated Apr. 2, 2015. |
| Official Communication for European Patent Application No. 14159535.5 dated May 22, 2014. |
| Official Communication for Great Britain Patent Application No. 1404457.2 dated Aug. 14, 2014. |
| Official Communication for New Zealand Patent Application No. 622181 dated Mar. 24, 2014. |
| Official Communication for U.S. Appl. No. 14/264,445 dated Apr. 17, 2015. |
| Official Communication for U.S. Appl. No. 14/278,963 dated Jan. 30, 2015. |
| Official Communication for U.S. Appl. No. 14/473,552 dated Feb. 24, 2015. |
| Official Communication for U.S. Appl. No. 14/486,991 dated Mar. 10, 2015. |
| Official Communication for U.S. Appl. No. 14/518,757 dated Apr. 2, 2015. |
| Official Communication for U.S. Appl. No. 14/518,757 dated Jul. 20, 2015. |
| Official Communication for U.S. Appl. No. 14/579,752 dated Aug. 19, 2015. |
| Official Communication for U.S. Appl. No. 14/579,752 dated May 26, 2015. |
| Official Communication for U.S. Appl. No. 14/639,606 dated May 18, 2015. |
| Official Communication for U.S. Appl. No. 14/639,606 dated Jul. 24, 2015. |
| “A First Look: Predicting Market Demand for Food Retail using a Huff Analysis,” TRF Policy Solutions, Jul. 2012, pp. 30. |
| “A Quick Guide to UniProtKB Swiss-Prot & TrEMBL,” Sep. 2011, pp. 2. |
| About 80 Minutes, “Palantir in a Number of Parts—Part 6—Graph,” Mar. 21, 2013, pp. 1-6. |
| Acklen, Laura, “Absolute Beginner's Guide to Microsoft Word 2003,” Dec. 24, 2003, pp. 15-18, 34-41, 308-316. |
| Alur et al., “Chapter 2: IBM InfoSphere DataStage Stages,” IBM InfoSphere DataStage Data Flow and Job Design, Jul. 1, 2008, pp. 35-137. |
| Amnet, “5 Great Tools for Visualizing Your Twitter Followers,” posted Aug. 4, 2010, http://www.amnetblog.com/component/content/article/115-5-grate-tools-for-visualizing-your-twitter-followers.html. |
| Ananiev et al., “The New Modality API,” http://web.archive.org/web/20061211011958/http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/modality/ Jan. 21, 2006, pp. 8. |
| Baker et al., “The Development of a Common Enumeration of Vulnerabilities and Exposures,” Presented at the Second International Workshop on Recent Advances in Intrusion Detection, Sep. 7-9, 1999, pp. 35. |
| Bluttman et al., “Excel Formulas and Functions for Dummies,” 2005, Wiley Publishing, Inc., pp. 280, 284-286. |
| Boyce, Jim, “Microsoft Outlook 2010 Inside Out,” Aug. 1, 2010, retrieved from the internet https://capdtron.files.wordpress.com/2013/01/outlook-2010-inside_out.pdf. |
| Bugzilla@Mozilla, “Bug 18726-[feature] Long-click means of invoking contextual menus not supported,” http://bugzilla.mozilla.org/show_bug.cgi?id=18726 printed Jun. 13, 2013 in 11 pages. |
| Canese et al., “Chapter 2: PubMed: The Bibliographic Database,” The NCBI Handbook, Oct. 2002, pp. 1-10. |
| Celik, Tantek, “CSS Basic User Interface Module Level 3 (CSS3 UI),” Section 8 Resizing and Overflow, Jan. 17, 2012, retrieved from internet http://www.w3.org/TR/2012/WD-css3-ui-20120117/#resizing-amp-overflow retrieved on May 18, 2015. |
| Chen et al., “Bringing Order to the Web: Automatically Categorizing Search Results,” CHI 2000, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, Apr. 1-6, 2000, The Hague, The Netherlands, pp. 145-152. |
| Chung, Chin-Wan, “Dataplex: An Access to Heterogeneous Distributed Databases,” Communications of the ACM, Association for Computing Machinery, Inc., vol. 33, No. 1, Jan. 1, 1990, pp. 70-80. |
| Conner, Nancy, “Google Apps: The Missing Manual,” May 1, 2008, pp. 15. |
| Crosby et al., “Efficient Data Structures for Tamper-Evident Logging,” Department of Computer Science, Rice University, 2009, pp. 17. |
| Definition “Identify”, downloaded Jan. 22, 2015, 1 page. |
| Definition “Overlay”, downloaded Jan. 22, 2015, 1 page. |
| Delcher et al., “Identifying Bacterial Genes and Endosymbiont DNA with Glimmer,” BioInformatics, vol. 23, No. 6, 2007, pp. 673-679. |
| Dramowicz, Ela, “Retail Trade Area Analysis Using the Huff Model,” Directions Magazine, Jul. 2, 2005 in 10 pages, http://www.directionsmag.com/articles/retail-trade-area-analysis-using-the-huff-model/123411. |
| “The FASTA Program Package,” fasta-36.3.4, Mar. 25, 2011, pp. 29. |
| FireEye—Products and Solutions Overview, <http://www.fireeye.com/products-and-solutions> Printed Jun. 30, 2014 in 3 pages. |
| FireEye, <http://www.fireeye.com/> Printed Jun. 30, 2014 in 2 pages. |
| Gesher, Ari, “Palantir Screenshots in the Wild: Swing Sightings,” The Palantir Blog, Sep. 11, 2007, pp. 1-12. |
| GIS-NET 3 Public _ Department of Regional Planning. Planning & Zoning Information for Unincorporated LA County. Retrieved Oct. 2, 2013 from http://gis.planning.lacounty.gov/GIS-NET3Public/Viewer.html. |
| Glaab et al., “EnrichNet: Network-Based Gene Set Enrichment Analysis,” Bioinformatics 28.18 (2012): pp. i451-i457. |
| Goswami, Gautam, “Quite Writly Said!,” One Brick at a Time, Aug. 21, 2005, pp. 7. |
| Griffith, Daniel A., “A Generalized Huff Model,” Geographical Analysis, Apr. 1982, vol. 14, No. 2, pp. 135-144. |
| Hansen et al., “Analyzing Social Media Networks with NodeXL: Insights from a Connected World”, Chapter 4, pp. 53-67 and Chapter 10, pp. 143-164, published Sep. 2010. |
| Hardesty, “Privacy Challenges: Analysis: It's Surprisingly Easy to Identify Individuals from Credit-Card Metadata,” MIT News on Campus and Around the World, MIT News Office, Jan. 29, 2015, 3 pages. |
| Hibbert et al., “Prediction of Shopping Behavior Using a Huff Model Within a GIS Framework,” Healthy Eating in Context, Mar. 18, 2011, pp. 16. |
| Hogue et al., “Thresher: Automating the Unwrapping of Semantic Content from the World Wide Web,” 14th International Conference on World Wide Web, WWW 2005: Chiba, Japan, May 10-14, 2005, pp. 86-95. |
| Huang et al., “Systematic and Integrative Analysis of Large Gene Lists Using DAVID Bioinformatics Resources,” Nature Protocols, 4.1, 2008, 44-57. |
| Huff et al., “Calibrating the Huff Model Using ArcGIS Business Analyst,” ESRI, Sep. 2008, pp. 33. |
| Huff, David L., “Parameter Estimation in the Huff Model,” ESRI, ArcUser, Oct.-Dec. 2003, pp. 34-36. |
| Hur et al., “SciMiner: web-based literature mining tool for target identification and functional enrichment analysis,” Bioinformatics 25.6 (2009): pp. 838-840. |
| IBM, “Determining Business Object Structure,” IBM, 2004, 9 pages. |
| Kahan et al., “Annotea: an Open RDF Infrastructure for Shared Web Annotations”, Computer Networks, Elsevier Science Publishers B.V., vol. 39, No. 5, dated Aug. 5, 2002, pp. 589-608. |
| Kitts, Paul, “Chapter 14: Genome Assembly and Annotation Process,” The NCBI Handbook, Oct. 2002, pp. 1-21. |
| Lee et al., “A Data Mining and CIDF Based Approach for Detecting Novel and Distributed Intrusions,” Lecture Notes in Computer Science, vol. 1907 Nov. 11, 2000, pp. 49-65. |
| Li et al., “Interactive Multimodal Visual Search on Mobile Device,” IEEE Transactions on Multimedia, vol. 15, No. 3, Apr. 1, 2013, pp. 594-607. |
| Li et al., “Identifying the Signs of Fraudulent Accounts using Data Mining Techniques,” Computers in Human Behavior, vol. 28, No. 3, Jan. 16, 2012. |
| Liu, Tianshun, “Combining GIS and the Huff Model to Analyze Suitable Locations for a New Asian Supermarket in the Minneapolis and St. Paul, Minnesota USA,” Papers in Resource Analysis, 2012, vol. 14, pp. 8. |
| Ma et al., “A New Approach to Secure Logging,” ACM Transactions on Storage, vol. 5, No. 1, Article 2, Published Mar. 2009, 21 pages. |
| Madden, Tom, “Chapter 16: The BLAST Sequence Analysis Tool,” The NCBI Handbook, Oct. 2002, pp. 1-15. |
| Manno et al., “Introducing Collaboration in Single-user Applications through the Centralized Control Architecture,” 2010, pp. 10. |
| Manske, “File Saving Dialogs,” <http://www.mozilla.org/editor/ui_specs/FileSaveDialogs.html>, Jan. 20, 1999, pp. 7. |
| Map of San Jose, CA. Retrieved Oct. 2, 2013 from http://maps.yahoo.com. |
| Map of San Jose, CA. Retrieved Oct. 2, 2013 from http://maps.bing.com. |
| Map of San Jose, CA. Retrieved Oct. 2, 2013 from http://maps.google.com. |
| Microsoft—Developer Network, “Getting Started with VBA in Word 2010,” Apr. 2010, <http://msdn.microsoft.com/en-us/library/ff604039%28v=office.14%29.aspx> as printed Apr. 4, 2014 in 17 pages. |
| Microsoft Office—Visio, “About connecting shapes,” <http://office.microsoft.com/en-us/visio-help/about-connecting-shapes-HP085050369.aspx> printed Aug. 4, 2011 in 6 pages. |
| Microsoft Office—Visio, “Add and glue connectors with the Connector tool,” <http://office.microsoft.com/en-us/visio-help/add-and-glue-connectors-with-the-connector-tool-HA010048532.aspx?CTT=1> printed Aug. 4, 2011 in 1 page. |
| Mizrachi, Ilene, “Chapter 1: GenBank: The Nuckeotide Sequence Database,” The NCBI Handbook, Oct. 2002, pp. 1-14. |
| “Money Laundering Risks and E-Gaming: A European Overview and Assessment,” 2009, http://www.cf.ac.uk/socsi/resources/Levi_Final_Money_Laundering_Risks_egaming.pdf. |
| Ngai et al., “The Application of Data Mining Techniques in Financial Fraud Detection: A Classification Frameworok and an Academic Review of Literature,” Decision Support Systems, Elsevier Science Publishers, Amsterdam, Netherlands, vol. 50, No. 3, Feb. 1, 2011. |
| Nierman, “Evaluating Structural Similarity in XML Documents”, 6 pages, 2002. |
| Nolan et al., “MCARTA: A Malicious Code Automated Run-Time Analysis Framework,” Homeland Security (HST) 2012 IEEE Conference on Technologies for, Nov. 13, 2012, pp. 13-17. |
| Olanoff, Drew, “Deep Dive with the New Google Maps for Desktop with Google Earth Integration, It's More than Just a Utility,” May 15, 2013, pp. 1-6, retrieved from the internet: http://web.archive.org/web/20130515230641/http://techcrunch.com/2013/05/15/deep-dive-with-the-new-google-maps-for-desktop-with-google-earth-integration-its-more-than-just-a-utility/. |
| Palantir Technologies, “Palantir Labs—Timeline,” Oct. 1, 2010, retrieved from the internet https://www.youtube.com/watch?v=JCgDW5bru9M. |
| Palmas et al., “An Edge-Bunding Layout for Interactive Parallel Coordinates” 2014 IEEE Pacific Visualization Symposium, pp. 57-64. |
| Perdisci et al., “Behavioral Clustering of HTTP-Based Malware and Signature Generation Using Malicious Network Traces,” USENIX, Mar. 18, 2010, pp. 1-14. |
| Quest, “Toad for ORACLE 11.6—Guide to Using Toad,” Sep. 24, 2012, pp. 1-162. |
| “Refresh CSS Ellipsis When Resizing Container—Stack Overflow,” Jul. 31, 2013, retrieved from internet http://stackoverflow.com/questions/17964681/refresh-css-ellipsis-when-resizing-container, retrieved on May 18, 2015. |
| Rouse, Margaret, “OLAP Cube,” <http://searchdatamanagement.techtarget.com/definition/OLAP-cube>, Apr. 28, 2012, pp. 16. |
| Schneier et al., “Automatic Event Stream Notarization Using Digital Signatures,” Security Protocols, International Workshop Apr. 1996 Proceedings, Springer-Veriag, 1997, pp. 155-169, https://schneier.com/paper-event-stream.pdf. |
| Schneier et al., “Cryptographic Support for Secure Logs on Untrusted Machines,” The Seventh USENIX Security Symposium Proceedings, USENIX Press, Jan. 1998, pp. 53-62, https://www.schneier.com/paper-secure-logs.pdf. |
| Shi et al., “A Scalable Implementation of Malware Detection Based on Network Connection Behaviors,” 2013 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, IEEE, Oct. 10, 2013, pp. 59-66. |
| Sigrist, et al., “PROSITE, a Protein Domain Database for Functional Characterization and Annotation,” Nucleic Acids Research, 2010, vol. 38, pp. D161-D166. |
| Sirotkin et al., “Chapter 13: The Processing of Biological Sequence Data at NCBI,” The NCBI Handbook, Oct. 2002, pp. 1-11. |
| Symantec Corporation, “E-Security Begins with Sound Security Policies,” Announcement Symantec, Jun. 14, 2001. |
| Thompson, Mick, “Getting Started with GEO,” Getting Started with GEO, Jul. 26, 2011. |
| Umagandhi et al., “Search Query Recommendations Using Hybrid User Profile with Query Logs,” International Journal of Computer Applications, vol. 80, No. 10, Oct. 1, 2013, pp. 7-18. |
| “Using Whois Based Geolocation and Google Maps API for Support Cybercrime Investigations,” http://wseas.us/e-library/conferences/2013/Dubrovnik/TELECIRC/TELECIRC-32.pdf. |
| VirusTotal—About, <http://www.virustotal.com/en/about/> Printed Jun. 30, 2014 in 8 pages. |
| Waters et al., “Building an Encrypted and Searchable Audit Log,” Published Jan. 9, 2004, 11 pages, http://www.parc.com/content/attachments/building_encrypted_searchable_5059_parc.pdf. |
| Wikipedia, “Federated Database System,” Sep. 7, 2013, retrieved from the internet on Jan. 27, 2015 http://en.wikipedia.org/w/index.php?title-Federated_database_system&oldid=571954221. |
| Wright et al., “Palantir Technologies VAST 2010 Challenge Text Records _ Investigations into Arms Dealing,” Oct. 29, 2010, pp. 1-10. |
| Yang et al., “HTML Page Analysis Based on Visual Cues”, A129, pp. 859-864, 2001. |
| Zheng et al., “GOEAST: a web-based software toolkit for Gene Ontology enrichment analysis,” Nucleic acids research 36.suppl 2 (2008): pp. W385-W363. |
| International Search Report and Written Opinion in Application No. PCT/US2009/056703 dated Mar. 15, 2010. |
| Notice of Acceptance for Australian Patent Application No. 2014250678 dated Oct. 7, 2015. |
| Notice of Allowance for U.S. Appl. No. 12/556,318 dated Nov. 2, 2015. |
| Notice of Allowance for U.S. Appl. No. 13/247,987 dated Mar. 17, 2016. |
| Notice of Allowance for U.S. Appl. No. 14/102,394 dated Aug. 25, 2014. |
| Notice of Allowance for U.S. Appl. No. 14/108,187 dated Aug. 29, 2014. |
| Notice of Allowance for U.S. Appl. No. 14/135,289 dated Oct. 14, 2014. |
| Notice of Allowance for U.S. Appl. No. 14/148,568 dated Aug. 26, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/192,767 dated Dec. 16, 2014. |
| Notice of Allowance for U.S. Appl. No. 14/192,767 dated Apr. 20, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/225,084 dated May 4, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/268,964 dated Dec. 3, 2014. |
| Notice of Allowance for U.S. Appl. No. 14/278,963 dated Sep. 2, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/294,098 dated Dec. 29, 2014. |
| Notice of Allowance for U.S. Appl. No. 14/323,935 dated Oct. 1, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/326,738 dated Nov. 18, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/473,860 dated Feb. 27, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/504,103 dated May 18, 2015. |
| Notice of Allowance for U.S. Appl. No. 14/579,752 dated Apr. 4, 2016. |
| Notice of Allowance for U.S. Appl. No. 15/151,904 dated Oct. 24, 2016. |
| Official Communication for Australian Patent Application No. 2014201511 dated Feb. 27, 2015. |
| Official Communication for Australian Patent Application No. 2014202442 dated Mar. 19, 2015. |
| Official Communication for Australian Patent Application No. 2014210604 dated Jun. 5, 2015. |
| Official Communication for Australian Patent Application No. 2014210614 dated Jun. 5, 2015. |
| Official Communication for Australian Patent Application No. 2014213553 dated May 7, 2015. |
| Official Communication for Australian Patent Application No. 2014250678 dated Jun. 17, 2015. |
| Official Communication for European Patent Application No. 14158861.6 dated Jun. 16, 2014. |
| Official Communication for European Patent Application No. 14159464.8 dated Jul. 31, 2014. |
| Official Communication for European Patent Application No. 14180142.3 dated Feb. 6, 2015. |
| Official Communication for European Patent Application No. 14180281.9 dated Jan. 26, 2015. |
| Official Communication for European Patent Application No. 14180321.3 dated Apr. 17, 2015. |
| Official Communication for European Patent Application No. 14180432.8 dated Jun. 23, 2015. |
| Official Communication for European Patent Application No. 14186225.0 dated Feb. 13, 2015. |
| Official Communication for European Patent Application No. 14187739.9 dated Jul. 6, 2015. |
| Official Communication for European Patent Application No. 14187996.5 dated Feb. 12, 2015. |
| Official Communication for European Patent Application No. 14187996.5 dated Feb. 19, 2016. |
| Official Communication for European Patent Application No. 14189344.6 dated Feb. 20, 2015. |
| Official Communication for European Patent Application No. 14189344.6 dated Feb. 29, 2016. |
| Official Communication for European Patent Application No. 14189347.9 dated Mar. 4, 2015. |
| Official Communication for European Patent Application No. 14189802.3 dated May 11, 2015. |
| Official Communication for European Patent Application No. 14191540.5 dated May 27, 2015. |
| Official Communication for European Patent Application No. 14197879.1 dated Apr. 28, 2015. |
| Official Communication for European Patent Application No. 14197895.7 dated Apr. 28, 2015. |
| Official Communication for European Patent Application No. 14197938.5 dated Apr. 28, 2015. |
| Official Communication for European Patent Application No. 14199182.8 dated Mar. 13, 2015. |
| Official Communication for European Patent Application No. 15155845.9 dated Oct. 6, 2015. |
| Official Communication for European Patent Application No. 15155846.7 dated Jul. 8, 2015. |
| Official Communication for European Patent Application No. 15156004.2 dated Aug. 24, 2015. |
| Official Communication for European Patent Application No. 15165244.3 dated Aug. 27, 2015. |
| Official Communication for European Patent Application No. 15175106.2 dated Nov. 5, 2015. |
| Official Communication for European Patent Application No. 15175151.8 dated Nov. 25, 2015. |
| Official Communication for European Patent Application No. 15180515.7 dated Dec. 14, 2015. |
| Official Communication for European Patent Application No. 15183721.8 dated Nov. 23, 2015. |
| Official Communication for European Patent Application No. 15201727.3 dated May 23, 2016. |
| Official Communication for European Patent Application No. 15201727.3 dated Sep. 27, 2017. |
| Official Communication for Great Britain Patent Application No. 1404574.4 dated Dec. 18, 2014. |
| Official Communication for Great Britain Patent Application No. 1408025.3 dated Nov. 6, 2014. |
| Official Communication for Great Britain Patent Application No. 1411984.6 dated Dec. 22, 2014. |
| Official Communication for Great Britain Patent Application No. 1413935.6 dated Jan. 27, 2015. |
| Official Communication for Netherlands Patent Application No. 2012433 dated Mar. 11, 2016. |
| Official Communication for Netherlands Patent Application No. 2012437 dated Sep. 18, 2015. |
| Official Communication for Netherlands Patent Application No. 2013306 dated Apr. 24, 2015. |
| Official Communication for New Zealand Patent Application No. 622513 dated Apr. 3, 2014. |
| Official Communication for New Zealand Patent Application No. 622517 dated Apr. 3, 2014. |
| Official Communication for New Zealand Patent Application No. 624557 dated May 14, 2014. |
| Official Communication for New Zealand Patent Application No. 627962 dated Aug. 5, 2014. |
| Official Communication for New Zealand Patent Application No. 628161 dated Aug. 25, 2014. |
| Official Communication for New Zealand Patent Application No. 628263 dated Aug. 12, 2014. |
| Official Communication for New Zealand Patent Application No. 628495 dated Aug. 19, 2014. |
| Official Communication for New Zealand Patent Application No. 628585 dated Aug. 26, 2014. |
| Official Communication for New Zealand Patent Application No. 628840 dated Aug. 28, 2014. |
| Official Communication for U.S. Appl. No. 12/556,318 dated Jul. 2, 2015. |
| Official Communication for U.S. Appl. No. 13/247,987 dated Apr. 2, 2015. |
| Official Communication for U.S. Appl. No. 13/247,987 dated Sep. 22, 2015. |
| Official Communication for U.S. Appl. No. 13/831,791 dated Feb. 11, 2016. |
| Official Communication for U.S. Appl. No. 13/831,791 dated Mar. 4, 2015. |
| Official Communication for U.S. Appl. No. 13/831,791 dated Aug. 6, 2015. |
| Official Communication for U.S. Appl. No. 13/835,688 dated Jun. 17, 2015. |
| Official Communication for U.S. Appl. No. 13/835,688 dated Sep. 30, 2015. |
| Official Communication for U.S. Appl. No. 13/839,026 dated Aug. 4, 2015. |
| Official Communication for U.S. Appl. No. 14/134,558 dated Oct. 7, 2015. |
| Official Communication for U.S. Appl. No. 14/139,628 dated Jan. 5, 2015. |
| Official Communication for U.S. Appl. No. 14/139,640 dated Dec. 15, 2014. |
| Official Communication for U.S. Appl. No. 14/139,713 dated Dec. 15, 2014. |
| Official Communication for U.S. Appl. No. 14/148,568 dated Oct. 22, 2014. |
| Official Communication for U.S. Appl. No. 14/148,568 dated Mar. 26, 2015. |
| Official Communication for U.S. Appl. No. 14/196,814 dated May 5, 2015. |
| Official Communication for U.S. Appl. No. 14/196,814 dated Oct. 7, 2015. |
| Official Communication for U.S. Appl. No. 14/223,918 dated Jun. 8, 2015. |
| Official Communication for U.S. Appl. No. 14/225,006 dated Sep. 10, 2014. |
| Official Communication for U.S. Appl. No. 14/225,006 dated Sep. 2, 2015. |
| Official Communication for U.S. Appl. No. 14/225,006 dated Dec. 21, 2015. |
| Official Communication for U.S. Appl. No. 14/225,006 dated Feb. 27, 2015. |
| Official Communication for U.S. Appl. No. 14/225,084 dated Sep. 11, 2015. |
| Official Communication for U.S. Appl. No. 14/225,084 dated Sep. 2, 2014. |
| Official Communication for U.S. Appl. No. 14/225,084 dated Feb. 20, 2015. |
| Official Communication for U.S. Appl. No. 14/225,084 dated Feb. 26, 2016. |
| Official Communication for U.S. Appl. No. 14/225,084 dated Jan. 4, 2016. |
| Official Communication for U.S. Appl. No. 14/225,160 dated Feb. 11, 2015. |
| Official Communication for U.S. Appl. No. 14/225,160 dated Aug. 12, 2015. |
| Official Communication for U.S. Appl. No. 14/225,160 dated May 20, 2015. |
| Official Communication for U.S. Appl. No. 14/225,160 dated Oct. 22, 2014. |
| Official Communication for U.S. Appl. No. 14/225,160 dated Jan. 25, 2016. |
| Official Communication for U.S. Appl. No. 14/225,160 dated Jul. 29, 2014. |
| Official Communication for U.S. Appl. No. 14/251,485 dated Oct. 1, 2015. |
| Official Communication for U.S. Appl. No. 14/268,964 dated Sep. 3, 2014. |
| Official Communication for U.S. Appl. No. 14/280,490 dated Jul. 24, 2014. |
| Official Communication for U.S. Appl. No. 14/289,596 dated Jul. 18, 2014. |
| Official Communication for U.S. Appl. No. 14/289,596 dated Jan. 26, 2015. |
| Official Communication for U.S. Appl. No. 14/289,596 dated Apr. 30, 2015. |
| Official Communication for U.S. Appl. No. 14/289,599 dated Jul. 22, 2014. |
| Official Communication for U.S. Appl. No. 14/289,599 dated May 29, 2015. |
| Official Communication for U.S. Appl. No. 14/289,599 dated Sep. 4, 2015. |
| Official Communication for U.S. Appl. No. 14/294,098 dated Aug. 15, 2014. |
| Official Communication for U.S. Appl. No. 14/294,098 dated Nov. 6, 2014. |
| Official Communication for U.S. Appl. No. 14/306,138 dated Sep. 14, 2015. |
| Official Communication for U.S. Appl. No. 14/306,138 dated Mar. 17, 2016. |
| Official Communication for U.S. Appl. No. 14/306,138 dated Feb. 18, 2015. |
| Official Communication for U.S. Appl. No. 14/306,138 dated Sep. 23, 2014. |
| Official Communication for U.S. Appl. No. 14/306,138 dated Dec. 24, 2015. |
| Official Communication for U.S. Appl. No. 14/306,138 dated May 26, 2015. |
| Official Communication for U.S. Appl. No. 14/306,138 dated Dec. 3, 2015. |
| Official Communication for U.S. Appl. No. 14/306,147 dated Feb. 19, 2015. |
| Official Communication for U.S. Appl. No. 14/306,147 dated Dec. 24, 2015. |
| Official Communication for U.S. Appl. No. 14/306,147 dated Aug. 7, 2015. |
| Official Communication for U.S. Appl. No. 14/306,147 dated Sep. 9, 2014. |
| Official Communication for U.S. Appl. No. 14/306,154 dated Feb. 1, 2016. |
| Official Communication for U.S. Appl. No. 14/306,154 dated Mar. 11, 2015. |
| Official Communication for U.S. Appl. No. 14/306,154 dated May 15, 2015. |
| Official Communication for U.S. Appl. No. 14/306,154 dated Nov. 16, 2015. |
| Official Communication for U.S. Appl. No. 14/306,154 dated Mar. 17, 2016. |
| Official Communication for U.S. Appl. No. 14/306,154 dated Jul. 6, 2015. |
| Official Communication for U.S. Appl. No. 14/306,154 dated Sep. 9, 2014. |
| Official Communication for U.S. Appl. No. 14/319,765 dated Feb. 1, 2016. |
| Official Communication for U.S. Appl. No. 14/319,765 dated Sep. 10, 2015. |
| Official Communication for U.S. Appl. No. 14/319,765 dated Jun. 16, 2015. |
| Official Communication for U.S. Appl. No. 14/319,765 dated Nov. 25, 2014. |
| Official Communication for U.S. Appl. No. 14/319,765 dated Feb. 4, 2015. |
| Official Communication for U.S. Appl. No. 14/323,935 dated Jun. 22, 2015. |
| Official Communication for U.S. Appl. No. 14/323,935 dated Nov. 28, 2014. |
| Official Communication for U.S. Appl. No. 14/323,935 dated Mar. 31, 2015. |
| Official Communication for U.S. Appl. No. 14/326,738 dated Dec. 2, 2014. |
| Official Communication for U.S. Appl. No. 14/326,738 dated Jul. 31, 2015. |
| Official Communication for U.S. Appl. No. 14/326,738 dated Mar. 31, 2015. |
| Official Communication for U.S. Appl. No. 14/473,860 dated Nov. 4, 2014. |
| Official Communication for U.S. Appl. No. 14/473,920 dated Aug. 10, 2017. |
| Official Communication for U.S. Appl. No. 14/473,920 dated Nov. 14, 2016. |
| Official Communication for U.S. Appl. No. 14/479,863 dated Dec. 26, 2014. |
| Official Communication for U.S. Appl. No. 14/487,021 dated Mar. 24, 2017. |
| Official Communication for U.S. Appl. No. 14/487,021 dated Jun. 30, 2017. |
| Official Communication for U.S. Appl. No. 14/490,612 dated Aug. 18, 2015. |
| Official Communication for U.S. Appl. No. 14/490,612 dated Jan. 27, 2015. |
| Official Communication for U.S. Appl. No. 14/490,612 dated Mar. 31, 2015. |
| Official Communication for U.S. Appl. No. 14/504,103 dated Mar. 31, 2015. |
| Official Communication for U.S. Appl. No. 14/504,103 dated Feb. 5, 2015. |
| Official Communication for U.S. Appl. No. 14/518,757 dated Dec. 1, 2015. |
| Official Communication for U.S. Appl. No. 14/579,752 dated Dec. 9, 2015. |
| Official Communication for U.S. Appl. No. 14/581,920 dated Mar. 1, 2016. |
| Official Communication for U.S. Appl. No. 14/631,633 dated Sep. 10, 2015. |
| Official Communication for U.S. Appl. No. 14/631,633 dated Feb. 3, 2016. |
| Official Communication for U.S. Appl. No. 14/639,606 dated Oct. 16, 2015. |
| Official Communication for U.S. Appl. No. 14/639,606 dated Apr. 5, 2016. |
| Official Communication for U.S. Appl. No. 14/726,353 dated Mar. 1, 2016. |
| Official Communication for U.S. Appl. No. 14/726,353 dated Sep. 10, 2015. |
| Official Communication for U.S. Appl. No. 14/813,749 dated Sep. 28, 2015. |
| Official Communication for U.S. Appl. No. 14/813,749 dated Apr. 8, 2016. |
| Official Communication for U.S. Appl. No. 14/857,071 dated Mar. 2, 2016. |
| Official Communication for U.S. Appl. No. 15/151,904 dated Jul. 29, 2016. |
| Restriction Requirement for U.S. Appl. No. 13/839,026 dated Apr. 2, 2015. |
| Restriction Requirement for U.S. Appl. No. 14/857,071 dated Dec. 11, 2015. |
| Arya et al., “A clustering based algorithm for network intrusion detection,” Oct. 2012, SIN '12: Proceedings of the Fifth International Conference on Security of Information and Networks, pp. 193-196. |
| Marschalek et al., “Classifying Malicious System Behavior Using Event Propagation Trees,” Dec. 2015, iiWAS '15, Proceedings of the 17th International Conference on Information Integration and Web-based Applications & Services, pp. 1-10. |
| Stack Overflow, How to use update trigger to update another table, May 2012, 2 pages. |
| Notice of Acceptance for Australian Patent Application No. 2014201599 dated Aug. 31, 2017. |
| Notice of Allowance for U.S. Appl. No. 14/473,920 dated Dec. 28, 2017. |
| Notice of Allowance for U.S. Appl. No. 14/487,021 dated Jan. 29, 2018. |
| Official Communication for European Patent Application No. 15175151.8 dated Jan. 3, 2018. |
| Official Communication for U.S. Appl. No. 14/948,936 dated Mar. 28, 2018. |
| Number | Date | Country | |
|---|---|---|---|
| 20190052648 A1 | Feb 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61800887 | Mar 2013 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14139640 | Dec 2013 | US |
| Child | 14928512 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13968265 | Aug 2013 | US |
| Child | 14139640 | US | |
| Parent | 13968213 | Aug 2013 | US |
| Child | 13968265 | US |
