The present disclosure relates generally to data processing, and particularly to methods and systems for keyword searching, i.e., detecting particular strings in input data.
Keyword searching techniques are used in a wide variety of applications. For example, in some applications, communication traffic is analyzed in an attempt to detect keywords that indicate traffic of interest. Some data security systems attempt to detect information that leaks from an organization network by detecting keywords in outgoing traffic. Intrusion detection systems sometimes identify illegitimate intrusion attempts by detecting keywords in traffic.
Various keyword searching techniques are known in the art. For example, Aho and Corasick describe an algorithm for locating occurrences of a finite number of keywords in a string of text, in “Efficient String Matching: An Aid to Bibliographic Search,” Communications of the ACM, volume 18, no. 6, June, 1975, pages 333-340, which is incorporated herein by reference. This technique is commonly known as the Aho-Corasick algorithm. As another example, Yu et al. describe a multiple-pattern matching scheme, which uses Ternary Content-Addressable Memory (TCAM), in “Gigabit Rate Packet Pattern-Matching using TCAM,” Proceedings of the 12th IEEE International Conference on Network Protocols (ICNP), Berlin, Germany, Oct. 5-8, 2004, pages 174-183, which is incorporated herein by reference.
There is provided, in accordance with some embodiments described herein, a method for searching input data. One or more newly-arrived keywords are added to a static dictionary, which is represented by a data structure stored in computer memory and needs to be recompiled in order to be updated, by recompiling the static dictionary. During the recompilation of the static dictionary, any keywords that arrive during the recompilation are added to both a first dynamic dictionary and a second dynamic dictionary, each of which is represented by a respective data structure stored in computer memory and does not need to be recompiled in order to be updated. Also during the recompilation of the static dictionary, input data is searched using both the first dynamic dictionary and the static dictionary. Following the recompilation, input data is searched using both the second dynamic dictionary and the static dictionary.
In some embodiments, the method further includes storing the respective data structures that represent the first and second dynamic dictionaries at least partly within a cache memory of a processor.
In some embodiments, storing the respective data structures at least partly within the cache memory of the processor includes:
In some embodiments, the method further includes, following the recompilation, rebuilding the first dynamic dictionary, such that the first dynamic dictionary does not include one or more keywords that the first dynamic dictionary previously included.
In some embodiments, the recompilation is a first recompilation, and the method further includes, following the first recompilation, adding the keywords that arrived during the first recompilation to the static dictionary, by performing a second recompilation of the static dictionary.
In some embodiments, the method further includes, during the second recompilation of the static dictionary:
In some embodiments, the method further includes adding the newly-arrived keywords to a keyword collection, and recompiling the static dictionary includes recompiling the static dictionary such that the static dictionary includes all keywords contained in the keyword collection at a start of the recompilation.
In some embodiments, the method further includes adding to the second dynamic dictionary, but not to the first dynamic dictionary, any keywords that arrive following the recompilation and prior to a subsequent recompilation.
In some embodiments, the method further includes repeatedly adding at least one respective newly-arrived keyword to the static dictionary, by repeatedly recompiling the static dictionary.
There is further provided, in accordance with some embodiments described herein, apparatus, including:
In some embodiments, the processors are further configured to store the respective data structures that represent the first and second dynamic dictionaries at least partly within a cache memory of one of the processors.
In some embodiments, the processors are configured to:
In some embodiments, the processors are further configured to, following the recompilation, rebuild the first dynamic dictionary, such that the first dynamic dictionary does not include one or more keywords that the first dynamic dictionary previously included.
In some embodiments, the recompilation is a first recompilation, and the processors are further configured to, following the first recompilation, add the keywords that arrived during the first recompilation to the static dictionary, by performing a second recompilation of the static dictionary.
In some embodiments, the processors are further configured to, during the second recompilation of the static dictionary:
In some embodiments, the processors are configured to add the newly-arrived keywords to a keyword collection, and to recompile the static dictionary such that the static dictionary includes all keywords contained in the keyword collection at a start of the recompilation.
In some embodiments, the processors are further configured to add to the second dynamic dictionary, but not to the first dynamic dictionary, any keywords that arrive following the recompilation and prior to a subsequent recompilation.
In some embodiments, the processors are configured to repeatedly add at least one respective newly-arrived keyword to the static dictionary, by repeatedly recompiling the static dictionary.
There is further provided, in accordance with some embodiments described herein, a computer software product including a tangible non-transitory computer-readable medium in which program instructions are stored. The instructions, when read by one or more processors, cause the processors to add one or more newly-arrived keywords to a static dictionary, which is represented by a data structure stored in computer memory and needs to be recompiled in order to be updated, by recompiling the static dictionary. During the recompilation of the static dictionary, the instructions cause the processors to (i) add to both a first dynamic dictionary and a second dynamic dictionary, each of which is represented by a respective data structure stored in computer memory and does not need to be recompiled in order to be updated, any keywords that arrive during the recompilation, and (ii) search input data using both the first dynamic dictionary and the static dictionary. Following the recompilation, the instructions cause the processors to search input data using both the second dynamic dictionary and the static dictionary.
In some embodiments, the instructions cause the processors to add to the second dynamic dictionary, but not to the first dynamic dictionary, any keywords that arrive following the recompilation and prior to a subsequent recompilation.
There is further provided, in accordance with some embodiments described herein, a method for maintaining and using a dynamic dictionary. Data structures that collectively represent at least one dynamic dictionary of keywords that does not need to be recompiled in order to be updated are stored in computer memory. The data structures include (i) a management automaton that includes a plurality of management nodes, and (ii) a runtime automaton that is derived from the management automaton and includes a plurality of runtime nodes. Input data is searched, using the runtime automaton.
In some embodiments, the runtime nodes are uniformly sized.
In some embodiments, storing the data structures includes storing data structures that collectively represent at least two dynamic dictionaries, each of the dynamic dictionaries including a management automaton and a runtime automaton.
In some embodiments, storing the data structures includes storing the data structures at least partly in a cache memory of a processor.
In some embodiments, storing the data structures at least partly in the cache memory of the processor includes storing the runtime automaton in the cache memory.
In some embodiments, storing the runtime automaton includes storing the runtime automaton in a contiguous portion of computer memory.
In some embodiments, the runtime automaton occupies less space in the computer memory than does the management automaton.
In some embodiments, the management automaton uses an alphabet of nibbles, such that each of the management nodes corresponds to a sequence of nibbles that is a portion of at least one of the keywords.
In some embodiments, at least one of the management nodes corresponds to a sequence of nibbles consisting of an odd number of nibbles.
In some embodiments, the method further includes, upon receiving a new keyword that is not included in the dictionary:
In some embodiments, updating the management automaton to include the new keyword includes adding one or more new management nodes to the management automaton, each of the new management nodes corresponding to at least a portion of the new keyword.
In some embodiments, the method further includes, for at least one of the new management nodes:
In some embodiments, the ascertaining includes ascertaining that the portion of the new keyword corresponded to by the new management node differs by an appendage of exactly one nibble from the sequence of symbols corresponded to by the other one of the management nodes.
In some embodiments, the method further includes storing, in the new management node, a parent pointer to the parent node.
In some embodiments, the method further includes:
In some embodiments, the method further includes:
In some embodiments, identifying the other one of the management nodes as the fallback node of the selected management node includes identifying the other one of the management nodes as the fallback node of the selected management node by following a shortcut pointer that is stored in the parent node and corresponds to the appended symbol.
In some embodiments, storing the child pointer to the new management node in the parent node includes replacing the shortcut pointer with the child pointer to the new management node.
In some embodiments, the method further includes storing, in the fallback node, a friend pointer to the selected management node.
In some embodiments, the method further includes:
In some embodiments, the method further includes storing, in one or more other management nodes, respective fallback pointers that point to a management node selected from the new management nodes, indicating that the selected management node corresponds to a largest suffix of respective sequences of symbols corresponded to by the other management nodes.
In some embodiments, storing the respective fallback pointers in the other management nodes includes:
In some embodiments, the method further includes storing, in one or more other management nodes, respective shortcut pointers that point to the new management node, indicating that the parent node corresponds to a largest suffix of respective sequences corresponded to by the other management nodes.
In some embodiments, storing the respective shortcut pointers in the other management nodes includes:
In some embodiments, updating the runtime automaton to include the new keyword includes updating the runtime automaton to include the new keyword while using the runtime automaton to search the input data.
In some embodiments, updating the runtime automaton to include the new keyword includes adding one or more new runtime nodes to the runtime automaton, each of the new runtime nodes corresponding to at least a portion of the new keyword.
In some embodiments, each of the new runtime nodes is derived from a respective one of the management nodes.
In some embodiments, the runtime automaton uses an alphabet of symbols, and the method further includes, for each of the new runtime nodes:
In some embodiments, adding the one or more new runtime nodes to the runtime automaton includes:
In some embodiments, the reporting runtime node further stores an index for the new keyword.
In some embodiments, the reporting runtime node stores the index in a location corresponding to a location which, in the regular runtime node, stores a pointer to another runtime node.
In some embodiments, searching the input data includes:
There is further provided, in accordance with some embodiments described herein, apparatus including a network interface and one or more processors. The processors are configured to store, in computer memory, data structures that collectively represent at least one dynamic dictionary of keywords that does not need to be recompiled in order to be updated, the data structures including (i) a management automaton that includes a plurality of management nodes, and (ii) a runtime automaton that is derived from the management automaton and includes a plurality of runtime nodes. The processors are further configured to search input data received via the network interface, using the runtime automaton.
There is further provided, in accordance with some embodiments described herein, a computer software product including a tangible non-transitory computer-readable medium in which program instructions are stored. The instructions, when read by one or more processors, cause the processors to store, in computer memory, data structures that collectively represent at least one dynamic dictionary of keywords that does not need to be recompiled in order to be updated, the data structures including (i) a management automaton that includes a plurality of management nodes, and (ii) a runtime automaton that is derived from the management automaton and includes a plurality of runtime nodes. The instructions further cause the processors to search input data, using the runtime automaton.
The present disclosure will be more fully understood from the following detailed description of embodiments thereof, taken together with the drawings, in which:
Security personnel who monitor communication traffic over a network (e.g., the Internet) may wish to search the traffic for particular keywords that may be indicative of a possible security threat. A dictionary of keywords is typically used for such searches. Some dictionaries are “static dictionaries” that are stored in computer memory as compiled data structures. To add a new keyword to a static dictionary, it is necessary to recompile the dictionary.
While some keywords of interest (e.g., “Trojan horse”) may be known a priori, other keywords (e.g., a virus signature identified in a “zero-day attack”) may not be known a priori, and therefore, it may be necessary to add such keywords to the dictionary “on the fly.” A problem with such additions, in the case of a static dictionary, is that during the recompilation, the communication traffic cannot be searched for the new keyword. Furthermore, some keywords may become irrelevant by the time the recompilation has finished. For example, in some cases, a first subject may transfer a particular file to a server on the network, and subsequently provide a second subject with a Uniform Resource Locator (URL) that points to the file. If the URL is identified as a keyword of interest, it may be necessary to add the URL to the dictionary. However, by the time the recompilation of the dictionary is finished (and hence, the URL has been added to the dictionary), the URL may have become irrelevant.
To address the problems described above, embodiments described herein provide apparatus and techniques for constructing and utilizing a “dynamic dictionary” that is not a compiled dictionary, and therefore does not need to be recompiled in order to be updated. The dynamic dictionary includes respective data structures that represent (i) a management automaton that includes a plurality of management nodes, and (ii) a runtime automaton that is derived from the management automaton and includes a plurality of runtime nodes. The runtime automaton may be used to search input data, such as communication traffic over a network, for keywords of interest, while the management automaton manages the addition of keywords to the dynamic dictionary. Typically, at least two (e.g., exactly two) such dynamic dictionaries are used in combination with a static dictionary, as described in detail hereinbelow.
System 20 may be used, for example, in security-related monitoring, as described above. Alternatively, system 20 may be used in an application that detects data leakage from a communication network. In applications of this sort, the presence of one or more keywords in a data item indicates that this data item should not be allowed to exit the network. Alternatively, system 20 can be used in any other suitable application in which input data is searched for occurrences of keywords, such as in intrusion detection and prevention systems, detection of spam in electronic mail (e-mail) systems, or detection of inappropriate content using a dictionary of inappropriate words or phrases.
Although the embodiments described herein refer mainly to processing of communication traffic, the disclosed techniques can also be used in other domains. For example, system 20 can be used for locating data of interest on storage devices, such as in forensic disk scanning applications. Certain additional aspects of keyword searching are addressed, for example, in U.S. Pat. No. 9,053,211, entitled “Systems and methods for efficient keyword spotting in communication traffic,” which is assigned to the assignee of the present patent applications and whose disclosure is incorporated herein by reference.
Network 24 may comprise any suitable public or private, wireless or wire-line communication network, e.g., a Wide-Area network (WAN) such as the Internet, a Local-Area Network (LAN), a Metropolitan-Area Network (MAN), or a combination of network types. The communication traffic, to be used as input data by system 20, may be provided to the system using any suitable means. For example, the traffic may be forwarded to the system from a network element (e.g., router) in network 24, such as by port tapping or port mirroring. In alternative embodiments, system 20 may be placed in-line in the traffic path. These embodiments are suitable, for example, for data leakage prevention applications, but can also be used in other applications.
Typically, network 24 comprises an Internet Protocol (IP) network, and the communication traffic comprises IP packets. In general, however, the methods and systems described herein can be used with any suitable network and/or packet types. Regardless of protocol, the packets searched by system 20 are referred to herein generally as input data.
In the example of
In some embodiments, processor 44, and/or processor 25, is embodied as a cooperatively networked or clustered set of processors. Alternatively or additionally, the tasks executed by processor 44, and/or by processor 25, may be executed multithreadedly. For example, as further described hereinbelow, the searching of the input data may be multithreaded.
Many of the processor-performed functions described below may be performed by either one of the processors alone, or cooperatively by both processors. Hence, in many cases, the description below refers generally to “the processor,” without specifying either processor 44 or processor 25.
Runtime computer 23 comprises a Network Interface Card (NIC) 36, which receives TCP packets from network 24. NIC 36 stores the incoming TCP packets in a memory 40, typically comprising a Random Access Memory (RAM). Processor 44 searches the TCP packets stored in memory 40 and attempts to identify occurrences of predefined keywords in the packets.
The keywords that processor 44 attempts to find in the input data are included in a system of dictionaries that comprises (i) a static dictionary 34, which needs to be recompiled in order to be updated, and (ii) a first dynamic dictionary 22a and a second dynamic dictionary 22b, each of which is not a compiled dictionary, and therefore does not need to be recompiled in order to be updated. Respective data structures that represent static dictionary 34 and first and second dynamic dictionaries 22a and 22b are stored in the memory of system 20. Typically, each of the dynamic dictionaries is stored as a collection of data structures that includes a management automaton 26, and a runtime automaton 27 that is derived from management automaton 26. In particular, runtime automaton 27 is typically a “slimmed-down” version of management automaton 26, such that the runtime automaton occupies less space in memory than does the management automaton. Processor 44 typically uses the runtime automaton, but not the management automaton, to search the input data. Hence, the runtime automata are stored in the runtime computer, while the management automata are stored in the management computer. (The memory of the management computer is not explicitly shown in
Techniques for updating and using the automata in the dynamic dictionaries are described in detail hereinbelow. In some embodiments, to further increase the access speed to the dynamic dictionaries, the management automaton and/or runtime automaton is stored in a respective contiguous portion of memory.
In some embodiments, one or both of the runtime automata, which are typically relatively small, are stored in an internal cache memory 48 of processor 44, while the static dictionary, which is typically larger than the dynamic dictionaries, is stored in memory 40, which is external to the processor. Since, typically, cache memory 48 has a much faster access speed than memory 40, the storing of the dynamic dictionaries in the cache memory facilitates quick and efficient modifications to, and/or usage of, the dynamic dictionaries. As described below, typically only one of the dynamic dictionaries is used at a time for searching; consequently, in some embodiments, only the runtime automaton that is currently being used is stored in the cache, while the other runtime automaton is stored in memory 40.
In some embodiments, processor 44 comprises multi-level cache circuitry, and cache 48 is of an appropriate cache level.
When processor 44 detects a given keyword in a given packet, it reports the detection to user 28 using an output device of terminal 32, such as a display 56. For example, the processor may issue an alert to the user and/or present the data item (e.g., packet or session) in which the keyword or keywords were detected. Alternatively or additionally, processor 44 may take various kinds of actions in response to detecting keywords. For example, in a data leakage or intrusion prevention application, processor 44 may block some or all of the traffic upon detecting a keyword. In some embodiments, the reporting action or other action taken by the processor is also in response to other conditions being satisfied. For example, the processor may report the detection of a particular keyword only if the packet originates from a particular set of IP addresses, or only if a related keyword was spotted nearby.
User 28 may interact with system 20 using an input device of terminal 32, e.g., a keyboard 60. For example, the user may enter or modify the list of keywords stored in a keyword collection 30 of the management computer, which is used to update static dictionary 34.
The configuration shown in
Typically, keywords arrive at different respective times to processor 25. The keywords may be received, for example, from user 28, or from any other source, e.g., some other source of data communicating over network 24. Upon receiving one or more newly-arrived keywords that are not included in the system of dictionaries, the processor typically adds the newly-arrived keywords to keyword collection 30. Subsequently, processor 44 adds the newly-arrived keywords to the static dictionary by recompiling the static dictionary such that the static dictionary includes all of the keywords that are contained in the keyword collection at the start of the recompilation. (In other words, the static dictionary is effectively recompiled “from scratch” to include all the keywords that have been received thus far.)
In addition to adding the newly-arrived keywords to the static dictionary, the processors add each of the newly-arrived keywords to one or both of the dynamic dictionaries, as described in detail hereinbelow.
As described hereinabove, static dictionary 34 is stored in memory as a compiled data structure. For example, the static dictionary may be represented as a compiled Aho-Corasick automaton. The static dictionary typically allows for faster and more effective searching than do the dynamic dictionaries. For example, in some embodiments, the static dictionary is capable of handling wildcards, whereas the dynamic dictionaries are not. On the other hand, the dynamic dictionaries do not need to be recompiled in order to be updated, whereas the static dictionary does. Hence, each type of dictionary has its own particular advantages. Embodiments described herein synergistically use the static dictionary together with the dynamic dictionaries, such that the dynamic dictionaries complement the static dictionary, and vice versa.
First, some terminology will hereby be defined, and a general overview will hereby be provided.
(a) A dynamic dictionary is said to be “active” during the time period in which any new keywords that arrive are added to the dynamic dictionary, and the dynamic dictionary is used to search any input data that arrives.
(b) A dynamic dictionary is said to be on “standby” during the time period in which any new keywords that arrive are added to the dynamic dictionary, but the dynamic dictionary is not used to search input data.
(c) A dynamic dictionary is said to be “inactive” during the time period in which it is neither updated nor used for searching.
As further described hereinbelow, each of the dynamic dictionaries transitions cyclically through the active, inactive, and standby states. Upon transitioning to the standby state from the inactive state, a dynamic dictionary is typically treated as if it were empty, such that, effectively, each of the dynamic dictionaries is repeatedly rebuilt from scratch. Hence, neither the first nor second dynamic dictionary is allowed to grow too large, thus facilitating fast and efficient usage of and updates to the dictionaries.
In general, during recompilation of the static dictionary, one of the dynamic dictionaries is active, while the other dynamic dictionary is on standby; hence, the processor adds any keywords that arrive during the recompilation to both of the dynamic dictionaries. Following the recompilation, the active dynamic dictionary becomes inactive, while the standby dictionary become active.
An advantage of adding the newly-arrived keywords to the active dynamic dictionary is that the active dynamic dictionary may then be used to search the input data for the newly-arrived keywords. An advantage of adding the newly-arrived keywords to the standing-by dynamic dictionary is that, following the recompilation, the previously-standing-by (and now active) dynamic dictionary may be used to search for the newly-arrived keywords, which won't be included in the static dictionary until after the end of the next recompilation of the static dictionary.
Reference is now made to
At T0, the recompiling of static dictionary 34 begins. The static dictionary may be recompiled, for example, in order to add one or more newly-arrived keywords, which were received prior to T0, to the static dictionary. (As described above, the recompilation of the static dictionary may effectively replace the data structure currently stored in memory with a new data structure that includes all of the keywords received thus far.) The new keywords are not actually included in the static dictionary until T1, when the recompilation ends. Between T0 and T1, the static dictionary may be used to search for “old” keywords, but not for any of the new keywords that arrived subsequently to the previous compilation of the static dictionary, whether prior to T0, or between T0 and T1.
For example,
To compensate for the above-described issue, the processor adds KW0 and KW1, typically immediately upon receipt, to active first dynamic dictionary 22a, such that the first dynamic dictionary may then be used to search the input data for KW0 and/or KW1 prior to T1. The processor also adds KW1 to standing-by second dynamic dictionary 22b, such that, following T1, the second dynamic dictionary may be used to search for KW1. (There is no need to add KW0 to the second dynamic dictionary, since (i) prior to T1, the first dynamic dictionary may be used to search for KW0, and (ii) after T1, the static dictionary may be used to search to KW0.)
Between T0 and T1, during recompilation of the static dictionary, the active first dynamic dictionary and the static dictionary are both used to search the input data for keywords. The standing-by second dynamic dictionary, on the other hand, is not used for searching between T0 and T1, at least because the second dynamic dictionary does not include any keywords that arrived prior to T0 (e.g., KW0). At T1, the recompilation ends, and the first dynamic dictionary becomes inactive, while the second dynamic dictionary becomes active.
Following the recompilation, any keywords that arrived during the recompilation (e.g., KW1), and/or any keywords that arrive following the recompilation (e.g., keyword KW2, shown in
Between T1 and T2, any keywords that arrive are added to the active second dynamic dictionary, but not to the inactive first dynamic dictionary. For example,
At T2, the first dynamic dictionary transitions to the standby state, and is subsequently rebuilt, such that the first dynamic dictionary does not include one or more keywords that the first dynamic dictionary previously included. As noted above, typically, the first dynamic dictionary is rebuilt from scratch, such that the first dynamic dictionary includes only those keywords that arrive subsequently to the transition to the standby state. As noted above, an advantage of rebuilding the first dynamic dictionary is that the first dynamic dictionary is not allowed to grow too large. Furthermore, there is no disadvantage to “losing” the keywords previously included in the first dynamic dictionary, since (i) any keywords received prior to T0 are included in the static dictionary, while (ii) any keywords received between T0 and T1 are included in the second dynamic dictionary.
Between T1 and T3, the active second dynamic dictionary and the static dictionary are both used to search the input data. The inactive or standing-by first dynamic dictionary, on the other hand, is not used to search the input data.
Between T2 and T3, during the second recompilation, any keywords that arrive are added to the first dynamic dictionary and to the second dynamic dictionary. In other words, the second recompilation is treated like the first recompilation, insofar as newly-arrived keywords are added to both of the dynamic dictionaries. However, with regards to searching, the respective statuses of the dynamic dictionaries are interchanged, relative to the first recompilation. In other words, as noted above, during the second recompilation, the processor uses both the second dynamic dictionary and the static dictionary, but not the first dynamic dictionary, to search the input data.
Typically, as shown in
Typically, when using the static dictionary and one of the dynamic dictionaries to search the input data, the dynamic dictionary is “consulted” first. For example, if the input data includes the string “ABC,” processor 44 first determines whether “ABC” is included in the dynamic dictionary, and only subsequently determines whether “ABC” is included in the static dictionary. If, on the other hand, the processor were to consult the static dictionary first, a recognized keyword might not be spotted in the input data, despite the presence of the keyword in the input data. For example, the keyword “ABC” may have been added to first dynamic dictionary 22a only shortly before T1, such that, by the time the processor ascertains that “ABC” is not in the static dictionary, the first dynamic dictionary may already be inactive.
Typically, the processor consults the static dictionary even if a particular string was already determined to be included in the dynamic dictionary, since, in the static dictionary, the particular string may lead to other keywords that are not included in the dynamic dictionary. For example, even if the string “ABC” was already determined to be included in the dynamic dictionary, the processor typically also looks for “ABC” in the static dictionary, since, for example, the keyword “ABCD” may be included in the static dictionary, but not in the dynamic dictionary. Furthermore, the dictionaries may include the same keyword under different respective “contexts.” For example, the dynamic dictionary may store an index for the keyword “virus” that refers to a particular computer virus, while the static dictionary may store an index for the same keyword, referring to a particular pathological virus. The searching of both dictionaries for the string “virus” may thus allow the user to be informed of both possible contexts of the spotted keyword “virus.”
The description below relates to the structure of dynamic dictionaries 22a and 22b. As the two dynamic dictionaries typically use the same data structure, and are typically used and maintained in the same manner, references below to “a dynamic dictionary” or “the dynamic dictionary,” in the singular, should be construed as applying to both of the dynamic dictionaries.
Reference is now made to
As shown in
In the context of the claims and description of the present application, the term “symbol” is to be construed broadly, to include within its scope any meaningful unit of information. For example, as noted above, the “symbols” of the alphabet of
For simplicity, in the description below, characters may be referred to as “symbols.” For example, for simplicity, reference may be made to “symbol ‘C’,” rather than “symbol ‘01000011’.”
Typically, each management node further stores a parent pointer to its parent node. For example,
Each of the management nodes may also store a respective fallback pointer to another, “fallback” management node. (The fallback pointers are depicted in the present figures as curved, solid arrows, upon which no symbols are overlaid.) A node that stores a fallback pointer to the fallback node may be referred to as a “friend” of the fallback node, and may be said to “fall back to” the fallback node. The fallback node corresponds to the largest suffix of the sequence that the friend node corresponds to, relative to the other nodes in the automaton. For example, as shown in
Typically, each management node stores a respective friend pointer (depicted in the figures by a dotted-and-dashed arrow) to each of its friend nodes. For example,
Each of the management nodes typically also stores one or more shortcut pointers. Such shortcut pointers are depicted as dashed arrows in
Typically, each management node stores either a child pointer or a shortcut pointer for each of the symbols in the alphabet.
Runtime automaton 27a is derived from management automaton 26a. The runtime automaton includes a plurality of runtime nodes 64. The runtime automaton shown in
Each runtime node is derived from a respective management node, at least in that (i) the sequence to which the runtime node corresponds is the same as the sequence to which the management node corresponds, and/or (ii) each of the pointers stored in the runtime node is derived from a pointer stored in the management node. In the context of the claims and specification of the present application, a runtime pointer is considered to be derived from a management pointer if the runtime pointer points to a runtime node that is derived from the management node to which the management pointer points. For example, with reference to
(a) The runtime “A” node is derived from the management “A” node, in that (i) the runtime “A” node also corresponds to the sequence “A,” and (ii) each of the pointers stored in the runtime “A” node is derived from either a child pointer or shortcut pointer stored in the management “A” node. For example, the pointer from runtime node “A” corresponding to “C” is derived from the child pointer corresponding to “C” that is stored in management node “A.”
(b) The runtime node “CDA” is derived from the management node “CDA,” in that (i) the runtime node “CDA” also corresponds to the sequence “CDA,” and (ii) each of the pointers stored in the runtime “CDA” node is derived from either a child pointer or shortcut pointer stored in the management “CDA” node. For example, the pointer from runtime node “CDA” corresponding to “A” is derived from the shortcut pointer corresponding to “A” that is stored in management node “CDA.”
In general, it is advantageous to maintain both a management automaton and a runtime automaton in the dynamic dictionary, as described herein, rather than maintaining only a single automaton. One reason for this is that in order to update the dynamic dictionary (as described hereinbelow), it is typically necessary to maintain at least some of the “extra” pointers described above—namely, parent pointers, friend pointers, and/or fallback pointers. Hence, if a single automaton were used, the automaton might be too large to fit inside the cache memory of the processor. Conversely, limiting the number and/or type of extra pointers stored by the automaton might reduce the speed at which the automaton is updated.
In embodiments described herein, on the other hand, the management automaton maintains the extra pointers, while the runtime automaton maintains only the pointers that are needed for traversing the automaton. Hence, the management automaton may be used to facilitate quick updates to the dictionary, while the runtime automaton may be small enough to fit inside the cache memory. Furthermore, as further described hereinbelow with reference to
Notwithstanding the above, in some embodiments (e.g., for applications in which it is assumed that the number of keywords will be relatively small), the dynamic dictionary may include only a single automaton, similar or identical to the management automaton described herein.
As noted above, the runtime automaton is typically used for searching the input data. For example, it will be assumed that the input data consists of the input string “CDAC.” To search this string for the keywords shown in
In the embodiment shown in
Reference is now made to
(a) Each of the keyword runtime nodes in automaton 27a is replaced with (i) a respective reporting node 66, and (ii) a corresponding “regular” runtime node 68. (Reporting nodes are indicated in
(b) In automaton 27b, all of the runtime nodes—including the reporting nodes—have the same size. (In contrast, in the embodiment of
For example, Table 1 below shows an example layout of a portion of memory storing automaton 27b. Each particular node shown in
In some cases, e.g., if the size of the index exceeds the amount of memory that is allocated to each of the pointers, an index may be spread over multiple memory locations. For example, with reference to Table 1, part of the index may be stored in the “Pointer C” location, with the rest of the index stored in the “Pointer D” location.
The traversal of runtime automaton 27b proceeds until a reporting node is reached. At that point, the processor ascertains that at least two pointers stored in the reporting node are equivalent to one another, and do not point to the root node (and—in relevant embodiments—also do not point to the parity node, described hereinbelow with reference to
For example, to search the input string “CDAC,” the processor traverses runtime automaton 27b, beginning at the runtime root note. The first symbol in the input string is “C,” such that the pointer from the root to node “C” is followed. The second symbol in the input string is “D,” such that the pointer from node “C” to node “CD” is followed. The third symbol in the input string is “A,” such that the pointer from node “CD” to reporting node “CDA” is followed. At this point, the processor ascertains that node “CDA” is a reporting node, and in response thereto, reports the spotting of the keyword “CDA” in the input string. The processor then follows any of the pointers that point to regular node “CDA.” The fourth symbol in the input string is “C,” such that the processor next follows the pointer from regular node “CDA” to reporting node “AC.” At this point, the processor reports the spotting of the keyword “AC” in the input string.
Typically, runtime automaton 27b differs from the management node from which it is derived only in that runtime automaton 27b does not include any of the extra pointers described above. Thus, typically, reporting nodes 66 are also stored in the management automaton from which runtime automaton 27b is derived. For example,
Typically, each of the automata in the dynamic dictionary supports only updates of accretion. There is typically no need to delete data from the dynamic dictionary, since, as described above with reference to
Reference is now made to
In general, adding a keyword to the management automaton comprises adding one or more new management nodes to the management automaton, each of the new management nodes corresponding to at least a portion of the new keyword. In other words, each of the new management nodes corresponds to a portion of the keyword, or to the entire keyword. (A reporting node always corresponds to an entire keyword.)
For example, as shown in
It is noted that if the new keyword begins a string that is already included in the automaton, only one new management node—a reporting node—is added. For example, upon the arrival of keyword “AB,” if the automaton already includes the string “ABC,” such that regular node “AB” already exists, the only new management node that is added is a reporting node “AB,” between node “A” and regular node “AB.”
The addition of a new management node to the management automaton typically comprises at least some of the following steps:
(a) The processor ascertains that the portion of the new keyword to which the new management node corresponds differs by an appendage of exactly one symbol from the sequence of symbols to which another one of the management nodes corresponds. In response thereto, the processor identifies the other one of the management nodes as the parent node of the new management node. In response thereto, the processor stores, in the parent node, a child pointer to the new management node, the child pointer corresponding to the appended symbol.
For example, in
The processor typically also stores, in the new management node, a parent pointer to the parent node. For example, in
It is noted that the new management node may be a reporting node. For example, in
In some embodiments, a new regular node that is preceded by one or more new reporting nodes stores a parent pointer that points to the immediately preceding reporting node. (In such embodiments, the parent node of the new regular node is found by following parent pointers from the regular node until the first regular node is reached.) In other embodiments, the parent pointer stored in the new regular node points directly to the parent of the new regular node. For example, in
(b) The processor identifies another one of the management nodes as the fallback node of the new management node. In response thereto, the processor stores, in the new management node, a fallback pointer to the fallback node. For example, in
The processor typically also stores, in the fallback node, a friend pointer to the new management node. For example, in
Typically, the processor identifies the fallback node of the new management node by either (i) following an appropriate child pointer or shortcut pointer stored in the fallback node of the parent node of the new node, or (ii) following an appropriate shortcut pointer from the parent node.
In the first embodiment, the processor first ascertains that a fallback pointer to another one of the management nodes is stored in the parent node of the new node. (The processor may identify the parent node of the new node by following the parent pointer from the new node.) In response thereto, the processor identifies the other one of the management nodes as a fallback node of the parent node. The processor then identifies the fallback node of the new node, by following, from the fallback node of the parent node, the pointer that corresponds to the symbol that is appended in the new node.
In the second embodiment, the processor first ascertains that a shortcut pointer, corresponding to the appended symbol, is stored in the parent node and points to another one of the management nodes. In response thereto, the processor identifies the other one of the management nodes as the fallback node of the new node. (As noted above, the shortcut pointer is subsequently replaced with a child pointer to the new node.)
For example:
Example (ii) above demonstrates that the fallback node of the new management node is not necessarily directly pointed to by the child pointer stored in the fallback node of the parent node, or by the shortcut pointer stored in the parent node. In particular, there may be one or more intervening reporting nodes. For example, as noted above, in
Example (ii) also demonstrates that during the course of identifying the fallback node of the new management node, the processor may also identify any keywords that are contained in the sequence to which the new management node corresponds. For example, as noted above, during the course of identifying regular node “AC” as the fallback node of new node “CDAC,” the processor identifies that “CDAC” contains the keyword “AC,” and thus, inserts a reporting node “AC” prior to new node “CDAC.” Notwithstanding the above, in some embodiments, the processor identifies the contained keywords only after identifying the fallback node, by following the parent pointer from the identified fallback node. For example, in
(c) The processor ascertains that the fallback node of the new node stores a child pointer to a child node of the fallback node of the new node. (Such a child node may be a reporting node or a regular node.) In response thereto, the processor stores, in the new management node, a shortcut pointer to the child node of the fallback node of the new node. For example, in
(d) The processor stores, in one or more other nodes, a respective fallback pointer to the new node, indicating that the new node corresponds to the largest suffix of the respective sequences stored in the other nodes. Typically, in performing this operation, the processor changes one or more previously-existing fallback pointers to point to the new node. For example, in
(e) The processor stores, in respective other nodes, one or more shortcut pointers that point to the new node, indicating that the new node is a child node of the fallback node of the other nodes (i.e., the parent node of the new node is the fallback node of the other nodes). Such storing typically comprises changing one or more existing shortcut pointers to point to the new node. For example, in
It is noted that fallback pointers typically always point to regular nodes, whereas shortcut pointers may point to reporting nodes. Hence, if the new nodes include both a regular node and a reporting node, step (d) will typically involve changing one or more fallback pointers to point to the new regular node, while step (e) will typically involve changing one or more shortcut pointers to point to the new reporting node.
Typically, the processor performs step (d) and/or step (e) by first identifying friend nodes of the parent node of the new node, and determining, for each of the friend nodes, whether the friend node stores a child pointer or shortcut pointer that corresponds to the symbol that is appended by the new node. Subsequently, the processor executes step (d) and/or step (e), as follows:
Step (d): If there is a child pointer corresponding to the appended symbol, the processor follows the child pointer. If the child node of the friend node is a regular node, the processor changes the fallback pointer stored in the child node of the friend node to point to the new node. Otherwise, the processor traverses to the first regular node, and changes the fallback pointer stored in the regular node to point to the new node.
For example, in
Likewise, in
Step (e): If there is a shortcut pointer, corresponding to the appended symbol, from the friend node to another node, the processor changes the shortcut pointer to point to the new node. For example, in
Typically, steps (d) and (e) are performed recursively, over the various levels of “friends” in the automaton, until all appropriate updates have been made. For example:
Step (d): With reference to
During the course of performing step (d), the processor may also identify that the sequence to which a particular node corresponds contains a new keyword, and, in response thereto, add a copy of the appropriate reporting node before the particular node. For example, with reference to
Step (e): In
An analogy to help understand steps (d) and (e) is as follows. The “new parent” node “notifies” all of its friends about the “birth” of a child node, this notification including an identifier (e.g., an address in memory) of the new child node, as well as the appended symbol. In response to the notification, each friend node of the “new parent” then modifies one of its shortcut pointers to point to the “newborn” child node, and, if relevant, tells its child node (or a descendant of its child node) to modify a fallback pointer to point to the “newborn” child node, as described above. Each friend node then “passes along the message” to its friend nodes, and so on. (In some embodiments, the implementation of steps (d) and (e) is roughly along the lines of the above analogy, i.e., a type of “notification” is passed between the relevant nodes.)
It is emphasized that the usage of friend pointers for updates, e.g., as described above, increases the speed of the updates, relative to if no friend pointers were used. Typically, due the use of friend pointers, the time required to update the automaton depends mainly on the length of the new keyword, and not on the size of the automaton.
As noted above, in some embodiments, each of the automata may store more than one reporting node corresponding to a given sequence. For example,
(i) Multiple indexes may be stored in a single runtime reporting node, by each of the indexes being stored in a respective “pointer location.” For example, a single runtime reporting node may precede runtime node “DAB,” such a reporting node storing one index for keyword “DAB” in the “Pointer C” location, and another index for keyword “AB” in the “Pointer D” location. (Analogously, multiple indexes may also be stored in a single management reporting node.)
(ii) A single runtime reporting node may precede runtime node “DAB,” storing a single index (in one of the pointer locations) that refers to the pair of keywords “DAB” and “AB.” (Analogously, a single index in a management reporting node may refer to multiple keywords.)
It is noted that the two alternate embodiments described immediately above are generally consistent with previously-described embodiments, except for the manner in which multiple keywords are reported for a single regular node.
Typically, following each addition of a management node to the management automaton, a runtime node, derived from the added management node, is added to the runtime automaton. Tables 2-7 show the incremental changes to runtime automaton 27b (
(i) Table 2 shows changes to runtime automaton 27b in response to the changes shown in
Table 2 depicts the addition of node “D.” First, pointers from the new node, collectively corresponding to every symbol in the alphabet, are stored in the new node (row 10). Subsequently, the relevant pointers to the new node are stored in the other nodes. Such an order of operations facilitates the updating of the runtime automaton without inhibiting the use of the runtime automaton during the update. In other words, by allowing traversal to the new runtime node only after the new runtime node stores a respective pointer corresponding to each one of the symbols in the alphabet, there is no chance of reaching a “dead end” while traversing the runtime automaton. Furthermore, the changes made to the runtime automaton are typically atomic, such that the processor may multithreadedly modify and/or use the runtime automaton without needing to resort to locks while modifications are made. Hence, typically, updates to the runtime automaton do not reduce the speed or effectiveness of the keyword searching.
(ii) Table 3 shows subsequent changes to runtime automaton 27b, in response to the changes shown in
(iii) Tables 4 and 5 show subsequent changes to runtime automaton 27b, in response to the changes shown in
(iv) Tables 6 and 7 show subsequent changes to runtime automaton 27b, in response to the changes shown in
Reference is now made to
Management automaton 26c, which uses an alphabet of bytes, includes three keywords, “A,” “B,” and “QA.” Management automaton 26d includes the same three keywords, but represents the keywords differently from the way in which management automaton 26c represents the keywords. In particular, management automaton 26d uses an alphabet of symbols that are nibbles (half-bytes), rather than bytes.
In such nibble-based embodiments, it is typically the case that at least one node corresponds to a sequence consisting of an odd number of nibbles. For example, management automaton 26d includes respective nodes corresponding to (i) the symbol “LA,” which is the left nibble of “A,” (ii) the symbol “LQ,” which is the left nibble of “Q,” and (iii) “Q(LA),” which is the sequence consisting of the byte “Q” followed by the left nibble of “A.”
Likewise, the pointers in management automaton 26d correspond to respective nibbles, rather than to respective bytes. For example, second management automaton 26d includes respective pointers corresponding to (i) the symbol “LA,” which is the left nibble of “A,” (ii) the symbol “RA,” which is the right nibble of “A,” and (iii) the symbol “RB,” which is the right nibble of “B.”
Some other particular features of management automaton 26d are as follows:
(i) A node corresponding to an even number of nibbles falls back to another node that corresponds to an even number of nibbles, or to the root node. Conversely, a node corresponding to an odd number of nibbles falls back to another node that corresponds to an odd number of nibbles, or to a parity node 74. Falling back to parity node 74, instead of to the root node, prevents the misinterpretation of a “right nibble” as a “left nibble.”
(ii) The child pointers and shortcut pointers stored in a particular node point to respective other nodes having the opposite parity. For example, the child pointer corresponding to the symbol “RA” points from the odd-parity node “LA” to the even-parity node “A.” (The root node is considered to have even parity, while the parity node is considered to have odd parity.)
(iii) As indicated by the thickened shortcut arrows in
An advantage of using an alphabet of nibble-symbols is that each node may store fewer pointers than might otherwise be necessary, and hence, each node may have a smaller size than in non-nibble-based embodiments. Even though the number of nodes may be greater than would otherwise be the case, the reduction in node size typically more than compensates for the increase in number of nodes. Hence, in nibble-based embodiments, the automata are typically smaller than in non-nibble-based embodiments, such that the runtime automaton is more likely to fit in processor cache 48 (
For example, for an alphabet of byte-symbols, each runtime node may need to store up to 2{circumflex over ( )}8=256 pointers. (For example, each runtime node may store exactly 256 pointers.) On the other hand, for an alphabet of nibble-symbols, each runtime node need not store more than 2{circumflex over ( )}4=16 pointers. (For example, each runtime node may store exactly 16 pointers.) Hence, even if the number of nodes in a nibble-based runtime automaton were to be double the number of nodes in the parallel byte-based runtime automaton, the nibble-based runtime automaton would still be approximately eight times smaller than the parallel byte-based runtime automaton. (Moreover, the number of nodes in the nibble-based automaton is typically less than twice the number of nodes in the parallel byte-based automaton.)
It is emphasized that methods relating to the use of a static dictionary in combination with dynamic dictionaries, as described above with reference to
It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof that are not in the prior art, which would occur to persons skilled in the art upon reading the foregoing description. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents in a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.
This application is a continuation of U.S. patent application Ser. No. 15/299,729, filed on Oct. 21, 2016, entitled “SYSTEM AND METHOD FOR MAINTAINING A DYNAMIC DICTIONARY,” the contents of which are hereby incorporated by reference in their entirety. The present application is related to U.S. patent application Ser. No. 15/299,735, filed on Oct. 21, 2016, entitled “SYSTEM AND METHOD FOR KEYWORD SEARCHING USING BOTH STATIC AND DYNAMIC DICTIONARIES.”
Number | Name | Date | Kind |
---|---|---|---|
4672571 | Bass et al. | Jun 1987 | A |
4701851 | Bass et al. | Oct 1987 | A |
4706212 | Toma | Nov 1987 | A |
5006849 | Baarman et al. | Apr 1991 | A |
5014327 | Potter et al. | May 1991 | A |
5276741 | Aragon | Jan 1994 | A |
5410475 | Lu et al. | Apr 1995 | A |
5448474 | Zamora | Sep 1995 | A |
5623406 | Ichbiah | Apr 1997 | A |
5689442 | Swanson et al. | Nov 1997 | A |
5748781 | Datta et al. | May 1998 | A |
5805911 | Miller | Sep 1998 | A |
5835924 | Maruyama et al. | Nov 1998 | A |
5850561 | Church et al. | Dec 1998 | A |
5924098 | Kluge | Jul 1999 | A |
6012057 | Mayer et al. | Jan 2000 | A |
6137911 | Zhilyaev | Oct 2000 | A |
6185524 | Carus et al. | Feb 2001 | B1 |
6240408 | Kaufman | May 2001 | B1 |
6360220 | Forin | Mar 2002 | B1 |
6385339 | Yokota et al. | May 2002 | B1 |
6404857 | Blair et al. | Jun 2002 | B1 |
6449657 | Stanback et al. | Sep 2002 | B2 |
6581063 | Kirkman | Jun 2003 | B1 |
6597812 | Fallon et al. | Jul 2003 | B1 |
6628777 | McIllwaine et al. | Sep 2003 | B1 |
6674447 | Chiang et al. | Jan 2004 | B1 |
6718023 | Zolotov | Apr 2004 | B1 |
6741992 | McFadden | May 2004 | B1 |
6757361 | Blair et al. | Jun 2004 | B2 |
6785677 | Fritchman | Aug 2004 | B1 |
6963425 | Nair et al. | Nov 2005 | B1 |
6963871 | Hermansen et al. | Nov 2005 | B1 |
7043690 | Bates et al. | May 2006 | B1 |
7058652 | Czarnecki et al. | Jun 2006 | B2 |
7134141 | Crosbie | Nov 2006 | B2 |
7203682 | Balogh | Apr 2007 | B2 |
7216162 | Amit et al. | May 2007 | B2 |
7225343 | Honig et al. | May 2007 | B1 |
7233699 | Wenzel et al. | Jun 2007 | B2 |
7287278 | Liang | Oct 2007 | B2 |
7308446 | Panigrahy et al. | Dec 2007 | B1 |
7451146 | Boehm | Nov 2008 | B2 |
7453439 | Kushler et al. | Nov 2008 | B1 |
7457404 | Hession et al. | Nov 2008 | B1 |
7466816 | Blair | Dec 2008 | B2 |
7467202 | Savchuk | Dec 2008 | B2 |
RE40634 | Blair et al. | Feb 2009 | E |
7587041 | Blair | Sep 2009 | B2 |
7877401 | Hostetter et al. | Jan 2011 | B1 |
7925498 | Baker et al. | Apr 2011 | B1 |
8015175 | Fairweather | Sep 2011 | B2 |
RE43103 | Rozman et al. | Jan 2012 | E |
8176527 | Njemanze et al. | May 2012 | B1 |
8201245 | Dewey et al. | Jun 2012 | B2 |
RE43528 | Rozman et al. | Jul 2012 | E |
RE43529 | Rozman et al. | Jul 2012 | E |
8224761 | Rockwood | Jul 2012 | B1 |
8327265 | Vogel | Dec 2012 | B1 |
RE43987 | Rozman et al. | Feb 2013 | E |
8402543 | Ranjan et al. | Mar 2013 | B1 |
8413244 | Nachenberg | Apr 2013 | B1 |
8447588 | Karttunen | May 2013 | B2 |
8499348 | Rubin | Jul 2013 | B1 |
8578493 | McFadden | Nov 2013 | B1 |
8682812 | Ranjan | Mar 2014 | B1 |
8762948 | Zaitsev | Jun 2014 | B1 |
8832036 | Xu et al. | Sep 2014 | B2 |
8838951 | Hicks et al. | Sep 2014 | B1 |
8839417 | Jordan | Sep 2014 | B1 |
8850579 | Kalinichenko | Sep 2014 | B1 |
8869268 | Barger | Oct 2014 | B1 |
8898063 | Sykes et al. | Nov 2014 | B1 |
9053211 | Goldfarb et al. | Jun 2015 | B2 |
9058813 | Blanksteen | Jun 2015 | B1 |
9135497 | Cerna et al. | Sep 2015 | B2 |
9230160 | Kanter | Jan 2016 | B1 |
9798714 | Yishay | Oct 2017 | B2 |
10546008 | Yishay | Jan 2020 | B2 |
20010034743 | Thomas | Oct 2001 | A1 |
20020016818 | Kirani et al. | Feb 2002 | A1 |
20020049742 | Chan et al. | Apr 2002 | A1 |
20020077808 | Liu et al. | Jun 2002 | A1 |
20020099744 | Coden et al. | Jul 2002 | A1 |
20020123882 | Mohammed | Sep 2002 | A1 |
20020129140 | Peled et al. | Sep 2002 | A1 |
20020163913 | Oh | Nov 2002 | A1 |
20020194159 | Kamath et al. | Dec 2002 | A1 |
20030061026 | Umpleby et al. | Mar 2003 | A1 |
20030097439 | Strayer et al. | May 2003 | A1 |
20030174891 | Wenzel et al. | Sep 2003 | A1 |
20030236783 | Eminovici | Dec 2003 | A1 |
20040122661 | Hawkinson et al. | Jun 2004 | A1 |
20040179477 | Lincoln et al. | Sep 2004 | A1 |
20040210434 | Wang et al. | Oct 2004 | A1 |
20040249631 | Harris | Dec 2004 | A1 |
20040249809 | Ramani et al. | Dec 2004 | A1 |
20050018618 | Mualem et al. | Jan 2005 | A1 |
20050080797 | Short | Apr 2005 | A1 |
20050108200 | Meik et al. | May 2005 | A1 |
20050120017 | Motoki | Jun 2005 | A1 |
20050251512 | McCauley et al. | Nov 2005 | A1 |
20050278309 | Evans et al. | Dec 2005 | A1 |
20060015317 | Nakagawa | Jan 2006 | A1 |
20060036561 | Aladahalli et al. | Feb 2006 | A1 |
20060085533 | Hussain | Apr 2006 | A1 |
20060161984 | Phillips et al. | Jul 2006 | A1 |
20060259462 | Timmons | Nov 2006 | A1 |
20060277190 | Fox et al. | Dec 2006 | A1 |
20070043714 | Stanton et al. | Feb 2007 | A1 |
20070044060 | Waller | Feb 2007 | A1 |
20070047457 | Harijono et al. | Mar 2007 | A1 |
20070074131 | Assadollahi | Mar 2007 | A1 |
20070129976 | Hochberg et al. | Jun 2007 | A1 |
20070159968 | Cutaia | Jul 2007 | A1 |
20070180509 | Swartz et al. | Aug 2007 | A1 |
20070185859 | Flowers et al. | Aug 2007 | A1 |
20070186284 | McConnell | Aug 2007 | A1 |
20070192863 | Kapoor et al. | Aug 2007 | A1 |
20070226248 | Darr | Sep 2007 | A1 |
20070239741 | Jordahl | Oct 2007 | A1 |
20070261112 | Todd et al. | Nov 2007 | A1 |
20070294768 | Moskovitch et al. | Dec 2007 | A1 |
20080010308 | Erdogan et al. | Jan 2008 | A1 |
20080014873 | Krayer et al. | Jan 2008 | A1 |
20080028463 | Dagon et al. | Jan 2008 | A1 |
20080141376 | Clausen et al. | Jun 2008 | A1 |
20080147801 | Foti | Jun 2008 | A1 |
20080162137 | Saitoh et al. | Jul 2008 | A1 |
20080178076 | Kritt et al. | Jul 2008 | A1 |
20080184371 | Moskovitch et al. | Jul 2008 | A1 |
20080196104 | Tuvell et al. | Aug 2008 | A1 |
20080201312 | Posner | Aug 2008 | A1 |
20080201772 | Mondaeev | Aug 2008 | A1 |
20080208850 | Boyce | Aug 2008 | A1 |
20080212514 | Chen | Sep 2008 | A1 |
20080228748 | Fairweather | Sep 2008 | A1 |
20080243484 | Mohri et al. | Oct 2008 | A1 |
20080261192 | Huang et al. | Oct 2008 | A1 |
20080267403 | Boult | Oct 2008 | A1 |
20080285464 | Katzir | Nov 2008 | A1 |
20080294656 | Bhat | Nov 2008 | A1 |
20080313214 | Duhig et al. | Dec 2008 | A1 |
20080319738 | Liu et al. | Dec 2008 | A1 |
20090019019 | Jones et al. | Jan 2009 | A1 |
20090028143 | Eswaran et al. | Jan 2009 | A1 |
20090052454 | Pourcher et al. | Feb 2009 | A1 |
20090063151 | Arrowood et al. | Mar 2009 | A1 |
20090089334 | Mohamed et al. | Apr 2009 | A1 |
20090106842 | Durie | Apr 2009 | A1 |
20090150999 | Dewey et al. | Jun 2009 | A1 |
20090157673 | Boyce | Jun 2009 | A1 |
20090158430 | Borders | Jun 2009 | A1 |
20090171936 | Kirk et al. | Jul 2009 | A1 |
20090182728 | Anderson | Jul 2009 | A1 |
20090192874 | Powles et al. | Jul 2009 | A1 |
20090216760 | Bennett | Aug 2009 | A1 |
20090220153 | Hall et al. | Sep 2009 | A1 |
20090249484 | Howard et al. | Oct 2009 | A1 |
20090282476 | Nachenberg et al. | Nov 2009 | A1 |
20100023318 | Lemoine | Jan 2010 | A1 |
20100030553 | Ball | Feb 2010 | A1 |
20100037314 | Perdisci | Feb 2010 | A1 |
20100071065 | Khan et al. | Mar 2010 | A1 |
20100100949 | Sonwane | Apr 2010 | A1 |
20100111287 | Xie et al. | May 2010 | A1 |
20100142698 | Spottiswoode et al. | Jun 2010 | A1 |
20100250497 | Redlich et al. | Sep 2010 | A1 |
20110035211 | Eden | Feb 2011 | A1 |
20110043387 | Abali et al. | Feb 2011 | A1 |
20110099620 | Stavrou et al. | Apr 2011 | A1 |
20110154497 | Bailey | Jun 2011 | A1 |
20110167494 | Bowen et al. | Jul 2011 | A1 |
20110196874 | Ittiachen | Aug 2011 | A1 |
20110271341 | Satish et al. | Nov 2011 | A1 |
20110302653 | Frantz et al. | Dec 2011 | A1 |
20110320816 | Yao et al. | Dec 2011 | A1 |
20120011153 | Buchanan et al. | Jan 2012 | A1 |
20120017281 | Banerjee | Jan 2012 | A1 |
20120030195 | Holt et al. | Feb 2012 | A1 |
20120053926 | Satpute | Mar 2012 | A1 |
20120066312 | Kandekar et al. | Mar 2012 | A1 |
20120109923 | Pasquero | May 2012 | A1 |
20120167221 | Kang et al. | Jun 2012 | A1 |
20120174225 | Shyamsunder et al. | Jul 2012 | A1 |
20120179694 | Sciacca et al. | Jul 2012 | A1 |
20120193424 | Al-Omari et al. | Aug 2012 | A1 |
20120203761 | Biran et al. | Aug 2012 | A1 |
20120209828 | Takenaka et al. | Aug 2012 | A1 |
20120222117 | Wong et al. | Aug 2012 | A1 |
20120233215 | Walker | Sep 2012 | A1 |
20120239667 | Vysyaraju et al. | Sep 2012 | A1 |
20120254210 | Dhulipala et al. | Oct 2012 | A1 |
20120296983 | Boehm | Nov 2012 | A1 |
20120304244 | Xie et al. | Nov 2012 | A1 |
20120311708 | Agarwal et al. | Dec 2012 | A1 |
20130014253 | Neou | Jan 2013 | A1 |
20130096917 | Edgar et al. | Apr 2013 | A1 |
20130282744 | Hartman et al. | Oct 2013 | A1 |
20130333038 | Chien | Dec 2013 | A1 |
20140075557 | Balabine et al. | Mar 2014 | A1 |
20140195474 | Anguera | Jul 2014 | A1 |
20140207917 | Tock et al. | Jul 2014 | A1 |
20140212028 | Ciarcia | Jul 2014 | A1 |
20140214407 | Yitshak | Jul 2014 | A1 |
20140250132 | Pollak | Sep 2014 | A1 |
20140298469 | Marion et al. | Oct 2014 | A1 |
20150046496 | Karmarkar et al. | Feb 2015 | A1 |
20150066963 | Macek et al. | Mar 2015 | A1 |
20150135326 | Bailey, Jr. | May 2015 | A1 |
20160226511 | Kataoka et al. | Aug 2016 | A1 |
20170116351 | Yishay | Apr 2017 | A1 |
Number | Date | Country |
---|---|---|
0989499 | Mar 2000 | EP |
2104044 | Sep 2009 | EP |
2437477 | Apr 2012 | EP |
2005024604 | Mar 2005 | WO |
Entry |
---|
Aho, A., et al., “Efficient String Matching: An Aid to Bibliographic Search,” Communication of the ACM, Jun. 1975, vol. 18, No. 6, pp. 333-340. |
Altshuler, Y., et al., “How Many Makes a Crowd? On the Evolution of Learning as a Factor of Community Coverage,” LNCS 7227, 2012, pp. 43-52. |
Altshuler, Y., et al., “Incremental Learning with Accuracy Prediction of Social and Individual Properties from Mobile-Phone Data,” IEEE, 2011, 10 pages. |
Altshuler, Y., et al., “Trade-Offs in Social and Behavioral Modeling in Mobile Networks,” LNCS 7812, 2013, pp. 412-423. |
Argamon, S., et al., “Automatically Profiling the Author of an Anonymous Text,” Communication of the ACM, vol. 52, No. 2, Feb. 2009, pp. 119-123. |
Argamon, S., et al., “Gender, Genre, and Writing Style in Formal Written Texts,” Text & Talk, vol. 23, Issue 3, 2003, 32 pages. |
Atkinson, M., et al., “Near Real Time Information Mining in Multilingual News,” World Wide Web Conference, Apr. 20-24, 2009, 2 pages. |
Bailey, M., et al., “Automated Classification and Analysis of Internet Malware,” RAID, 2007, pp. 178-197. |
Bayer, U., et al., Scalable, Behavior-Based Malware Clustering, Secure Systems Lab, Technical University, Vienna, 2009, 18 pages. |
Bilge, L., et al., “EXPOSURE: Finding Malicious Domains Using Passive DNS Analysis,” Feb. 2011, 17 pages. |
Cloudshield, Inc., “Lawful Intercept Next-Generation Platform,” 2009, 6 pages. |
Coffman, T., et al., “Graph-Based Technologies for Intelligence Analysis,” CACM, Mar. 2004, 12 pages. |
Corney, M., et al. “Gender-Preferential Text Mining of E-mail Discourse,” Proceedings of the 18the Annual Computer Security Applications Conference, 2002, 8 pages. |
de Vel, O., et al., “Language and Gender Author Cohort Analysis of E-mail for Computer Forensics,” Defence Science and Technology Organisation, Australia, 2002, 16 pages. |
Dharmapurikar, S., et al., “Fast and Scalable Pattern Matching for Network Intrusion Detection Systems,” IEEE Journal on Selected Areas in Communications, Oct. 2006, vol. 24, Issue 10, pp. 1781-1792. |
Dietrich, C.J., et al., “CoCoSpot: Clustering and recognizing botnet command and control channels using traffic analysis,” 2012, pp. 475-486. |
Eagle, N., et al., “Inferring friendship network structure by using mobile phone data,” PNAS, vol. 106, No. 36, 2009, pp. 15274-15278. |
Eslahi, M., “botAnalytics: Improving HTTP-Based Botnet Detection by Using Network Behavior Analysis system,” Dissertation, Faculty of Computer Science and Information Technology, University of Malaya, 2010, 124 pages. |
Estival, D., et al., “Author Profiling for English Emails,” Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics, 2007, pp. 263-272. |
Fisk, M., et al., “Applying Fast String Matching to Intrusion Detection,” Los Alamos National Laboratory and University of California San Diego, Jun. 1975, 22 pages. |
FoxReplay Analyst, Fox Replay BV, http//www.foxreplay.com, Revision 1.0, Nov. 2007, 5 pages. |
FoxReplay Analyst Product Brochure, Fox-IT BV, http//www.foxreplay.com, 2006, 2 pages. |
Goldfarb, E., “Mass Link Analysis: Conceptual Analysis,” Jun. 24, 2007, Version 1.1, 21 pages. |
Goswami, S., et al., “Stylometric Analysis of Bloggers' Age and Gender,” Proceedings of the Third International ICWSM Conference, 2009, pp. 214-217. |
Gu, G., et al., “BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure-Independent Botnet Detection,” USENIX Security Symposium, vol. 5, No. 2, XP61009228, 2008, 16 pages. |
Gu, G., et al., “BotSniffer: Detecting Botnet Command and Control Channels in Network Traffic,” Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS'08), San Diego, California, 2008, 18 pages. |
Jacob, G., et al., “JACKSTRAWS: Picking Command and Control Connections from Bot Traffic,” Proceedings of the 20th Usenix Security Symposium, San Francisco, 2011, 16 pages. |
Lakhina, A., et al., “Mining Anomalies Using Traffic Feature Distributions,” SIGCOMM, 2005, pp. 217-228. |
Liu, R-T., et al., “A Fast Pattern-Match Engine for Network Processor-based NIDS,” Proceedings of the 20th International Conference on Information Technology (ITCC'04), 2006, 23 pages. |
Livadas, C., et al., “Using Machine Learning Techniques to Identify Botnet Traffic,” In 2nd IEEE LCN Workshop on Network Security (WoNS'2006), 2006, pp. 967-974. |
Mohrehkesh, S., et al., “Demographic Prediction of Mobile User from Phone Usage,” Proceedings Mobile Data Challenge by Nokia Workshop, Newcastle, United Kingdom, 2012, 4 pages. |
Navarro, G., et al., “Flexible Pattern Matching in Strings: Practical On-Line Search Algorithms for Texts and Biological Sequences,” Cambridge University Press, 2002, 166 pages. |
Netronome SSL Inspector Solution Overview White Paper, “Examining SSL-encrypted Communications,” 2010, 8 pages. |
Pan, L., “Effective and Efficient Methodologies for Social Network Analysis,” Dissertation submitted to faculty of Virginia Polytechnic Institute and State University, Blacksburg, Virginia, Dec. 11, 2007, 148 pages. |
Pandiselvam, P., et al., “A Comparative Study on String Matching Algorithms of Biological Sequences,” 2013, 5 pages. |
Rangel, F., et al., “Overview of the Author Profiling Task at PAN 2013,” CLEF 2013 Evaluation Labs, 2013, 13 pages. |
Rieck, K., et al., “Botzilla: Detecting the ‘Phoning Home’ of Malicious Software,” Proceedings of the ACM Symposium on Applied Computing (SAC), Sierre, Switzerland, 2010, 7 pages. |
Rohde & Schwarz GmbH & Co. KG, “ACCESSNET-T, DMX-500 R2, Digital Mobile eXchange,” Product Brochure, Secure Communications, Mar. 2000, 4 pages. |
Rohde & Schwarz GmbH & Co. KG, “ACCESSNET-T IP,” Product Brochure, Secure Communications, Jan. 2000, 4 pages. |
Rohde & Schwarz GmbH & Co. KG, “R&S AllAudio Integrated Digital Audio Software,” Product Brochure, Radiomonitoring & Radiolocation, Feb. 2000, 12 pages. |
Rohde & Schwarz GmbH & Co. KG, “R&S AllAudio Integrierte digitale Audio-Software,” Product Brochure, Feb. 2002, 12 pages. |
Rohde & Schwarz GmbH & Co. KG, “R&S AMMOS GX425 Software,” http://www2.rohde-schwarz.com/en/products/radiomonitoring/Signal_Analysis/GX425, Jul. 30, 2010, 1 page. |
Rohde & Schwarz GmbH & Co. KG, “R&S AMMOS GX430 PC-Based Signal Analysis and Signal Processing Standalone software solution,” http://www2.rohde-schwarz.com/en/products/radiomonitoring/Signal_Analysis/GX430, Jul. 30, 2010, 1 page. |
Rohde & Schwarz GmbH & Co. KG, “Digital Standards for R&S SMU200A, R&S SMATE200A, R&S SMJ100A, R&S SMBV100A and R&S AMU200A,” Data Sheet, Test & Measurement, May 2000, 68 pages. |
Rohde & Schwarz GmbH & Co. KG, “Integrated Digital Audio Software R&S AllAudio,” Specifications, 2000, 8 pages. |
Rohde & Schwarz GmbH & Co. KG, “R&S RA-CM Continuous Monitoring Software,” Product Brochure, Radiomonitoring & Radiolocation, Jan. 2001, 16 pages. |
Rohde & Schwarz GmbH & Co. KG, “R&S RAMON COMINT/CESM Software,” Product Brochure, Radiomonitoring & Radiolocation, Jan. 2000, 22 pages. |
Rohde & Schwarz GmbH & Co. KG, “R&S TMSR200 Lightweight Interception and Direction Finding System,” Technical Information, Aug. 14, 2009, 8SPM-ko/hn, Version 3.0, 10 pages. |
Schulzrinne, H., et al., “RTP: A Transport Protocol for Real-Time Applications,” Standards Track, Jul. 2003, 89 pages. |
Sheng, L., et al., “A Graph Query Language and Its Query Processing,” IEEE, Apr. 1999, pp. 572-581. |
Soghoian, C., et al., “Certified Lies: Detecting and Defeating Government Interception Attacks Against SSL,” 2010, 19 pages. |
Stamatatos, E., “Author identification: Using text sampling to handle the class imbalance problem,” Science Direct, Information Processing and Management, vol. 44, 2008, pp. 790-799. |
Svenson, P., et al., “Social network analysis and information fusion for anti-terrorism,” CIMI, 2006, 8 pages. |
Thonnard, O., et al., “Actionable Knowledge Discovery for Threats Intelligence Support Using a Multi-Dimensional Data Mining Methodology,” 2008 IEEE International Conference on Data Mining Workshops, 2008, pp. 154-163. |
Tongaonkar, A., “Fast Pattern-Matching Techniques for Packet Filtering,” Stony Brook University, May 2004, 44 pages. |
Verint Systems Inc., “Mass Link Analysis: Conceptual Analysis,” Jun. 2007, 21 pages. |
Verint Systems Inc., “Mass Link Analysis: Solution Description,” Dec. 2008, 16 pages. |
Wang, W., et al., “Breaking Bad: Detecting malicious domains using word segmentation,” 2015, 7 pages. |
Wang, H., et al., “NetSpy: Automatic Generation of Spyware Signatures for NIDS,” Proceedings of the 22nd Annual Computer Security Applications Conference, Miami Beach, Florida, Dec. 2006, ten pages. |
Yu, F., et al., “Fast and Memory-Efficient Regular Expression Matching for Deep Packet Inspection,” ANCS'06, San Jose, California, Dec. 3-5, 2006, 10 pages. |
Yu, F., et al., “Gigabit Rate Packet Pattern-Matching Using TCAM,” Proceedings of the 12th IEEE International Conference on Network Protocols (ICNP'04), 2004, 10 pages. |
Number | Date | Country | |
---|---|---|---|
20200159758 A1 | May 2020 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 15299729 | Oct 2016 | US |
Child | 16752955 | US |