FAST AND SCALABLE PROCESS FOR REGULAR EXPRESSION SEARCH

Abstract

A method includes reducing a deterministic finite automata DFA representative of an expression to provide a smaller DFA, and subjecting information that matches the smaller DFA to non-deterministic finite automata NFA representative of the expression for reducing memory required for pattern matching of the information.

Description

BRIEF DESCRIPTION OF DRAWINGS

These and other advantages of the invention will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.

FIG. 1 shows state diagrams illustrating delayed deterministic finite automata (D2FA) according to the prior art.

FIG. 2 shows state diagrams illustrating real time deterministic finite automata (RDFA) according to the prior art.

FIG. 3A is a state diagram of a deterministic finite automata (DFA) representing the expression (a [b-e] [g-i]| f [g-h]j) k+.

FIG. 3B is a state diagram illustrating a state merged equivalent of the DFA in FIG. 3A, in accordance with the invention.

FIG. 4 depicts state diagrams illustrating abstracting a DFA so that the resulting smaller DFA can produce false positives but no false negatives, in accordance with the invention.

FIG. 5 is a block diagram illustrating a hybrid finite automata (FA) having the performance of a deterministic finite automata (DFA) and the memory requirements of a non-deterministic finite automata (NFA), in accordance with the invention.

DETAILED DESCRIPTION

The invention addresses the memory blow-up of deterministic finite automatas DFAs and the slow speed of non-deterministic finite automatas NFAs. One aspect of the invention is reduction of a DFA, such as state merging, where two or more non-equivalent states in a DFA can be merged into a single state using transition labels. Coupled with an enhanced data structure, this merger compresses the DFA by an order of magnitude in practice. The second aspect of the invention is an abstracted hybrid automaton where a DFA is abstracted and combined with an NFA to build an automaton that has the speed of a DFA and the compactness of an NFA.

State Merging. The inventive state merging is a technique that allows non-equivalent states in a DFA to be merged using a scheme where the transitions in the DFA are labeled. By carefully labeling transitions, in effect, we are transferring information from the nodes to the edges of the graph representing the DFA. A data structure for representing a DFA with merged states and labeled transitions is a lossless compression method that can achieve significant memory reductions in practice.

Two or more states in a DFA or NFA can be merged into a single state by introducing labels on their transitions. For every transition connecting two merged states, we define source labels and destination labels. A transition, represented by c.l_d/l₀,l₁ . . . , thus has three attributes: (1) a character c upon which the transition is taken; (2) a single destination label Id that indicates to the destination state which underlying original state this transition is meant for; and (3) one or more source labels l₀,l₁ . . . that indicate to the source state upon which label to take this transition.

Each time a transition c.l_d/l₀,l₁ . . . is taken, a label Id is produced and stored. Transition c.l_d/l₀,l₁ . . . will be taken if the current input character is ‘c’ and the stored label is any of l₀,l₁. . . . If either the source or destination states are not merged, those labels are absent from the transition. Clearly, labels cause an overhead in terms of memory since they need to be stored. The number of required labels is bounded and small, and therefore their introduction only marginally affects memory usage. Such a transformation on the DFA is legal and does not affect correctness. FIGS. 3A and 3B show an example of the state merging transformation on a DFA. The DFA on the right 30B is the state-merged equivalent of the original DFA on the left 30A. Transition labels are the “.0” and “.1” in the “a.0” and “f.1” labels indicated by reference arrows and the merged state S1_—2 is indicated by reference arrows.

Merged-state DFAs can be realized in two major ways. First, they can be realized purely in software. It has been demonstrated that, for real security rule-sets, state merging can reduce software memory requirements by 10× over basic data structures, and by over 2× over the more advanced bitmap-based data structure. The bitmap-based data structure is discussed in more detail in priority claimed U.S. Provisional Application No. 60/821,192, entitled “Memory-Efficient Regular expression Search for Intrusion Detection”, filed on Aug. 2, 2006, the contents of which is incorporated by reference herein.

Second, they may be realized using specialized hardware, implemented using field programmable gate arrays (FPGAs) or custom chips. The specialized hardware consists of a lookup table to implement the state-to-next-state mapping of the DFA. With specialized hardware, the memory reduction possible is over 10×. In addition to this, there is a considerable reduction in the hardware logic complexity.

Hybrid Finite Automata. Two key ideas are used to realize hybrid finite automata. The first is the notion of “abstracting a DFA” to build a smaller DFA that allows false positives in a regulated manner. The second is the well-known architectural principle of “making the common-case fast”. We describe these below.

DFA Abstraction. The goal of DFA abstraction is to remove states from the DFA in such a manner that the resulting, smaller DFA can produce false positives but no false negatives. The state diagrams 41, 42, 43 of FIG. 4 show an example. From the original DFA 41, state 4 is removed, and all its transitions changed to state 3 (see reference 47 in FIG. 4) resulting in the first abstracted DFA 42. From the first abstracted DFA 42, state 3 is removed, and all its transitions changed to state 5 (see reference 48 in FIG. 4) resulting in the second abstracted DFA 43. Notice how the example input “afgjm” fails 44 in the original DFA 441, and must fail 45, 46 in all the abstracted DFAs 42, 43 (by construction) to avoid false negatives 49.

For the purpose of outlining how to systematically build a reduced DFA, let d be the transition function of the DFA, and d(S, c) indicate the state to which state S transitions to upon receiving input character c. We want to find two states A and B such that, for all possible strings w, d(A, w) is an accepting state if d(B, w) is an accepting state. Once we find A and B, we move B's incoming and outgoing transitions to A and then delete B. The resulting DFA can have false positives but no false negatives.

While in practice it may not be possible to build a reduced DFA with no false negatives, we propose a probabilistic approach where the reduced DFA will have very few false positives and very few false negatives. We do this by profiling the input traffic and removing those transitions from the original DFA that have the least likelihood of being traversed. This may be done during a training period. After the training period, the reduced DFA that is built may be deployed. During operation, if a transition that was removed is traversed, we revert to the NFA for resolution.

One method of realizing the above reduced DFA is to maintain an additional bitmap for each state. (Refer to the bitmap discussion/references provided before). The new bitmap tells us which transition was removed. For example, in a 4-character alphabet, if state S0 had valid transitions on characters a, b and c, it's bitmap would be 1110. If we remove transition c during training, the second bitmap would be 0010. The third bit being ‘1’ indicates that transition c was present in the original DFA but removed (so we must consult the NFA if this transition is traversed).

A DFA provides high performance (O(1) processing time per input character) but can require considerable memory (up to O(2ⁿ), where n is the number of characters in the regular expression). On the other hand, an NFA is slow (up to O(n) time per input character), but has small memory requirements (O(n)). The goal is to build a hybrid finite automata (FA) that combines the benefits of both an NFA and DFA. In other words, the hybrid FA aims to have the performance of a DFA and the memory requirements of an NFA.

We realize this by combining an reduced DFA with an NFA in such a manner that all matches from the DFA (including false positives) are checked by the NFA. In networking security applications where very few packets contain malicious information, matches will be few and far between. Therefore most of the packets will be processed quickly by the abstracted DFA, and a few will be checked by the slower NFA. Since the abstracted DFA is typically much smaller than a regular DFA, overall memory requirements are mitigated.

The block diagrams 50 of FIG. 5 detail the hybrid FA in accordance with the invention. A non-deterministic finite automata NFA is built from a regular expression set 51. The expression is converted to a definite finite automata DFA 52 from which a reduced DFA is built 53. A packet payload can be inspected by the reduced DFA and passed through if no match is found. As noted previously, the reduced DFA build allows for high speed, low latency, and low memory but the possibility of false matches 55. If a possible match is found, the packet is inspected by the full NFA 56. The full NFA build allows for low speed, high latency, low memory and no false matches. If there is no match in the full NFA build 56 then the packet payload is passed through. However, if a match is found then a malicious packet has been identified and a potential intrusion may have been detected. The advantages of the combined NFA with a DFA are a high speed in the common-case, low latency in the common-case and overall low memory.

In summary, the invention teaches reducing a DFA is to decrease the memory usage by removing states and transitions. In doing so, we try to MINIMIZE false positives and false negatives. In the ideal case, we want no false negatives, but this may not be practically achievable. The two methods of reducing a DFA are: (i) state merging with transition labeling and (ii) deleting states and transitions based on their probabilities (obtained by profiling network traffic). A reduced DFA, however it is generated, is always coupled with an NFA. When we encounter a false positive or a false negative, we resolve it using the NFA.

The present invention has been shown and described in what are considered to be the most practical and preferred embodiments. It is anticipated, however, that departures may be made therefrom and that obvious modifications will be implemented by those skilled in the art. It will be appreciated that those skilled in the art will be able to devise numerous arrangements and variations which, although not explicitly shown or described herein, embody the principles of the invention and are within their spirit and scope.

Claims

1. A method comprising the steps of: reducing a deterministic finite automata DFA representative of an expression to provide a smaller DFA, andsubjecting information that matches said smaller DFA to non-deterministic finite automata NFA representative of said expression for reducing memory required for pattern matching of said information.

2. The method of claim 1, wherein said smaller DFA can produce both false positives and false negatives.

3. The method of claim 2, wherein said false positives and false negatives are resolved using said NFA.

4. The method of claim 1, wherein said smaller DFA can produce false positives and no false negatives.

5. The method of claim 4, wherein said reducing said DFA comprises building a reduced said DFA according to: (i) where d is a transition function of said DFA,(ii) d(S,c) indicate the state to which S transitions to upon receiving input character c,(iii) finding two sates A and B such that, for all possible strings w, d(A,w) is an accepting state if d(B,w) is an accepting state, and(iv) once finding A and B, moving B's incoming and outgoing transitions to A and then deleting B.

6. The method of claim 4, wherein said information is packet information and matching of said packet information to both said smaller DFA and said NFA is indicative of a malicious packet.

7. The method of claim 1, wherein said reducing of said DFA comprises sate merging where at least two non-equivalent states in said DFA are merged into a single state using transition labels.

8. The method of claim 7, wherein said state merging is a non-lossy transformation of the original DFA producing neither false positives nor false negatives.

9. The method of claim 7, wherein said sate merging of said DFA is realized in at least one of software and hardware for reducing memory requirements.

10. The method of claim 9, wherein said hardware comprises a look up table for implementing state-to-next-sate mapping of said DFA.

11. A method comprising the steps of: removing states from a discriminate finite automata DFA for deriving a smaller said DFA that can produce false positives and no false negatives,building a non-discriminate finite automata NFA, andsubjecting packet information that matches said DFA to a check by said NFA for pattern matching that combines processing rate of said DFA with memory requirements of said NFA.

12. The method of claim 11, wherein said step of removing said states comprises building a reduced said DFA according to an outline where: (i) d is a transition function of said DFA,(ii) d(S,c) indicate the state to which S transitions to upon receiving input character c,(iii) finding two sates A and B such that, for all possible strings w, d(A,w) is an accepting state if d(B,w) is an accepting state,(iv) once finding A and B, moving B's incoming and outgoing transitions to A and then deleting B.

13. The method of claim 11, wherein matching of said packet information to both said smaller DFA and said NFA is indicative of a malicious packet.

14. A method comprising the steps of: subjecting network information to pattern matching combining reduced deterministic finite automata DFA producing false positives and no negatives followed by non-deterministic finite automata NFA for detecting network information that is malicious.