Patent Grant
7865954
This invention relates to methods to detect an attack on a web site, and in particular, a denial of service attack, executed through SYN flooding.
Recent Denial-of-Service (DoS) attacks on major e-commerce web sites, like yahoo.com, amazon.com and ebay.com, have been very successful in disrupting their Internet activity for a considerable period of time. The popularity of DoS attacks in the hacker-community may be attributed to the vulnerability of interconnected computer systems and the ease with which DoS attacks can be launched over the Internet. SYN Flooding attacks are one of the most popular forms of DoS attacks that exploit the TCP's 3-way handshake connection mechanism and its limitation in maintaining the ‘Incomplete Connection Queue.’
To initiate a normal TCP connection, the client sends a SYN packet to the server. The server creates a new entry on the ‘incomplete connection queue’ for the SYN packet that has arrived, and responds by sending a SYN+ACK packet back to the initiating client. The client then acknowledges the server by sending an ACK packet (generally, ACK=SYN+ACK+1), thus completing the TCP 3-way handshake. See
To launch a SYN Flooding attack, an attacker floods the victim server with a huge number of SYN packets originating from spoofed IP sources. Attempted connection has an associated entry in the ICQ. The server responds to the SYN packet with the return ACK. However, the initiating “client” fails to respond to the ACK with the SYN+ACK, hence, leaving the entry in the ICQ for this attempt until time out. With a sufficient number of attempts, the ICQ will completely fill, overwhelming the system due to the inability to log new incoming calls in the ICQ. Due to a filled ICQ, the server rejects all incoming connections, even those originating from legitimate sources, causing a denial of service. To clear the system, it may be necessary to shut down and restart, resulting in downtime and data loss.
Prior art in attack detection schemes can be classified into two categories: (1) ‘Stateful’ mechanisms, which maintain a per-connection state, and (2) ‘Stateless’ mechanisms which do not maintain a per-connection state. SYN Cache, SYN Cookies, Synkill and SynProxying are some examples of ‘Stateful’ approaches. The SYN Cache replaces the per-socket linear incomplete connection queue with a global hash table. The ‘Cachelimit’ parameter imposes an upper bound on the memory that the SYN Cache uses and the ‘BucketSize’ parameter limits number of entries per hash bucket, bounding the time required for searching the entries. An entry overflow is handled by performing a FIFO drop of an entry on the hash list. SYN Cookies replace the Syn Cache's overflow handling mechanism by sending a SYN Cookie instead of dropping an entry from the hash list. A cookie contains an Initial Sequence Number, which is returned in the final phase of the TCP's three way handshake. As connection establishment is performed by the returning ACK, a secret is used to validate the connection. The Synkill algorithm classifies the addresses of all incoming packets into ‘good’ or ‘evil’ classes based on observed network traffic and input supplied by the administrator. A decision process based on a finite state machine determines the correct state membership of each incoming packet and sends RST packets in response to deter connection establishment attempts from evil IP sources. SynProxying sets a threshold on the number of SYN packets per second passing through the firewall. On reaching the threshold, the firewall proxies all incoming SYN packets by storing the incomplete connections in a queue. The incomplete connections remain in the firewall until the connection is completed or the request times out. All ‘Stateful’ approaches are by themselves vulnerable to flooding attacks.
‘Stateless’ non-parametric algorithms based on Sequential and Batch-Sequential Change Point Detection theory to detect SYN Flooding attacks have been developed by Wang et al. “Detecting SYN Flooding Attacks,” Proceedings of IEEE Infocom, June 2002. This method uses the discrepancy between the SYN-FIN (RST) pairs to detect SYN flooding. The weakness in using the discrepancy between SYN-FIN pairs as a criterion for detecting SYN flooding is that an attacker can paralyze the detection mechanism by flooding a mixture of SYN and FIN (RST) packets. Moreover, change point detection based algorithms may be sensitive to daytime variations in the Internet traffic, increasing the number of false positives when the attack triggering threshold is inappropriately selected.
The SFD (SYN Flood Detection) System is an adaptive sliding-window algorithm to detect SYN Flooding attacks based on the discrepancy between the packet arrivals and departures in the ICQ. The SFD System algorithm is based on the observation that there is an inherent delay between arrivals and departures. A function-estimate of the arrivals is a reasonably good estimate of the departures with a time lead of ‘α’, and inversely, the function-estimate of the departures is a reasonably good estimate of the arrivals with a time lag ‘α’, where ‘α’ is the service time or lag time of the queuing system. The implemented algorithm uses polynomial estimation for extrapolating the number of arrivals and departures.
Arrivals' are the number of SYN packets arriving into the Incomplete Connection Queue in a designated time period (i.e. either the number of SYN's per designated unit time, or the number of SYN+ACK, per designated unit time). SYN Packet Departures are the number of SYN packets “leaving” the queue on receiving an acknowledgement from the client in a designated period of time (i.e. the number of ACK's received per designated period of time). In general, arrivals will track departures, but the departures will be “delayed” from its arrival by the time to complete the handshaking between the client/server. For instance, in period of heavy traffic, upon initiation of heavy traffic, arrivals will exceed departures. As the handshaking process is completed, departures will “catchup” will arrivals, and when the heavy traffic ceases, departures will exceed arrivals as the queue empties.
When data traffic is normal, with normal fluctuations, actual (arrivals—“delayed departures”) will hover around zero. As traffic increases the actual (arrivals—“delayed departures”) should also hover around zero, if the delay approximates the delays need to complete handshaking. In the presence of an attack, arrivals will exceed departures, as residence time in the queue escalates.
The technique uses extrapolation of a fitted function (polynomials have been implemented) to (1) predict arrivals “forward in time” to a predicted time interval, and (2) predict departures “backward in time” to a reconstruction time interval. The prediction forward or backward in time accounts for the handshaking delays. In the presence of an attack: (a) arrivals will increase, but without a consequent increase in departures, even after accounting for handshaking delays (departures will increase after system timeout as unanswered ACKs are dropped, but the attack may overwhelm the system by this time); and (b) departures will decrease as “attack” attempts begin to populate or fill the queue.
A function is fitted to the arrivals a(t) in a designated estimation window, and similarly, a function is fitted to the departures d(t) in another designated estimation window. For a given “detection window,” the estimation window for departures is delayed from that for the arrivals by a predetermined time period. The “estimation window” length for arrivals does not have to match that of the “estimation window” length for departures. Alternatively, the fitted function could be the average value, a running sum, or some other mathematical manipulation.
Polynomials have been used as the fitted function ƒa (for arrivals) and ƒd (for departures). Given a set of data points (x1, y1), . . . , (xi, yi), a polynomial ƒ(x)=Pm=a+bx+cx2+dx3+ . . . +lxm−1 is fitted to the data points using the Method of Least Squares where the best fitting curve has the least square error
Polynomial fitting requires that we use a ‘Estimation Window Size’ of at least N+1 to fit a polynomial of degree N.
As shown in
Erra-di denotes the maximum signed difference in the prediction window Pa-di of (ƒai(t)−d(t)), for t ε Pd-ai; and Errd-ai is the maximum signed difference in the reconstruction window Rd-ai of (a(t)−ƒdi(t)), for t ε Rd-ai, again as shown in
The System is an adaptive sliding-window technique. The technique is an iterative procedure based on two hypothesis: (1) the arrival function ‘ƒai’ obtained from estimating a function for the samples in Arrival Estimation Window Eai should reasonably predict departures in the forward looking Prediction Time-window Pa-di, and (2) the departure function ‘ƒdi’ obtained from estimating a function for the samples in Departure Estimation Time-window Edi should reasonably reconstruct the arrivals in the rearward looking Reconstruction Time-window Rd-ai. This hypothesis is based upon the observation that arrivals—delayed departures should hover around the value of 0. The technique will base its “attack” analysis on the difference between the calculated arrivals and actual departures, or the calculated arrivals and actual departures, or both. That is (ƒdi(t)−a(t)) or (ƒai(t)−d(t)), where ƒdi and ƒai are the fitted prediction functions of the departures and arrivals in the prediction window; and a(t) and d(t) are the actual arrivals and departures respectively, during the prediction window.
The prediction and reconstruction errors Erra-di and Errd-ai are initialized to 0. The first step of the algorithm is to estimate functions ƒai and ƒdi for the samples in the Eai and Edi respectfully (in this example, a third degree polynomial is fitted). The second step is to calculate the differences (ƒai(t)−d(t)) in the window Pa-di, and the differences (ƒdi(t)−a(t)) in the window Rd-ai. The third step involves calculation of prediction error Erra-di as maximum of the singed value of (ƒai(t)−d(t)) in the window Pa-di, and calculation of reconstruction error Errd-ai as the maximum of the signed value of (ƒdi(t)−a(t)) in the window Rd-ai. In the presence of an attack, we anticipate Erra-di to be positive and to grow, and Errd-ai to be negative and to grow small. Instead of the signed maximum (which chooses the maximum of the absolute value, and then assigns the sign (+ or −) of the number chosen to the result) other metrics or distance functions can be used to determine the “distance” between (ƒdi(t)−a(t)) and (ƒai(t)−d(t)).
High values of Erra-di and Errd-ai indicate the failure of the algorithm to predict and/or reconstruct arrivals and departures respectively, and Erra-di and Errd-ai can be used to identify a SYN flooding attack (for instance, if abs(Erra-di)>α, or if abs(Errd-ai)>β, identify an attack). Each “Err” function is an indicator of discrepancies between actual data and estimated or predicted data. Hence, both “Err” functions can be used to identify a DOS attack. A more robust measure for identifying an attack is believed to be the difference between Erra-di and Errd-ai calculated using the Manhattan distance metric, (or some other chosen metric), and identifying an attack if the difference is greater than some threshold r. Because the two error functions are expected to move in opposite directions in the presence of an attack, combining the two Err functions should predict an attack earlier than either Err function alone. If another metric is used to calculate the Err functions and the result is both Err functions are positive, then Erra-di+Errd-ai should be used in the attack analysis, instead of Erra-di−Errd-ai. Alternatively, the absolute value of the two Err functions could be combined. Once an attack is predicted, notification of an attack should be provided to a user, where a user can include another program that would analyze the incoming requests to help analyze the nature of the threat.
After completing the technique in time window Di, the Detection window ‘Di’ is slid forward to become ‘Di+1’ and the procedure repeated in the new window. As shown in the example of
As can be seen, the technique can be implemented to work in almost real time. The algorithm may include a “backward” looking component (e.g., the calculation of ƒdi) and hence, when ƒdi is calculated, the algorithm is not real time, but would only be delayed from real time calculations by the departure estimation window length, as the samples in this window are used to compute ƒdi.
The technique was tested using seven Pentium 7 PC's out of which one PC was configured as an Apache Web server hosting about 2000 files; five PCs were used as Client Systems that generate different Web workloads; and a PC was configured to launch SYN Flooding attack. See
A SYN Flooding attack was generated using ‘sendip’ program. See
Table 1 summarizes the details of the Tcpdump datasets. With the experimental setup discussed above, we generated seven datasets containing normal background TCP traffic and labeled attack traffic. The ‘clients’ column in Table 1 indicates the Web workloads in terms of number of clients simultaneously accessing the Web server. The datasets with ‘n+m’ clients represent simulations which start with HTTP requests generated by ‘n’ clients. The request workload of the server is increased during the simulation run by adding ‘m’ clients to represent the day-time variations that occur in the workloads of an operational Web server. The ‘Duration’ column indicates the duration of each dataset. Number of attacks in each dataset and the duration of each attack are also indicated in Table 1.
The behavior of the incomplete connection queue is simulated on the dataset. The incomplete connection queue can either be implemented as a linear list or a hash table. We chose to implement it as a linear list of size 1024 which corresponds to the SOMAXCONN parameter at the server. We classified all packets collected at the Web server into SYN+ACK packets, ACK, and RST packets by inspecting the TCP packet headers. The flow chart in
The size of the detection window Di was set to 9. The sizes of estimation windows Eai and Edi were set to 4. The prediction window sizes Pa-di and Rd-ai were set to 1. The number of arrivals into and departures from the incomplete connection queue were sampled at 15 second intervals. The attack detection threshold has been set to τ=3000 for all the datasets.
The results of the technique are analyzed using three factors: (1) detection accuracy, (2) detection delay, and (3) false positive rate to measure the performance of our algorithm. Detection accuracy is the ratio of the total number of attacks detected by an FDS to the number of attacks in the test datasets. Detection delay is the time taken by the FDS to recognize an attack after the attack has actually occurred. The false positive rate is the number of normal traffic instances that an FDS flags as an attack. Table 2 gives the attack start times, attack detection times, attack detection delays, and the percentage of false alarms obtained when the technique was run on the seven datasets. The false alarm rate and detection delay for the datasets was recorded at 100% detection accuracy i.e., when the technique detected all occurrences of SYN flooding attack in the datasets. From the values in Table 2 we calculated the average number of false positives and the average detection latency of the technique. The average number of false alarms recorded over seven hours of normal TCP activity was 0.021 and the average detection latency over 21 attack instances was 116 seconds.
The plot in
The algorithm as implemented: (1) is stateless, i.e., it does not allocate resources on a per-connection basis to monitor SYN flooding attacks (the statelessness of the algorithm makes it resilient to SYN flooding); (2) is reliable because it uses the discrepancy between the number of packets entering and leaving the incomplete connection queue to detect SYN flooding (this discrepancy is a foolproof signature of an effective SYN flooding attack in progress); and (3) does not require separate signature-learning sessions to identify SYN flooding attacks and is therefore immediately operational when deployed on a TCP/IP based network.
To implement the technique, the code will be associated with a server of interest, either Web server, email server, FTP server, or any other server type that communicates to the outside world. The code implementation may be constructed to work with all servers in a multiple server configuration. The code is constructed to monitor the external communication ports for SYN (or SYN+ACK) and ACK. Generally, each server communicates over separate ports, so in a multi server embodiment, the installed algorithm would monitor several ports.
The code can be implemented in several embodiments. For instance, the embodiment can contain and maintain a “real time” ICQ queue, or several ICQ queues (as later describes, these queues can be dispensed with if the detection algorithm to be implemented will not perform an analysis on the incoming/outgoing data to identify the source or other aspects of the attack). The queue may be implemented as a hash table or a table queue. Each queue represents either a SYN, or a SYN+ACK, (it is easier to track the SYN+ACK, but the SYN alone can be used) and each entry can have the associated IP address of the SYN request, the port received over (in a multi port configuration), and a time stamp. Hence, as a SYN is received through a designated port (or SYN+ACK sent through the designated port) an entry is made in the ICQ. If the ACK is later received, the embodiment would find the ICQ entry for the associated SYN (or SYN+ACK), and indicate that that entry should be ignored (freeing up the slot in the ICQ) or remove the entry from the ICQ. As can be seen, the constructed queue(s) can be used in a post attack analysis, as the ICQ holds time and IP address information.
To implement the algorithm, a window size is chosen, say 10 seconds. An arrival counter Arr (or Arr(p), where p represents the port) and departure counter Dpt (or Dpt(p)) are established. These counters are incremental counters, that is, during each window, if a SYN is received, Arr=Arr+1, and if an ACK is received, Dpt=Dpt+1. Additionally, the Dpt counter can also account for RST (a ‘reset’ packet) if desired. A RST packet operates to reset a connection, and to the ICQ, an RST operates as an ACK, e.g., frees a slot in the ICQ. It is preferred that the counter account for an RST, but given that the number of RST packets is much less compared to the ACK numbers, RST can be ignored without substantial impact to the method.
At the end of the window, a(t)=Arr and d(t)=Dpt and the counters are reset to zero for the next window (alternatively, instead of resetting, the counter, the next window could start where the counter left off, that is, the time series would represent a running sum. With a running sum time series, the per window samples can be recovered if needed). In this fashion, we obtain two time series: (1) the time series ‘a(t)’ gives the number of SYN (or SYN+ACK) packets arriving every T seconds (here 10 seconds) and (2) the time series ‘d(t)’ gives the number of ACKs received (e.g. representing the number of packets leaving the incomplete connection queue) every T seconds.
As described, the ICQ's are not necessarily required for the algorithm as the incremental counters alone can be used to generate the arrival time series and the departure time series. Required parameters to the algorithm include:
(1) Sample window size for the counters and length of the counters;
(2) Estimation window size for ƒa;
(3) Estimation window size for ƒd (note, (2) and (3) can be the same));
(4) Delay or δ (delta) between an the end of the Estimation window for arrivals and the start of the estimation window of departures;
(5) prediction window size;
(6) relationship of the prediction window to the Estimation Windows;
(7) polynomial degree (assuming a polynomial is the fitted function);
(8) Err size or threshold that indicates attack;
As an attack indicator, the program can be configured to use Erra-di or Errd-ai alone, or a combination (e.g. abs(Erra-di−Errd-ai), abs(Erra-di+Errd-ai), or some other combination, such as max(Erra-di, Errd-ai), a linear combination of the err functions, etc). The delays and window size must account for the sample window size. For instance, if the sample window size is 10 seconds, the delay parameter now represents increments of 10 seconds for actual system response. Selection of parameters (extent of sampling or data decimation, prediction window sizes, estimation window sizes, relationship of the windows, degree of polynomial fit, relationship of adjacent detection windows (i.e. how the detection window is slid forward) and thresholds) can be varied during the day or during the week. For instance, if a system experiences heavy traffic during business hours, and light traffic in non-business hours, estimation windows may be shorter in heavy traffic situations and delta delays longer that in a light traffic pattern.
In general, the estimation windows size should be kept small, as longer sizes are more computationally inefficient, and longer estimation windows (or longer sampling decimation) are more likely to reflect smoothed “average conditions” rather than a granular snapshot condition, and hence not as responsive to rapid changes in traffic conditions. Additionally, longer estimation windows result in delayed prediction of an attack, as the backward looking window Edi must be captured to be used for analysis. For instance, if Edi is 20 samples long, (each 10 seconds), then the earliest this window could be used to predict an attack is 200 seconds after the attack. The actual chosen values will depend on expected peak traffic and the associated delays in the ACK which can be determined form experience with the server in question.
Generally, for a four sample prediction window, a third degree polynomial is used for the fit. For longer windows, (say 5-7 samples), a third or fourth degree polynomial will generally be suitable. Higher degree polynomial fits raise the computational inefficiency of the technique.
Supported in part by the US Army Research Office under Grant No. DAAD 19-01-1-0646 and the government has certain rights in the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6487666 | Shanklin et al. | Nov 2002 | B1 |
| 6725378 | Schuba et al. | Apr 2004 | B1 |
| 6772334 | Glawitsch | Aug 2004 | B1 |
| 6816910 | Ricciulli | Nov 2004 | B1 |
| 6823387 | Srinivas | Nov 2004 | B1 |
| 6973040 | Ricciulli | Dec 2005 | B1 |
| 7043756 | Tsafnat et al. | May 2006 | B2 |
| 7058718 | Fontes et al. | Jun 2006 | B2 |
| 7114182 | Robert et al. | Sep 2006 | B2 |
| 7137144 | Attwood et al. | Nov 2006 | B1 |
| 7143180 | Chaar et al. | Nov 2006 | B2 |
| 7190671 | D'Souza et al. | Mar 2007 | B2 |
| 7203961 | Dalal et al. | Apr 2007 | B1 |
| 7219228 | Lin | May 2007 | B2 |
| 7234161 | Maufer et al. | Jun 2007 | B1 |
| 7283461 | D'Souza et al. | Oct 2007 | B2 |
| 7284272 | Howard et al. | Oct 2007 | B2 |
| 7363652 | Yang et al. | Apr 2008 | B2 |
| 7464410 | Halasz et al. | Dec 2008 | B1 |
| 7512980 | Copeland et al. | Mar 2009 | B2 |
| 7657934 | Poletto et al. | Feb 2010 | B2 |
| 20030226035 | Robert et al. | Dec 2003 | A1 |
| 20040153669 | Yang et al. | Aug 2004 | A1 |
| 20070226239 | Johnson et al. | Sep 2007 | A1 |
