Patent Application
20160191549
NOT APPLICABLE
NOT APPLICABLE
This invention relates to tools for network administration and more particularly to a method and apparatus for monitoring and analysis of a packet-based digital communication network to protect against external threats.
Today's enterprise networks face cyber attacks of increasing intensity and complexity. Almost every day there are reports of cyber attacks and data breaches despite billions of dollars already spent on enterprise security solutions. Clearly there are shortcomings in the current set of cyber security solutions.
Packet capture and analysis tools such as Wireshark (Wireshark Foundation) are counted as some of the most valuable ones in security analysts' toolbox. These tools provide great details for forensic analysis. In a high speed data communication environment, the amount of data quickly overwhelms anyone attempting to look through the network traffic over a span of more than a few minutes or a a few seconds. The sheer volume of traffic renders impractical if not impossible the monitoring and analysis of the network on a long-term and continuous basis.
SIEM-based solutions are widely used by enterprises to detect attacks. SIEM applications use application logs or security logs to find anomalous or suspicious activities that happened on network nodes. Network nodes can be PCs, servers, switches, routers, etc. SIEM-based solutions are fundamentally limited by how rich the logs are designed and implemented. Their effectiveness is further reduced if logging is not enabled on some network nodes.
Firewall, IDS, IPS and sandbox-based threat detection systems are the most important part of today's enterprise network defense systems. They are designed to create a secure perimeter to protect enterprise networks. When it works, they represent a great solution to guard against attacks. Unfortunately, these systems typically detect threats using known signatures and pre-defined rules. Nowadays, threat actors become increasingly sophisticated. They have learned how to evade detection by perimeter-based security systems. As a result, over 30% of cyber attacks succeed in passing through perimeter-based security systems. New solutions are needed to counter increasingly sophisticated attacks.
A significant portion of cyber attacks succeed in passing through the perimeter defense of enterprise networks. Once inside the network, attackers have a free hand to conduct malicious activities: to steal sensitive information, paralyze the operation of parts of the network, etc. These malicious activities are sometimes undetected for months or even years because they are not under the watch of any perimeter-based security systems. Their activities are often invisible to SIEM-based systems.
Most of internal network activities are not monitored today. Monitoring the internal network activities would give security analysts great visibility into the parts of the network they typically do not observe. This increased visibility would give security analysts much-needed help to detect and stop malicious activities that would otherwise be unnoticed and undisturbed for months. It is technically possible to monitor internal networks, but is a daunting prospect from cost, performance, and policy management/false positive standpoints.
Internal network monitoring has been done before using full packet capture approach. What is preferred is to capture and examine every packet flowing through the internal network. In reality this is not practical. Several problems exist with full packet capture based solution: 1) the amount of the data captured would be too voluminous to be effective. (For one single 1 Gb/s full-duplex link, at peak rate, there will be 250 MB of data captured per second. For one hour, 900 GB of data would be captured. Over 20 TB of storage space would be needed to store one day worth of data. Storage space would cost an exorbitant amount of money); 2) In addition to the storage problem, huge computing power needs to be available to process the amount of data captured in order to detect the threats “buried” in mountains of data. A different approach is needed.
According to the invention, network security monitoring is provided that is based on “rich metadata” collected from internal network traffic that is analyzed for anomalies to detect threats. To do so, network traffic is tapped at critical points of the internal network. Direct links bring tapped traffic to metadata probes. Metadata of every traffic flow is extracted automatically on a continuous basis by the probes. The extracted data are then aggregated into a big data cluster to provide instant insights to security analysts without requiring time-consuming searching through a huge amount of data. The same data can be used for real-time detection of anomalies and network attacks by analytics software. The solution also protects sensitive data and provides insight into the use of content within the enterprise network. It helps organizations better understand their data traffic and improved their ability to classify network activities and manage content. An embodiment of the invention targets smaller enterprise networks to simplify management as well as reduce the system cost and improve performance based on a consolidated architecture and the novel metadata-based analysis under an unified system control management.
Although end-to-end encryption may protect the content of the message, metadata still can be captured even when encryption is applied. By “rich metadata” it is meant at least information found in the headers of every layer of protocols associated with digital communication. This information describes the communication between two or more network entities. Such communication can be the result of human user actions such as a user browsing a web page. It can also be an autonomous action taken by the software running on a computer, such as a DHCP request automatically sent to acquire a dynamic IP address for a computer. Metadata contains critical information exchanged between network entities that can help security analysts quickly understand at a high level what type of communications happened and between which network entities. Such metadata typically represent up to 5% of total flow traffic. By going as deep as possible into all layers of an OSI stack, critical information about all network traffic flows can be extracted, thus enabling the understanding of behavior patterns not only at individual network entity level but also at entire logical network level. When connecting internal network traffic metadata to network users' information, one can enable the development of capabilities that detect human users' behavior on the internal enterprise network. This opens up a set of analysis possibilities that can lead to fast and accurate detection of network attacks while reducing false positives to a minimum. Herein after is a high level architecture view of a possible end-to-end network security monitoring and threat detection solution based on continuous rich metadata flows extracted from internal network traffic.
Further according to the invention, key points of the internal network are monitored and rich metadata of network flows are extracted. In addition, techniques of keeping track of unique IP addresses observed in the internal network are provided that give security analysts the ability to see new actors in their networks in real time.
Further, techniques are provided to automatically capture the mapping from IP address to MAC address and to domain name using extracted DNS and DHCP flow metadata. The organization information can be used to map the hostnames, MAC addresses, and phone numbers to real network users who are assigned to the network devices where traffic is originated or terminated. This rich set of mapping information enables a new way of detecting the suspicious actors early before harm is done. Furthermore, creating traffic distribution graphs, traffic patterns, and relationship maps for a network or a particular entity of interest not only paints a much more accurate picture of the monitored network or entity, but also highlights their characteristics, functionality and normality. By combining these aspects, one can derive and analyze the internal network characteristics over time, which is crucial to develop a powerful and sophisticated anomaly detection system with low false positives. Generally, the anomaly behaviors can be discovered by the following methods:
1. Analytics: The analyst views the analytics provided by the cyber security tool. The analytics provides visualizations of the traffic over time, the applications and protocols, device statistics, relationships, etc. Often, the analyst can “spot” anomalous behaviors from these analytics.
2. Policies/Rules: The analyst creates rules to detect anomalous behaviors. In our solution, these rules can be created against any of the captured metadata. Rules are often categorized as follows:
a. Simple (event driven)
b. Volumetric
c. Temporal
d. Spatial
e. Comparison
f. Dependent
g. Boolean
h. Nested
(The details for each of these are outside the scope of this document. Rules can be complex and may require a strong understanding of networking and security. For this reason, solutions often include templates to help the analyst create comprehensive rules.)
3. Machine Learning and Automation: The cyber security solution “learns” the normal behavior of the network users and entities. Once this “baseline” is established, the machine can also be employed to detect deviations from the normalcy, thus automating the threat detection process. The analyst can still create policy engine rules, but they can become much more sophisticated. For instance, a rule could issue an alert upon traffic levels dropping by a specified percentage. Most often, both machine learning and sophisticated policy rules are used with such solutions.
In a specific embodiment, the proposed solution is to monitor internal network activities among network entities at critical points by continuously extracting a rich set of metadata. By analyzing the extracted metadata, one can create and archive:
Determining the baseline behaviors for a sophisticated network can be a daunting task. This behavior analysis must consider the time span over which the baseline is determined, the parameters to be baselined, and how those parameters will be categorized. There are at least two choices for the time span:
The choice of parameters to be baselined and how those parameters are categorized greatly affect the complexity and responsiveness of the solution. We propose a unique baselining solution that will pivot around users. Specifically, all learned parameters will be categorized to a user.
Other baselining approaches can quickly become very complex and result in large databases, which could become unwieldy for large networks. For instance, one could envision an approach that attempts to baseline every parameter according to every category. So, the challenge is to develop an efficient method to baseline a network user according to a selected set of attributed parameters.
The present baselining approach is a user-centric method. The user is defined to be the entity that creates network traffic. The entity may have a user name (a credential tied to an employee account, for instance); he may have multiple devices that he “normally” uses; he may be associated with “normal” activities, etc.
The invention will be better understood by reference to the following detailed description in connection with the accompanying drawings.
A metadata probe according to the invention is operative to look into packets as wide and deep as possible to extract the important attributes of all traffic flows under monitoring. As herein defined, it produces a rich set of metadata for the network traffic flows that the probe monitors. Instead of using IP address as a node in an internal network, according to the invention, the probe looks at the network at a more abstract point of view. The probe defines a network entity as either an employee or a device. An employee can be responsible for multiple devices such as laptops, desktops, tablets, and phones. A device can be a web server, DNS server, LDAP server, or any type of machine that has network access. LDAP (Lightweight Directory Access Protocol) servers have an important role in enterprise networks, and LDAP is commonly used by medium-to-large organizations. It not only provides the authentication service, it also holds the enterprise information such as organization information, user information, and server information. When the server starts, the server creates an LDAP client instance to obtain the organization, user, server information to compose a list of network entities, providing directory-like information. For each network server, the application compares the assigned role (such as web server, LDAP server, mail server, file server) to its actual behavior. For example, a server A is expected to be a file server and operate based on the network activity. It behaves as an HTTP server, so it is a suspicious activity; hence, will be flagged. Each user has a telephone number, the information of his or her assigned devices, and a role in the organization. For example, one can create a user Alice whose phone number is 123-456-7890; she has a laptop with the name DOG, and she is a software engineer in Group A in the organization. Alice's information is then used to gather all of the flows related to her, such as SIP phone calls, HTTP traffic, SSH traffic, etc. For each flow, the inventive application compares Alice's behavior against other software engineers as a way of baselining for our anomaly detection. Once can obtain an accurate mapping between network entities to IP addresses, MAC addresses, host names, phone numbers for a given time range while minimizing the traffic generated by the probe itself.
By using organization information (e.g., LDAP), one can obtain the roles of employees and devices within an organization using an internal data network. By comparing behavior of a network entity as herein defined with similar entities, anomalies can be detected. The richness of the metadata based on this approach can be easily illustrated by the following example of an extraction from an HTTP flow:
The foregoing metadata set is much richer than a conventional NetFlow type of metadata commonly used by other known security software-based tools. NetFlow essentially gives analysts what is commonly called a 5-tuple: source IP address and port number, destination IP address and port number and Layer 4 protocol. However, the present metadata collection may go as deep as the OSI stack, where its critical information is extracted from each traffic flow composed of a sequence of packets sent from a particular source to a particular unicast, anycast, or multicast destination that the source desires to label as a flow. The basic metadata set specifically collected is the flow's:
For DNS flows, the metadata collected also includes DNS queries, number of queries, time between each query, server error message, answers, canonical names and IP addresses in addition to the basic set.
For HTTP flows, the metadata collected also includes session history entries, such as method, referrer, host, path, cookie, and content type in addition to the basic set.
For DHCP flows, the metadata collected also includes transaction ID, server IP address, subnet, requested IP address, requested lease duration, requested renewal of lease duration, requested rebinding of lease duration, time DHCP_DISCOVER was made, time offer packet was made, time DHCP_REQUEST packet was made, time server declined request, time server replied with ACK, time server replied with NACK, time client sent DHCP_INFORM packet, and time client sent a release packet in addition to the basic set.
For SIP flows, the metadata collected includes uri of the caller, uri of the callee, call ID of the call in addition to the basic set.
For mail flows, the metadata collected includes the login user name, the password, sender of the email, recipient(s), cc recipients, bcc recipients, subject, date, initial sender, email header, comments, resent date, resent sender, SMTP tags, SMTP server reply, pop3 commands, and commands in addition to the basic set.
For SSL flows, the metadata collected includes SSL certificate information such as range of validity, country, postal code, city, organization name, and organizational unit of the certificate, and the primary domain of the SSL encryption in addition to the basic set.
Continuous extraction and storage of rich metadata enables network security professionals to quickly gain insights into their own network in many different ways. The following sub-sections provide details on different types of insights that can be derived by performing analytics on metadata on entities as collected from internal networks.
Network communication can happen using many different protocols and various implementations of protocols. Knowing the types of protocols and applications present on the network helps spot problem areas quickly without having to go through tedious searching through billions of packets or thousands of log files. As an example of network traffic patterns a metadata probe can provide consider the various network traffic patterns of
The images of
Rich metadata provides insights into the complex communication relationships between network entities, external or internal, in any given time period for any combinations of protocols and applications.
Knowing the “actors” on a particular network can go a long way in helping security analysts quickly identify potential threats coming to that network or already in that network. By continuously monitoring the internal network and extracting the metadata of all traffic flows, one can keep track of a complete set of unique IP addresses. Using the GeoIP lookup tools, one can quickly identify where the network entity is geographically located. By leveraging the IP reputation information available from other third party sources, one can also automatically raise flags on certain new IP addresses observed for further investigation. It can help quickly detect malicious actors before they even do any harm to your network.
Based on a set of metadata records captured, one can also deduce the roles of network entities on the network. Examples of roles are: web server, file server, DNS server, DHCP server, LDAP server, HTTP client, etc. This determination is based purely on the type of network “conversations” (protocols and applications) and which side of the communication the network entity is on (server or client). This information, although simple, can contribute to identification of suspicious entities or traffic flows on the network. If a known dedicated file server is observed engaging in HTTP communication with another entity, such action would be a good reason to flag it for further monitoring or investigation. This flag function is built into the system according to the invention.
When certain network entities are deemed to be suspicious, further investigation would be useful. These entities are automatically monitored by the metadata probe. Automatic monitoring can take multiple forms including: selective full packet capture of the traffic from/to this entity. Another example would be the generation of alerts/notifications when the entity is observed by metadata probes communicating with other entities using certain application/protocol. The network rules or policies can be specifically designed for use only on an entity of interest, enabling capabilities to detect and notify any violations when the user is taking part into sensitive activities or attempting to hide it from detection, such as by encrypting it or modifying source documents.
There are two commonly used types of Internet Protocol (IP) traffic. These are TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). Devices on a network use IP to communicate over the Internet or a local network. Much of the communication between these devices is done using various protocols, e.g., DNS, LDAP, DHCP, etc. These protocols incorporate the use of either TCP or UDP. As an example, both DNS and DHCP use UDP while LDAP uses TCP to communicate. As a result of identifying TCP and UDP on the network, protocols, application, and usage can be identified in network traffic flows. Network traffic flows in an IP network are fundamentally identified by IP addresses. IP addresses are important information to understand the details of network “conversations.” However, they are of limited use when the detection of security problems can only rely on more accurate information such as the identities of the true network devices that communicate with each other or the owners of those devices involved. This limitation is caused by the simple fact that IP addresses are often dynamic. Today's enterprise networks generally support hundreds or thousands of network devices. Manual static IP address assignment is a very time-consuming and error-prone operation. Adding to this fact, most computing devices are mobile, such as laptops, smart phones and tablets, where it is largely impractical to assign static IP addresses. Enterprise IT operations typically rely on DHCP as a mechanism to dynamically assign IP addresses. As a result, the association between an IP address and a network entity is rarely fixed. Using an IP address to determine the associated network entity is not reliable as the same IP address may be assigned to different entities at different times. Physical MAC addresses, which are by definition unique to all devices, and logical domain names assigned to network entities are more reliable information to help understand which entities are involved in network conversations. Most network entities are assigned to individual network users. Being able to trace back to the owners of network entities will truly help get to the bottom of the critical question: “who is talking to whom?” An accurate answer to this question enables the fast and accurate detection of security problems while making it possible to keep the false positives low.
By extracting metadata buried deep in network traffic, one can extract a valuable amount of IP-address-to-domain-name-mapping information that can help better understand network entities at domain name level rather than at IP level without having to perform explicit DNS reverse lookup. Deep metadata extraction method according to the invention enables the automatic discovery of relationships between IP addresses and the true network entity they represent. In the case of multiple IP addresses used for the same host, the inventive process removes yet another layer of ambiguity.
The following is an example of DNS flow metadata record captured according to the invention:
This metadata of a captured DNS flow shows that a device at IP address 192.168.2.143 issued a DNS query on “daisy.ubuntu.com”. A local DNS server responded using cached information or its own query to the next level DNS server. The domain name is mapped to two different IP addresses: 91.189.92.55 and 91.189.92.57. Noting the similarity of the IP addresses, it will be concluded that any communication originating or terminating at either of these two addresses is actually from the same network entity. This relationship definitely helps bringing additional visibility into network traffic flowing through the enterprise networks.
Typical enterprise networks use DHCP to dynamically assign IP addresses to network devices attached to the network. The same IP address may be assigned to different devices at different times. Metadata extraction of network traffic flows enable automatic capturing of this assignment information dynamically and in real time. The following is an example of a DHCP flow metadata that was captured according to the invention:
By continuously capturing all DHCP flow metadata, a dynamic IP address at any given time can be mapped to the true network device identified by the source MAC address.
Insights into network activities can be gained by correlating flow metadata with network user information including the internal organization they belong to, the network devices they are assigned to and the logged-in users on each device. This information is available in existing directory services or IT auditing services widely used in enterprise networks. Correlating the network user information with traffic flows provides true insights and opens up possibilities for more powerful analysis and more accurate detection of security problems.
The rich metadata extracted from DHCP flows gives the lease duration as well as IP and MAC address attached to the flow. Also extracted are the metadata for DNS flows to keep track of association between IP address-to-host name and MAC-address-to-domain name. From domain name, the employee that is responsible for the device can be associated or related. Also extracted are the SIP flows to obtain the phone numbers involved in a call. Hence, by using the above information, one can track the activities between network entities using phone number, IP address, MAC address, hostname for a given period of time.
The baselining approach of the present invention is a user-centric method. The user is defined to be the entity that creates network traffic. The entity may have a user name (a credential tied to an employee account, for instance); he may have multiple devices that he “normally” uses; he may be associated with “normal” activities, etc.
The initial list of parameters to be baselined by user is:
Within the learning process, packets and flows are analyzed (classified and parsed), the attribute values are extracted, and those values are written to a database according to the user that they are associated with. Then a series of algorithms is provided to determine the normal behavior for the system. There is a finite set of algorithms and these can be easily added to over time. These algorithms determine such behavior baselines as what is the normal volume of X that occurs over time Y. In general, these algorithms are related to collecting numbers of events, volumes, and time. Some examples:
The final step is to detect the abnormal behaviors that may signify a network threat. As mentioned above, this can be automated or rules can be created to look for specific anomalies. Further, analyst feedback could be employed to mark certain alerts as false, increasing the accuracy of the detection over time.
The four-step process according to the invention as described above is shown in
1. DPI on the monitored traffic to extract the various events.
2. Write the baseline parameters to the database according to the user they are associated with.
3. Analyze the baseline parameters according to the algorithms to determine the baseline behaviors.
4. Detect deviations from the baseline by automation or analyst-defined rules.
To keep costs low for a device implementing the metadata probe function, a standard x86-based server may be used. Such devices can be manufactured and assembled by commercial suppliers such as SuperMicro or SMC. Key components of the server platform are a multi-core dual CPU such as the Intel Xeon E5-2695v2, 2.4 GHz or similar. Each CPU has 12 cores with a 30 MB cache. Each core supports two HyperThreads. This is to enable a reasonable number of true parallel processes. RAM size of 128 GB and a disk size of 16 TB raw disk capacity with RAID 10 configuration provides capacity and reliability. The internal bus is a type Gen 2 PCI-e bus and the operating system is for example Centos 6.5 installed on dual solid state drives. As explained below, one or more high-speed accelerator cards, such as the NT4E-NEBS four-port or the NT100E3-1-PTP high-speed single port cards (Napatech, Soeborg, Denmark), may be used to capture packets.
The architecture of the hardware as described is shown in
The extraction module (
The MetadataProducer class (
When a file is ingested, the file is placed within a sharedQueue to pass it to the MetadataConsumer class. The MetadataConsumer class proceeds to read the file line by line since the records are written in that format. Each line read is placed within the parsingQueue to prepare it for parsing. After every line is read, the file is then passed to the injectionQueue. If the backup setting is enabled, the GeoLocationInjector class takes the file, injects GeoLocation data into each record, and writes or appends the backup file into the specified backup folder. The original file is then destroyed.
Referring again to
As discussed, it is a daunting prospect to compile and process metadata from cost, performance, and policy management/false positive standpoints. However according to the invention, by targeting smaller enterprise networks, the task is manageable. Shown in
It is necessary to distinguish between internal and external actors in a network. Internal IP addresses have the prefix 192.168.x.x and 10.1.x.x. DNS flows have the domain name trailing the host name. A GEO-location look-up tool also indicates if an IP address is local or external. Key points in a network are the tap points. The tap points are typically at the switching location where sub-networks meet.
The invention has been explained with reference to specific embodiments. Other embodiments will be evident to those of skill in the art. It is therefore not intended that this invention be limited, except as indicated by the appended claims.
The present application claims benefit under 35 USC 119(e) of U.S. Provisional Application No. 62/061,845, filed on Oct. 9, 2014, entitled “RICH METADATA-BASED NETWORK SECURITY MONITORING AND ANALYSIS,” the content of which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62061845 | Oct 2014 | US |
