Patent Grant
7171450
This invention relates generally to electronic messaging and, more particularly, relates to filtering undesired electronic mail.
Electronic messaging, particularly electronic mail (“e-mail”) carried over the Internet, is rapidly becoming not only quite pervasive in society but also, given its informality, ease of use and low cost, a preferred method of communication for many individuals and organizations.
Unfortunately, e-mail recipients are increasingly being subjected to unsolicited mass mailings. With the growth of Internet-based commerce, a wide and growing variety of electronic merchandisers are repeatedly sending unsolicited mail advertising their products and services to an ever-expanding universe of e-mail recipients. Most consumers who order products or otherwise transact with a merchant over the Internet expect to and, in fact, do regularly receive such solicitations from those merchants.
However, electronic mailers are continually expanding their distribution lists to reach an increasing number of recipients. For example, recipients who merely provide their e-mail addresses in response to perhaps innocuous appearing requests for visitor information generated by various web sites often receive unsolicited mail and much to their displeasure, they find that they have been included on electronic distribution lists. This occurs without the knowledge, let alone the assent, of the recipients. Furthermore, an electronic mailer will often disseminate its distribution list, whether by sale, lease or otherwise, to another such mailer for its use, and so forth with subsequent mailers. Consequently, over time, e-mail recipients often find themselves increasingly barraged by unsolicited mail resulting from separate distribution lists maintained by a wide and increasing variety of mass mailers. An individual can easily receive hundreds, and even thousands, of pieces of unsolicited e-mail over the course of a year. Individuals on e-distribution lists can expect to receive a considerably larger number of unsolicited messages over a much shorter period of time.
Furthermore, while many unsolicited e-mail messages are benign, such as offers for discount office or computer supplies, mortgage rate quotes, or invitations to attend conferences of one type or another, others, such as pornographic, inflammatory and abusive material, are offensive to their recipients. These unsolicited messages are known as “junk” mail or as “spam.” The e-mail load from spam can be equivalent to the load generated from legitimate e-mail.
Similar to the task of handling junk postal mail, an e-mail recipient must sift through his incoming mail to remove the spam. The computer industry recognized this problem and has developed techniques to automate the removal of spam. For example, one technique is turf lists. E-mail recipients subscribe to turf lists, which identifies and refuses to accept mail using a defined rule based set of characteristics. Unfortunately, the choice of whether a given e-mail message is spam or not is highly dependent on the particular recipient and the actual content of the message. What may be spam to one recipient may not be spam to another, which limits the functionality of turf lists. Additionally, an electronic mailer (i.e., a spam generator) will prepare a message such that its true content is not apparent from its subject line and can only be discerned from reading the body of the message.
Another technique developed is known as a black hole list. The black hole list is a list of known spam addresses from which spam is sent. The e-mail sender's address is checked against the black hole list. If the address is on the list, the e-mail is not accepted. Spam generators simply change their address to bypass this technique. Other techniques have also been developed. None of the techniques are 100% effective. Innovations by e-mail servers to prevent spam are met with innovations by spam creators to overcome the innovations.
The present invention provides a framework that enables multiple spam detection solutions to be deployed to work together in a manageable and rational manner and enables new innovations to be created and deployed under a rapid deployment model.
A method is presented that determines if an e-mail message is spam using anti-spam modules. The method invokes one of the anti-spam modules and receives a spam confidence level from the anti-spam module. A tuning factor is applied to the spam confidence level to create a tuned spam confidence level. The tuned spam confidence level is added to a summed spam confidence level and the summed spam confidence level is compared to at least one threshold. If the summed spam confidence level is greater than the threshold, an action associated with the at least one threshold is invoked. The process repeats until either the summed spam confidence level is greater than the threshold or all of the anti-spam modules have been invoked.
In one embodiment, a plurality of thresholds including a top threshold is used and the summed spam confidence level is compared to each threshold. If the summed spam confidence level is higher than one or more of the thresholds; the action associated with the threshold that has been exceeded that is closest to the top threshold is invoked.
The tuning factor can range from a simple scaling factor such as multiplying the spam confidence level by one or scaling the spam confidence level by a user's confidence level in the anti-spam module that provided the spam confidence level to a complex tuning factor that normalizes the spam confidence level using a non-linear confidence level normalization.
The actions invoked includes dropping a connection if the summed spam confidence level exceeds a first threshold level, returning a non-delivery message to a sender if the summed spam confidence level exceeds a second threshold level and is below the first threshold level, and delivering the message to a junk mail folder if the message exceeds a third threshold level and is below the second threshold level.
Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments which proceeds with reference to the accompanying figures.
While the appended claims set forth the features of the present invention with particularity, the invention, together with its objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
Turning to the drawings, wherein like reference numerals refer to like elements, the invention is illustrated as being implemented in a suitable computing environment. Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a personal computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multi-processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
With reference to
Computer 110 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 110. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the personal computer 110, although only a memory storage device 181 has been illustrated in
When used in a LAN networking environment, the personal computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the personal computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
In the description that follows, the invention will be described with reference to acts and symbolic representations of operations that are performed by one or more computers, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by the processing unit of the computer of electrical signals representing data in a structured form. This manipulation transforms the data or maintains it at locations in the memory system of the computer, which reconfigures or otherwise alters the operation of the computer in a manner well understood by those skilled in the art. The data structures where data is maintained are physical locations of the memory that have particular properties defined by the format of the data. However, while the invention is being described in the foregoing context, it is not meant to be limiting as those of skill in the art will appreciate that various of the acts and operation described hereinafter may also be implemented in hardware.
The Simple Mail Transfer Protocol (SMTP) with an Exchange server shall be used to describe the invention. Exchange is an e-mail server produced by Microsoft Corporation. SMTP is the predominant email protocol used on the Internet. While SMTP and Exchange will be used, the invention may be used with other transfer protocols and mail servers. SMTP is a Transmission Control Protocol/Internet Protocol (TCP/IP) communication protocol that defines the message formats used for transfer of mail from one e-mail server such as Exchange via the Internet to another e-mail server. According to SMTP, an email message is typically sent in the following manner. A user runs an e-mail program to create an email message and the e-mail program places the message text and control information in a queue of outgoing messages. The queue is typically implemented as a collection of files accessible to the e-mail server.
The Exchange server establishes a Transmission Control Protocol (TCP) connection to the reserved SMTP port on the destination e-mail server and uses the SMTP to transfer the message across the Internet. The SMTP session between the sending and receiving servers results in the message being transferred from a queue on the sending host to a queue on the receiving host in stages. The stages range from the sending server providing the IP address of the connection being established to receiving all the message headers and message content. When the message transfer is completed, the receiving server closes the TCP connection used by SMTP, the sending host removes the message from its mail queue, and the recipient uses his configured e-mail program to read the message in the mail queue.
Turning now to
Turning now to
The real time black hole list module 300 compares the message sender's IP address against a known list of spam addresses. If the IP address is on the known list, the Exchange server 204 does not accept the mail. The non-linear module 302 normalizes confidence levels of filter 210 using functions such as an s-shaped curve, a Bayesian function, and the like that forces separation between spam and legitimate messages. For example, if a filter 210 returns a confidence level of 95%, the non-linear module 302 may scale the confidence level to 96% while a confidence level of 40% may be scaled to a confidence level of 30%. The turf list module 308 rejects mail during the SMTP protocol exchange between the sender and the Exchange server based on information available, including the senders mail address and/or domain, the target recipient(s) of the mail, and characteristics of the actual message body such as message id, date, subject, attachment type and name.
The framework 210 manages invocation of one or more of the anti-spam filters 210, normalizes the results of each invocation, evaluates the normalized result, and takes action on the result. The framework 210 is most typically deployed to the servers 204 at the edge of a network (i.e. the mail servers that are the first to receive emails from the Internet). Some of the technologies used, such as text classification, can be used for different uses, such as identifying importance or sensitivity of a message. As a result of this, the framework can also be usefully deployed on internal servers. The framework 210 can be used solely as a library of utilities that are invoked by existing standalone spam detection implementations to assist in migration from the standalone implementations, or more preferably as a wrapper that provides an abstraction from the underlying eventing mechanism (described below) that is used to invoke the anti-spam filters 210. The wrapper embodiment enables anti-spam filters 210 developed for email to also be used for other messaging solutions, such as Instant Messaging, determining harassment messages, etc. In either case, the framework is delivered as a library that is linked into the anti-spam technology at build or run time.
The architecture of the SMTP stack 200 in Exchange is such that events are fired by (i.e., sourced from) the stack 200 to event sinks, which are typically implemented as COM objects. When a new anti-spam technology is deployed, it implements a COM object which registers with the protocol event system at install time. The registration code is delivered by the framework 210. Installation of the framework 210 includes installing the software on the server in question, registering the event sink, enabling or disabling particular techniques for the particular server via a system administrator console, and establishing the evaluation and action strategies to be followed when spam is received. The enablement/disablement of a particular technique improves the manageability of the framework 210 by allowing all servers in a network to contain identical software binaries.
Turning now to
If the particular anti-spam filter 212 is “enabled,” the framework 210 obtains whatever information is available for the anti-spam filter 212 to examine and sends the information to the filter 212 (step 406). The amount and type of information available will change depending at the stage of the protocol at which the filter 212 is invoked. For example, the first time invoked there may only be information about the IP address of the connection being established. At the last call before the message is accepted by the system, all the message headers and content will be available. The framework 210 will crack the encoded content of the message into a form more readily useable by the anti-spam filter 212: When the framework is implemented as a wrapper, this information will be automatically available. When implemented as a library, the information will only be available if specifically requested by the anti-spam filter 210. In one embodiment, the framework 210 also provides a lookup of recipient addresses in the message that the filter 212 may use as part of its evaluation of a piece of mail as spam.
Upon completion of its evaluation, the filter 212 passes back to the framework 210, either by return value or reference (or by calling into the framework library), an evaluation of the confidence that the solution has that the particular mail message is spam (step 408). The framework 210 typically expects an answer in the range of 0–100%, where 0% represents clearly not spam, and 100% represents clearly spam. In one embodiment, the percentage is indicated as a number between 0 and 1000. To accommodate the various anti-spam technologies 300–310 that use different measures, the framework 210 provides a scaling or tuning factor to be applied to the results from each individual filter 212 invoked to create a normalized or tuned spam confidence level (step 410). The scaling or tuning factor is configured by the administrator of the server 204. This constitutes the normalization that framework must perform in order to compare the results of different filters 210. This normalized number will be referred to as the spam confidence level. The spam confidence level is added to the summary of spam confidence levels (step 412). The framework 210 will store the calculated spam confidence level as a running sum either-on the message itself for persistence and/or in memory for performance. Evaluation results of successive solutions are added to the summary of spam confidence levels.
The normalization (i.e., application of the scaling factor) can be implemented in a variety of ways. For example, one way to normalize the results is to trust the results of each filter 210 equally and simply sum the results (e.g., 0.5+0.7+0.8+ . . . =summary). Another way is to apply a scale to each one. For example, if the administrator likes the way a particular filter detects spam, the administrator would scale the spam confidence level of that filter by a relatively high number (e.g., 0.9). Similarly, if there is a filter that the administrator does not feel very confident about, the administrator scales the spam confidence level of that filter by a relatively low number (e.g., 0.3). Another example of a scaling factor is to use a non-linear confidence level normalization using a weighted curve such as an s-shaped curve. For example, if a filter 210 returns a spam confidence level of 95%, the scaling factor scales it higher to, for example, 96%. If the spam confidence level is in the middle of the range (e.g., 50–55%), a more radical scaling is applied to scale it lower (e.g., 30–35%).
The framework 210 provides the administrator with the ability to set several thresholds enabling the administrator to define different actions to be taken with a message based on the maximum threshold exceeded by the summary of spam confidence levels. The actions can be keeping the message from being delivered until a better idea of whether the message is spam is known, dropping the connection, sending a non-delivery message to the sender, deleting the message, passing it to another filter 210 based on the summary of spam confidence levels, sending the message to the recipient, etc. A default set of thresholds and corresponding actions is provided in the framework 210.
The summary of scaled spam confidence levels is compared to the top threshold set by the administrator (step 414). If the summary of spam confidence levels exceeds the top threshold, the action configured for the top threshold is taken (step 416). If the summary of spam confidence levels does not exceed the top threshold level and if there are more filters 210 that can be used to evaluate the message (step 418), steps 404 to 416 are repeated until either the top threshold has been exceeded or the end of the message has been received (step 420). If the end of the message has not been received, the SMTP stack 200 moves to the next message acceptance state (422) and steps 406 to 420 are repeated for the next message acceptance state. If the end of the message has been received and all of the enabled filters have analyzed the message, the summary of spam confidence levels is compared to the remaining thresholds in order from the top threshold to the bottom threshold (step 424) until the summary of spam confidence levels exceeds a threshold (step 426). If the summary of spam confidence levels exceeds a threshold, the action configured for that threshold is taken.
In summary, after a filter 212 has completed it analysis, the framework 210 evaluates the summary of spam confidence levels against a set of thresholds defined by the administrator. If the summary of spam confidence levels is greater than the highest threshold set by the administrator, then the action specified for the highest threshold is taken with the message. Otherwise, subsequent filters are used to evaluate the message until either the maximum threshold is exceeded, or all filters have evaluated the message. After all filters have evaluated the message, the summary of spam confidence levels is compared against all thresholds, and the matching threshold selected. The action associated with that threshold is then taken. For example, if the summary of span confidence levels exceeds a 99% confidence level threshold, the message connection may be dropped silently. If the summary of span confidence levels exceeds a 70% confidence level threshold, a non-delivery report may be returned to the sender. If the summary of span confidence levels exceeds a 40% confidence level threshold, the message may be delivered to a “junk mail” folder in the user's mailbox. If the summary of span confidence levels does not exceed any of the thresholds, the message may be considered legitimate and delivered to the user's inbox.
At all points, the actions taken with a particular message can be logged or added to a message tracking table, depending on the level of information that the admisistrator chooses to record. A default set of actions is available to the administrator with the framework 210. Additional actions may be added by delivering additional action execution code in a manner similar to that used to deploy new anti-spam filters.
Any message accepted for delivery to a mailbox by the framework 210 will have the summary of span confidence levels of that message stored on it in a well-known property. A delivery agent processing the message may choose to evaluate this property as part of its own logic. A client viewing the message, or a table of such messages, may choose to list the messages in ascending or descending order of the summary of span confidence levels, as an aid to identifying those messages that may have been miscalculated.
It can be seen that a platform to enable spam and viruses to be detected and handled at the edge of the network boundary using a variety of existing and future anti-spam filters and technologies has been described. The platform enables the solutions and technologies to interact and be managed in a rational way, thereby providing the ability to deploy rapid innovation on the server side in the warlike environment of detecting spam.
In view of the many possible embodiments to which the principles of this invention may be applied, it should be recognized that the embodiment described herein with respect to the drawing figures is meant to be illustrative only and should not be taken as limiting the scope of invention. For example, those of skill in the art will recognize that the elements of the illustrated embodiment shown in software may be implemented in hardware and vice versa or that the illustrated embodiment can be modified in arrangement and detail without departing from the spirit of the invention. Therefore, the invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6052709 | Paul | Apr 2000 | A |
| 6161130 | Horvitz et al. | Dec 2000 | A |
| 6321267 | Donaldson | Nov 2001 | B1 |
| 6615242 | Riemers | Sep 2003 | B1 |
| 6654787 | Aronson et al. | Nov 2003 | B1 |
| 6732157 | Gordon et al. | May 2004 | B1 |
| 6772196 | Kirsch et al. | Aug 2004 | B1 |
| 6868498 | Katsikas | Mar 2005 | B1 |
| 6901398 | Horvitz et al. | May 2005 | B1 |
| 6931433 | Ralston et al. | Aug 2005 | B1 |
| 6965919 | Woods et al. | Nov 2005 | B1 |
| 20020116463 | Hart | Aug 2002 | A1 |
| 20020199095 | Bandini et al. | Dec 2002 | A1 |
| 20030229672 | Kohn | Dec 2003 | A1 |
| 20040024823 | Del Monte | Feb 2004 | A1 |
| 20040093384 | Shipp | May 2004 | A1 |
| 20050080855 | Murray | Apr 2005 | A1 |
| 20050080856 | Kirsch | Apr 2005 | A1 |
| 20050102366 | Kirsch | May 2005 | A1 |
| Number | Date | Country |
|---|---|---|
| 2373130 | Sep 2002 | GB |
| Number | Date | Country | |
|---|---|---|---|
| 20040139160 A1 | Jul 2004 | US |
