Patent Application
20150220646
The disclosed embodiments generally relate to online social networks. More specifically, the disclosed embodiments relate to techniques for efficiently identifying member profiles containing disallowed content, such as obscenity or pornography, in an online social network.
Perhaps the most significant development on the Internet in recent years has been the rapid proliferation of online social networks, such as LinkedIn™ and Facebook™. Billions of users are presently accessing such social networks to connect with friends and acquaintances and to share personal and professional information. As these online social networks become increasingly popular, some enterprising individuals are beginning to exploit these online social networks for financial gain by populating member records with disallowed content, such as pornographic images, to attract customers to websites selling pornography, sexual services, drugs or counterfeit goods.
This disallowed content is offensive to many users of online social networks, and these users often become upset when such content appears while they browse the online social network. Also, some users of these online social networks are children, who should be protected from this type of content.
Hence, what is needed is a mechanism for efficiently filtering out disallowed content from online social networks.
The following description is presented to enable any person skilled in the art to make and use the disclosed embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed embodiments. Thus, the disclosed embodiments are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.
The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing code and/or data now known or later developed.
The methods and processes described in the detailed description section can be embodied as code and/or data, which can be stored on a non-transitory computer-readable storage medium as described above. When a system reads and executes the code and/or data stored on the non-transitory computer-readable storage medium, the system performs the methods and processes embodied as data structures and code and stored within the non-transitory computer-readable storage medium.
Furthermore, the methods and processes described below can be included in hardware modules. For example, the hardware modules can include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), and other programmable-logic devices now known or later developed. When the hardware modules are activated, the hardware modules perform the methods and processes included within the hardware modules.
The disclosed embodiments relate to a system for identifying member profiles containing disallowed content in an online social network. During operation, the system scans member profiles looking up words from one or more fields in “bad-word” and/or “good-word” dictionaries. These dictionaries were previously generated by examining member profiles that were judged by a human reviewer to be “good” profiles or “bad” profiles, and determining whether a word belongs in a specific dictionary based on frequencies of occurrence of the word in the good profiles and the bad profiles. Next, the system computes a score for each member profile based at least partially on occurrences of words from the dictionaries in the member profile. Finally, the system identifies a member profile as containing disallowed content based on the score.
Next, the system performs a verification operation involving a human reviewer on the identified member profiles. During this verification operation, the system presents member profiles that were identified as containing disallowed content to the human reviewer, wherein each identified profile is presented to the human reviewer along with an image and associated text from the member profile, wherein multiple member profiles are presented to the human reviewer in a single view through a user interface (UI). Next, the system allows the human reviewer to select and/or deselect a member profile through the UI, wherein a selection of a member profile verifies that the member profile contains disallowed content.
Before describing details about how the system analyzes member profiles, we first describe a computing environment in which the system operates.
More specifically, mobile devices 104 and 108, which are operated by users 102 and 106 respectively, can execute mobile applications that function as portals to an online application, which is hosted on mobile server 110. Note that a mobile device can generally include any type of portable electronic device that can host a mobile application, including a smartphone, a tablet computer, a network-connected music player, a gaming console and possibly a laptop computer system.
Mobile devices 104 and 108 communicate with mobile server 110 through one or more networks (not shown), such as a WiFi® network, a Bluetooth™ network or a cellular data network. Mobile server 110 in turn interacts through proxy 122 and communications bus 124 with a storage system 128. These interactions allow users to generate and update member profiles 130, which are stored in storage system 128. Although the illustrated embodiment shows only two mobile devices, in general a large number of mobile devices and associated mobile application instances (possibly thousands or millions) can simultaneously access the online application.
Member profiles 130 include various types of information about each member. For example, if the online social network is an online professional network, such as LinkedIn™, a member profile can include: first and last name fields containing a first name and a last name for a member; a headline field specifying a job title and a company associated with the member; and one or more position fields specifying prior positions held by the member.
The disclosed embodiments also allow users to interact with the online social network through desktop systems. For example, desktop systems 114 and 118, which are operated by users 112 and 116, respectively, can interact with a desktop server 120, and desktop server 120 can interact with storage system 128 through communications bus 124.
Note that communications bus 124, proxy 122 and storage device 128 can be located on one or more servers distributed across a network. Also, mobile server 110, desktop server 120, proxy 122, communications bus 124 and storage device 128 can be hosted in a virtualized cloud-computing system.
During operation of the system illustrated in
During the analysis process, a dictionary-generation module 203 receives the member profile text 202 and profile reviews 206 as inputs, and uses these inputs to produce one or more word-scoring dictionaries 129 as is described in more detail below. Note that word-scoring dictionaries 129 can include both bad-word and good-word dictionaries, wherein a bad-word dictionary contains “bad words,” which are correlated with bad profiles containing disallowed content, and a good-word dictionary contains “good words,” which are correlated with good profiles that do not contain disallowed content.
Disallowed content can generally include any type of content that violates a terms of service agreement for members of the online social network. For example, the disallowed content can include: profanity, pornography, copyright violations, trademark violations, solicitations for sexual services, advertisements for drugs, and advertisements for counterfeit goods.
After the word-scoring dictionaries 129 are created, a profile-scoring module 208 uses the word-scoring dictionaries 129 (along with associated dictionary-specific coefficients) to generate profile scores 210. These profile scores 210 are used to identify a set of profiles containing disallowed content.
Next, the identified profiles are presented through a UI 131 to a human reviewer 134 along with an image and associated text from the member profile. Note that multiple member profiles can be presented to the human reviewer in a single view through a user interface (UI). This allows the human reviewer to select and/or deselect each member profile through UI 131, wherein a selection of a member profile by the human reviewer verifies that the member profile contains disallowed content.
Next, the system performs a number of operations, including categorizing the badness 310 of the words and profiles, quantifying the words and categories in the content 312, and performing a logistic-regression operation based on content scoring 314 to produce model content output 316. These operations involve using the set of profiles as a training data set for a logistic-regression operation, which optimizes parameters for a dictionary-based scoring function to score the profiles in a way that differentiates profiles containing disallowed content from the other profiles. These operations are described in more detail below.
Next, the system performs a number of operations to enable a human reviewer to verify profiles that are identified as containing disallowed content. These operations include a content-review operation 318, a verification/flagging operation 320, and a data-validation operation 322 that produces verified content output 323. These operations present profiles that have been identified as containing disallowed content to a human reviewer along with a profile photograph, and then allow the human reviewer to verify whether the profiles contain disallowed content. The verified content output 323 can then be used as additional inputs to retrain the model 324 to recognize the disallowed content.
Next, for each word, the system computes a percentage indicating how frequently the word appears in bad profiles, wherein bad profiles were judged by a human reviewer to be bad profiles (step 404). Next, the system categorizes words into good and bad dictionaries based on the computed percentages for the words and also a minimum number of instances for the word (step 406). For example, the system can place words into three bad dictionaries and one good dictionary based on the following rules:
For example, using the four dictionaries listed above, the profile-scoring function might look like the following:
The system considers each profile in the training set to be a four-dimensional vector having integer values for each of the four dictionaries along with a good/bad determination for the profile. These vectors and the profile-scoring function are used by a logistic-regression technique to determine a set of coefficients that enables the profile-scoring function to select profiles that are likely to include disallowed content. Note that logistic regression is a type of probabilistic classification model that is used to predict a binary response (e.g., a good or bad profile) based on one or more features (e.g., counts for each dictionary).
For example, in the case of an online professional network such as LinkedIn™, the system can maintain good and bad dictionaries for (1) first and last names (represented by “fl_”), (2) profile headlines (represented by “prof”), and (3) positions (represented by “pos_”). In this example, the coefficients may look like the coefficients that appear in
For example,
Human reviewers tend to be very good at recognizing images in the profiles. For example, a human reviewer can easily determine whether an image contains pornography. Also, the human reviewer can scan a large number of images in a small amount of time. Hence, if UI 131 presents 40 profiles with images in one view, the human reviewer will be able to select the bad images in a short amount of time. Thus, this visual review of profile images facilitates rapid review of profiles by a human reviewer.
The above-described system can be extended in a number of ways. For example, the system is not limited to English and can be used to detect disallowed content in foreign languages. In this case, the dictionaries need to be extended to include non-English words.
The system can also be extended to look at other types of data to determine whether the associated profile contains disallowed content. For example, the system can examine an IP address that a member used to access the online social network to determine whether the IP address is associated with other members whose records contain disallowed content. The system can also examine the email address in a member record to determine whether the email address originates from a domain that is associated with member records containing disallowed content. The system can also examine the text of email addresses to look for bad words. (A good profile will generally not contain an email address that includes a bad word.)
In additional to examining individual words, the system can examine N-grams comprising two or more adjacent words to see if specific N-grams are correlated with profiles containing disallowed content.
The system can also be extended to look for different categories of badness. For example, words associated with pornography belong to a different category of badness than words associated with counterfeit goods. Hence, the system can include words belonging to different categories of badness in the dictionaries.
In an online professional network, the system can possibly count the number of “start dates” for jobs in the member records and can determine from the number of start dates whether the associated profile is likely to contain disallowed content. For example, if the number of start dates in a member record is zero or is extremely large, say over 10, the member record is likely to contain disallowed content.
The foregoing descriptions of disclosed embodiments have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit the disclosed embodiments to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the disclosed embodiments. The scope of the disclosed embodiments is defined by the appended claims.
