Patent Application
20170068842
Social networks, or social utilities that track and enable connections between users (including people, businesses, and other entities), have become prevalent in recent years. In particular, social networking websites allow users to share and consume information more efficiently and in real-time. A social networking website may also allow users to associate themselves with other users, thus creating a web of connections among the users of the social networking website. These connections among the users can be leveraged by the website to offer more relevant information to each user in view of the users' shared information. Recognition of faces in images can be performed by any of a number of conventional methods. Such conventional methods for face recognition typically do not take advantage of the wealth of additional information available via social networks.
The present disclosure relates generally to social networking websites and other systems in which users can form connections with other users, and, more specifically, relates to recognition of faces in video, photos, or other media hosted on the social network.
The figures depict various embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.
Particular embodiments relate to a social networking environment including a social networking system and related systems that include conventional stationary as well as mobile embodiments. A social networking system offers its users the ability to communicate and interact with other users of the social networking system. Users join the social networking system and then add connections to a number of other users to whom they desire to be connected. Once they are members of the social network, the users may interact with the social network itself, by viewing and posting images and video on their own page, or by using mobile devices to view and post location-based social networking information.
In particular embodiments, a user may capture photos and/or video that includes real-time facial recognition with respect to the people in their social network. As each image in a video clip or a set of related images is processed, faces are detected and tracked. For a given face in an image, different potential candidates for identification of the face are selected (if available) from the user's social network. Each candidate for identification of the face in the image is scored, based on any number of factors, including but not limited to a confidence level, relationship coefficients with respect to the user and the candidate, physical proximity of the candidate to the user at the time the video and/or images were captured, etc. As the face is tracked through a series of images or video frames, the individual candidate scores in each image are computed and incorporated into an overall candidate score.
In one embodiment, as a user captures video or otherwise views their environment using their mobile device, facial recognition is performed in real-time, to provide identification suggestions for any detected faces in the video which may correspond to friends in the user's social network. In one embodiment, when a user views previously captured video on a computing device, facial recognition is performed in real-time, as the video is played, to provide identification suggestions for any detected faces in the video which may correspond to friends in the user's social network. Some embodiments perform real-time facial recognition for conventional photographs, including, for example, a series of related photographs. In some embodiments, an identification suggestion is not provided unless a face is detected in a minimum threshold number of images and/or video frames. Other embodiments are readily envisioned by the disclosure and are described in detail below.
In one embodiment, the user device 140 is used by a user initiating the actions being monitored by the server 110. The user device 150 is used by a recipient of the monitored actions. User devices 140 and 150 may each be any suitable computing device, such as a desktop computer, or a mobile device, e.g., a mobile phone, a cell phone, a smart phone, a personal digital assistant, a notebook computer, a tablet computer, a laptop computer, a camera, a video camera, or a hand-held game console.
In some embodiments, the server 110 comprises a face detection and tracking module 220 that operates upon images received by image receiving module 210 to detect and/or track faces in the images. Any conventional technology may be used for detection of faces in images and tracking of faces in series of images, such as video clips. Face detection and tracking module 220 may perform detection and tracking of faces itself, or it may be an interface that interacts with another server or a third-party service that performs those functions.
In some embodiments, the server 110 comprises a recognition module 230 that operates upon faces detected by face detection and tracking module 220 to select candidates for identification of the faces. Any conventional technology may be used for recognition of faces in images. In some embodiments, the pool of users from which candidates are selected comprises the set of users of a social network who are connected to the primary user. In some embodiments, the pool of users from which candidates are selected comprises the set of users of a social network who have some affinity with the primary user (e.g., belonging to a certain group, sharing a hobby/interests, or matriculating from the same school). In some embodiments, the pool of users from which candidates are selected may also include public figures, celebrities, and/or other well-known people. Recognition module 230 may perform face recognition itself, or it may be an interface that interacts with another server or a third-party service that performs those functions. In some embodiments, face detection and tracking module 220 and recognition module 230 may comprise a single interface that interacts with another server or a third-party service that performs all of those functions.
Server 110 may also comprise a candidate-scoring module 240. Candidate-scoring module 240 scores the candidates selected by recognition module 230, based on an algorithm that incorporates any of one or more factors. In some embodiments, the algorithm for candidate-scoring module 240 incorporates social-networking information (e.g., user identity and profile, degree of separation, relationship coefficients with the primary user) for each candidate who is connected to the primary user within a social network.
Coefficient scores may be calculated based on any number of variables. For example, increased interaction between any two users may result in a higher coefficient score, lending to the inference that the two are closer real-world friends. Similarly, a large number of mutual friends may increase the coefficient score. Methods for calculating coefficient scores between two or more users are well-known. Regardless of how the coefficient scores are calculated, once candidate-scoring module 240 obtains coefficient scores between the primary user and each of the candidates, candidate-scoring module 240 may incorporate the coefficient scores into the algorithm. For example, a candidate with a coefficient score above a minimum threshold, indicating that the users are close friends in real-life, is more likely to be in the same photo as the primary user. In some embodiments, coefficient information is retrieved only for candidates within a maximum threshold for degrees of separation from the primary user. In some embodiments, information is retrieved only for candidates having certain relationship coefficients with the primary user. Further description of relationship coefficients can be found in U.S. patent application Ser. No. 12/978,265, filed 23 Dec. 2010, titled “Contextually Relevant Affinity Prediction in a Social Networking System,” the contents of which are hereby incorporated by reference in its entirety.
In some embodiments, candidate-scoring module 240 may incorporate other information from recognition module 230, such as, for example, a confidence level that quantifies the level of confidence with which each candidate was selected as a possible identification of a given detected face. In some embodiments, social-networking information is only retrieved for those candidates selected with a minimum confidence level. In some embodiments, where the confidence level for a particular candidate selection is above a certain threshold (e.g., 0.55), that candidate's score is increased by a particular factor (e.g., a confidence level of 0.55 is increased to 0.70).
In some embodiments, server 110 may also comprise a candidate score registry 250. Candidate score registry 250 may be used by candidate-scoring module 230 to keep track of scores for each candidate. In some embodiments, the algorithm for candidate-scoring module 240 determines individual image scores for each face detected in an image—such individual image scores may be stored in candidate score registry 250. For series of images, such as frames of a video clip, or a series of still photographs all taken around the same time or approximately at the same location, candidate score registry 250 may store not only individual image scores, but also an overall score for each candidate with respect to all images in which the associated detected face appears. In some embodiments, the algorithm for candidate-scoring module 240 is based on a hidden Markov model.
In some embodiments, server 110 also comprises the social network data store 120; in some embodiments, server 110 is communicably connected to social network data store 120 and has access to profile information and relationship information associated with the primary user. Server 110 may also include or have access to an image store 260 that stores images with faces for which an identification has been confirmed (e.g., tag suggestion was confirmed). In some embodiments, server 110 is trained using machine learning techniques and images from image store 260 to accurately recognize faces and/or accurately score candidates for a detected face.
The server 110 may include a memory 270 to store applications, modules, user data, and/or other information. In some embodiments, applications, modules, and/or data store 120 may reside in memory on several servers (not shown).
In one example embodiment, candidates for the face detected in region 315 of image 300 are recognized and selected (the candidates having been selected from a pool of users connected to the primary user in the social network within a minimum degree of separation). In some embodiments, the number of candidates selected is limited to a maximum threshold number of candidates. Scores are then determined for the three candidates “Dan,” “Bob,” and “Bill” according to a candidate-scoring algorithm. As illustrated in
The winning-candidate algorithm for determining whether candidate “Dan” is the winner for identification of the face detected in region 315 may also take into account other factors, such as: (1) whether “Dan” was determined to be in close physical proximity to the primary user at the time the image was captured, (2) for a related series of images, such as a number of high-speed still photos taken at the same location or frames in a video of a scene, whether “Dan” was the winning candidate in the last N of M images, where N is a minimum threshold number of images and M comprises all or some subset of the series of related images, (3) relationship coefficients as between candidate “Dan” and the primary user, and (4) other relevant factors. In some embodiments, for example, for a series of related images, a winning candidate for a detected face is not determined unless the detected face appears in and is determined to be the winning candidate in the last N of M images.
In some embodiments, “older” or less recent candidate scores may expire. For example, if “Dan” is determined to be the winning candidate for a given face detected in video frames 1-5, 44-67, and 70-72, the face that “Dan” won in frames 1-5 may be disregarded when evaluating the winning candidate for frames 77-90.
In some embodiments, for a series of related images, once the same winning candidate for a detected face has been selected for the last P of Q images, any subsequent image in the series of related images including the same detected face is presumed to be the same winning candidate, where P is a maximum threshold number of images and Q comprises all or some subset of the series of related images.
In some embodiments, the functionality of candidate-scoring algorithm and the functionality of winning-candidate algorithm may be integrated into or operate as a single process.
In some embodiments, after analyzing candidates for the detected face in all three frames, “Dan” is selected as the final winning candidate with respect to all images in the series. In some embodiments, where the winning candidate of the previous image is selected as a candidate for the current image, the candidate-scoring algorithm increases the score of the candidate with respect to the current image. In some embodiments, the final winning candidate is determined by storing each candidate's score for each image in a series of related images and then determining the sum for each candidate of all of their image-based scores. In some embodiments, the final winning candidate is determined by using a hidden Markov model or any other similar algorithm.
In some embodiments, known series of related images with previously-identified faces may be used to train the recognition module and/or the candidate-scoring algorithm and/or the winning-candidate algorithm.
In particular embodiments, the methods illustrated in
This disclosure contemplates any suitable number of computer systems 700. This disclosure contemplates computer system 700 taking any suitable physical form. As example and not by way of limitation, computer system 700 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 700 may include one or more computer systems 700; be unitary or distributed; span multiple locations; span multiple machines; span multiple datacenters; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 700 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems 700 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 700 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.
In particular embodiments, computer system 700 includes a processor 702, memory 704, storage 706, an input/output (I/O) interface 708, a communication interface 710, and a bus 712. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.
In particular embodiments, processor 702 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 702 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 704, or storage 706; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 704, or storage 706. In particular embodiments, processor 702 may include one or more internal caches for data, instructions, or addresses. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.
In particular embodiments, memory 704 includes main memory for storing instructions for processor 702 to execute or data for processor 702 to operate on. As an example and not by way of limitation, computer system 700 may load instructions from storage 706 or another source (such as, for example, another computer system 700) to memory 704. Processor 702 may then load the instructions from memory 704 to an internal register or internal cache. To execute the instructions, processor 702 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 702 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 702 may then write one or more of those results to memory 704. In particular embodiments, processor 702 executes only instructions in one or more internal registers or internal caches or in memory 704 (as opposed to storage 706 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 704 (as opposed to storage 706 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 02 to memory 704. Bus 712 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 702 and memory 704 and facilitate accesses to memory 704 requested by processor 702. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.
In particular embodiments, storage 706 includes mass storage for data or instructions. Storage 706 may include removable or non-removable (i.e., fixed) media, where appropriate. Storage 706 may be internal or external to computer system 700, where appropriate. In particular embodiments, storage 706 is non-volatile, solid-state memory. Where appropriate, storage 706 may include one or more storages 706. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.
In particular embodiments, I/O interface 708 includes hardware, software, or both providing one or more interfaces for communication between computer system 700 and one or more I/O devices. Computer system 700 may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system 700. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touchscreen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces 708 for them. Where appropriate, I/O interface 708 may include one or more device or software drivers enabling processor 702 to drive one or more of these I/O devices. I/O interface 708 may include one or more I/O interfaces 708, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.
In particular embodiments, communication interface 710 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 700 and one or more other computer systems 700 or one or more networks. As an example and not by way of limitation, communication interface 710 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a Wi-Fi network. This disclosure contemplates any suitable network and any suitable communication interface 710 for it. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.
In particular embodiments, bus 712 includes hardware, software, or both coupling components of computer system 700 to each other. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.
Herein, reference to a computer-readable storage medium encompasses one or more non-transitory, tangible, computer-readable storage media possessing structure. As an example and not by way of limitation, a computer-readable storage medium may include a semiconductor-based or other integrated circuit (IC) (such, as for example, a field-programmable gate array (FPGA) or an application-specific IC (ASIC)), a hard disk, an HDD, a hybrid hard drive (HHD), an optical disc, an optical disc drive (ODD), a magneto-optical disc, a magneto-optical drive, a floppy disk, a floppy disk drive (FDD), magnetic tape, a holographic storage medium, a solid-state drive (SSD), a RAM-drive, a SECURE DIGITAL card, a SECURE DIGITAL drive, or another suitable computer-readable storage medium or a combination of two or more of these, where appropriate. Herein, reference to a computer-readable storage medium excludes any medium that is not eligible for patent protection under 35 U.S.C. §101. Herein, reference to a computer-readable storage medium excludes transitory forms of signal transmission (such as a propagating electrical or electromagnetic signal per se) to the extent that they are not eligible for patent protection under 35 U.S.C. §101. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.
This disclosure contemplates one or more computer-readable storage media implementing any suitable storage. In particular embodiments, a computer-readable storage medium implements one or more portions of processor 702 (such as, for example, one or more internal registers or caches), one or more portions of memory 704, one or more portions of storage 706, or a combination of these, where appropriate. In particular embodiments, a computer-readable storage medium implements RAM or ROM. In particular embodiments, a computer-readable storage medium implements volatile or persistent memory. In particular embodiments, one or more computer-readable storage media embody software. Herein, reference to software may encompass one or more applications, bytecode, one or more computer programs, one or more executables, one or more instructions, logic, machine code, one or more scripts, or source code, and vice versa, where appropriate. In particular embodiments, software includes one or more application programming interfaces (APIs). This disclosure contemplates any suitable software written or otherwise expressed in any suitable programming language or combination of programming languages. In particular embodiments, software is expressed as source code or object code.
Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.
This disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments herein that a person having ordinary skill in the art would comprehend. Similarly, where appropriate, the appended claims encompass all changes, substitutions, variations, alterations, and modifications to the example embodiments herein that a person having ordinary skill in the art would comprehend. Moreover, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative.
The foregoing description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure. For example, although the foregoing embodiments have been described in the context of a social network system, it will apparent to one of ordinary skill in the art that the invention may be used with any electronic social network service and, even if it is not provided through a website. Any computer-based system that provides social networking functionality can be used in accordance with the present invention even if it relies, for example, on e-mail, instant messaging or other form of peer-to-peer communications, and any other technique for communicating between users. The invention is thus not limited to any particular type of communication system, network, protocol, format or application.
Some portions of this description describe the embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.
Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.
Embodiments of the invention may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a tangible computer readable storage medium or any type of media suitable for storing electronic instructions, and coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
While the foregoing processes and mechanisms can be implemented by a wide variety of physical systems and in a wide variety of network and computing environments, the server or computing systems described below provide example computing system architectures for didactic, rather than limiting, purposes.
The present invention has been explained with reference to specific embodiments. For example, while embodiments of the present invention have been described as operating in connection with a social network system, the present invention can be used in connection with any communications facility that allows for communication of messages between users, such as an email hosting site. Other embodiments will be evident to those of ordinary skill in the art. It is therefore not intended that the present invention be limited, except as indicated by the appended claims.
Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.
This application is a continuation under 35 U.S.C. §120 of U.S. patent application Ser. No. 14/743,840, filed 18 Jun. 2015, which is a continuation under 35 U.S.C. §120 of U.S. patent application Ser. No. 13/297,074, filed 15 Nov. 2011, issued as U.S. Pat. No. 9,087,273.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14743840 | Jun 2015 | US |
| Child | 15353066 | US | |
| Parent | 13297074 | Nov 2011 | US |
| Child | 14743840 | US |
