Patent Grant
10652592
The present invention relates to systems and methods for enriching a viewing experience of a user watching video content on a screen of a client terminal by increasing the relevance of additional media content proposed to the user. In particular, the present invention is suitable for improving the efficacy of disambiguation of named entities detected in the watched video content, the disambiguation carried out for selecting related content to be recommended or provided to the user.
Natural Language Processing (NLP) is a well-known technology field which deals with analysis and understanding of natural language texts by computers. Extensive research is already going on in this field for several decades and good progress had been achieved, even though there is still much to be improved until computers will get close to humans in understanding free and general-purpose texts.
A specific topic within the NLP field is Named Entity Recognition (NER). A named entity may be a real-world object, such as a person, a location, an organization, a product, etc., that can be denoted with a proper name. It can be abstract or have a physical existence. Examples of named entities include Barack Obama, New York City, Volkswagen Golf, and anything else that can be named. Named entities can simply be viewed as instances of classes of entities (e.g., New York City is an instance of a city). Many natural language sentences contain named entities, and understanding such sentence requires (i) identifying the existence of named entities, and (ii) for each term identified to be a named entity, finding what does that named entity refer to.
For example, in order to correctly understand the sentence “I am familiar with New Jersey” a computer must detect that the word “Jersey” is part of the named entity “New Jersey” (a state of the US) and does not refer to its stand-alone meaning of “knitted clothing”. One may argue that the fact that in the above example Jersey starts with a capital letter makes the task trivial. However, it should be remembered that in many cases the text to analyze is received from a speech-to-text automatic converter, where no capital letters can be known. Additionally, not all languages capitalize first letters of names as English does. If the above sentence would be written in Hebrew, there would be no detectable difference between the two interpretations of “Jersey” Similarly, in a language such as German wherein all nouns are capitalized (and not just ‘proper nouns’ as in English), there would be no detectable difference between two interpretations of any given noun that has an everyday meaning in addition to its meaning as a named entity.
Much research has gone into solving the Named Entity Recognition task and reasonably good solutions exist. Typically, NER implementations make extensive use of a dictionary, an encyclopedia, a database or a knowledge base for identifying a term as a named entity and for extracting its meaning. Wikipedia is the most commonly used source that is used for that purpose, because of its large size and diversity of covered fields.
However, recognizing a term in a sentence to be a named entity does not always bring with it immediate understanding of what it refers to. Consider the sentence “I admire Washington”. Prior art Named Entity Recognition systems should have no difficulty in identifying “Washington” to be a named entity, but will face difficulties when having to determine which entity the sentence refers to—(i) Washington D.C., (ii) Washington State, (iii) Washington Irving, (iv) George Washington, or (v) another person called Washington. The problem of distinguishing between multiple candidate interpretations of a given named entity is called Named Entity Disambiguation (NED).
As for the NER problem, much research was done for solving the NED problem. Several examples of such research papers are:
The issue of identifying the existence of a NED problem is straightforward—for example, a NER implementation that uses Wikipedia for identifying named entities can easily detect the existence of multiple Wikipedia pages referring to the same name. (Wikipedia actually provides an explicit “disambiguation page” for every name having multiple pages matching it, with the disambiguation page pointing to the different candidates for the name). The really difficult issue is determining which of the multiple candidates matching the same name is the one referred to in the analyzed text.
Prior art NED implementations use a variety of approaches. The simplest ones take their decision by looking only at the competing Wikipedia entries, without referring to additional inputs. For example, some NED systems determine which of the potential Wikipedia pages is most “popular” and pick it to be the intended meaning. Popularity of a given Wikipedia page can be determined, for example, by counting how many other Wikipedia pages contain a link pointing to the given page. Thus, in such NED systems the candidate Wikipedia page having the highest number of incoming links from other Wikipedia pages will always be selected for a given named entity.
More advanced NED systems do not always select the same candidate for all occurrences of a given named entity, but use the context in which the named entity appears in the analyzed text to customize the selection for each specific occurrence. The context of an occurrence of a named entity is typically taken to be the full sentence in which it appears, the full paragraph in which it appears, the full chapter in which it appears, or even the full document in which it appears. The competing Wikipedia pages are analyzed against the context of the named entity, and the “relatedness” between each of the competing pages and the context text is evaluated. The page found to be the most “related” to the context text is picked to be the intended interpretation of the named entity in its current occurrence.
“Relatedness” between two blocks of text obviously does not have a clear-cut measurement scale. Consequently, many algorithms were proposed in the research literature for measuring relatedness.
Some relatedness measuring algorithms are based on counting words that appear in both blocks.
Other relatedness algorithms are based on counting links from other blocks of text into the blocks whose relatedness is evaluated, working under the assumption that a third block containing links pointing to the two blocks whose relatedness is evaluated is a proof of a relation between them.
Other relatedness algorithms allocate different weights to different words, with some words contributing more to the relatedness when appearing in both text blocks. For example, words that appear in headers or in links may be considered more “important” and should contribute more, or words that appear multiple times in one or two of the compared text blocks should contribute more than words that appear only once in each block.
Going back to the NED disambiguation issue, the currently most advanced disambiguation algorithms use a semantic approach that takes into account the fact that there is usually some inter-dependence between multiple named entities appearing in the same context. When an analyzed block of text contains multiple ambiguous named entities, each having multiple candidates to select from, it is reasonable to assume the correct selections for the multiple named entities depend on each other. This is in contrast to all previously described disambiguation algorithms that deal separately with each ambiguous named entity and pick the most reasonable candidate for it independently of the disambiguation selections made for the other ambiguous named entities.
Ignoring inter-relations between adjacent ambiguous named entities may result in a clearly incorrect interpretation of the analyzed sentence, even when each one of the named entities in question is assigned its most reasonable interpretation. For example, a text about a football game may say “The game between Amsterdam and Barcelona will take place in Madrid”. Each of the three named entities (“Amsterdam”, “Barcelona” and “Madrid”) is ambiguous—it may refer either to a city or to a football club associated with a city. As the text is known to be about football, most NED algorithms disambiguating each ambiguous named entity on its own will resolve all three named entities as referring to a football club, which is indeed the most reasonable decision for each of the three ambiguities. The analyzed sentence would then be assumed to mean “The game between Ajax Amsterdam and FC Barcelona will take place in Real Madrid,” which is obviously wrong. Thus, ignoring the semantic relations between named entities might lead to easy-to-detect failures in disambiguation.
The semantic approach to NED resolves the multiple ambiguities of the above example by searching for solutions in a single joint vector space combining the possible selections for all three named entities. In this case, a solution vector has a length of three and the solution space has 2×2×2=8 possible values from which we can choose.
In some implementations of the semantic approach, the disambiguation is achieved using semantic relatedness scores obtained with a graph-based model, taking into account the semantic relationships between all named entities. For each Wikipedia page that is a candidate for one of the ambiguous named entities, a list of features is extracted—words in the page title, most frequently used words in the page, words from categories of the page, words from outgoing links in the page, etc. We then construct a graph whose nodes are the candidates and the features, and the graph is used for determining semantic relatedness. The easier it is to move on the graph between two nodes, the more related are the nodes. The entities disambiguation algorithm is then based on a random walk of the graph. Applying such a semantic NED algorithm to the above example should produce the correct interpretation of “The game between Ajax Amsterdam and FC Barcelona will take place in Madrid.”
Most NED implementations of the prior art rely for the disambiguation task only on (i) entries of the dictionary, encyclopedia, database or knowledge-base in use (e.g. Wikipedia) corresponding to potential candidates for the ambiguous named entities, and (ii) the textual context of the ambiguous named entities, usually making use only of the other named entities appearing within the textual context, but in some implementations also making use of words in the textual context which are not named entities. At least one prior art NED implementation (KORE, see the second research paper listed above) adds another source of information for disambiguating an ambiguous named entity—Internet websites that are associated with the candidates for the ambiguous named entity. For example, if the candidate entity is a person, then we may use his Internet home page. If the candidate entity is a company, then we may use its Internet website. If the candidate entity is performer, then we may use his/her fan website.
The use of NLP is widespread and the technology is applied in many fields of use. Consequently, the use of NER and NED is also widespread, as practically all NLP implementations require named entities recognition and disambiguation.
For example, NED is widely used in understanding search queries. When a user asks Google's search engine “What is the height of Washington?” the search engine needs to determine what is the meaning of the named entity “Washington” in the query—does the question refer to the height of a person or to the level above sea level of a city.
Another use of NED is in the field of content enrichment for video content consumers. When a user watches video content (a movie, a program, a news broadcast, etc.) on a viewing device (a TV set, a laptop, a smartphone, a tablet, etc.) it is common to present to him recommendations for related video content or other related information he may be interested in watching. The related content may be other movies or programs dealing with similar topics, biographical information about people mentioned or seen in the watched content, etc.
In many cases, the determination of what enriching content to recommend to the user is derived from the text heard in the sound track of the currently watched content. That text is obtained either from the subtitles of the movie or program that are provided in the stream of the watched content, or from an automatic conversion of the spoken text as it is heard in the sound track into written text using a speech-to-text conversion engine. In the case of subtitles in old movies, the text might be burned into the video, in which case extracting it from the video may require OCR technology. Regardless of the way by which the analyzed text is obtained, NED may be required. For example, if the text contains the named entity “Washington” there is a need to know if this refers to Washington State, Washington D.C. or George Washington. This determination will decide whether the TV system will recommend to the user the movie “Disclosure” (which was filmed in Washington State and takes place in Washington State), the TV series House of Cards (which takes place in Washington D.C.) or a documentary about George Washington.
The success rate of prior art NED implementations is not satisfactory. Even a success rate of 85% is considered to be very good (see the Hoffart Ph.D. thesis mentioned above). This is certainly not good enough for many real-world applications. A TV user may become highly frustrated when 15% of the recommendations he gets from his TV system turn out to be completely non-related to what he is currently watching.
Therefore, there is clearly a need for NED implementations that provide better success rates than what is achievable with prior art NED solutions.
The following United States published patent applications are incorporated herein by reference in their entirety: United States Patent Publication 20170161367, United States Patent Publication 20170153782, United States Patent Publication 20170147924, United States Patent Publication 20170147635, United States Patent Publication 20170147557, United States Patent Publication 20170124065, United States Patent Publication 20170060835, United States Patent Publication 20170039272, United States Patent Publication 20170011092, United States Patent Publication 20160335234, United States Patent Publication 20160306984, United States Patent Publication 20160306789, United States Patent Publication 20160275148, United States Patent Publication 20160203130, United States Patent Publication 20160188597, United States Patent Publication 20160124937, United States Patent Publication 20160117360, United States Patent Publication 20160110350, United States Patent Publication 20160085740, United States Patent Publication 20160078245, United States Patent Publication 20160055845, United States Patent Publication 20160048655, United States Patent Publication 20160012040, United States Patent Publication 20160012021, United States Patent Publication 20160012020, United States Patent Publication 20150332049, United States Patent Publication 20150331850, United States Patent Publication 20150286629, United States Patent Publication 20150269139, United States Patent Publication 20150161237, United States Patent Publication 20150095306, United States Patent Publication 20150081281, United States Patent Publication 20140337372, United States Patent Publication 20140316768, United States Patent Publication 20140297252, United States Patent Publication 20140282219, United States Patent Publication 20140214820, United States Patent Publication 20140195532, United States Patent Publication 20140142922, United States Patent Publication 20140136184, United States Patent Publication 20140101542, United States Patent Publication 20140074886, United States Patent Publication 20140046653, United States Patent Publication 20140039879, United States Patent Publication 20130346421, United States Patent Publication 20130311467, United States Patent Publication 20130275438, United States Patent Publication 20130238312, United States Patent Publication 20130198268, United States Patent Publication 20130173604, United States Patent Publication 20130166303, United States Patent Publication 20130080152, United States Patent Publication 20120324350, United States Patent Publication 20120271624, United States Patent Publication 20120203772, United States Patent Publication 20120117078, United States Patent Publication 20120102045, United States Patent Publication 20110258556, United States Patent Publication 20110246442, United States Patent Publication 20110246076, United States Patent Publication 20110225155, United States Patent Publication 20110125735, United States Patent Publication 20100235313, United States Patent Publication 20100185689, United States Patent Publication 20100145902, United States Patent Publication 20100145678, United States Patent Publication 20100076972, United States Patent Publication 20100004925, United States Patent Publication 20090319257, United States Patent Publication 20090204596, United States Patent Publication 20090192968, United States Patent Publication 20090164431, United States Patent Publication 20090157705, United States Patent Publication 20090144609, United States Patent Publication 20080319978, United States Patent Publication 20080301112, United States Patent Publication 20080208864, United States Patent Publication 20080154871, United States Patent Publication 20080126076, United States Patent Publication 20080071519, United States Patent Publication 20080065621, United States Patent Publication 20080040352, United States Patent Publication 20070233656, United States Patent Publication 20070214189, United States Patent Publication 20070106493, United States Patent Publication 20070067285, United States Patent Publication 20070016580, United States Patent Publication 20060247983, United States Patent Publication 20060149555, United States Patent Publication 20060136385, United States Patent Publication 20060136208, United States Patent Publication 20060136196, United States Patent Publication 20060010138, United States Patent Publication 20050251382, United States Patent Publication 20050216443, United States Patent Publication 20050080613, United States Patent Publication 20050049852, and United States Patent Publication 20030217052.
The present invention relates to systems and methods for enriching a viewing experience of a user watching video content on a screen of a client terminal by increasing the relevance of additional media content proposed and/or provided to the user.
Embodiments of a method for enriching a viewing experience of a user watching video content on a screen of a client terminal by increasing the relevance of additional media content proposed to the user, are disclosed. The method comprises providing at least a portion of a video content item to the client terminal, thereby causing playing the at least a portion of the video content item on the screen of the client terminal. The method also comprises obtaining a segment of text that is spoken in a sound track of the video content item and identifying an occurrence of an ambiguous reference to a named entity in the segment of text, the ambiguous reference matching multiple candidate named entities. The method also comprises disambiguating the ambiguous reference, the disambiguating comprising identifying the video content item, identifying an information source, the information source being directly associated with the identified video content item; and assigning one candidate named entity of the multiple candidate named entities to the ambiguous reference, the assigning being based on information from the identified information source. The method also comprises selecting one or more media content items that are related to the video content item, the selecting being based on the assigned one candidate named entity, and providing one media content item of the one or more media content items, thereby causing displaying the one media content item on the screen of the client terminal.
In some embodiments, the disambiguating can additionally comprise analyzing visual content of a segment of video from the video content item, and the assigning can also be based on a result of the analyzing.
In some embodiments, the information from the identified information source can comprise at least one of graphics and video, the disambiguating can additionally comprise analyzing visual content of the at least one of graphics and video, and the information from the identified information source on which the assigning is based can include the analyzed visual content.
In some embodiments, the obtaining of the segment of text can include obtaining the segment of text from subtitles of the video content item. In some embodiments, the obtaining of the segment of text can include converting speech appearing in the sound track of the video content item into text using a speech-to-text conversion engine.
In some embodiments, the disambiguating of the named entity can be done in parallel to the playing of the at least a portion of the video content item.
In some embodiments, the providing of the one media content item can include causing the displaying on the screen of the client terminal of a proposal for displaying the one media content item, and receiving from the client terminal a request to display the one media content item. The request can be caused by an input provided by the user subsequent to the displaying of the proposal.
The displaying of the one media content item can be done in parallel to the playing of the at least a portion of the video content item. The displaying of the one media content item can be done subsequent to the playing of the at least a portion of the video content item. The displaying of the one media content item can be done while the playing of the at least a portion of the video content item is paused.
In some embodiments, the information source directly associated with the identified video content item can be a synopsis of the identified video content item. The synopsis can be obtained from an electronic program guide (EPG). The synopsis can be obtained from a video-on-demand (VOD) library. In some embodiments, the information source directly associated with the identified video content item can be an advertisement for the identified video content item. In some embodiments, the information source directly associated with the identified video content item can be a review of the identified video content item. In some embodiments, the information source directly associated with the identified video content item can be an entry in a source that is one of a group consisting of a database, a knowledge base, a dictionary and an encyclopedia, the entry corresponding to the identified video content item. The source that is one of the group can be one of the IMDb database and the Wikipedia encyclopedia. In some embodiments, the information source directly associated with the identified video content item can be an account in a social network, the account being dedicated to the identified video content item.
In some embodiments, the client terminal can include one of a smart TV, a Set-Top Box and a TV screen, a smartphone, a tablet, and a personal computer.
In some embodiments, the information source directly associated with the identified video content item can be in a first language, and the segment of text can be in a second language that is different from the first language.
Embodiments of a method for enriching a viewing experience of a user watching video content on a screen of a client terminal by increasing the relevance of additional media content proposed to the user, are disclosed. The method comprises providing at least a portion of a video content item to the client terminal, thereby causing playing the at least a portion of the video content item on the screen of the client terminal, obtaining a segment of text that is spoken in a sound track of the video content item, and identifying an occurrence of an ambiguous reference to a named entity in the segment of text, the ambiguous reference matching multiple candidate named entities. The method also comprises disambiguating the ambiguous reference, wherein the disambiguating comprises analyzing visual content of a segment of video from the video content item, and assigning one candidate named entity of the multiple candidate named entities to the ambiguous reference to a named entity, the assigning being based on a result of the analyzing. The method also comprises selecting one or more media content items that are related to the video content item, the selecting being based on the assigned one candidate named entity, and providing one media content item of the one or more media content items, thereby causing displaying the one media content item on the screen of the client terminal.
In some embodiments of the method, the disambiguating can additionally comprise identifying the video content item and identifying an information source, the information source being directly associated with the identified video content item; in addition, the assigning can also be based on information from the identified information source.
In some embodiments, the obtaining of the segment of text can include obtaining the segment of text from subtitles of the video content item. In some embodiments, the obtaining of the segment of text can include converting speech appearing in the sound track of the video content item into text using a speech-to-text conversion engine.
In some embodiments, the segment of video can include the identified occurrence of the ambiguous reference. In some embodiments, the segment of video can exclude the identified occurrence of the ambiguous reference. In some embodiments, the segment of video may include the segment of text. In some embodiments, it can be that at least a portion of the segment of text is not included in the segment of video.
In some embodiments, the disambiguating of the ambiguous reference can be done in parallel to the playing of the at least a portion of the video content item. In some embodiments, the providing of the one media content item can include causing the displaying on the screen of the client terminal of a proposal for displaying the one media content item, and receiving from the client terminal a request to display the one media content item. The request can be caused by an input provided by the user subsequent to the displaying of the proposal.
The displaying of the one media content item can be done in parallel to the playing of the at least a portion of the video content item. The displaying of the one media content item can be done subsequent to the playing of the at least a portion of the video content item. The displaying of the one media content item can be done while the playing of the at least a portion of the video content item is paused.
In some embodiments, the client terminal can include one of a smart TV, a Set-Top Box and a TV screen, a smartphone, a tablet, and a personal computer.
Disclosed herein are embodiments of a system for enriching a viewing experience of a user watching video content on a screen of a client terminal by increasing the relevance of additional media content proposed to the user. The system comprises one or more computer processors, non-transitory computer-readable storage medium storing program instructions for execution by the one or more computer processors; and one or more media content servers in data communication with the client terminal and operable to provide at least a portion of a video content item to the client terminal, so as to cause playing the at least a portion of the video content item on the screen of the client terminal, and also operable to provide a media content item, so as to cause displaying the media content item on the screen of the client terminal, wherein the media content item is one of one or more media content items selected by executing, by the one or more computer processors, the program instructions stored in the non-transitory computer-readable storage medium. The stored program instructions comprise first program instructions for identifying an occurrence of an ambiguous reference to a named entity in a segment of text that is spoken in a sound track of the video content item, the ambiguous reference matching multiple candidate named entities; second program instructions for disambiguating the ambiguous reference, the disambiguating comprising identifying the video content item, identifying an information source that is directly associated with the identified video content item, and assigning one candidate named entity of the multiple candidate named entities to the ambiguous reference, the assigning being done based on information obtained from the identified information source; and third program instructions for selecting the one or more media content items, the one or more media content items being related to the video content item, the selecting being based on the assigned one candidate named entity.
Disclosed herein are embodiments of a system for enriching a viewing experience of a user watching video content on a screen of a client terminal by increasing the relevance of additional media content proposed to the user. The system comprises one or more computer processors, non-transitory computer-readable storage medium storing program instructions for execution by the one or more computer processors, and one or more media content servers in data communication with the client terminal and operable to provide at least a portion of a video content item to the client terminal, so as to cause playing the at least a portion of the video content item on the screen of the client terminal, and also operable to provide a media content item, so as to cause displaying the media content item on the screen of the client terminal, wherein the media content item is one of one or more media content items selected by executing, by the one or more computer processors, the program instructions stored in the non-transitory computer-readable storage medium. The stored program instructions comprise first program instructions for identifying an occurrence of an ambiguous reference to a named entity in a segment of text that is spoken in a sound track of the video content item, the ambiguous reference matching multiple candidate named entities; second program instructions for disambiguating the ambiguous reference, the disambiguating comprising analyzing visual content of a segment of video from the video content item, and assigning one candidate named entity of the multiple candidate named entities to the ambiguous reference to a named entity, the assigning being done based on a result of the analyzing; and third program instructions for selecting the one or more media content items, the one or more media content items being related to the video content item, the selecting being based on the assigned one candidate named entity.
The invention will now be described further, by way of example, with reference to the accompanying drawings, in which the dimensions of components and features shown in the figures are chosen for convenience and clarity of presentation and not necessarily to scale. In the drawings:
The proposed solutions described herein are based on the observation that one of the reasons for the relatively low success rate of prior art NED solutions is that they are intended to be general-purpose and target any application in which NED functionality is required.
The proposed solutions are further based on the observation that in the field of TV content enrichment there is extra information that is known at the time of conducting the NED analysis, information that does not exist in other NED application fields and is currently not used by NED algorithms. In some embodiments, this extra information is (i) we know that the analyzed text is taken from a sound track of a video content item that the user is currently watching, and (ii) we know the identity of that video content item. In some embodiments, this extra information is the video data of the video content item to which the analyzed text is related (as subtitles or audio sound track).
Consequently, the proposed solution deals only with NED conducted for the purpose of TV content enrichment, and is not a general-purpose solution.
In some embodiments, a first mode of NED implementation operates by performing the following steps:
In step a, the obtaining of the text can be from subtitles that are part of the video content item and can be displayed on the screen, providing either the text spoken in the video content item or a translation of it. Alternatively, the obtaining of the text can be from an automatic conversion of the spoken text to written text by a speech-to-text conversion engine.
In step b, the identifying of the one or more named entities is done according to the prior art methods of Named Entity Recognition.
In step c, the determining of which named entity is ambiguous is done according to the prior art methods of Named Entity Disambiguation.
In step e, the identity of the video content item is obtained. In some embodiments, the identity is obtained by the local playing terminal (e.g. smart TV or STB), which usually knows what is currently playing. This is always possible when the user picks his currently playing content item from local content in the terminal or from a VOD library of a TV operator. But even when the user is watching a linear TV channel, it is typically the case that the local terminal is aware of the identity of the currently playing video content item, as the local terminal has a copy of the EPG (Electronic Program Guide) which can tell the local terminal what is the currently playing program in any linear channel.
But even if the local terminal is for some reason not directly aware of the identity of the currently playing video content item, that information can be obtained by querying the TV operator's server, which always knows what is currently playing (except for the case of local content, in which case there is no question the local terminal knows the identity of the currently playing video content item). Therefore, in these embodiments, in all cases the local terminal is able to obtain the identity of the video content item. If any of the next steps, which require knowing the identity, is carried out by the TV operator's server and the identity is not known to the server, then the local terminal reports the identity of the video content item to the TV operator's server.
In other embodiments, the identity of the video content item is obtained by the TV operator's server. This is always possible when the currently playing video content item is picked from a VOD library or from a linear TV channel. But even if the video content item is picked from local content stored in the terminal, the server can obtain the identity from the local terminal. Therefore, in these other embodiments, in all cases the TV operator's server is able to obtain the identity of the video content item. If any of the next steps which requires knowing the identity is carried out by the local terminal and the identity is not known to the local terminal, then the server reports the identity of the video content item to the local terminal.
In step f, an information source directly associated with the currently playing video content item is obtained. As explained in greater length in the Definitions section under Definition “ee”, the expression ‘information source directly associated with a media content item’ wherever used herein means an ‘information source that includes information about the media content item when viewed as a whole.’ This is different from the prior art in which all the information sources used by the NED algorithms are associated either with potential candidates for the ambiguous named entities or with words appearing in the context of the ambiguous named entities (i.e. in the analyzed text). For example, the websites proposed as extra sources of information in the KORE system mentioned above are always directly associated with the candidates for the ambiguous named entities and never with a video content item from which the named entities in question are extracted.
The obtained information source can be any information source satisfying the condition of being directly associated with the video content item from which the analyzed text is taken. The paragraphs below disclose several examples of such information sources, which examples do not limit the above statement or the scope of the invention.
An information source directly associated with the currently playing video content item which is applicable to linear TV channels can be obtained from the EPG of the TV system. Most EPGs contain not only names of the broadcasted programs, but also a brief summary (typically called “synopsis”) providing information about the storyline, the cast, the location of filming, and the like.
Another information source directly associated with the currently playing video content item which is applicable to VOD content can be obtained from the VOD library of the TV operator, which also contains program summaries (synopses) similar to those in an EPG.
Another information source directly associated with a video content item is an advertisement for the video content item, which advertisement may be published on the Internet. In some embodiments, the advertisement can be a graphic advertisement, which can undergo visual analysis for its content.
Another information source directly associated with a video content item is a review of the video content item, which review may be published on the Internet. The review may be a critique from a professional critic or from ordinary viewers. In some embodiments, the review can be a video review, which can undergo visual or textual analysis for its content.
Still another information source directly associated with a video content item is an entry associated with the video content item in a dictionary, an encyclopedia, a database or a knowledge-base. For example, the TV series “House of Cards” has an entry in Wikipedia which can be used as an information source by the proposed NED algorithm when disambiguating an ambiguous reference to a named entity appearing in the sound track of an episode of that TV series. The dictionary, encyclopedia, database or knowledge-base containing the information source directly associated with the video content item need not be the same as the dictionary, encyclopedia, database or knowledge-base containing information sources used for other stages of the NED algorithm. For example, the information source directly associated with the currently playing video content item can be obtained from the IMDb database, while the rest of the NED algorithm may be using Wikipedia pages.
Still another information source directly associated with a video content item is an account in a social network, where the account is dedicated to the video content item. For example, the creators of a movie may open and advertise a Facebook account for the movie, in which they post news and anecdotes about it. Or a fan of the movie may open a Facebook account for the movie, in which he posts his views about it.
In step g, use is made of information obtained from the information source directly associated with the video content item, in a way that enhances the NED algorithm. The examples and explanations below provide more details about this step, but do not limit the above statement or the scope of the invention.
For example, information obtained from the information source directly associated with the video content item may be used for computing relatedness scores of competing candidates for a given ambiguous reference to a named entity. Such relatedness scores are herein called “content-item-relatedness scores,” as they are based on information directly associated with the video content item from which the ambiguous reference to named entity is taken. For each of the competing candidates a comparison is made against text obtained from the information source (which can be, for example, any one of the examples of information sources directly associated with the video content item described above), and a content-item-relatedness score is computed. The content-item-relatedness scores obtained this way can be used in computing the total relatedness scores, with other relatedness scores (that may be similar to relatedness scores of the prior art) also contributing to the total relatedness scores.
The algorithms used to calculate the content-item-relatedness scores may be similar to algorithms used by prior art NED systems for calculating regular relatedness scores between two blocks of text. For example, a NED algorithm may calculate a regular relatedness score between two Wikipedia entries by counting shared incoming links. Similarly, if the text obtained from the information source directly associated with the video content item for the purpose of calculating content-item-relatedness scores is a Wikipedia page corresponding to the video content item, then the same relatedness computation algorithm used for calculating regular relatedness scores (e.g. counting shared incoming Wikipedia links) may be used for calculating the content-item-relatedness scores. If, however, the text obtained from the information source directly associated with the video content item for the purpose of calculating content-item-relatedness scores is not a Wikipedia page, as is the case when it is taken from an EPG, then the content-item-relatedness scores may be computed by other algorithms. For example, the text obtained from the information source (e.g. the synopsis from the EPG) and the Wikipedia page of a candidate named entity may both be treated as bags-of-words and the words shared by both of them are counted: The higher the count, the higher the content-item-relatedness score assigned to the candidate.
In some embodiments, a second mode of NED implementation operates by performing the following steps:
In step a, the obtaining of the text can be from subtitles that are part of the video content item and can be displayed on the screen, providing either the text spoken in the video content item or a translation of it. Alternatively, the obtaining of the text can be from an automatic conversion of the spoken text to written text by a speech-to-text conversion engine.
In step b, the identifying of the one or more named entities is done according to the prior art methods of Named Entity Recognition.
In step c, the determining of which named entity is ambiguous is done according to the prior art methods of Named Entity Disambiguation.
In step e, video data of the video content item from which the analyzed text was taken is obtained. In some embodiments, the step is carried out by the local terminal (e.g. smart TV or STB). If the user picked his currently playing content item from local content stored in the local terminal, then the video data is available to the processor of the local terminal from the local storage device of the local terminal. But if the user picked the currently playing content from a VOD library or from a linear channel of the TV operator, the content item is typically streamed in real-time from the TV operator and is not usually locally stored by default. In such case it is preferable that a local terminal implementing this step of the NED algorithm (or at least part of it) records the streamed video content, possibly using a connected device with digital video recording functionality, so that it is available to the local terminal for analysis. The storing does not have to last long, as the video content is required to be available only until the disambiguating of the currently playing video segment ends. If any of the next steps, which require having access to the video data of the video content item, is carried out by the TV operator's server and the video data is not accessible to the server (e.g. when the video content item is picked from local storage), then the local terminal provides the video data to the TV operator's server.
In other embodiments, the step is carried out by the TV operator's server. If the user picked his currently playing content item from a VOD library or from a linear TV channel, then the TV operator's server has access to video data of the currently playing video content item. If the user picked his currently playing content item from local storage, then the server obtains the video data from the local terminal. If any of the next steps which requires having access to the video data of the video content item is carried out by the local terminal and the video data is not accessible to the local terminal (e.g. when the video content item is picked from a VOD library and the local terminal does not record it), then the server provides the video data to the local terminal.
In step f, for each ambiguous reference to a named entity in the analyzed text, video data of a video segment corresponding in time to the appearance of the reference in the analyzed text is analyzed, in order to determine the relatedness or connection between the video segment and each candidate for the ambiguous reference to a named entity in the analyzed text.
For example, when the ambiguous reference is known to refer to a person and the multiple candidates are different people who may match the ambiguous reference (i.e. having the same name), we may obtain pictures of each of the candidates (for example: from Wikipedia). Then a visual analysis of the video segment (or of a portion of it or of one or more still images of the segment that serve as a representative sample of the segment) is performed in order to find out (for each candidate) whether the relevant picture (or a similar picture) appears somewhere in the segment. A relatedness score may be assigned to each candidate, when a good match between the video segment and the candidate's picture gets a high score and a weak match gets a low score.
As another example, when the ambiguous reference is known to refer to a location (e.g. a town, a neighborhood, a street, etc.) and the multiple candidates are different locations that may match the ambiguous reference (i.e. towns in different states having the same name, neighborhoods in different towns having the same name, streets in different neighborhoods having the same name, etc.), we may search the video data for textual information that may shed light on the intended named entity. For example, if the candidates are streets of the same name that exist in multiple cities, then the analysis of the video data may detect a street sign or a shop sign that discloses the name of the city, thus resolving the correct identification of the ambiguous reference. Or alternatively, the visual analysis may try to match a picture of a candidate location (for example obtained from Wikipedia) to visual scenery shown in the video segment.
It should be noted that it is not always the case that the class of the ambiguous reference is known, as was the case in the previous examples. For example, it may be the case that the ambiguous reference may be either a person or a location, as in the “Washington” example previously mentioned. In such case the analysis of the visual content might require applying multiple methods, both those applicable for people and those applicable for locations.
As stated above, this step determines relatedness or connection between a candidate and the relevant video segment. By “connection” it is meant here reaching a result that specifies the similarity or relevance by a value that is one of a discrete set of possible values having no more than three alternatives. For example {Yes, No} or {Yes, No, Undecided}. By “relatedness” it is meant here reaching a result that specifies the similarity or relevance in finer detail on a scale having more than three possible values. This may be an analog scale or a digital scale having at least four possible values.
It should be noted that the analyzed text might not have a detectable correlation to a narrow video segment that directly corresponds in time to an appearance of an ambiguous reference to a named entity currently under analysis. It might be, for example, that a person is shown in the video content at a specific moment but the audio track mentions his name only after a minute or two. Or that a location is shown in the video content at a specific moment but the audio track mentions its name one or two minutes earlier. Therefore, it is important that the selection of the video segment to analyze for a given ambiguous reference to a named entity (i.e. the video segment corresponding in time to the mentioning of the relevant reference in the analyzed text) will be broad enough to contain early and late occurrences of the reference in the video content. On the other hand, it is undesirable to select the analyzed video segment too broadly, because that might badly affect the performance of the system because of the heavier processing required for analyzing a longer video segment. Reasonable coverage can typically be obtained with segments that are between plus/minus ten seconds to plus/minus three minutes around the ambiguous reference appearance. The video segment to analyze need not necessarily be symmetric around the ambiguous reference appearance, for example covering an interval starting one minute before and ending two minutes after the appearance of the reference.
It should also be noted that if the currently playing content item is streamed in real-time from the TV operator and is not locally stored, then there is difficulty in providing both real-time analysis of the visual content and symmetry of the analyzed video segment around the ambiguous reference appearance. This is because at the time of appearance of the ambiguous reference the video immediately following it is still not available.
This issue may be resolved, for example, in one of the following two ways:
i. Delaying the NED decision. For example, if the video segment analyzed for relatedness to a candidate is plus/minus one minute around the appearance of the named entity, then carrying out the analysis with a one-minute delay, so that the desired two-minute segment can be analyzed.
ii. Giving up detection of early appearances of the named entity. With this approach the analysis takes place immediately with or closely after the appearance of the ambiguous reference in the analyzed text, and only the portion of the relevant video segment that is already known at that time is used. This implies that if the audio track mentions the named entity before it appears in the visual content, then the matching might not be detected. In some embodiments, it is thus possible that the appearance of the named entity will fall outside the time interval corresponding to the analyzed video segment. In other words, it is possible that the analyzed segment of text in which the ambiguous reference is found will fully or partially fall outside the time interval corresponding to the analyzed video segment.
In step g, use is made of the relatedness or connection information found in the previous step, in a way that enhances the NED algorithm. The examples and explanations below provide more details about this step, but do not limit the above statement or the scope of the invention.
For example, when the ambiguous reference is known to refer to a person and there are three persons that are candidates for matching the reference to a named entity, the visual analysis step may determine a binary conclusion for the appearance of pictures of each of the candidates in the video content, resulting for example in a {No, Yes, No} visual-connections vector. In this case the visual analysis clearly directs us to a conclusion that the second candidate is the right one, as it is the only person out of the three whose face appears in the video segment. Therefore, the NED implementation may give heavy weight to the visual analysis and use it as the sole deciding factor.
If, however, the connections vector was found to be {No, Yes, Yes}, then the NED algorithm may conclude the correct solution is either the second or third candidate, but certainly not the first. In this case the decision cannot be taken based solely on the visual analysis, and the NED algorithm would have to apply its non-visual methods for deciding.
As another example, when the ambiguous reference is known to be a location and there are three locations that are candidates for matching the reference to a named entity, the visual analysis step may use a high-resolution relatedness scale for measuring similarity between images of the candidate locations and the video content, resulting for example in a {0.34, 0.56, 0.49} visual-relatedness vector. In this case the visual analysis does not provide a clear-cut conclusion, and the NED algorithm should treat the visual analysis as one more input contributing to the decision process, on top of all other inputs used in prior art NED algorithms.
We will now present detailed examples of how, according to various embodiments, content-item-relatedness scores and/or visual-relatedness scores may be used for disambiguating an ambiguous reference to a named entity appearing in a sound track of a video content item, which disambiguating is in turn used for content enrichment decisions.
In a first example illustrating, inter alia, the use of content-item-relatedness scores, a reference to the named entity “Washington” is detected to exist in the sound track of the movie “Disclosure” watched by a user. For the sake of simplicity, we assume that this is the only reference to a named entity detected within a currently analyzed segment of text (e.g. corresponding to a one-minute long audio segment taken from the audio track of the movie).
The content enrichment process for the first example may then include the following steps:
In the above example equal importance is given to relatedness of the candidates to the context of the reference to the ambiguous named entity and to relatedness of the candidates to the information source directly associated with the video content item in which the reference appears. It should be noted that the two relatedness scores measure different characteristics. A context-relatedness score measures a “local” relatedness that represents relatedness between a candidate interpretation and the immediate vicinity of the reference. A content-item-relatedness score measure a “global” relatedness that represents relatedness between a candidate interpretation and the video content item as a whole.
Therefore, the context-related scores tend to be “jumpy” and change with time while a video content item is playing, while the content-item-related scores are stable while playing a single video content item. For many content enrichment applications, stability of recommendations is a desired feature. In such cases the content-item-relatedness scores should get a heavier weight than the context-relatedness scores. For example, the content-item-relatedness scores may be given three times the weight of the context-relatedness scores. Applying this weight to the example above results in total-relatedness scores of {210, 290, 70, 70}. In this example the use of weights did not change the bottom-line conclusion of selecting Washington State, but it is easy to realize that in other examples the use of weights may result in a different selection than when no weights are used.
It should be noted that the proposed solution of improving the effectiveness of NED tasks is applicable to all types of NED algorithms, whether semantic or not. However, as explained above, the solution is intended for NED tasks which serve the goal of enhancing the viewing experience of video content items, when the sound tracks of the video content items contain the references to named entities which should be disambiguated.
It should also be noted that the language of the information source directly associated with the video content item may be different from the language of the text that is being analyzed. For example, the video content item may be an episode of the American television show “Friends” with a Spanish sound track, so that the analyzed text is in Spanish. The information source selected for supporting the disambiguation task may be the “Friends” page in the English Wikipedia.
In such a case it is preferred, even though not absolutely necessary, that for the computation of the content-item-relatedness scores, the texts corresponding to the candidate named entities will be in the language of the information source, so that the same relatedness calculation algorithms used for the single-language case will also be applicable without modifications. In the above example, if the content-item-relatedness scores are computed by comparing the text obtained from the information source to Wikipedia pages of the candidates, then it is preferred to use the English Wikipedia pages of the candidates for that purpose, even if the Spanish Wikipedia pages of the candidates are the ones used for determining the context-relatedness scores of the candidates.
In a second example illustrating, inter alia, the use of visual-relatedness, the example shows that it can be that visual-relatedness scores alone do not provide a clear-cut conclusion, in which case additional relatedness inputs are required (as in the previous example). The additional inputs in this example are context-relatedness scores, which indicate the level of relatedness of a candidate (e.g. the Wikipedia page corresponding to the candidate) to the context in which the ambiguous reference appears (e.g. the paragraph of the analyzed text containing the ambiguous reference). Algorithms for calculating relatedness scores between two blocks of text are well known in the art, for example based on treating both texts as bags-of-words and counting the words shared by both texts. The larger is the count, the higher is the relatedness score.
In this second example, a reference to the named entity “Washington” is again (i.e., like in the first example) detected to exist in the sound track of the movie “Disclosure” watched by a user. For the sake of simplicity, we assume that this is the only reference to a named entity detected within a currently analyzed segment of text (e.g. corresponding to a one minute long audio segment taken from the audio track of the movie).
The content enrichment process for the second example may then include the following steps:
Steps a through d—the same as in the first example above.
e. A video segment is extracted from the movie “Disclosure”, the video segment including the point in time in which the reference to “Washington” appears.
f. A visual-relatedness score is computed for each of the four remaining candidates, measuring its relatedness to the visual content of the extracted video segment. In order to determine these scores, the Wikipedia pages of all four candidates are accessed and relevant pictures are downloaded—a picture of George Washington, a picture of the flag of Washington State, a picture of the seal of Washington state, an aerial view of Seattle, a view of Downtown Seattle from Queen Anne Hill, a picture of the flag of Washington D.C., a picture of Lincoln Memorial in Washington D.C., a picture of the White House, a picture of the Capitol, a picture of Arlington National Cemetery, etc. Each of the downloaded pictures is matched against the extracted video segment, looking for appearances of the picture or part of it in the video segment, including geometrically distorted appearances.
In addition, the extracted video segment is analyzed for appearances of Latin characters. Such appearances may provide hints for the correct interpretation of the ambiguous reference.
In this example, the flag of Washington State is found to appear in three locations within the video segment, the view of downtown Seattle appears in two locations, the picture of George Washington appears twice, and all the other pictures are not detected at all. Additionally, the characters “Seat” are determined to appear on a building in one location in the video segment. Based on those findings, the resulting visual-relatedness scores for {president, state, capital, metropolitan area} are determined to be {50, 90, 0, 0} on a scale of 0 to 100.
g. The context-relatedness scores and the visual-relatedness scores are combined into one set of total-relatedness scores. This can be done by averaging the two scores for each remaining candidate, resulting in {55, 70, 12.5, 5}. Alternatively and equivalently, this can be done by adding the two scores for each remaining candidate, resulting in {110, 140, 25, 10}. As each of the numbers in the last computation is by definition equal to twice its corresponding number in the previous result, both computations always lead to the same end result, which in this example is selection of the second candidate interpretation (state) to be the correct one, as it has the highest total-relatedness score of all remaining candidates.
h. As in the first example, it is now determined that the reference to “Washington” is a reference to Washington State, and the content enrichment engine looks for one or more media content items that are related to Washington State in order to recommend them to the user. Once again, in this example the media content items that are proposed to the user are the movies “Free Willy” and “Sleepless in Seattle”, which were either (at least partially) filmed in Washington State or have a plot that (at least partially) occurs in Washington State.
In the above second example, equal importance is given to relatedness of the candidates to the context of the ambiguous reference to the named entity and to relatedness of the candidates to the visual image of the video content item in which the reference appears. It should be noted that the two relatedness scores measure different characteristics. A context-relatedness score measures an “aural” relatedness that represents relatedness between a candidate interpretation and the audio track of the video content item. A visual-relatedness score measures a “visual” relatedness that represents relatedness between a candidate interpretation and the video track of the video content item.
For many content enrichment applications, visual relatedness is considered more important than aural relatedness. In such cases the visual-relatedness scores should get a heavier weight than the context-relatedness scores. For example, the visual-relatedness scores may be given three times the weight of the context-relatedness scores. Applying this weight to the example above results in total-relatedness scores of {210, 320, 25, 10}. In this example the use of weights did not change the bottom-line conclusion of selecting Washington State, but it is easy to realize that in other examples the use of weights may result in a different selection than when no weights are used.
It should be noted that the proposed solution of improving the effectiveness of NED tasks is applicable to all types of NED algorithms, whether semantic or not. As explained above, the solution is intended for NED tasks which serve the goal of enhancing the viewing experience of video content items, when the sound tracks of the video content items contain the references to named entities which should be disambiguated.
Methods of analyzing visual content for identifying appearances of text characters are well known in art, and any such method may be used in implementing the proposed solution. Examples for such methods can be seen in U.S. Pat. Nos. 6,608,930, 6,823,084, 7,031,553, 8,320,674, and US Patent Publication 2011/0123115, all of which are incorporated herein by reference in their entirety.
Methods of analyzing visual content for identifying appearances of faces are well known in the art, and any such method may be used in implementing the proposed solution. Examples for such methods can be seen in U.S. Pat. Nos. 5,012,522, 5,715,325, US Patent Publication 2008/0080743, and US Patent Publication 2012/0250950, all of which are incorporated herein by reference in their entirety.
Methods of performing general-purpose Named Entity Disambiguation that are not specific to enhancing the viewing experience of video content items are well known in the art, and any such method may be used in implementing the proposed solution. Examples for such methods can be seen in U.S. Pat. No. 7,672,833, US Patent Publication 2007/0233656, US Patent Publication 2008/0208864, US Patent Publication 2009/0144609, and US Patent Publication 2011/0106807, all of which are incorporated herein by reference in their entirety.
It should be clear to the skilled practitioner that parallel elements of the two detailed examples can be combined in various ways. For example, for some content enrichment applications, a total relatedness score can be a function of all three of: a context-relatedness score, a content-relatedness score, and a visual-relatedness score. As another example, if two (or more) candidates are tied after the tallying of a total relatedness score based on a content-relatedness score and a context-relatedness score, as used in the first detailed example, then a visual-relatedness score calculated as described in the second detailed example might be used as a ‘tie-breaker’ in order to complete the disambiguating. Similarly, if two (or more) candidates are tied after the tallying of a total relatedness score based on a visual relatedness score and a context-relatedness score, as used in the second detailed example, then a content-relatedness score calculated as described in the first detailed example might be used as a ‘tie-breaker’ in order to complete the disambiguating.
Referring now to the figures, and specifically to
As shown in
The one or more media server(s) 110 is/are operable to (i) provide at least a portion of a video content item 201 to the client terminal 140, so as to cause playing the at least a portion of the video content item 201 on the screen of the client terminal 140, and (ii) provide a media content item 202, so as to cause displaying the media content item 202 on the screen of the client terminal 140. In some embodiments, as illustrated in
As also shown in
The exemplary system illustrated in
In some other embodiments, the one or more computer processors 145 are operative to execute program instructions 161 stored in the storage medium 130. The program instructions 161, which are represented schematically in
The exemplary system illustrated in
Referring now to
The disambiguating can be done in parallel to the playing of the at least a portion of the video content item of Step S01. The information source directly associated with the identified video content item can be, to name a few non-limiting examples, a synopsis (which can be obtained from an electronic program guide or from a video-on-demand library), an advertisement, or a review. Other non-limiting examples include: an entry in a database, a knowledge base, a dictionary or an encyclopedia (where the entry corresponds to the identified video content item), including, for example IMDB or Wikipedia, and an account dedicated to the identified video content item in a social network. In some embodiments, the information source directly associated with the identified video content item can be in a different language than the segment of text obtained in Step S02.
Any of the steps in the method, and in fact any of the steps in any of the methods disclosed herein, can be implemented either in the system 100, i.e., on the server side, or in the client terminal.
In some embodiments, as illustrated in the flow chart of
In some embodiments, as illustrated in
Referring now to
In some embodiments, as illustrated in the flow chart of
In some embodiments, as illustrated in
The present invention has been described using detailed descriptions of embodiments thereof that are provided by way of example and are not intended to limit the scope of the invention. The described embodiments comprise different features, not all of which are required in all embodiments of the invention. Some embodiments of the present invention utilize only some of the features or possible combinations of the features. Variations of embodiments of the present invention that are described and embodiments of the present invention comprising different combinations of features noted in the described embodiments will occur to persons skilled in the art to which the invention pertains.
This disclosure should be interpreted according to the definitions below.
In case of a contradiction between the definitions in this Definitions section and other sections of this disclosure, this section should prevail.
In case of a contradiction between the definitions in this section and a definition or a description in any other document, including in another document included in this disclosure by reference, this section should prevail, even if the definition or the description in the other document is commonly accepted by a person of ordinary skill in the art.
dd. “identifying a media content item”—finding an attribute of the media content item that allows searching for and locating the media content item or information related to the media content item. The attribute may be a name of the media content item, a web address (URL) pointing to a copy of the media content item, an identification number of the media content item within a given library or collection of media content items, an identification number of the media content item that is globally unique, etc. The attribute value does not have to be globally unique as long as it is practically possible to use it for the searching and locating. For example, a movie can be identified by a name even where there are several movies having the same name, as long as it is possible to search in some database or search engine using the name and thus find the media content item or information related to it (even if a manual selection between several alternatives is required).
This patent application claims the benefit of U.S. Provisional Patent Application No. 62/530,905 filed on Jul. 11, 2017, and U.S. Provisional Patent Application No. 62/528,104 filed on Jul. 2, 2017, both of which are incorporated herein by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5012522 | Lambert | Apr 1991 | A |
| 5715325 | Bang et al. | Feb 1998 | A |
| 6608930 | Agnihotri et al. | Aug 2003 | B1 |
| 6823084 | Myers et al. | Nov 2004 | B2 |
| 7031553 | Myers et al. | Apr 2006 | B2 |
| 7386438 | Franz | Jun 2008 | B1 |
| 7672833 | Blume et al. | Mar 2010 | B2 |
| 8320674 | Guillou et al. | Nov 2012 | B2 |
| 8700604 | Roseman et al. | Apr 2014 | B2 |
| 8713604 | Newell | Apr 2014 | B2 |
| 9137580 | Scott et al. | Sep 2015 | B2 |
| 9462342 | Krishnamurthy | Oct 2016 | B2 |
| 9542395 | Agarwal et al. | Jan 2017 | B2 |
| 9582482 | Sharifi et al. | Feb 2017 | B1 |
| 10225625 | Homyack | Mar 2019 | B2 |
| 20030217052 | Rubenczyk et al. | Nov 2003 | A1 |
| 20050049852 | Chao | Mar 2005 | A1 |
| 20050080613 | Colledge et al. | Apr 2005 | A1 |
| 20050216443 | Morton et al. | Sep 2005 | A1 |
| 20050251382 | Chang et al. | Nov 2005 | A1 |
| 20060010138 | Huerta et al. | Jan 2006 | A1 |
| 20060136196 | Brun et al. | Jun 2006 | A1 |
| 20060136208 | Chung | Jun 2006 | A1 |
| 20060136385 | Bobrow et al. | Jun 2006 | A1 |
| 20060149555 | Fabbrizio et al. | Jul 2006 | A1 |
| 20060247983 | Dalli | Nov 2006 | A1 |
| 20070016580 | Mann et al. | Jan 2007 | A1 |
| 20070067285 | Blume | Mar 2007 | A1 |
| 20070106493 | Sanfilippo et al. | May 2007 | A1 |
| 20070169148 | Oddo et al. | Jul 2007 | A1 |
| 20070214189 | Groble | Sep 2007 | A1 |
| 20070233656 | Bunescu et al. | Oct 2007 | A1 |
| 20080040352 | Ellis | Feb 2008 | A1 |
| 20080065621 | Ellis | Mar 2008 | A1 |
| 20080071519 | Brun et al. | Mar 2008 | A1 |
| 20080080743 | Schneiderman et al. | Apr 2008 | A1 |
| 20080082510 | Wang | Apr 2008 | A1 |
| 20080126076 | Ming et al. | May 2008 | A1 |
| 20080154871 | Leidner et al. | Jun 2008 | A1 |
| 20080208864 | Cucerzan et al. | Aug 2008 | A1 |
| 20080301112 | Wu | Dec 2008 | A1 |
| 20080319978 | Brun et al. | Dec 2008 | A1 |
| 20090019474 | Robotham | Jan 2009 | A1 |
| 20090144609 | Liang et al. | Jun 2009 | A1 |
| 20090157705 | Nomiyama | Jun 2009 | A1 |
| 20090164431 | Livkovic et al. | Jun 2009 | A1 |
| 20090192968 | Tunstall-Pedoe | Jul 2009 | A1 |
| 20090204596 | Brun et al. | Aug 2009 | A1 |
| 20090319257 | Blume et al. | Dec 2009 | A1 |
| 20100004925 | Ah-Pine et al. | Jan 2010 | A1 |
| 20100046842 | Conwell | Feb 2010 | A1 |
| 20100076972 | Baron et al. | Mar 2010 | A1 |
| 20100145678 | Csomai et al. | Jun 2010 | A1 |
| 20100145902 | Boyan et al. | Jun 2010 | A1 |
| 20100185689 | Hu et al. | Jul 2010 | A1 |
| 20100235313 | Rea et al. | Sep 2010 | A1 |
| 20100293195 | Houghton | Nov 2010 | A1 |
| 20110106807 | Srihari et al. | May 2011 | A1 |
| 20110123115 | Lee et al. | May 2011 | A1 |
| 20110125735 | Petrou | May 2011 | A1 |
| 20110225155 | Roulland et al. | Sep 2011 | A1 |
| 20110246076 | Su et al. | Oct 2011 | A1 |
| 20110246442 | Bartell | Oct 2011 | A1 |
| 20110258556 | Kiciman | Oct 2011 | A1 |
| 20110282888 | Koperski et al. | Nov 2011 | A1 |
| 20120102045 | Cucerzan et al. | Apr 2012 | A1 |
| 20120117078 | Morton et al. | May 2012 | A1 |
| 20120203772 | Cucerzan et al. | Aug 2012 | A1 |
| 20120250950 | Papakipos et al. | Oct 2012 | A1 |
| 20120271624 | Chen et al. | Oct 2012 | A1 |
| 20120324350 | Rosenblum et al. | Dec 2012 | A1 |
| 20130080152 | Brun et al. | Mar 2013 | A1 |
| 20130166303 | Chang et al. | Jun 2013 | A1 |
| 20130173604 | Li et al. | Jul 2013 | A1 |
| 20130198268 | Hyman | Aug 2013 | A1 |
| 20130238312 | Waibel | Sep 2013 | A1 |
| 20130275438 | Ajmera et al. | Oct 2013 | A1 |
| 20130311467 | Galle et al. | Nov 2013 | A1 |
| 20130346421 | Wang et al. | Dec 2013 | A1 |
| 20140039879 | Berman | Feb 2014 | A1 |
| 20140046653 | Gopalakrishnan et al. | Feb 2014 | A1 |
| 20140074886 | Medelyan et al. | Mar 2014 | A1 |
| 20140101542 | Albrecht et al. | Apr 2014 | A1 |
| 20140136184 | Hatsek et al. | May 2014 | A1 |
| 20140142922 | Liang et al. | May 2014 | A1 |
| 20140195532 | Dheap et al. | Jul 2014 | A1 |
| 20140214820 | ODonnell | Jul 2014 | A1 |
| 20140282219 | Haddock | Sep 2014 | A1 |
| 20140297252 | Prasad et al. | Oct 2014 | A1 |
| 20140316768 | Khandekar | Oct 2014 | A1 |
| 20140337372 | Lee et al. | Nov 2014 | A1 |
| 20150081281 | Bustelo et al. | Mar 2015 | A1 |
| 20150095306 | Zivkovic et al. | Apr 2015 | A1 |
| 20150106157 | Chang | Apr 2015 | A1 |
| 20150161237 | Agarwal et al. | Jun 2015 | A1 |
| 20150179228 | Oisel | Jun 2015 | A1 |
| 20150212697 | Nordstrom | Jul 2015 | A1 |
| 20150269139 | McAteer et al. | Sep 2015 | A1 |
| 20150286629 | Abdel-Reheem et al. | Oct 2015 | A1 |
| 20150296228 | Chen | Oct 2015 | A1 |
| 20150312633 | Masciarotte | Oct 2015 | A1 |
| 20150331850 | Ramish | Nov 2015 | A1 |
| 20150332049 | Chen et al. | Nov 2015 | A1 |
| 20150334446 | Krikorian et al. | Nov 2015 | A1 |
| 20160012020 | George et al. | Jan 2016 | A1 |
| 20160012021 | Rosenblum et al. | Jan 2016 | A1 |
| 20160012040 | Hamada | Jan 2016 | A1 |
| 20160048655 | Maitra et al. | Feb 2016 | A1 |
| 20160055845 | Ikawa et al. | Feb 2016 | A1 |
| 20160078245 | Amarendran et al. | Mar 2016 | A1 |
| 20160085740 | Ikawa et al. | Mar 2016 | A1 |
| 20160110350 | Waibel | Apr 2016 | A1 |
| 20160117360 | Kunc et al. | Apr 2016 | A1 |
| 20160124937 | Elhaddad | May 2016 | A1 |
| 20160188597 | Moore | Jun 2016 | A1 |
| 20160203130 | Roque et al. | Jul 2016 | A1 |
| 20160275148 | Jiang | Sep 2016 | A1 |
| 20160306789 | Cougias et al. | Oct 2016 | A1 |
| 20160306984 | Amarendran et al. | Oct 2016 | A1 |
| 20160335234 | Baker et al. | Nov 2016 | A1 |
| 20160337609 | Zhou | Nov 2016 | A1 |
| 20170011092 | Huddleston et al. | Jan 2017 | A1 |
| 20170026705 | Yeh et al. | Jan 2017 | A1 |
| 20170039272 | Roseman et al. | Feb 2017 | A1 |
| 20170041644 | Dalrymple et al. | Feb 2017 | A1 |
| 20170060835 | Radford et al. | Mar 2017 | A1 |
| 20170085947 | Garrison et al. | Mar 2017 | A1 |
| 20170124065 | Cougias et al. | May 2017 | A1 |
| 20170147557 | Ajmera et al. | May 2017 | A1 |
| 20170147635 | McAteer et al. | May 2017 | A1 |
| 20170147924 | Ajmera et al. | May 2017 | A1 |
| 20170153782 | Sharifi et al. | Jun 2017 | A1 |
| 20170161367 | Kemp | Jun 2017 | A1 |
| 20170161796 | Clark | Jun 2017 | A1 |
| 20170243244 | Trabelsi et al. | Aug 2017 | A1 |
| 20170251265 | Perez | Aug 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 3127339 | Jan 2017 | EP |
| 2452519 | Mar 2009 | GB |
| 2007063468 | Jun 2007 | WO |
| 2007078846 | Jul 2007 | WO |
| 2012051620 | Apr 2012 | WO |
| WO2013082142 | Jan 2017 | WO |
| 2017149447 | Sep 2017 | WO |
| Entry |
|---|
| Co-pending U.S. Appl. No. 15/935,000, filed Mar. 25, 2018. |
| Co-pending U.S. Appl. No. 16/053,106, filed Aug. 2, 2018. |
| Co-pending U.S. Appl. No. 16/120,972, filed Sep. 4, 2018. |
| Co-pending U.S. Appl. No. 16/149,175, filed Oct. 2, 2018. |
| Co-pending U.S. Appl. No. 16/223,149, filed Dec. 18, 2018. |
| Co-pending U.S. Appl. No. 16/228,821, filed Dec. 21, 2018. |
| Number | Date | Country | |
|---|---|---|---|
| 20190007711 A1 | Jan 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62528104 | Jul 2017 | US | |
| 62530905 | Jul 2017 | US |
