Patent Grant
10796089
The following discussion generally relates to the processing of closed captioning or other timed text information associated with television broadcasts, media streams and/or other media programming.
Viewers now watch television content received from many different sources. In addition to conventional direct broadcast satellite (DBS), cable or terrestrial television broadcasts, content is now available from any number of streaming services (e.g., video on demand (VOD) services) and the like. Moreover, many viewers now make use of time shifting devices (e.g., local storage or remote storage digital video recorders (DVRs)) and/or placeshifting devices. In a typical placeshifting setting, a placeshifting device encodes a live media stream for real-time (or even faster than real-time) delivery to a viewer. The live stream may encode a previously-stored program, a live broadcast, and/or any other content as desired. With the advent of these new technologies, media content is now viewable not only on televisions, but also on mobile phones, tablets, computer systems, video game players, media players and any number of other devices.
With the emergence of new media player types and capabilities, it is now possible to provide additional services and/or to enhance the viewer's media experience. Further, modern “smart” televisions are capable of providing more advanced features beyond mere playback of received broadcasts. It is therefore desirable to provide systems, devices and/or processes that improve the experience of viewing media content on a television or other device. These and other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background section.
According to various exemplary embodiments, systems, devices and methods are described to process timed text that is associated with television, media stream and/or other video programming.
Various embodiments provide a process executable by a media player device having a processor, memory and an interface to a network to process timed text that is associated with television programming. The process suitably comprises: receiving, at the media player, a program stream that comprises video content associated with a particular program and a timed text stream that comprises textual data associated with the particular program; providing the video content associated with the particular program in a first window to a display for presentation to the viewer; and simultaneously providing the timed text stream in a second window different from the first window for presentation to the viewer, wherein the timed text stream is presented in the second window to be scrollable such that the viewer is able to scroll back to view dialog occurring prior to a current scene of the particular program.
Other embodiments relate to processes to automatically replace certain words (e.g., slang) within the timed text stream, or to enhance the timed text stream with links or other additional information from a web site or other source. Still other embodiments relate to processes by which the media player, television receiver or another computing device is able to automatically construct a summary of a program from the timed text data of the program. The automatically-generated summary may be presented to the viewer as the program is in progress so that the viewer is quickly brought current as to earlier developments in the program, for example.
Still other embodiments relate to media player applications and devices that implement the various processes described herein, or to other computing systems and devices that implement the various automated processes set forth herein.
Additional embodiments, aspects and other features are described in more detail below.
Example embodiments will hereinafter be described in conjunction with the following drawing figures, wherein like numerals denote like elements, and
The following detailed description of the invention is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any theory presented in the preceding background or the following detailed description.
Timed text that is provided in a television broadcast or media stream can be used to provide an improved user experience. A scrollable text window that presents the timed text can be provided, for example, that can allow the user to quickly “catchup” from a missed moment or to see a word or phrase that might have been difficult to understand. In some embodiments, timed text may be enhanced to allow URLs or other links to dictionaries, Wikipedia, online sources, thesauruses, translating services, and/or the like. Further embodiments could use automated tools to automatically generate program summaries for watched content. Various features may be particularly useful in a placeshifting setting wherein media streams are encoded and transmitted across a network for remote viewing; other embodiments may be used in set top boxes, television receivers, media players and/or other devices and systems as desired.
According to various examples, processes for presenting one or more running streams of closed captioning (CC) or other timed text (TT) data in vertical or other windows are described. These windows may appear similar to an instant message chat window, in some embodiments, and may be shown along with the running video on a media player client (e.g. a mobile phone, tablet or other device) or other display as desired. The timed text window may be activated or deactivated by the user, as desired.
The caption transcript can be used by the user to quickly get a grasp of any missed conversation, dialog or other events in the video without the need to seek backward to the missed location in the video (which can be disruptive to other viewers). This may be particularly useful for viewers who are not fluent in the language of the program, since may such viewers may be able to read certain languages at a faster rate and/or a more fluent level than they can listen and understand spoken language. The caption transcript may also be helpful in recognizing or comprehending slang terms that even a more fluent speaker may not understand.
In further embodiments, the transcript can be post-processed so that interesting keywords are detected and highlighted as smart tags that provide a quick info/link/glossary for the word. That is, the viewer may be able to obtain more information about a word or phrase that is presented in the TT content. The word may be a player name in a sporting event, for example, which could link to sports statistics or other information about that player. Other embodiments could simply link to thesaurus or dictionary entries for certain words, or to synonyms for certain words or phrases to improve reader/viewer comprehension. To that end, some embodiments could connect a keyword in the caption text to an HTML, XML and/or other link that is clickable by the user to open a web search, dictionary entry, Wikipedia or other informational webpage, and/or other information about the linked word.
The thesaurus function could be further automated, as desired. In some embodiments, certain words (e.g., slang) may be replaced entirely with words that have clearer meanings. If a movie or TV program uses American slang, for example, certain words may be ambiguous or unclear when the program is broadcast or streamed in other countries. Even fluent English speakers may not be used to current slang from other geographic locations, so certain words that would be unfamiliar or otherwise difficult to understand in the local region could be automatically replaced in the caption window with words that are more common or that are more likely to be understood. Difficult words can therefore be replaced with other words in the same or a different language to make the meaning clearer.
Various embodiments alternately or additionally provide an automatic transcript feature that is prepared from the caption text. This transcript can be further processed using automated tools to generate a summary description of the watched content. Such summaries may be especially useful for sports programs, although automatic summaries could be equivalently created for other programs as well.
To that end, a typical television viewer often subscribes to multiple channels each having different types of programs (sports, movies, games, comedies etc.). If a user is interested to know about a particular program, the electronic program guide (EPG) typically provides only a vague description about the program in just a few lines of text that may be inadequate to satisfy the viewer's desire for information about the program. Moreover, if a user joins a particular program that is already in progress, he or she has no idea of the sequence of events that may have occurred during the earlier programming. By generating a summary (e.g., a précis) that is extracted from the close captioning or the subtitles associated with the program, a more detailed description of the part of the program that the user has missed can be provided. This summary can be generated by the television receiver or media client, if desired, and presented to potential viewers before or during a broadcast to give a more accurate description of the content. Other embodiments could generate automatic summaries at a separate server (e.g., a network server) that distributes summaries as desired. Each of these different embodiments is described more fully below.
Turning now to the drawing figures and with initial reference to
The example system 100 illustrated in
Placeshifting device 110 is similarly implemented using conventional hardware, such as a processor 111, memory 112 and input/output interfaces 113 as desired to implement the various functions described herein. Typically, placeshifting device 110 executes a placeshifting server application 116 that controls the encoding of video and the delivery of encoded video to the remote player device 110. In various embodiments, placeshifting device no is a standalone device such as any of the various SLINGBOX devices available from Sling Media Inc. of Foster City, Calif., although any number of other devices could be equivalently used. Other placeshifting devices no are integrated within set top boxes (STBs), television receivers, media players and/or other consumer-type hardware components that are available from any number of different sources.
Generally speaking, the placeshifting device 120 receives one or more media streams from a content source (e.g., a DVR, television receiver, stored media player, VOD source, and/or the like). Device 120 suitably converts the received stream(s) into a format that can be routed over network 125 to media player application 126 executing on media player device 120.
In the example of
The media player application 126 suitably receives the streams 130-132, decodes the content, and presents the content as appropriate. In various embodiments, timed text streams are presented in separate windows from the video content; these separate windows may be activated or deactivated by the viewer as desired. In various embodiments, each of the timed text windows suitably includes a scroll feature that allows the viewer to scroll back and view dialog or other events that have already occurred. Various embodiments could present multiple caption windows so that multiple programs can be monitored on the same display. Still other embodiments may use the secondary text streams to generate summaries of programs that may be viewed in the future, or for other purposes as desired.
Caption data may be processed using any format. All broadcast television programming in the United States and many other nations must now include closed caption information. For conventional digital (e.g., ATSC) television broadcasts, caption streams are typically encoded in a well-known format such as EIA-608 format, EIA-708 format, and/or the like. Other closed captioning formats or standards may be implemented in the United States or elsewhere.
Closed captioning typically refers to text delivered in conjunction with a television program or other programming content that allows viewers to access additional or interpretive information about programming content. Closed captions often provide textual transcriptions of the audio portions of programs as the programs progress, thereby allowing the hearing impaired and others to follow the program content without listening to the audio track typically associated with the programming. Motion pictures, video disk (e.g., DVD, Blu-ray) content, streamed audio/video, video games and the like may similarly incorporate closed captions using any number of standard or non-standard techniques. Closed caption/timed text often uses the presentation time stamps (PTSs) of conventional MPEG bundles to synchronize the text to the rest of the media program, although other standards may use other synchronization features, as desired.
As more and more consumers receive their television programming from broadband sources, the Society of Motion Picture and Television Engineers (SMPTE) and others have recommended a transition from the older standards (e.g., EIA 608/708) to more advanced “timed text” formats such as SMPTE-TT. This newer timed text format incorporates syntax, structures and tags similar to the extensible markup language (XML) to provide more advanced features and to improve flexibility in comparison to the more traditional legacy formats. The SMPTE-TT format is described in SMPTE Standard ST 2052-1-2010, which was approved on Dec. 3, 2010. The concepts set forth herein may be applied using any type of timed text/closed caption formatting, as appropriate.
The example 200 of
The summary window 210 presents summary data that may be automatically generated at media player 120 and/or provided from any other source. An example process to create a summary is presented below with respect to
Timed text is transmitted to the media player 120 in any manner. In the example of
The media player application 126 decodes the received content and formats an appropriate display (function 308). In various embodiments, the timed text is presented in a separate scrollable window 205 that is adjacent to the video display window 202, as described above. Other timed text windows 206, summary windows 210 and/or other information may be additionally provided.
In various embodiments, at least some of the words or phrases in the timed text window 205 may be marked with an HTML link or otherwise highlighted to indicate that additional information is available. In such implementations, the media player 120 typically contacts an external web service 140 for additional information or processing. The HTML link may reference a uniform resource locator (URL) on the Internet or another network, for example, that provides additional information about the word, phrase, or other item identified by the text. An athlete's name, for example, could be hot linked to a web page providing statistics or other information about that athlete. Actor/actress names, director names, or other persons could be similarly tagged to provide ready access to additional information. Other items could be linked to an encyclopedia page or other reference source for more information about the item, or words could be hot linked to a dictionary or thesaurus service (e.g., web service 140) that provides additional information about the linked words. Content could be selected for highlights or links in any manner; in various embodiments, an electronic program guide (EPG) or the like provides a table of keywords to be highlighted, along with URLs or other addresses to be referenced when the highlighted keyword is activated by the viewer.
In some implementations, the media player 120 simply contacts the external web service (function 310) as desired to resolve URL links or to obtain requested data. Web service 140 processes the request (function 312) and returns the requested data as appropriate (function 314). A dictionary or thesaurus service 140, for example, could receive a web query 310 that identified a particular word or phrase, and response 314 could include additional information about that particular word or phrase. The media player 120 would format the received information as appropriate for presentation to the viewer.
Other services 140 could provide translation or word replacement services, as appropriate. As noted above, some viewers may not be familiar with dialects, slang or other words of a particular language. This can be particularly true when a program is shown in a country other than where it was created. Text relating to slang terms (e.g., “totes cray”, “amazeballs”) from one country or region, for example, could be automatically replaced with more readily understood terms (e.g., “totally crazy”, “amazing”) for viewers in other regions. This would allow the viewer to readily ascertain the meaning from the timed text even though the original audio still contains the more obscure term. Replacement terms may be automatically supplied to media player 120 in response to media player application 126 noticing certain terms in the timed text. In some embodiments, media player application 126 could tag/highlight terms that do not appear in a dictionary of conventional words so that additional information 314 is obtained when the user clicks on a highlighted term. Other embodiments could implement the automatic word replacement feature within application 126 (e.g., by supplying a table of words and suitable replacements to application 126). Word replacement in this manner could also be useful for removing or softening harsh language, words or phrases that are offensive, or other words or phrases in the timed text that are undesirable for any reason.
The decode video and text data is presented to the viewer in any manner (function 316). In various embodiments, an interface such as that shown in
A program summary may also be automatically created, as desired (function 318). The summary may be created by the receiver or media player as the program is received by simply aggregating text entries that are of particular relevance. Alternately, summaries may be generated by the placeshifter, by broadcast or distribution services, or by any other service that is remotely accessible to the media player or other viewing device.
Typically, various types of programs (e.g., American football game, European football game, cricket match, soap opera, adventure show, news broadcast, etc.) can be associated with a set of key words or events that may occur during the course of the program and that would be relevant to include in a program summary. A sporting match, for example, might summarize scoring events, near scoring events, penalties, turnovers or other events of significance. These events could be associated with trigger words that are likely to occur within the timed text. Trigger words for American football, for example, could include “touchdown”, “fumble”, “turnover”, “third” or “fourth down”, “interception” and/or the like. Finding trigger words in the timed text, then, can lead to updates of a program summary.
In the example of
As the program text is received, an extractor module receives the timed text (function 404) and monitors the text for an occurrence of the key words/events identified in the database (function 406). In various embodiments, the summary event may be triggered using any sort of artificial intelligence that considers multiple timed text entries (e.g., corresponding to commentator dialog, or the like).
If a match is found in the timed text (function 408), then summary text may be extracted and/or generic text associated with the identified event may be added (function 412). The generic text may be a simple description of the event (e.g., “touchdown scored by TeamA”), which may be provided along with the time the event occurred in some embodiments.
Automatic summary generation may continue as desired (function 410). In various embodiments, summaries may be automatically generated even for programs that are not viewed. If a viewer comes to a program already in progress, for example, an automatic text summary can allow the viewer to catch up very quickly. Alternatively, text summaries can be used for programs that the viewer does not want to watch, but does want to know the sequence of events occurring (e.g., sporting games, etc.).
To generate a précis/summary, at least two software modules could be created: a closed captioning (CC)/subtitles extractor and a summary generator. The extractor module would go through the video and parse out the CC/subtitle data, in the case of a live channel this will happen on the fly. The summary generator module will then go through the extracted text and create a summary using the key text, key text can be found based on the usage of the text or by referring to key moments from Video and audio data. Whenever the viewer requests program info, the detailed summary can be provided so the user knows exactly what events that have occurred. This can be further improved to provide user more comprehensive summary of a TV show when a user starts watching a program mid-season.
Placeshifting of text streams may therefore be used to supplement or enhance the media viewing experience. Since only the timed text needs to be transported to the remote viewer, the amount of bandwidth consumed is substantially less than would be consumed for placeshifted video. Even if video content is not provided with the stream, the media player (or another source) can process the text to provide any number of useful features for the viewer.
The various embodiments described herein may be independently implemented and/or interchangeable mixed and matched with each other in any manner. Scrolling timed text windows could be augmented with word replacement features, for example, as well as automated summary generation capabilities. Conversely, automated summaries and/or word replacement or highlighting may be implemented even when separate timed text windows or other displays are not available.
The general concepts set forth herein may be equivalently applied in any number of different embodiments. Displaying of scrollable timed text, automatic word substitution and/or automatic summary features are not limited to placeshifting applications, but may be used in any television or other media viewing environment. The various interfaces and other features described herein are not limited to placeshifting and media streaming applications, but may be used in conventional television viewing or media watching, as desired. Other applications, modifications and enhancements may be considered in any number of other equivalent embodiments.
The term “exemplary” is used herein to represent one example, instance or illustration that may have any number of alternates. Any implementation described herein as “exemplary” should not necessarily be construed as preferred or advantageous over other implementations. While several exemplary embodiments have been presented in the foregoing detailed description, it should be appreciated that a vast number of alternate but equivalent variations exist, and the examples presented herein are not intended to limit the scope, applicability, or configuration of the invention in any way. To the contrary, various changes may be made in the function and arrangement of the various features described herein without departing from the scope of the claims and their legal equivalents.
This application claims priority to U.S. Provisional Application Ser. No. 62/098,675 filed on Dec. 31, 2014, which is incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6751776 | Gong | Jun 2004 | B1 |
| 8682672 | Ha | Mar 2014 | B1 |
| 8745683 | Maeng | Jun 2014 | B1 |
| 20030065503 | Agnihotri | Apr 2003 | A1 |
| 20030192044 | Huntsman | Oct 2003 | A1 |
| 20040006767 | Robson | Jan 2004 | A1 |
| 20050251832 | Chiueh | Nov 2005 | A1 |
| 20090185074 | Streijl | Jul 2009 | A1 |
| 20100082658 | Athsani | Apr 2010 | A1 |
| 20100322391 | Michaelis | Dec 2010 | A1 |
| 20110283243 | Eckhardt | Nov 2011 | A1 |
| 20120159537 | Casagrande et al. | Jun 2012 | A1 |
| 20120320267 | Landow | Dec 2012 | A1 |
| 20140071342 | Winograd | Mar 2014 | A1 |
| 20150046148 | Oh | Feb 2015 | A1 |
| Entry |
|---|
| Slang, n.3, Oxford English Dictionary, available at http://www.oed.com/view/Entry/181318?isAdvanced=false&result=3&rskey=fgVCYA& retrieved on Sep. 18, 2017. |
| Number | Date | Country | |
|---|---|---|---|
| 20160191959 A1 | Jun 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62098675 | Dec 2014 | US |
