Patent Application
20150312652
Instead of watching an entire event such as a sporting event, users often prefer to watch a summary, or highlight reel, of the event. It is known to have highlight shows that compile highlights from sporting and other events. These highlight shows are manually produced and choreographed by a television production team to have high production value. Users who record or download a video at present do not have an easy way to browse the video for highlights. It would be desirable to have a system for automatically producing a highlight reel from a recorded or downloaded video which had the look and feel of a high quality, manually produced television production.
The present technology, roughly described, relates in general to a system for automatically generating a highlight reel of video content. In embodiments, this highlight reel may be augmented with features providing the highlight reel with a high quality production appearance. In embodiments, the present system works in tandem with a segment list which includes a list of different segments of an event. One typical example of a segment list is a play-by-play (PBP) which is prepared contemporaneously with a sporting event and describes features and what went on during respective segments of the sporting event.
In accordance with the present technology, segments from a segment list may be associated with, or indexed to, corresponding sequences from a video of an event for which the segment list is prepared. Thereafter, also using the segment list, segments may be scored using a variety of predefined criteria to come up with segments which are likely to be of greatest interest to a particular user. The video sequences associated with the highest scored segments are used as the video highlight reel.
In one example, the present technology relates to a method of generating a video highlight reel, comprising: (a) indexing a video to a segment list setting forth the video sequences in the video to identify positions of different video sequences within the video; (b) comparing data from segments in the segment list against one or more predefined rules to identify one or more segments that satisfy a rule of the one or more predefined rules; and (c) selecting one or more video sequences into the highlight reel, the one or more selected video sequences having corresponding segments from the segment list that satisfied the rule of the one or more predefined rules in said step (b).
In another example, the present technology relates to a computer readable medium for programing a processor to perform a method of generating an interactive video highlight reel, comprising: (a) correlating segments in a segment list to video sequences in a video; (b) identifying one or more video sequences for inclusion in the video highlight reel, a video sequence included in the video highlight reel where a segment, correlated to the video sequence, satisfies one or more predefined rules; (c) displaying an interactive script including a plurality of script segments, a script segment of the plurality of script segments matched to a video sequence identified for inclusion in the video highlight reel in said step (b); (d) receiving selection of the script segment displayed in said step (c); and (e) displaying the video sequence matched to the script segment upon selection of the script segment in said step (d).
In a further example, the present technology relates to a system for generating a video highlight reel, comprising: a video including a plurality of videos sequences from one or more events, a group of one or more video sequences selected for inclusion in a highlight reel; one or more segment lists including a listing of segments corresponding to the video sequences from the one or more events; and an interactive script including script segments, displayed on a display of a computing device, the interactive script generated based on the segments of the segment list corresponding to the video sequences selected into the highlight reel, selection of a script segment from the interactive script displaying a corresponding highlight reel video sequence.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The present technology will now be described with reference to
In accordance with one aspect of the present technology, segments from a segment list may be associated with, or indexed to, corresponding points or segments in a video of an event for which the segment list is prepared. A length of the video sequence associated with each segment may also be defined. In the example of a football game, a single segment from the segment list and a sequence from the video may be a single play (kickoff, running play, passing play, punt, etc). The present technology indexes segments from the segment list to their corresponding sequences in the video where those segments occur and are displayed.
Either during or after the indexing of a video, the segment list may be analyzed for interesting or noteworthy segments for inclusion in a highlight reel. These may be segments which are determined to be of general interest, or of specific interest to a user for whom the highlight reel is created. The video sequences associated with the noteworthy segments may be processed into the highlight reel together with voice overlay and contextual content. The highlight reel may then be rendered and interactively browsed as explained below.
Referring to
The segment list 116 and video 118 may be received and stored in the computing device 100 from remote sources via a network connection such as the Internet 117. The video may alternatively or additionally arrive via an alternate source 119 in further embodiments, such as for example via cable TV, satellite TV, terrestrial broadcast etc. The received segment list 116 may include a segment-by-segment description of different sequences from the video, where one segment from the segment list corresponds to one sequence from the stored video.
The result of the operation of the indexing engine 110 may be a table correlating sequences from the video 118 of determined lengths to their respective corresponding segments from the segment list 116. This table may be stored in a memory 113, which may resident within computing device 100. Alternatively or additionally, the indexing table may be stored remotely from computing device 100, for example on remote storage 122. Details relating to the operation of the indexing engine 110 to generate the indexing table are explained below with reference to the flowchart of
In embodiments, a segment list is indexed to a single, stored video of an event. However, it is conceivable that a segment list may be indexed to multiple stored videos of the same event. In particular, it may happen that more than one video feed is captured of a given event. For example, more than one network or content provider may capture video of the same event such as a football game. Alternatively or additionally, the same network may capture the event using multiple cameras. Each video feed in these examples will capture the same sequences of the event, but the actual video from the different feeds may differ from each other (different perspectives, focus, etc.). It is conceivable that both videos be stored, and that sequences from both videos be indexed to a single segment list as explained below. When a user browses to sequences from the stored highlight reel of the video event as also explained below, the user may be shown sequences from both stored videos, or be given the option to choose one video sequence or another from the different stored videos of the event.
In accordance with a further aspect of the present technology, after the stored video is indexed and a highlight reel has been created, users may interactively watch, or browse, the highlight reel video by jumping to desired sequences in the video for playback.
The computing device 120 may for example be a hand-held computing device such as a mobile phone, laptop or tablet displaying a user interface 104. It may be a computing device other than a hand-held device in further embodiments, such as a desktop computer. The computing device 130 may be a desktop computer, media center PC, a set-top box and the like. It may be a portable computer similar to computing device 120 in further embodiments.
The computing device 130 may be connected to an audio/visual (A/V) device 136 having a display 138 (
In embodiments, the computing device 130 may further include a device such as a digital video recorder (DVR) 128 for recording, storing and playing back video content, such as sports and other events. The video content may be received from an external computer-readable medium such as a DVD, or it may be downloaded to the DVR 128 via a network connection such as the Internet 117. In further embodiments, the DVR 128 may be a standalone unit. Such a standalone unit may be connected in line with the computing device 130 and the A/V device 136.
It is conceivable that the present technology not operate with a DVR within or directly connected to the computing device 130. In such an embodiment, video content may be stored on a remote content server, such as for example remote storage 122, and downloaded via the Internet 117 to the computing device 130 based on selections made by the user as explained below.
In embodiments including two computing devices such as computing devices 120 and 130, the system may be practiced in a distributed computing environment. In such embodiments, devices 120 and 130 may be linked through a communications network implemented for example by communications interfaces 114 in the computing devices 120 and 130. One such distributed computing environment may be accomplished using the Smartglass™ software application from Microsoft Corporation which allows a first computing device to act as a display and/or other peripheral to a second computing device. Thus, the computing device 120 may provide a user interface for browsing video content stored on the computing device 130 for display on the A/V device 136. In such a distributed computing environment, a browsing engine 124 for implementing video browsing aspects of the present technology may be located on one or both computing devices 120 and 130 (in the embodiment shown in
Browsing engine 124 generates a user interface 134 (
In embodiments, the computing device 100 and the computing device 130 may be the same or different computing devices. In embodiments where the devices 100 and 130 are the same, an indexed video may be recorded and saved on DVR 128, and then played back from DVR 128. Additionally, the indexing table generated by the indexing engine 110 may be stored on local memory 113, and accessed from local memory 113 when browsing a video.
It is understood that the functions of computing devices 100, 120 and/or 130 may be performed by numerous other general purpose or special purpose computing system environments or configurations. Examples of other well-known computing systems, environments, and/or configurations that may be suitable for use with the system include, but are not limited to, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, distributed computing environments that include any of the above systems or devices, and the like.
In the embodiments described above, browsing of videos may be accomplished using multiple (two or more) computing devices in a distributed computing environment. In further embodiments, a single computing device may be used to implement the browsing aspects of the present technology. In such an embodiment, shown in
The description of the present technology below often uses an example where the event is a sporting event having a running clock associated with each sequence from a video (such as running clock 140 shown on
A high-level description of aspects of the present technology will now be explained with reference to the schematic block diagram of
In embodiments, the segment list 116 prepared by the third-party service may be a structured data feed including known fields of data categories. For example, where the segment list 116 is a structured feed from a football game, a first data field may describe the down (i.e., first, second, third or fourth) and the yards needed for a first down; a second data field may provide the game clock time at which the play started or ended; and a third data field may describe the play and result. These fields are a way of example only, and the segment list may include alternative and/or additional data fields. Structured data fields may be easily searched for information (such as a running game clock) that may be used to index segments from the segment list to stored video sequences of the event. In further embodiments, the segment list 116 prepared by the third-party service may alternatively be parsed into a text file in step 206 so that it may be searched for information that may be used to index segments to the stored video.
In step 208, the indexing engine 110 may confirm that the received segment list 116 corresponds to the received video 118. The video 118 may have certain descriptors or other metadata which may also be included as part of segment list 116 confirm that they correspond to each other. As used herein, metadata is a type of data.
In step 212, the indexing engine 110 may analyze frames of the stored video for display of a game clock having the running time for the event. For example, in a football game, the running game clock is generally displayed for each down that is played. An example of such a game clock 140 is shown
In the description above and that follows, the example of a game clock is used to match a segment in the segment list to a sequence from the stored video. However, it is understood that, more generally, identifiers other than a game clock may appear in both the segments of the segment list and sequences from the video, and these other identifiers may be used to index segments to sequences of the video. It may be some other form of sequential alphanumeric text (ascendingly sequential or decendingly sequential) may be displayed in different sequences of the stored video and this alphanumeric text may also appear in respective segments of the segment list to mark the start or end of a sequence of the video that is described by the segment of the segment list. In this instance, the sequential alphanumeric text may be used to index the segments of the segment list to sequences of the video as described above and hereafter.
The indexing engine checks whether sequential alphanumeric text, such as a running game clock, was found in step 214. If not, the present technology employs various methods for identifying video frames at the start or end of a segment, as explained hereinafter with respect to the flowchart of
However, if a game clock was identified in step 214, the indexing engine may take a game clock time of a segment from the segment list in step 216 and then determine whether a video frame is found having a game clock that matches the segment list time in step 220. For example, the indexing engine 110 may start with the first segment listed in the segment list. In a football game, this may be the opening kickoff starting with the game clock showing “15:00”. The indexing engine 110 searches for a video frame including a clock time 140 of “15:00”. If none is found, the indexing engine 110 may skip to step 228 to see if there more segments in the list. On the other hand, if a matching video frame is found, the indexing engine 110 may next perform step 222 of determining a length of video to index to the matched segment from the segment list as explained below.
In step 220, the indexing engine 110 may start with the first segment in the segment list 116 and proceed in succession through all segments in the segment list. However, the indexing engine 110 need not start with the first segment in further embodiments. Moreover, it is understood that several frames may have the same game clock time. For example, if the video has a frame rate of 30 frames per second, 30 frames should (ideally) have the same game clock time. In embodiments, the indexing engine 110 may take the first frame having a game clock time found to match the time of a segment from the segment list in step 220. Other frames having the matched time may be selected in further embodiments.
In step 222, the indexing engine 110 may index the segment from the segment list to a sequence of video including the video frame having the matched clock time. Further details of step 222 are now described with reference to the flowchart of
There are a number of ways the indexing engine 110 may recognize whether the identified video frame is at the beginning or end of a sequence. In embodiments, the indexing engine may receive information (for example from the received segment list) as to the type of event that the segment list and video relate to. The indexing engine 110 may the apply heuristics which have been stored (locally or remotely) which have been developed and stored for that type of event.
As an example, in football, individual plays are generally characterized by the players being relatively still before a play begins, then frantically moving during the play, and then moving slowly at the completion of the play. Where the indexing engine 110 receives information that the type of event is a football game, the indexing engine can examine frames before and after the identified video frame to determine the type of movement that is occurring (amount of change from frame to frame). From this, the indexing engine 110 may make a determination as to whether the identified frame is at the start or end of a play.
As another example, it is known how long football plays generally take. If the time listed in the preceding segment on the segment list is too large for a typical play, that can indicate that the current time is from the end of the next subsequent play. Other football-specific and generalized heuristics can be employed for football games, and other event-specific and generalized heuristics can be applied to events other than football games.
Another example is to use shot and/or scene detection on the video to break it into shots. In this example, the indexing engine 110 can find a video frame containing the clock time (or some other identifiable signature), thereafter find the sequence that the clock time or signature is contained in and then use the start/end time of the found sequence.
If the indexing engine 110 determines in step 234 that the matched clock time is at the start of a segment, the indexing engine 110 may determine the end of the video sequence in step 238. It may do so using the same or similar heuristics applied in step 234. For example, in a football game, the video may be examined to find the start of the play and then thereafter when the players slow down and stop. This may be considered the end of the play. The determined start and end times of the video sequence determined to correspond to the current segment may be stored.
Conversely, if the indexing engine 110 determines in step 234 that the matched clock time is at the end of sequence, the indexing engine 110 may determine the start of the sequence in step 240. Again, the same or similar heuristics as applied in step 234 may be applied in step 240 to work backwards from the end the sequence to determine the start of sequence. The determined start and end times of the video determined to correspond to the current segment may be stored.
When a user views a video sequence as explained below, it may be desirable to have a buffer at the start and end of the video sequence to provide a lead in and lead out for the video sequence. As such, in step 244, the indexing engine 110 may add a buffer of for example a few seconds to start and end of determined length of the video sequence. The buffer at the start or end of a video sequence may be omitted in further embodiments.
Referring again to
In step 228, the indexing engine 110 may check if there are more segments in the segment list for matching to sequences in the video. If so, the indexing engine 110 return to step 216 to get the time of another segment from the segment list, and steps 220, 222, 224 and 228 are repeated. On the other hand, if all segments in the segment list 116 have been accounted for in step 228, the indexed video 160 is completed and stored in step 230. The video may be indexed by time-stamping the corresponding segments in the segment list to their corresponding video sequence in the video.
As noted above, not all video events have a running game clock. Certain athletic contests such as baseball and tennis matches may be played to completion regardless of how long they take. Additionally, the present technology may operate on stored video events unrelated to sports, and which have no running game clock. If no game clock is discerned from examining frames of a video in step 214, the indexing engine 110 may next look for a segment signature as shown in step 250 of
A segment signature may be data describing a particular frame of video from the stored video of the event. This segment signature may be generated by the third-party at the time they prepare the segment list, and may describe a video frame at the start of a segment, though it may be the end of a segment in further embodiments.
As one example, a segment signature may be stored image data (jpeg, gif, etc.) from a single frame of the video which the third-party provider grabs from the video at the start of a video sequence and stores in association with the segment from the segment list. Thus, each segment from the segment list will have an associated segment signature which describes a single point in the video of the event. In further embodiments, the segment signature may be a time in the video. That is, the video of the event begins at time t0, a first sequence starts at video run time t1, a second sequence starts at video run time t2, etc. The segment signature for a particular sequence may thus be the video run time at which that sequence begins (or ends).
In step 250, the indexing engine 110 may check whether the segments in the segment list received from a third party include associated segment signatures. If not, the indexing engine 110 may not generate the interactive script for that video and the operation of the indexing engine 110 may end.
On the other hand, if segment signatures are included with the segment list, in step 252 the indexing engine 110 may determine whether a given stored segment signature matches to a point in the video. As indicated above, this comparison may involve comparing stored image data against the image data of successive frames of the video until a match is found.
A number of technologies exist for abstracting or summarizing data of a signature and video frames for the purposes of comparison and finding a match. One example of such technology is disclosed in US Patent Publication No. 2012/0008821 entitled “Video Visual and Audio Query.” That patent publication describes different examples for extracting image signatures from video frames, to be compared with each other in order to find matching video frames. In one such example, the system divides each video frame image into 64 (8×8) equal size rectangular ordered cells. In each cell, the system can generate two ordered bits. For example:
Instead of image data from the video, it is conceivable that the signature be audio data, for comparison against audio data in the video. An example for comparing audio data from a signature and video is disclosed in US Patent Publication No. 2012/0296458, entitled “Background Audio Listening for Content Recognition.” One example disclosed in that patent publication, a signature comprised of sequence of audio can be processed using a feature extraction algorithm in any of a variety of ways including for example applying a Hamming window to the audio data, zero padding the audio data, transforming the data using fast or discrete Fourier transform, and applying a log power. This processed audio signature may then be compared against audio segments from the video, which may be processed in a similar manner.
Other known technologies for processing image and/or audio data from the video and four performing the comparison of the data may be used. As indicated above, the matching may instead involve finding the video run time corresponding to the video run time stored as the segment signature for a given segment.
If no match is found in step 252, the indexing engine may check for more segments in the list in step 228 (as above). On the other hand, if a match is found, the segment associated with the segment signature is indexed to the video including the matched video point. In particular, the indexing engine may determine a length of the video sequence to index to the matched segment from the segment list (step 222), and the length of the indexed video sequence may then be stored in association with the matched segment from the segment list (step 224), which steps have been explained above. In step 228, the indexing engine 110 may check if there are more segments in the segment list for matching to a point in the video. If so, the indexing engine 110 may return to step 252 to get the next segment signature from the segment list, and steps 252, 222, 224 and 228 are repeated. If there are no more segments in the segment list, the indexed video 160 is completed and stored in step 230 as explained above.
It may happen that a stored video, such as that for sporting event, may include video replays of certain sequences. For example, in a football game, networks often show replays of eventful passes, runs, defensive plays, penalties, etc. In embodiments, it may be advantageous to identify a video replay of the sequence as opposed to the video of the underlying sequence itself For example, when indexing segments from the segment list to video sequences, it may be beneficial to index to the video sequence itself instead of or in addition to a replay of the video sequence. It may also be beneficial to include a replay in the highlight reel in addition to the underlying video sequence.
Various generalized and event-specific heuristics may be employed by the indexing engine 110 to identify a replay of a sequence and distinguish that replay from the underlying sequence. For example, in football games, replays are generally shown without display of the running game clock. Additionally, networks typically flash a network logo or some other graphic at the start and end of a replay to highlight that it is a replay that is being shown. Replays are also often shown at slower than normal speeds. The indexing engine 110 may include rules to look for these and other characteristics of a replay so as to determine when the video is of a replay. Once a replay is identified, it can be omitted or included from a video highlight reel by the selection module 162 explained below.
It is possible that a segment in the segment list can be indexed to two different parts of the video. In the example of a football game, if a play is shown live, and then a replay is shown later in the broadcast, the play are replay may be covered by a single segment. However that segment may be indexed to two different sections of the video (the first being the underlying play and the second being the replay of the play).
Referring again to the schematic block diagram of
The operation of the HLR generation engine 112 to form a highlight reel includes two modules. As shown in
Referring now to the flowcharts of
User preferences 170 may include a list, for example built up by a user over time, of the type of content that a user is interested in. In a sports context, user preferences may include for example a user's favorite sports, sports teams and players; channels and sporting events he/she would like to watch and sports content he/she would like to receive; fantasy teams, rosters and schedules, etc. This information may additionally or alternatively include a wide variety of other non-sports related information. User preferences 170 may be stored locally within memory 113 (
The statistical history 176 may provide a probabilistic outcome for a wide variety of combinations of statistical data. For example, in a football game, the statistical data may indicate that, if the score is 17:14 in favor of the home team, there is 3 minutes left in the 3rd quarter, and the home team is on offense at the other team's 33 yard line with a 1st down and 10 yards to go, the probability is that the home team will win 55% of the time. Each of the preceding items of data, referred to herein as state data, is by way of example only, and varying one or more of these items of state data may change the probabilistic outcome according to the statistical history data. It is also understood that this type of statistical data may exist a wide variety of other events, enabling probabilistic outcomes for these events for a given set of statistical data. The statistical history for different events covering a plurality of years may be stored, either on remote storage 122 or locally within memory 113 of computing device 100.
In step 304, the selection module 162 may evaluate the state data for a given segment. In one example, the selection module start with the first segment in the segment list and proceed sequentially, though it may be otherwise in further embodiments. In step 306, the selection module 162 may determine the probabilistic outcome of the event described by the segment list using the state data for the segment then under consideration together with the statistical history for the type of event covered by the segment list.
In step 308, the selection module 162 may determine any change in the probabilistic outcome of the current segment relative to the probabilistic outcome of one or more previous segments. In step 310, the selection module 162 determines whether this change is greater than some predefined threshold. Step 308 and 310 (as well as other steps in
On the other hand, if the segment does appreciably change the outcome, the flow may next check in step 312 whether a user has set the length of the highlight reel 166. Where a user chooses a length, the selection module 162 may check in step 314 whether the segment under consideration (in particular, the video sequence associated with that segment) is too long for the defined length. The selection module 162 may make this determination in a number of ways. In one example, the selection module 162 has data relating to the average length of video sequences in the indexed video 160. This may vary depending on the type of event captured in the indexed video. Where the length of a video sequence is longer than the average, that video sequence may be omitted from the highlight reel 166 in step 314, especially where a user has set a particularly short overall length of the highlight reel. The selection module 162 may alternatively or additionally look at how much time remains in the user-specified length of the highlight reel, and the number of segments still to consider, and make a determination as to whether to add the current segment to the highlight reel.
In further embodiments, the selection module 162 may consider how significant a segment is (the degree to which it changes the probabilistic outcome of the event) in combination with how long it is when determining whether the associated video sequence is too long to include in the highlight reel. Thus, where a video sequence is particularly long, but also particularly significant, the selection module 162 may include it in the highlight reel, even where the user has set a short length of the highlight reel. Determination of the significance of a segment is described below.
If the segment appreciably change the outcome, and is not too long (or no length was set) the segment to be added to the highlight reel in step 316. In step 318, the selection module 162 checks for more segments. If more segments are found, the next segment is called in step 320, and the flow returns to step 304 to evaluate that segment as described above. Where there are no further segments in step 318, the selection module 162 stores the segments selected for the highlight reel in step 322, and generates and stores an interactive script in step 324. As explained below, the interactive script may be displayed on a user interface to allow browsing of the highlight reel videos.
A user may set the length of the highlight reel 166 in step 336. Where a user chooses a length, the selection module 162 may check in step 338 whether the segment under consideration (in particular, the video sequence associated with that segment) is too long for the defined length. The selection module may use the methods described above for making this determination or other methods.
Assuming a user has not set the length of the highlight reel in step 336, or a video sequence is not determined to be too long in step 338, the selection module 162 next looks to whether a segment correlates to a user preference in step 342. As one of many examples, a segment may name a particular player as being involved in the segment, and that same player may be stored as one of the user's favorites or on the user's fantasy team in the user preferences. As another example, a user may have a saved preference to see sequences involving particular results, such as for example quarterback sacks. Segments having state data relating to quarterback sacks may then be selected in step 342. As noted, the segment may relate to a wide variety of other topics which may be of particular interest to a user and stored in the user preferences, including a variety of sports and non-sports related features. This step may be performed by a keyword search of the segment and the user stored preferences to find matches. Where a match is found, the segment may be added to the highlight reel in step 352. Where no match is found, the flow may proceed to step 346.
Even where segment is unrelated to a user preference, the segment may be noteworthy or significant in and of itself. For example, in a football game, a play involving a long pass or run, a touchdown, interception, sack or fumble may be considered noteworthy independent of other factors. In baseball, segments involving various hits, a run scored or good fielding play may be considered noteworthy. In soccer, a goal, penalty kick or good defensive play may be considered noteworthy, etc. Where segment is determined to be noteworthy in step 346 by the selection module 162, the segment may be added to the highlight reel in step 352. Otherwise, the flow may proceed to step 350.
In step 346, the selection module 162 may employ a variety of different stored criteria or rules for determining a threshold noteworthiness for segment to be added to the highlight reel. In examples, these thresholds may be quantitative. For example, in a football game, net yardage gains of at least a predetermined number of yards may meet the quantitative threshold of being considered noteworthy. Plays longer than a predetermined length of time may relate to long runs or long scrambles, and thus many considered noteworthy. Plays resulting in touchdowns or field goals may also be considered to meet the threshold level of noteworthiness. Plays resulting in at least a predefined number of fantasy points for one or more of the user's fantasy players may be considered to meet the threshold level of noteworthiness. A wide variety of other criteria may be employed for determining a threshold noteworthiness for segments to be added to the highlight reel in step 346.
It may happen that a segment is not interesting in and of itself, but given the context in which the segment occurs that segment may be considered noteworthy and added to the highlight reel. For example, a short run or pass play in a football game may not be that interesting. However, given the context in which it occurs, such as to gain a first down at a critical time late in a close game, it may be noteworthy and worthwhile including in the highlight reel. In step 350, the selection module 162 determines whether a segment is contextually noteworthy. If so, the segment is added to the highlight reel in step 352.
As above, the selection module 162 may employ a variety of different criteria or rules for determining a threshold contextual noteworthiness for segment to be added to the highlight reel. For example in a football game, all plays resulting in a first down may be considered contextually noteworthy where they occur late in the game (e.g., last two minutes) and where the teams are tied or within a predetermined point differential of each other (e.g., 7 points). A wide variety of other rules may be employed for determining a threshold contextual noteworthiness for segments to be added to the highlight reel.
It is understood that other factors may be used in determining how and when to add segments to a highlight reel in step 352. For example, in a further embodiment, each of the segments may be scored, using a combination of the above-described factors. In such an embodiment, user preferences, the significance of a segment, and the context in which the segment occurred may each be considered and each factor may be quantified using predefined scoring rules to result in a net score for each factor. The score for each factor may be summed, and segments above some threshold may result in a noteworthy segment. The predefined scoring rules may use any of the above-described criteria, such as for example, net yardage in a play, length of the play, whether the play resulted in game or fantasy player points. Other factors such as whether there was a turnover on the play, penalty or whether there was a replay shown of the play may also factor into the score.
In embodiments described above, the segment list 116 is used to determine whether a segment should be added to the highlight reel. In further embodiments, the indexed video 160 may additionally be used. For example, the indexed video 160 may include an audio soundtrack. It may be assumed that crowd noise increases for interesting plays in a game. Thus, where crowd noise rises above a predefined decibel level for a predefined period of time, the video sequence occurring at that time may also be added to the highlight reel (or receive a high score for this particular factor).
If a segment is not contextually noteworthy in step 350, or where a segment has been added to the highlight reel of step 352, the flow proceeds to step 354 to determine whether there are additional segments to analyze for possible addition to the highlight reel. If there are more segments, the flow returns to step 334, and the above-described steps are repeated. If there are no further segments, the selection module 162 stores the video sequences selected into the highlight reel in step 356, and generates and stores an interactive script in step 360.
As explained below, the interactive script may be displayed to a user on the user interface 134. The user may select script segments from the interactive script, and then be shown the video sequence from the highlight reel which has been indexed to that script segment. Each script segment from the interactive script may be generated from the respective segments from the segment list. For example, a script segment may be populated with some or all of the data fields and/or parsed text from a segment in the segment list. Thus, the video sequences can be said to be indexed to the script segments, in that the video sequences are indexed to segments from the segment list, which segments are in turn used to generate corresponding script segments in the interactive script.
Additionally, each script segment may include hypertext or otherwise be hyperlinked to the index table created and stored in step 224. Thus, when a user selects a particular script segment from the interactive script, the index table may be accessed to determine which indexed video sequence is associated with that script segment. Instead of being created by the indexing engine 110, some or all of the interactive script may be generated by browsing engine 134 as explained below.
In embodiments, the video segments selected into the highlight reel come from coverage of a single event, such as a single football game, baseball game, talk show, etc. However, in further embodiments, the video segments selected into the highlight reel may come from coverage of multiple events. The events may or may not be related to each other. In such an embodiment, multiple videos and segments list may be used in forming the video highlight reel.
Referring again to
Further details of the operation of the augmentation module 164 will now be explained with reference to the flowchart of
The opening video clip 180 may render broadcast-style graphics that includes for example a highlight reel title and video previews, possibly with titling graphics, of the upcoming clips. The opening video clip may include an audio track of music or talk as well. This audio may be taken from the videos 172 in the highlight reel 166, or from audio overlay store 174 as explained below.
The augmentation module 164 may employ one or more software templates using a markup language to set the overall layout, appearance and animation flow of the opening video clip. The markup language templates can dynamically change, or swap, assets to customize the opening video clip for a given highlight reel. Swappable assets may come from a stored stock of assets and/or from metadata associated with videos in the highlight reel or the segment list 116. The augmentation module 164 can choose which assets to include in the markup language template based on the content of the highlight reel videos and from the metadata associated with the video and segment list.
Some of the elements that may be swappable include background, midground, title field, portraits and other types of graphics. In embodiments, the augmentation module 164 examines the selected highlight reel videos and associated metadata and makes a determination as to which assets to include in the template. At a rendering step of the highlight reel (or before), the template displays the opening video clip with the selected assets. The markup language templates for the transitional video clips and closing video clips, described below, may function in a similar manner.
The one or more markup language templates of the augmentation module 164 for the opening video clip 180 may be populated with textual graphics, such as for example a user's name and other profile information. Other textual graphics such as for example a general subject matter of the selected highlight reel videos may be included. For example, if all of the clips have a common overarching theme (a user's fantasy team, favorite team or player, current events, etc.), a title for the playlist may be included in the metadata for the selected highlight reel videos 172 or segment list 116 and used by the markup language template(s) for the opening sequence. Other textual graphics such as the date, length of playlist, source of the playlist, etc. may be included.
The markup language template(s) may further receive the opening or highlight frames from the metadata of one some or all of the video clips for display in the opening video clip as a preview of what is to come in the highlight reel. These may play in succession (0.5 to 1.5 seconds each, though the length of time the frames are shown may be shorter or longer than this in further embodiments). These frames may play after the textual graphics, or together with the textual graphics, for example below the textual graphics, off to the side of the textual graphics or as a background behind the textual graphics. Instead of playing in succession, the frames may be displayed all at once, for example as thumbnails below the textual graphics, off to the side of the textual graphics or as a background behind the textual graphics.
After creation of the opening video clip, the augmentation module 164 may create transitional video clips 182 in step 368 introducing the first (and then subsequent) video clips in the highlight reel. The markup language template for the transitional video clip may be populated with textual graphics, such as for example a title of the upcoming video clip received from the metadata from the upcoming video clip or associated segment from the segment list 116. Other textual graphics such as the date, countdown clock to the start of the video clip, length of the video clip, countdown clock showing the time to the next video clip, source of the video clip, etc. may be included. Other non-textual graphics may be included, such as for example team logos and/or logos from remote storage 122.
The markup language template(s) for the transitional video clips 182 may further receive the opening or highlight frames from the metadata of the upcoming video clip/segment list as a preview of what is to come in the video clip. These one or more frames may play after the textual graphics, or together with the textual graphics, for example below the textual graphics, off to the side of the textual graphics or as a background behind the textual graphics.
The content included in the transitional video clips 182 may vary depending on the associated highlight reel video sequence. For example, if the upcoming video clip focuses on a player, the transitional video clip may provide statistics and other information for the player.
In step 370, the augmentation module 164 may further generate a closing video clip 184. The closing video clip 184 may be similar to the closing of a traditional broadcast television show, and may instill in the user a feeling of the user viewing a professionally choreographed television show. The closing video sequence may render broadcast-style graphics that includes any of the textual graphics and/or frames described above. It may include a further closing salutation textual graphic indicating the highlight reel is over, such as for example displaying “End,” or “Your Highlight Reel Entitled [title of highlight reel from metadata] Has Completed.” Other closing text may be used in further embodiments.
The closing video clip 184 may be created by one or more markup language templates of the augmentation module 164. The software templates receive assets from the metadata associated with one, some or all of the video sequences/segment list to create the closing video sequence.
In addition to opening, closing and transitional clips, the selected highlight reel videos 172 may be augmented themselves by adding an audio track over the videos 172 in step 372. The augmentation module 164 may work with one or more software templates as described above which have access to a number of canned audio phrases, which vary depending on the underlying context of the highlight reel. For example, there would be a separate library of canned audio for football highlight reels, baseball highlight reels, basketball highlight reels, etc. There could be separate libraries of canned audio for a variety of other non-sports related highlight reels as well.
The markup language templates fuse together contextual audio data from segment list with these audio phrases to provide a contextually relevant voice overlay for a given portion of one or more of the highlight reel videos. The templates can dynamically change, or swap, audio data assets to customize the voice overlay for a given video segment of the highlight reel. Swappable audio assets may come from a stored stock of assets and/or from audio metadata associated with videos in the highlight reel or the segment list 116. The augmentation module 164 can choose which assets to include in the markup language template based on the content of the highlight reel videos and from the metadata associated with the video and segment list. Where a voice layover is provided by the augmentation module 164, the audio recorded with the video sequence can be muted during the voice overlay, and then return upon completion of the voice overlay.
After the selected highlight reel videos 172 have been augmented with an opening clip, transitional clips, closing clip and/or audio overlays, the finished highlight reel 166 may be rendered in step 376. Thereafter, the highlight reel 166 is stored and made available for viewing. The highlight reel may alternatively be rendered at the time it is displayed. As mentioned above, the addition of these features provides a highlight reel with a professionally choreographed look and feel. However, it is understood that one or more of the above described opening clip, transitional clips, closing clip and voice overlays may be omitted in further embodiments. It is conceivable that the augmentation module 164 may be omitted altogether, at which point the selected highlight reel videos 172 are used as is as the finished highlight reel 166.
The augmentation engine 164 has been described as adding features that play in the finished highlight reel together with videos 172 selected into the highlight reel. However, in further embodiments, the videos 172 selected for the highlight reel may themselves be altered and/or augmented. For example, in a sporting event highlight reel, the ball or puck or other object within the video may be highlighted. It is also possible to highlight key-frames in the video with a “flash+hold” effect. The key-frame is highlighted to simulate a camera flash. The key-frame is then repeated for around a second in the video. This provides a compelling graphics effect to the highlight video 172.
Referring again to the schematic diagram of
When a highlight reel is displayed, it may be displayed straight through as a linear video without the user interacting with the video. However, in embodiments, the present technology may further provide a user interface 134 including an interactive script 150 that allows a user to interactively browse the highlight reel. Using the interactive script 150, a user may instantly jump forward or backward to a desired highlight reel video 172 in the highlight reel 166.
The user interface 134 and interactive script 150 may be implemented by the browsing engine 124. Operation of an embodiment of the browsing engine will now be explained with reference to the flowchart of
In step 260, a user may access an interactive script for a stored highlight reel 166 for display on a user interface 134. The interactive script includes a listing, or script, of the segments in the highlight reel 166, set up with hypertext or with hyperlinks. The links are set up so that, once a specific script segment is selected, the indexing table retrieves and displays the associated highlight reel video sequence from memory.
The user interface 134 may be displayed on a display of the computing device 120 as shown in
The interactive scripts of
As seen in
In step 262, the browsing engine 124 may look for user selection of a script segment from the interactive script 150. Once a selection is received, the browsing engine 124 finds the highlight reel video sequence indexed to the selected script segment in step 264 using the stored index table. That video sequence is then displayed to the user in step 266, for example on display 138. It is conceivable that a user may be able to select multiple script segments. In this instance, the multiple video sequences indexed to the multiple selected script segments may be accessed, and then played successively. Upon completion of a displayed video sequence, the closing video clip may be displayed and/or the video may end. Alternatively, the stored video may continue to play forward from that point.
As an example, referring to
Similarly, in the example of
Referring again to
In step 274, the browsing engine 124 determines whether any highlight reel script segments satisfy the search query. If not, a message may be displayed to the user that no script segments were found satisfying the search. On the other hand, if script segments were found satisfying the search query, those script segments may be displayed to the user, or otherwise highlighted in the overall interactive script 150. Thereafter, the flow returns to step 262 where the browsing engine 124 looks selection of a script segment.
Thus, in the football game example of
As noted, the appearance of the interactive scripts 150 shown in
In embodiments described above, the indexing engine 110, the highlight reel generation engine 112 and browsing engine 124 operate separately. However, in further embodiments, two or more of these software engines 110, 112 and 124 may be integrated together. In such an embodiment, the segment list may be used as the interactive script. That is, the segment list may be received from the third-party with hyperlinks, or otherwise augmented with hyperlinks, so that each segment in the segment list may be displayed on the user interface 134 as a selectable link.
In this embodiment, a user may select one of the segments displayed on the user interface 134. At that point, the combined indexing/browsing engine may examine the selected segment for a running clock time or a segment signature as described above. If found, the indexing/browsing engine may then examine the stored video to find the corresponding video sequence. That video sequence may then be displayed to the user, possibly adding a buffer at the start and/or end of the video segment.
A system as described above provides several advantages for a user viewing a stored event. First, the present technology can generate a highlight reel automatically and which is customized for a user. The highlight reel may include a variety of overlays to give it the look and feel of a high quality, manually produced television broadcast. Another benefit of the present technology is that a user may quickly and easily browse directly to points in the generated highlight reel that are of particular interest to the user, and the user can skip over other less interesting portions of the stored highlight reel.
With reference to
Computer 710 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 710 and includes both volatile and nonvolatile media, removable and non-removable media. Computer readable media can be any available tangible media that can be accessed by computer 710, including computer storage media. Computer readable media does not include transitory, modulated or other transmitted data signals that are not contained in a tangible media. By way of example, and not limitation, computer readable media may comprise computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the information and which can accessed by computer 710.
The system memory 730 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 731 and random access memory (RAM) 732. A basic input/output system 733 (BIOS), containing the basic routines that help to transfer information between elements within computer 710, such as during start-up, is typically stored in ROM 731. RAM 732 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 720. By way of example, and not limitation,
The computer 710 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
The computer 710 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 780. The remote computer 780 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 710, although a memory storage device 781 has been illustrated in
When used in a LAN networking environment, the computer 710 is connected to the LAN 771 through a network interface or adapter 770. When used in a WAN networking environment, the computer 710 typically includes a modem 772 or other means for establishing communications over the WAN 773, such as the Internet. The modem 772, which may be internal or external, may be connected to the system bus 721 via the user input interface 760, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 710, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
CPU 801, memory controller 802, and various memory devices are interconnected via one or more buses (not shown). The details of the bus that is used in this implementation are not particularly relevant to understanding the subject matter of interest being discussed herein. However, it will be understood that such a bus might include one or more of serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus, using any of a variety of bus architectures. By way of example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus also known as a Mezzanine bus.
In one implementation, CPU 801, memory controller 802, ROM 803, and RAM 806 are integrated onto a common module 814. In this implementation, ROM 803 is configured as a flash ROM that is connected to memory controller 802 via a PCI bus and a ROM bus (neither of which are shown). RAM 806 is configured as multiple Double Data Rate Synchronous Dynamic RAM (DDR SDRAM) modules that are independently controlled by memory controller 802 via separate buses (not shown). Hard disk drive 808 and portable media drive 805 are shown connected to the memory controller 802 via the PCI bus and an AT Attachment (ATA) bus 816. However, in other implementations, dedicated data bus structures of different types can also be applied in the alternative.
A graphics processing unit 820 and a video encoder 822 form a video processing pipeline for high speed and high resolution (e.g., High Definition) graphics processing. Data are carried from graphics processing unit (GPU) 820 to video encoder 822 via a digital video bus (not shown). Lightweight messages generated by the system applications (e.g., pop ups) are displayed by using a GPU 820 interrupt to schedule code to render popup into an overlay. The amount of memory used for an overlay depends on the overlay area size and the overlay preferably scales with screen resolution. Where a full user interface is used by the concurrent system application, it is preferable to use a resolution independent of application resolution. A scaler may be used to set this resolution such that the need to change frequency and cause a TV resync is eliminated.
An audio processing unit 824 and an audio codec (coder/decoder) 826 form a corresponding audio processing pipeline for multi-channel audio processing of various digital audio formats. Audio data are carried between audio processing unit 824 and audio codec 826 via a communication link (not shown). The video and audio processing pipelines output data to an A/V (audio/video) port 828 for transmission to a television or other display. In the illustrated implementation, video and audio processing components 820-828 are mounted on module 814.
In the implementation depicted in
MUs 840(1) and 840(2) are illustrated as being connectable to MU ports “A” 830(1) and “B” 830(2) respectively. Additional MUs (e.g., MUs 840(3)-840(6)) are illustrated as being connectable to controllers 804(1) and 804(3), i.e., two MUs for each controller. Controllers 804(2) and 804(4) can also be configured to receive MUs (not shown). Each MU 840 offers additional storage on which games, game parameters, and other data may be stored. In some implementations, the other data can include any of a digital game component, an executable gaming application, an instruction set for expanding a gaming application, and a media file. When inserted into console 800 or a controller, MU 840 can be accessed by memory controller 802. A system power supply module 850 provides power to the components of multimedia console 800. A fan 852 cools the circuitry within console 800. A microcontroller unit 854 is also provided.
An application 860 comprising machine instructions is stored on hard disk drive 808. When console 800 is powered on, various portions of application 860 are loaded into RAM 806, and/or caches 810 and 812, for execution on CPU 801, wherein application 860 is one such example. Various applications can be stored on hard disk drive 808 for execution on CPU 801.
Multimedia console 800 may be operated as a standalone system by simply connecting the system to audio/visual device 16, a television, a video projector, or other display device. In this standalone mode, multimedia console 800 enables one or more players to play games, or enjoy digital media, e.g., by watching movies, or listening to music. However, with the integration of broadband connectivity made available through network interface 832, multimedia console 800 may further be operated as a participant in a larger network gaming community.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
