Patent Grant
10999649
A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
The present disclosure relates, in general, to a method and system for automatically summarizing video and/or audio content.
Today, movie and television show summaries are required to be produced by people watching the movie and/or television shows, and selecting the most popular scenes. Once the scenes are selected, they are compiled together to create a new video sequence for playback. This is a manual and labor intensive process.
Hence, there is a need for more robust solutions for developing video summaries of television shows and the like.
Various embodiments provide systems and techniques for auto-summarizing video and/or audio content. In some embodiments, a remote terminal, at a service provider, might retrieve video and/or audio content from a content database, and might transmit the video and/or audio content to one or more user devices (including, but not limited to, a tablet computer, a smart phone, a mobile phone, a personal digital assistant (“PDA”), a portable gaming device, a gaming console, a desktop computer, a laptop computer, a set-top box (“STB”), a television set (such as a broadcast TV, a high-definition TV (“HDTV”), an Internet Protocol TV (“IPTV”), and/or a digital TV, or the like). A user associated with each user device might enter, using the user device, one or more inputs, including one or more triggers, which might correspond to the user's favorite scene(s) or favorite segment(s) of the video and/or audio content. In some embodiments, the one or more inputs might be associated with actions or sounds recorded from a live audience at a live presentation of a show (e.g., a movie, a play, a live TV broadcast, etc.). In other embodiments, the one or more inputs might include cue tones provided by a broadcaster, a distributor, or a producer of the video and/or audio content. A summary server, which might be located at the service provider, might receive the one or more inputs, might associate the one or more inputs with one or more time codes corresponding to the video and/or audio content, and might store the one or more time codes in a time code database.
The summary server might retrieve the one or more time codes from a time code database, and might analyze the one or more time codes to determine at least one selected time code among the one or more time codes that exceeds a predetermined number of selections from a plurality of users associated with a plurality of user devices. The one or more triggers might include user-inputted triggers, triggers associated with actions by a live audience, or broadcaster/distributor/producer-provided triggers (i.e., cue tones). The summary server might determine one or more selected segments of video or audio content corresponding to the at least one selected time code, and might create one or more summary tracks, where each of the one or more summary tracks might comprise each of the one or more selected segments of the video or audio content. The one or more summary tracks may be compiled into a summary sequence, which might be stored in a summary server and/or might be transmitted to one or more user devices for streaming the summary sequence thereon.
In an aspect, a method might be provided for auto-summarizing video and audio content. The method might comprise a remote terminal, located at a service provider, retrieving video and audio content from a content database, transmitting the video and audio content to at least one user device among a plurality of user devices over a network, and streaming the video and audio content on the at least one user device. The method might further comprise a summary server, located at the service provider, receiving one or more inputs from at least one user among a plurality of users. The plurality of users might be associated with the plurality of user devices, and the one or more inputs might comprise one or more triggers.
The method might comprise the summary server determining one or more time codes each corresponding to each of the one or more triggers, and storing the one or more time codes in a time code database. The method might further comprise the summary server analyzing the one or more time codes to determine at least one selected time code among the one or more time codes that exceeds a predetermined number, and determining one or more selected segments of the video and audio content corresponding to the at least one selected time code. Each of the at least one selected time code might include a central selected time code and a range of time codes spanning a predetermined number of seconds before and after the central selected time code.
The method might further comprise the summary server, creating one or more summary tracks, compiling the one or more summary tracks into a summary sequence, and storing the summary sequence in a summary database. Each of the one or more summary tracks might comprise each of the one or more selected segments. Each of the one or more summary tracks might further comprise at least one of a back-stepped segment of the video and audio content or a forward-stepped segment of the video and audio content.
In another aspect, a method might be provided for auto-summarizing video or audio content. The method might comprise a summary server retrieving one or more time codes from a time code database, and analyzing the one or more time codes to determine at least one selected time code among the one or more time codes that exceeds a predetermined number of selections. Each of the one or more time codes might correspond to each of one or more triggers. The method might further comprise the summary server determining one or more selected segments of video or audio content corresponding to the at least one selected time code, and creating one or more summary tracks. Each of the one or more summary tracks might comprise each of the one or more selected segments of the video or audio content.
In some embodiments, the method might further comprise compiling, with the summary server, the one or more summary tracks into a summary sequence. In some examples, the method might further comprise transmitting, with the summary server, the summary sequence to at least one user device among a plurality of user devices over a network. The plurality of user devices might be associated with a plurality of users.
In yet another aspect, a system might be provided for auto-summarizing video or audio content. The system might comprise a time code database on which is stored one or more time codes. Each of the one or more time codes might correspond to each of one or more triggers. The system might further comprise a summary server operatively coupled with the time code database. The summary server might be configured to retrieve the one or more time codes from the time code database, and analyze the one or more time codes to determine at least one selected time code among the one or more time codes that exceeds a predetermined number of selections. The summary server might be further configured to determine one or more selected segments of video or audio content corresponding to the at least one selected time code, and create one or more summary tracks. Each of the one or more summary tracks might comprise each of the one or more selected segments of the video or audio content.
In some embodiments, the summary server might be further configured to compile the one or more summary tracks into a summary sequence. According to some examples, the summary server might be configured to transmit the summary sequence to at least one user device among a plurality of user devices over a network. The plurality of user devices might be associated with a plurality of users.
Various modifications and additions can be made to the embodiments discussed without departing from the scope of the invention. For example, while the embodiments described above refer to particular features, the scope of this invention also includes embodiments having different combination of features and embodiments that do not include all of the above described features.
A further understanding of the nature and advantages of particular embodiments may be realized by reference to the remaining portions of the specification and the drawings, in which like reference numerals are used to refer to similar components. In some instances, a sub-label is associated with a reference numeral to denote one of multiple similar components. When reference is made to a reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components.
While various aspects and features of certain embodiments have been summarized above, the following detailed description illustrates a few exemplary embodiments in further detail to enable one of skill in the art to practice such embodiments. The described examples are provided for illustrative purposes and are not intended to limit the scope of the invention.
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the described embodiments. It will be apparent to one skilled in the art, however, that other embodiments of the present invention may be practiced without some of these specific details. In other instances, certain structures and devices are shown in block diagram form. Several embodiments are described herein, and while various features are ascribed to different embodiments, it should be appreciated that the features described with respect to one embodiment may be incorporated with other embodiments as well. By the same token, however, no single feature or features of any described embodiment should be considered essential to every embodiment of the invention, as other embodiments of the invention may omit such features.
Unless otherwise indicated, all numbers used herein to express quantities, dimensions, and so forth used should be understood as being modified in all instances by the term “about.” In this application, the use of the singular includes the plural unless specifically stated otherwise, and use of the terms “and” and “or” means “and/or” unless otherwise indicated. Moreover, the use of the term “including,” as well as other forms, such as “includes” and “included,” should be considered non-exclusive. Also, terms such as “element” or “component” encompass both elements and components comprising one unit and elements and components that comprise more than one unit, unless specifically stated otherwise.
Herein, the term “adaptive bitrate streaming” (“ABR streaming”) might refer to a method of video and/or audio content streaming over the Internet or other network (e.g., using HTTP or similar protocol) where the source content is encoded at different bit rates (including, without limitation, high bit rate, medium bit rate, and low bit rate, etc.) and streamed as multiple streams according to bit rate. A streaming client may be made aware of the available streams at different bit rates as well as the segments of the streams by a manifest file. Conditions—including, but not limited to, download speed, current available bandwidth, and the like—might cause a download client to select segments (or packets) from one or the other of the different bit rate bit rate streams. For example, for the first 30 seconds of a video stream, the download speed and/or current available bandwidth might be high. As such, the download client might select multiple segments having segment sizes between 2 and 10 seconds long from the high bit rate stream corresponding to the first 30 seconds of the video stream. In the next 15 seconds of the video stream, the download speed and/or currently available bandwidth might deteriorate. Accordingly, the download client might select multiple segments having segment sizes between 2 and 10 seconds long from the medium bit rate stream corresponding to the video stream between 30 seconds and 45 seconds. In the next 15 seconds of the video stream, the download speed and/or currently available bandwidth might further deteriorate, resulting in the download client selecting from the low bit rate stream corresponding to the video stream between 45 seconds and 60 seconds. In the next 60 seconds of the video stream, the download speed and/or bandwidth might improve significantly, resulting in the download client selecting again from the high bit rate stream corresponding to the video stream between 60 seconds and 120 seconds.
Various embodiments provide systems and techniques for auto-summary video and/or audio content. In some embodiments, a remote terminal, at a service provider, might retrieve video and/or audio content from a content database, and might transmit the video and/or audio content to one or more user devices (including, but not limited to, a tablet computer, a smart phone, a mobile phone, a personal digital assistant (“PDA”), a portable gaming device, a gaming console, a desktop computer, a laptop computer, a set-top box (“STB”), a television set (such as a broadcast TV, a high-definition TV (“HDTV”), an Internet Protocol TV (“IPTV”), and/or a digital TV, or the like). A user associated with each user device might enter one or more inputs, including one or more triggers, which might correspond to the user's favorite scene or favorite segment of the video and/or audio content. In some embodiments, the one or more inputs might be associated with actions or sounds recorded from a live audience at a live presentation of a show (e.g., a movie, a play, a live TV broadcast, etc.). In other embodiments, the one or more inputs might include cue tones provided by a broadcaster, a distributor, or a producer of the video and/or audio content. A summary server, which might be located at the service provider, might receive the one or more inputs, might associate the one or more inputs with one or more time codes corresponding to the video and/or audio content, and might store the one or more time codes in a time code database.
The summary server might retrieve the one or more time codes from a time code database, and might analyze the one or more time codes to determine at least one selected time code among the one or more time codes that exceeds a predetermined number of selections (which might include a set number of selections, a percentage of user selections from among participating users, or the like) from a plurality of users associated with a plurality of user devices. The one or more triggers might include, without limitation, user-inputted triggers, triggers associated with actions by a live audience, or broadcaster/distributor/producer-provided triggers (i.e., cue tones such as SCTE 35 or SCTE 130 cue tones, or the like). In other words, the one or more triggers might correspond to popular scenes or popular segments of the video and/or audio content, and/or might correspond to what the broadcast/distributor/producer deems to be scenes or segments that audiences might enjoy. The summary server might determine one or more selected segments of video or audio content corresponding to the at least one selected time code, and might create one or more summary tracks, where each of the one or more summary tracks might comprise each of the one or more selected segments of the video or audio content. The one or more summary tracks may be compiled into a summary sequence, which might be stored in a summary server and/or might be transmitted to one or more user devices for streaming the summary sequence thereon.
In some embodiments, the one or more user devices 105 might comprise one or more of a tablet computer 105a, a smart phone 105b, a mobile phone 105c, a personal digital assistant (“PDA”) 105d, and/or a portable gaming device 105e.
The one or more user devices 105 might, in some cases, comprise one or more of a telephone set 105f, an image capturing device 105g, a wired audio input device 105h, a wireless audio input device 105i, and a remote control device 105j. The telephone set 105f might include one of more of a plain old telephone system (“POTS”) telephone, a voice over Internet Protocol (“VoIP”) telephone, a land-line telephone, or the like, any of which might be a table-top, counter-top, or wall-mounted telephone. The image capturing device 105g might include, without limitation, a webcam, a digital camera, an SLR camera, a video camera, or any other suitable image capturing device. The wired audio input device 105h might include, but is not limited to, a wired microphone, or the like, while the wireless audio input device 105i might include, but is not limited to, a wireless microphone, or the like. The remote control device 105j might include, without limitation, a television remote controller, a set-top box remote controller, a universal remote controller, or any other suitable remote control device for any household electronic device, and the like.
In some embodiments, the one or more user devices 105 may comprise one or more of a gaming console 105k, a high-definition (“HD”) television 1051, a desktop computer 105m, a laptop computer 105n, a broadcast television 105o, an Internet Protocol television (“IPTV”) 105p, a set-top box (“STB”) 105q, and a cable television 105r. According to some examples, the HD TV 1051, the broadcast TV 105o, the IPTV 105p, and the cable TV 105r may be interchangeable.
The portable devices 110 may, in some cases, include any one or more of the tablet computer 105a, the smart phone 105b, the mobile phone 105c, the personal digital assistant (“PDA”) 105d, and/or the portable gaming device 105e, or any other suitable portable electronic device, and the like.
The user input devices 115, according to some embodiments, might include any one or more of the telephone set 105f, the image capturing device 105g, the wired audio input device 105h, the wireless audio input device 105i, and the remote control device 105j, as well as any of the portable user devices 110, the desktop computer 105m, the laptop computer 105n, or any other suitable electronic, audio (including voice recognition), image-capturing, or mechanical input devices, and the like.
The display/presentation user devices 120 might comprise any of the portable user devices 110, as well as any of desktop computer 105m, laptop computer 105n, or any of the televisions (including, without limitation, HDTV 1051, broadcast TV 105o, IPTV 105p, cable TV 105r, or the like), or any suitable electronic or other device capable of display video images, outputting audio recordings (e.g., music, soundtracks, voice data, and the like).
In operation, there might be two distinct stages for auto-summarizing video and/or audio content. The first stage might include tracking, compiling, tallying, and/or analyzing user input, while the second stage might include creating and compiling a summary sequence of favorite or popular segments of the video and/or audio content, based on the user input or a statistical analysis of user inputs. According to some embodiments, the first and second stages might be performed in sequence by one or more associated parties. In other embodiments, only the first stage or only the second stage may be performed one or more associated parties, while the other of the first or second stage may be performed by other parties.
In some cases, the first stage might comprise one or more users among a plurality of users entering one or more user inputs into one or more user devices 105. The one or more user inputs might comprise one or more triggers. In some embodiments, each trigger might comprise an event including, without limitation, at least one user actuating a button on at least one user device, at least one user providing sound input into a microphone or suitable audio input device of at least one user device, and/or at least one user providing a physical gesture(s) captured by an image or video capturing device, or the like.
In some examples, actuating a button might comprise at least one of depressing a hard button 180 on the at least one user device, interacting with a touchscreen display of the at least one user device, toggling a switch on the at least one user device, and/or actuating a lever on the at least one user device. Interacting with a touchscreen display of the at least one user device might include at least one of actuating one or more soft buttons 180a on the touchscreen display of the at least one user device, swiping a surface or interactive bar 180b of the touchscreen display, and/or drawing a predetermined pattern 180c on the touchscreen display. The sound input, according to some examples, might be received by the wired audio recording device 105h, the wireless audio recording device 105i, or any other suitable sound recording device that is either part of any other user device 105 or used in conjunction with the other user devices 105. The sound input might comprise at least one of one or more user-designated keywords, one or more default keywords, clapping sounds, sounds of laughter, sounds of cheering, sounds of screaming, or sounds of shouting, wherein each of the clapping sounds, laughter, cheering, screaming, and/or shouting exceeds a predetermined decibel level (e.g., exceeding 65 decibels (dB), preferably exceeding 70 dB, more preferably exceeding 80 dB, and in some cases exceeding 90 dB, etc.).
Alternatively, or additionally, the one or more triggers might be associated with actions by a live audience that might be captured by the one or more recording devices 195. For example, the one or more triggers might comprise an action (including, but not limited to, clapping, standing, pointing, mass movement toward person or object, or the like) that might be captured by video cameras, that when analyzed by suitable image/action recognition software executed by a computer system, might function as a trigger. In some cases, the one or more triggers might comprise an action (including, without limitation, an audience clapping, an audience laughing, an audience cheering, an audience screaming, an audience shouting, and an audience whistling, or the like) that might cause a sound level to exceed a predetermined decibel level, where the sound level might be recorded and/or detected by an audio input device. The predetermined decibel level might be a level exceeding 65 dB, preferably exceeding 70 dB, more preferably exceeding 80 dB, and in some cases exceeding 90 dB, or the like.
According to some embodiments, the video or audio content may be encoded with a set of time codes based on clock 170. In other embodiments, the clock 170 might track the run time of the video or audio content being displayed/presented on a user device 105; in such cases, the clock 170 might be the clock of the user device 105 displaying/presenting the video or audio content. In some cases, particularly involving the live audience, the clock 170 might be associated with the recording devices 190 that record the actions/sounds of the live audience, record the show that is presented before the live audience, or both.
In some embodiments, the one or more triggers might be associated with one or more time codes corresponding to segments of the video or audio content. In particular, the one or more triggers might be associated with what the one or more users (or live audience members) consider favorite scenes or segments, or otherwise deem to be of interest in the video or audio content (or in the show being presented), while the one or more time codes might be associated with the particular segments of the video or audio content (or the show being presented) at which the one or more triggers are made. The summary server 155 might receive the one or more triggers from one or more users (i.e., from one or more user devices) or from the venue 185. In some embodiments, the summary server 155 might determine one or more time codes associated with the video or audio content (or show being presented to the live audience 190), each corresponding to the one or more triggers, and might store the one or more time codes in the time code database 160. In some cases, the one or more triggers from one or more users (preferably from a plurality of users) may be stored on a database (e.g., in the time code database 160 with one or more time codes, or in a similar database, or the like).
The summary server 155 might, in some embodiments, analyze the one or more time codes to determine at least one selected time code among the one or more time codes that exceeds a predetermined number. The predetermined number might be a fixed number (e.g., 100, 200, 500, 1000, 10000 users, or the like) or might be a percentage of the number of users sending inputs (e.g., 5%, 10%, 15%, 25%, 50%, 70%, or the like). For example, if 100 users entered user inputs into respective user device 105 in response to viewing a particular movie, and the predetermined number is 30% of users sending inputs for that particular movie, then 30 or more triggers (from 30 or more users) corresponding to a selected time code of about 10 minutes into the movie might result in the summary server 155 determining that the time code of about 10 minutes is popular or otherwise interesting. In some cases, each selected time code might include a central selected time code (based, e.g., on average time codes or mean time codes close to the 10 minute mark) and a range of time codes spanning a predetermined number of seconds (e.g., 1, 2, 5, 10, 15, 30, 45, and 60 seconds, or the like) before and after the central selected time code. The range of time codes takes into account user lag in entering the user input (i.e., the trigger), while more accurately tracking segments of the video or audio content that might be deemed popular or interesting. Once the one or more selected time codes have been determined, the summary server 155 might store the selected one or more time codes in the time code database 160.
In the second stage, the summary server 155, the video/audio analyzer 165, or the summary server 155 using the video/audio analyzer 165 might analyze the video or audio content to determine one or more selected segments of the video and audio content corresponding to the at least one selected time code. In some cases, this might involve retrieving the selected one or more time codes from the time code database 160. Based on this determination, the summary server 155 might create one or more summary tracks, each comprising each of the one or more selected segments. According to some embodiments, each summary track might include a back-stepped segment, a forward-stepped segment, both, or none. Each of the back-stepped segment and the forward-stepped segment might be a segment of the video or audio content that spans a particular period and might have a particular duration, which may include, without limitation a duration of 30 seconds, 1 minute, 1.5 minutes, 2 minutes, 3 minutes, 5 minutes, or 10 minutes, or may include, but is not limited to, a range between 30 seconds and 10 minutes, a range between 30 seconds and 3 minutes, and a range between 2 minutes and 5 minutes, or the like. Embodiments having the back-step and/or forward-step are described in greater detail with respect to
According to some embodiments, the summary server 155 might compile the one or more summary tracks into a summary sequence. In some cases, the summary sequence might be stored in the summary database 175. The remote terminal 135 might, according to some examples, retrieve the summary sequence from the summary database 175 and transmit the summary sequence to at least one user device 105 (such as the portable user devices 110 and/or the display/presentation user devices 120).
In some examples, the creating and compiling of the summary tracks into a summary sequence might take advantage of adaptive bitrate (“ABR”) streaming techniques or concepts, as described in greater detail below with respect to
In some embodiments, in addition or alternative to user-inputted triggers, the one or more triggers might comprise one or more cue tones provided by a broadcaster, distributor, producer, or the like of the video or audio content. In some cases, each of the one or more cue tones might be based on a cue tone standard such as Society of Cable Telecommunications Engineers (“SCTE”) standards, including, without limitation the SCTE 35 standard, the SCTE 130 standard, or the like. The use of cue tones is described in greater detail with respect to
The network 125 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of commercially-available (and/or free or proprietary) protocols, including without limitation TCP/IP and the like. Merely by way of example, the network 125 can include a local area network (“LAN”), including without limitation a fiber network, an Ethernet network, a Token-Ring™ network and/or the like; a wide-area network; a wireless wide area network (“WWAN”); a virtual network, such as a virtual private network (“VPN”); the Internet; an intranet; an extranet; a public switched telephone network (“PSTN”); an infra-red network; a wireless network, including without limitation a network operating under any of the IEEE 802.11 suite of protocols, the Bluetooth™ protocol known in the art, and/or any other wireless protocol; and/or any combination of these and/or other networks.
Embodiments can also include one or more server computers. In some cases, one or more of the remote terminal or pedestal 135, the media center 140, media server 145, summary server 155, and/or video/audio analyzer 165 may each be embodied as one or more server computers. Each of the server computers may be configured with an operating system, including without limitation any of those discussed above, as well as any commercially (or freely) available server operating systems. Each of the servers may also be running one or more applications, which can be configured to provide services to one or more clients 105a through 105r, and/or other servers.
Merely by way of example, one of the servers 145 (or remote terminal 135, media center 140, summary server 155, video/audio analyzer 165, etc.) may be a web server, which can be used, merely by way of example, to process requests for web pages or other electronic documents from the user devices (105a-105r). The web server can also run a variety of server applications, including HTTP servers, FTP servers, CGI servers, database servers, Java servers, and the like. In some embodiments of the invention, the web server may be configured to serve web pages that can be operated within a web browser on one or more of the user devices 105 to perform methods of the invention. The servers 135, 140, 145, 155, and 165 may also function as a middleware server that provides content. For example, middleware servers may include a server for accessing media content and/or data, among other middleware servers. In addition, databases (e.g., one or more of databases 150, 160, and 175) may be embodied as middleware or database servers.
The server computers 135, 140, 145, 155, and 165, in some embodiments, might include one or more application servers, which can be configured with one or more applications accessible by a client running on one or more of the user devices (105a-105r). Merely by way of example, the server(s) 135, 140, 145, 155, and 165 can be one or more general purpose computers capable of executing programs or scripts in response to instructions/requests from the user devices 105, including without limitation web applications (which might, in some cases, be configured to perform methods provided by various embodiments). Merely by way of example, a web application can be implemented as one or more scripts or programs written in any suitable programming language, such as Java™, C, C#™ or C++, and/or any scripting language, such as Perl, Python, or TCL, as well as combinations of any programming and/or scripting languages. The application server(s) can also include database servers, including without limitation those commercially available from Oracle™, Microsoft™, Sybase™, IBM™ and the like, which can process requests from clients (including, depending on the configuration, dedicated database clients, API clients, web browsers, etc.) running on one or more user devices 105. In some embodiments, an application server can create web pages dynamically for displaying the information in accordance with various embodiments, such as displaying a selection screen for an on-demand channel(s) as a web page(s). Data provided by an application server may be formatted as one or more web pages (comprising HTML, HTML5, JavaScript, etc., for example) and/or may be forwarded to user devices 105 (as described above, for example). Similarly, a web server might receive web page requests and/or input data from the user devices 105 and/or forward the web page requests and/or input data to an application server. In some cases, a web server may be integrated with an application server.
In accordance with further embodiments, one or more servers 135, 140, 145, 155, and 165 can function as a file server and/or can include one or more of the files (e.g., application code, data files, etc.) necessary to implement various disclosed methods, incorporated by an application running on user devices 105. Alternatively, as those skilled in the art will appreciate, a file server can include all necessary files, allowing such an application to be invoked remotely by one or more the user devices 105.
It should be noted that the functions described with respect to various servers herein (e.g., media content server, etc.) can be performed by a single server and/or a plurality of specialized servers, depending on implementation-specific needs and parameters.
In certain embodiments, the system can include one or more databases including, but not limited to, databases or DBs 150, 160, and 175, as described in greater detail with respect to the figures below. The location of the database(s) 150, 160, and 175 is discretionary: merely by way of example, content database 150 might reside on a storage medium local to (and/or resident in) media center 140 or video server 145. Alternatively, database 150 can be remote from any or all of media center 140 or video server 145, so long as it can be in communication (e.g., via the network 125 or similar network) with one or more of these. Likewise, time code database 160 and/or summary database 175 might reside on a storage medium local to (and/or resident in) summary server 155. Alternatively, database 160 and/or 175 can be remote from summary server 155, as long as they can be in communication (e.g., via network 125 or similar network, with summary server 155.
In a particular set of embodiments, a database 150, 160, and/or 175 can reside in a storage-area network (“SAN”) familiar to those skilled in the art. In one set of embodiments, the database 150, 160, and/or 175 can be a relational database, such as an Oracle database, that is adapted to store, update, and retrieve data in response to SQL-formatted commands. The database might be controlled and/or maintained by a database server, as described above, for example.
In addition, various user devices (including, but not limited to, user devices 105a-105r shown in
In some embodiments, such as with broadcast movies or television shows, the triggers received from the users might be associated with time codes corresponding to the actual broadcast times, whereas in other embodiments, because time codes are associated with the video/audio content, the time codes can be separate from any actual broadcast times. As such, issues with download speeds and the like need not be considered. In such cases, each of the responding user devices need not receive the video/audio content 205 or transmit the triggers 215 at the same time as other user devices streaming the same video/audio content 205, for the same segment(s) of video/audio content 205.
Summary tracks 220, which might be created based on an association of each peak or each central peak portion 245 with time codes 210 and the video/audio content 205, might include summary tracks A 220a and summary tracks B 220b. In
Likewise, summary tracks B 220b might include one or more selected segments 250b of video/audio content 205, each corresponding to one of the six peaks 240, and each of the one or more selected segments 250b might be compiled to create summary sequence B 225b. Each selected segment 250b might have a duration x″ corresponding to a nominal width of the segment 250b centered about the central peak portion 245. In some cases, the nominal width (i.e., duration x″) might include a few seconds (e.g., 1, 2, 3, 5, 10, or 15 seconds, etc.) of the video/audio content 205. Similar to summary sequence A 225a, summary sequence B 225b might comprise intervals 255b between each selected segment 250b. In some embodiments, the intervals 255b might be zero seconds in length or 1, 2, or a few seconds in length.
In
In some embodiments, summary tracks with BackStep 320a might comprise a back-stepped segment 360 that extends backward in time from each selected segment 350, which might correspond to either selected segments 250a or 250b as described above with respect to
According to some embodiments, instead of a back-stepped segment 360, a forward-stepped segment 365 might be included that extends forward in time from each selected segment 350. For example, summary tracks with ForwardStep 320b might comprise one or more selected segment-forward-stepped segment pairs each corresponding to each peak 340. Compiling the summary tracks 320b, the resultant summary sequence with ForwardStep 325b might comprise one or more selected segment-forward-stepped segment pairs and intervals 355 between each selected segment-forward-stepped segment pair.
In other embodiments, both a back-stepped segment 360 and a forward-stepped segment 365 might be included that extends backward in time and forward in time, respectively, from each selected segment 350. For example, summary tracks with BackStep/ForwardStep 320c might comprise one or more back-stepped segment-selected segment-forward-stepped segment combinations each corresponding to each peak 340. Compiling the summary tracks 320b, the resultant summary sequence with BackStep/ForwardStep 325c might comprise one or more back-stepped segment-selected segment-forward-stepped segment combinations and intervals 355 between each back-stepped segment-selected segment-forward-stepped segment combination.
Each selected segment 350 might have a duration x, which might correspond to duration x′ or duration x″ as described above with respect to
Today, the most common format for mobile video content is ABR streaming. These systems are now beginning to be offered for set-top box (“STB”) and television viewing systems. Various embodiments herein take advantage of the nature of standard ABR streams to make it very easy to change (full) live streamed video (or audio) content into video (or audio) on demand assets. For example, a live video stream might be made up of a series of 2-10 second segments, which might be played back from a playlist called a “manifest file.” The manifest file contains all the segments that make up the stream. The various embodiments identify the selected segments 250 of the video/audio 405, and capture the ABR segments 480, 495 corresponding to these selected segments 250 as they are sent for playback and uses these to “build” the video on demand (“VOD”) file or audio on demand (“AOD”) file. The system then creates a new manifest file 490 with the captured ABR segments 480, 495 to create the “whole” or entire VOD or AOD file. As with standard ABR streaming, there is no need to re-encode the file. It is simply a matter of capturing the segments and making a new manifest file. This process of utilizing the nature of ABR streaming is shown in greater detail in
With reference to
Taking the examples as shown in
Similarly, for each selected segment 250b in summary tracks B 220b, corresponding ABR segments 495 may be compiled, where each of the ABR segments might comprise different bit rate segments 475. For example, as shown in
At block 520, the summary server (e.g., summary server 155) might receive one or more inputs, which might include one or more triggers (as described in detail above), from at least one user associated with the at least one use device. The summary server, at block 525, might determine one or more time codes each corresponding to each of the one or more triggers, and might store, at block 530, the one or more time codes in a time code database (e.g., time code database 160). The summary server might analyze the one or more time codes, in some cases using a video/audio analyzer (e.g., video/audio analyzer 185) to determine at least one selected time code among the one or more time codes that exceeds a predetermined number (block 535). The at least one selected time code might be determined within a range of tolerance, in order to account for user input lag or the like. For example, each selected time code might include a central selected time code (which might correspond to an average, a mean, a maximum or other statistically number of triggers, or the like), and a range of time codes spanning a predetermined number of seconds (e.g., 1, 2, 3, 4, 5, or 10 seconds, or the like) before and after the central selected time code. According to some embodiments, the predetermined number might be a fixed number (e.g., 100, 200, 500, 1000, 10000 users, or the like) or might be a percentage of the number of users sending inputs (e.g., 5%, 10%, 15%, 25%, 50%, 70%, or the like).
At block 540, the summary server might determine one or more selected segments of the video and/or audio content corresponding to the at least one selected time code. The summary server might, at block 545, create one or more summary tracks, each comprising each of the one or more selected segments. Each summary track might include a back-stepped segment, a forward-stepped segment, or both segments (as shown in
The summary server might compile the one or more summary tracks into a summary sequence (block 550), and might, in some cases, store the summary sequence in a summary database (e.g., summary database 175) (block 555). According to some embodiments, the summary server might, at block 560, transmit the summary sequence to at least one user device among a plurality of user devices for streaming the summary sequence thereon. In some cases, the at least one user device at block 560 might be the same as the at least one user device at block 510. In other examples, the at least one user device at block 560 might be different from the at least one user device at block 510.
At block 615, a video or audio recording might be made of the live audience (e.g., using video/audio recording devices 195). A summary server (e.g., summary server 155) might, at block 620, analyze the video or audio recording of the live audience to identify one or more triggers (e.g., laughter, applause, cheering, shouting, screaming, or the like) that exceed a predetermined threshold level. In some embodiments, video analysis software programs might be used to analyze video recordings of the live audience to determine a trigger action including, but not limited to, clapping, standing, pointing, mass movement toward a person(s) or object(s) or the like that exceed a predetermined threshold level, which might include, without limitation, a certain pre-set number of people performing acts that might be classified as one or more of the trigger actions mentioned above, or the like. In other embodiments, the one or more triggers might comprise an action that might cause a sound level to exceed a predetermined decibel level; such actions might include, without limitation, an audience clapping, an audience laughing, an audience cheering, an audience screaming, an audience shouting, and an audience whistling, or the like. The predetermined decibel level might be a level exceeding 65 dB, preferably exceeding 70 dB, more preferably exceeding 80 dB, and in some cases exceeding 90 dB, or the like.
The summary server, at block 625, might determine one or more time codes each corresponding to each of the one or more triggers, and might store, at block 630, the one or more time codes in a time code database (e.g., time code database 160). The summary server might analyze the one or more time codes, in some cases using a video/audio analyzer (e.g., video/audio analyzer 185) to determine at least one selected time code among the one or more time codes that exceeds a predetermined number (block 635). The at least one selected time code might be determined within a range of tolerance, in order to account for user input lag or the like. For example, each selected time code might include a central selected time code (which might correspond to an average, a mean, a maximum or other statistically number of triggers, or the like), and a range of time codes spanning a predetermined number of seconds (e.g., 1, 2, 3, 4, 5, or 10 seconds, or the like) before and after the central selected time code. According to some embodiments, the predetermined number might be a fixed number (e.g., 100, 200, 500, 1000, 10000 users, or the like) or might be a percentage of the number of users sending inputs (e.g., 5%, 10%, 15%, 25%, 50%, 70%, or the like).
At block 640, the summary server might determine one or more selected segments of the video recording corresponding to the at least one selected time code. The summary server might, at block 645, create one or more summary tracks, each comprising each of the one or more selected segments. Each summary track might include a back-stepped segment, a forward-stepped segment, or both segments (as shown in
The summary server might compile the one or more summary tracks into a summary sequence (block 650), and might, in some cases, store the summary sequence in a summary database (e.g., summary database 175) (block 655). According to some embodiments, the summary server might, at block 660, transmit the summary sequence to at least one user device among a plurality of user devices (e.g., user devices 105 shown in
At block 715, the summary server (e.g., summary server 155) might identify one or more segments of the video and/or audio content associated with the one or more triggers (e.g., SCTE 35 or SCTE 130 cue tones, or the like). The summary server might, at block 720, create one or more summary tracks, each comprising each of the one or more segments associated with the one or more triggers (e.g., cue tone or the like). In some embodiments, each summary track might include a back-stepped segment, a forward-stepped segment, or both segments (as shown in
At block 725, the summary server might compile the one or more summary tracks into a summary sequence, and might, at block 730, store the summary sequence in a summary database (e.g., summary database 175). In some cases, the summery server might transmit the summary sequence to at least one user device among a plurality of user devices (e.g., user devices 105 shown in
The computer system 800 is shown comprising hardware elements that can be electrically coupled via a bus 805 (or may otherwise be in communication, as appropriate). The hardware elements may include one or more processors 810, including without limitation one or more general-purpose processors and/or one or more special-purpose processors (such as digital signal processing chips, graphics acceleration processors, and/or the like); one or more input devices 815, which can include without limitation a mouse, a keyboard, and/or the like; and one or more output devices 820, which can include without limitation a display device, a printer, and/or the like.
The computer system 800 may further include (and/or be in communication with) one or more storage devices 825, which can comprise, without limitation, local and/or network accessible storage, and/or can include, without limitation, a disk drive, a drive array, an optical storage device, solid-state storage device such as a random access memory (“RAM”) and/or a read-only memory (“ROM”), which can be programmable, flash-updateable, and/or the like. Such storage devices may be configured to implement any appropriate data stores, including without limitation, various file systems, database structures, and/or the like.
The computer system 800 might also include a communications subsystem 830, which can include without limitation a modem, a network card (wireless or wired), an infra-red communication device, a wireless communication device and/or chipset (such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a WWAN device, cellular communication facilities, etc.), and/or the like. The communications subsystem 830 may permit data to be exchanged with a network (such as the network described below, to name one example), with other computer systems, and/or with any other devices described herein. In many embodiments, the computer system 800 will further comprise a working memory 835, which can include a RAM or ROM device, as described above.
The computer system 800 also may comprise software elements, shown as being currently located within the working memory 835, including an operating system 840, device drivers, executable libraries, and/or other code, such as one or more application programs 845, which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the method(s) discussed above might be implemented as code and/or instructions executable by a computer (and/or a processor within a computer); in an aspect, then, such code and/or instructions can be used to configure and/or adapt a general purpose computer (or other device) to perform one or more operations in accordance with the described methods.
A set of these instructions and/or code might be encoded and/or stored on a non-transitory computer readable storage medium, such as the storage device(s) 825 described above. In some cases, the storage medium might be incorporated within a computer system, such as the system 800. In other embodiments, the storage medium might be separate from a computer system (i.e., a removable medium, such as a compact disc, etc.), and/or provided in an installation package, such that the storage medium can be used to program, configure, and/or adapt a general purpose computer with the instructions/code stored thereon. These instructions might take the form of executable code, which is executable by the computer system 800 and/or might take the form of source and/or installable code, which, upon compilation and/or installation on the computer system 800 (e.g., using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc.) then takes the form of executable code.
It will be apparent to those skilled in the art that substantial variations may be made in accordance with specific requirements. For example, customized hardware (such as programmable logic controllers, field-programmable gate arrays, application-specific integrated circuits, and/or the like) might also be used, and/or particular elements might be implemented in hardware, software (including portable software, such as applets, etc.), or both. Further, connection to other computing devices such as network input/output devices may be employed.
As mentioned above, in one aspect, some embodiments may employ a computer system (such as the computer system 800) to perform methods in accordance with various embodiments of the invention. According to a set of embodiments, some or all of the procedures of such methods are performed by the computer system 800 in response to processor 810 executing one or more sequences of one or more instructions (which might be incorporated into the operating system 840 and/or other code, such as an application program 845) contained in the working memory 835. Such instructions may be read into the working memory 835 from another computer readable medium, such as one or more of the storage device(s) 825. Merely by way of example, execution of the sequences of instructions contained in the working memory 835 might cause the processor(s) 810 to perform one or more procedures of the methods described herein.
The terms “machine readable medium” and “computer readable medium,” as used herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using the computer system 800, various computer readable media might be involved in providing instructions/code to processor(s) 810 for execution and/or might be used to store and/or carry such instructions/code (e.g., as signals). In many implementations, a computer readable medium is a non-transitory, physical, and/or tangible storage medium. Such a medium may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical and/or magnetic disks, such as the storage device(s) 825. Volatile media includes, without limitation, dynamic memory, such as the working memory 835. Transmission media includes, without limitation, coaxial cables, copper wire and fiber optics, including the wires that comprise the bus 805, as well as the various components of the communication subsystem 830 (and/or the media by which the communications subsystem 830 provides communication with other devices). Hence, transmission media can also take the form of waves (including without limitation radio, acoustic and/or light waves, such as those generated during radio-wave and infra-red data communications).
Common forms of physical and/or tangible computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read instructions and/or code.
Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to the processor(s) 810 for execution. Merely by way of example, the instructions may initially be carried on a magnetic disk and/or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and/or executed by the computer system 800. These signals, which might be in the form of electromagnetic signals, acoustic signals, optical signals, and/or the like, are all examples of carrier waves on which instructions can be encoded, in accordance with various embodiments of the invention.
The communications subsystem 830 (and/or components thereof) generally will receive the signals, and the bus 805 then might carry the signals (and/or the data, instructions, etc. carried by the signals) to the working memory 835, from which the processor(s) 805 retrieves and executes the instructions. The instructions received by the working memory 835 may optionally be stored on a storage device 825 either before or after execution by the processor(s) 810.
While certain features and aspects have been described with respect to exemplary embodiments, one skilled in the art will recognize that numerous modifications are possible. For example, the methods and processes described herein may be implemented using hardware components, software components, and/or any combination thereof. Further, while various methods and processes described herein may be described with respect to particular structural and/or functional components for ease of description, methods provided by various embodiments are not limited to any particular structural and/or functional architecture but instead can be implemented on any suitable hardware, firmware and/or software configuration. Similarly, while certain functionality is ascribed to certain system components, unless the context dictates otherwise, this functionality can be distributed among various other system components in accordance with the several embodiments.
Moreover, while the procedures of the methods and processes described herein are described in a particular order for ease of description, unless the context dictates otherwise, various procedures may be reordered, added, and/or omitted in accordance with various embodiments. Moreover, the procedures described with respect to one method or process may be incorporated within other described methods or processes; likewise, system components described according to a particular structural architecture and/or with respect to one system may be organized in alternative structural architectures and/or incorporated within other described systems. Hence, while various embodiments are described with—or without—certain features for ease of description and to illustrate exemplary aspects of those embodiments, the various components and/or features described herein with respect to a particular embodiment can be substituted, added and/or subtracted from among other described embodiments, unless the context dictates otherwise. Consequently, although several exemplary embodiments are described above, it will be appreciated that the invention is intended to cover all modifications and equivalents within the scope of the following claims.
This application is a continuation application of U.S. patent application Ser. No. 15/495,385, filed Apr. 24, 2017 by Steven M. Casey et al. and titled, “Auto-Summarizing Video Content System and Method”, which is a continuation application of U.S. patent application Ser. No. 13/826,810 (Now U.S. Pat. No. 9,667,937), filed Mar. 14, 2013 by Steven M. Casey et al. and titled, “Auto-Summarizing Video Content System and Method”, the disclosures of both of which are hereby incorporated by reference in their entirety for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8860882 | Bloch et al. | Oct 2014 | B2 |
| 9967937 | Robin et al. | May 2018 | B2 |
| 20070109446 | Jeong | May 2007 | A1 |
| 20120189273 | Folgner | Jul 2012 | A1 |
| 20120216121 | Lin | Aug 2012 | A1 |
| 20120219271 | Vunic | Aug 2012 | A1 |
| 20130343729 | Rav-Acha | Dec 2013 | A1 |
| 20140270699 | Casey et al. | Sep 2014 | A1 |
| 20170230730 | Casey et al. | Aug 2017 | A1 |
| 20180192158 | Smith | Jul 2018 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20180242053 A1 | Aug 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15495385 | Apr 2017 | US |
| Child | 15956469 | US | |
| Parent | 13826810 | Mar 2013 | US |
| Child | 15495385 | US |
