Patent Application
20230269446
The present application is generally related to implementing dynamic content serving using a media device.
Systems and techniques are described herein for implementing dynamic content serving using a media device. For instance, the dynamic content serving systems and techniques can substitute (e.g., in real-time or in near-real-time) one media segment for another media segment within a media stream with high accuracy and low latency. The media segment can include a video segment or other type of media (e.g., audio). In some cases, the media segment to be substituted into the media stream can be stored locally (e.g., stored in a cache memory or other local storage) on a media device playing the media stream. In some cases, the entire media segment or a portion of the media segment to be substituted into the media stream can be obtained (e.g., in real-time or near real-time) over a network from a content source. In some implementations, the media segment to be substituted into the media stream can be marked with an identifying watermark in a portion of the video data and/or audio data of the media segment.
In one illustrative example, the systems and techniques can perform dynamic content insertion, where a first item of third party content can be “spliced” into a video stream being displayed by a display device (e.g., a television) to replace a second item of third party content that was originally part of the original video programing. By substituting the first item of third party content into the video stream, the viewer of the video stream can see different content than what other viewers of the same video stream might see on their respective display devices. The determination of the third party content that is to be substituted for a specific viewer can be determined by a demographic profile of the viewer, a zip code of the viewer, and/or any other information.
Content substitution for various types of content requires precise timing (e.g., one-tenth of a second or other time period), since additional delay can cause the inserted content to lack visual quality. To achieve the strict timing required to substitute one media segment for another media segment, a watermark embedded in the program stream (e.g., a video watermark embedded into the video data of a video stream) can be used as a trigger to initiate certain actions of the dynamic content serving system. As noted above, in some cases, content that can be used for substitution can be stored locally (e.g., in cache memory) on a client device (e.g., a television, a mobile device, a set-top box, and/or other suitable client device), so that the content can be quickly accessed and used as a substitute for existing content. In some cases, part of the content or the entire content can be obtained from a content source (e.g., from one or more servers storing the content).
Furthermore, the electronic circuits of modern display devices and media processing devices (e.g., televisions, set-top boxes, etc.) are limited in their ability to rapidly switch video signals. To address this, the dynamic content serving systems and techniques described herein can utilize a picture-in-picture (PIP) video path available in display devices and media processing devices. In general, a PIP video path allows one media stream (e.g., a television channel, a movie, or other media) to be displayed in a PIP window at the same time (e.g., as an overlay, an inset window, or the like) as a second media stream being displayed on the full display in a full screen mode. In preparation for displaying one or more substitute media segments, the techniques and systems described herein can establish a PIP frame with a dimension of 0x0 so that a PIP window is not visible when displayed with a full screen media stream. A media segment to be substituted at a given watermark trigger time is cued to play in the PIP window. When the watermark trigger is detected, the system can expand the existing PIP window from the 0x0 dimension to a larger dimension or to full-screen (in which case the PIP window is to be displayed in substitution of the original video segment).
In some examples, the techniques and systems described herein can perform additional processes prior to playout of the substitute media segment, such as decoding in advance the compressed video stream and then applying the stream to the PIP input process. Further details are described herein.
This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.
The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.
Illustrative embodiments of the present application are described in detail below with reference to the following figures:
Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of embodiments of the application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive.
The ensuing description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing an exemplary embodiment. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.
During early years of television and the growth of third party content-supported programming (e.g., advertising-supported programming), messages displayed on home televisions by necessity only promoted products or services with a relatively broad appeal. Because all viewers viewed the same third party content, companies generally targeted broad markets that would likely be tuned-in during a respective daypart. A daypart is a part of a day in which media content (e.g., television content) is divided. As cable television networks developed, and eventually became known as “multichannel video programming distributors” (MVPD), the ability to target some third party content on a regional, instead of a nationwide, basis enabled the presentation of third party content for more localized products and services, and attracted attempts to exploit known demographic preferences.
By the early 1990's, technologies were being developed with the intent to further divide-up or “cellularize” certain large MVPD systems, which enabled the targeting of specific neighborhoods with more relevant third party content. The shift to digital broadcasting of TV programing in the early 2000's and the parallel proliferation of more-advanced video distribution methodologies, such as digital cable, digital satellite, and broadband Internet service, was combined with network-connected televisions (referred to as smart televisions or smart TVs), smartphones, tablets, laptops, and other devices for viewing video programming.
Such advances in media distribution and display technologies have created an infrastructure that offers the opportunity for the development of systems and methods to display third party content messages on multiple devices and coordinate certain messages to the same user across those devices. Further, third party content may be substituted in real-time for the existing message, based on location or demographic information about a certain household; a process known as dynamic content insertion (DCI) and third party content redirection (also referred to as dynamic advertisement insertion and advertisement redirection).
Dynamic content serving systems, methods (also referred to as processes), and computer-readable media are described herein for implementing dynamic content serving. An example of the dynamic content servicing includes dynamic content insertion (DCI) and dynamic content re-direction. For example, the techniques described herein can be used to substitute (e.g., in real-time or in near-real-time) one media segment for another media segment within a media stream with high accuracy and low latency. The media segment to be substituted into the media stream can be stored locally (e.g., stored in a cache memory or other local storage) on a media device playing the media stream. In some examples, the entire media segment or part of the media segment (e.g., a portion of the media segment after an initial part of the media segment) can be obtained in real-time over a network from a content source. In some implementations, the media segment to be substituted into the media stream can be marked with an identifying watermark in a portion of the video data and/or audio data of the media segment.
The media content for which segments are substituted can include any type of media content, such as video, audio, media that includes both audio and video content, or other type of media content. While examples are described herein using video as an example, one of ordinary skill will appreciate that the techniques described herein can apply to other types of media content. Further, when discussing video content herein, the video can include audio content in some cases. The media segments that can be substituted for other media segments can include any media content, such as third party content (e.g., an advertisement or commercial message), a movie, a television show, an on-demand show (e.g., offered through a streaming video service, broadcast video service, etc.), or other media content.
In one illustrative example, the dynamic content serving system and associated techniques can perform DCI in response to a triggering event, such as watermark data. For example, in response to detecting a watermark in a video frame of a video stream being displayed by a display device (e.g., a television), a first video segment (e.g., a first item of third party content) can be “spliced” into the video stream to replace a second video segment (e.g., a second item of third party content) that was originally part of the original video programing. For example, the dynamic content serving system can identify video segments, such as third party content items (e.g., advertisements) arriving at a display device, can identify the specific video segment being received (e.g., by identifying a video watermark and/or audio watermark), and can determine in real-time (or near-real-time) an alternative video segment to be substituted for the original video segment.
By substituting the first item of third party content into the video stream, the viewer of the video stream can see different content than what other viewers of the same video stream might see on their respective display devices. For instance, the display of a certain television third party content item being broadcast as part of a television program can be substituted with a different third party content item, such as an item of third party content that is predetermined to be more relevant for that particular household or user. The determination of the third party content that is to be substituted for a specific viewer can be determined by a demographic profile of the viewer, a zip code of the viewer, and/or any other information.
Such third party content substitution is dynamic so that it can occur any time a third party content item that is eligible to be replaced is detected to be currently displayed by a media device (e.g., a network-connected or smart TV, a set-top box, a personal computer, a mobile phone, a tablet computer, and/or other media device). In one example, techniques described herein address an existing need to more narrowly target third party content to specific audiences to keep television third party content offerings (e.g., advertising) competitive with alternative digital media platforms, among other uses.
Using watermark data or other means for triggering substitution of media segments can be used to satisfy the precise timing requirements for performing content substitution for some content, where any additional delay (also referred to as latency), can be visually objectionable to a TV viewer and/or can cause visual confusion, among other undesirable artifacts. A video watermark embedded in program stream where the substitution is to occur is an effective triggering mechanism for precise timing of the dynamic content serving system when performing DCI. Another example of a method of triggering a media segment substitution event can include a video matching system internal to the media device to identify video segments being displayed, product logos, entire video frames, or other portion of a video stream being displayed. Another example of a method of triggering a media segment substitution event can include decoding an audio watermark signal to trigger. Other methods can also be used to trigger substitution of media segments.
While the dynamic content serving system 100 is shown to include certain components, one of ordinary skill will appreciate that the dynamic content serving system 100 can include more or fewer components than those shown in
In another example, the media system 101 can include, or can be part of a computing device that includes, one or more input devices (not shown), one or more output devices (not shown), one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and/or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and/or other processing devices) in communication with and/or electrically connected to the one or more memory devices, one or more wireless interfaces (e.g., including one or more transceivers and a baseband processor for each wireless interface) for performing wireless communications, one or more wired interfaces (e.g., a serial interface such as a universal serial bus (USB) input, a lightening connector, and/or other wired interface) for performing communications over one or more hardwired connections, and/or other components that are not shown in
In one illustrative example, a computing device used to implement the media system 101 is a media device, such as a television (e.g., a network-connected or smart television), a set-top box, a personal computer, a tablet computer, a mobile device (e.g., a mobile phone or other mobile device), a wearable device (e.g., a smart watch, a virtual reality headset, an augmented reality headset, and/or other wearable device), a server (e.g., in a software as a service (SaaS) system or other server-based system), and/or any other computing device with the resource capabilities to perform the techniques described herein.
The third party content database 121 can include candidate media segments (referred to herein as third party content items) that can be substituted for other media segments (third party content items) that are scheduled to be presented with media content 108. The media content 108 can include any type of media content, such as television programming, streaming (or over-the-top (OTT)) content that can include shows, movies, or other content, any combination thereof, and/or other media content. The third party content items in the third party content database 121 can include advertisements, recommended media content (e.g., a movie, show, song, or other media to recommend to a user for viewing or listening), or other type of content. In one illustrative example, the media content 108 includes television programming and the third party content items in the third party content database 121 include commercials that can be substituted to replace commercials that are originally scheduled to play with the television programming.
In some examples, to achieve low latency in the provisioning of content substitution, the candidate third party content items can be stored locally in the media system 101 in the content/data cache 107. In some implementations, a local storage device other than cache can be used to store the third party content items, such as a database, random access memory (RAM), read only memory (ROM), or other storage mechanism. Storing the third party content items locally can allow the media system 101 to quickly retrieve and display a third party content item in response to detecting a trigger such as a watermark. In some cases, all available candidate third party content items can be received and stored locally by the media system 101. In some cases, only candidate third party content items that can be substituted for a currently-selected media content item (e.g., a television channel, a movie, a show, etc.) are received and stored locally by the media system 101. In one illustrative example, the media content 108 includes television programming as noted above. In response to the media system 101 changing to a particular television channel (e.g., in response to a user input), the DCI client 102 or the DCI media processor 106 can request all candidate third party content items that relate to the television channel from the third party content server 120. The third party content server 120 can then send the requested candidate third party content items and can store them in the content/data cache 107 or other local storage. In some cases, in the example of a television channel, the candidate third party content items that relate to a certain daypart or time period of the television channel (e.g., a next hour, from a certain starting time point to an ending time point such as from Noon to 5:00 μm, or other time period) can be requested from the third party content server 120.
In some cases, there can be too many third party content items for the media system 101 to store locally in the content/data cache 107 or other local storage. For instance, on a contemporary Multichannel Video Programming Distributor (MVPD) system, there can be many third party substitution opportunities across many television program channels that can be too numerous for a media device (e.g., a television) to store in an internal cache 107 or other local storage. In such cases, for example, techniques are provided that can advantageously assist in performing dynamic content insertion. In one illustrative example, the DCI client 102 or the DCI media processor 106 of the media system 101 can request from the third party content server 120 a set of partial media segments (e.g., video segments) representing a first portion of a plurality of candidate third party content items that can be substituted for a particular item of media content. The third party content server 120 can send the set of partial media segments to the DCI media processor 106 or the DCI client 102, which can send the partial media segments to the content/data cache 107 or other local storage. In one illustrative example, the media content 108 includes television programming as noted above. In response to the media system 101 changing to a particular television channel (e.g., in response to a user input), the DCI client 102 or the DCI media processor 106 can send a request to the third party content server 120 to return a set of partial media segments representing a first portion of all candidate third party content items that relate to the television channel. In some cases, in the example of a television channel, the partial media segments of candidate third party content items that relate to a certain daypart or time period of the television channel can be requested from the third party content server 120. The third party content server 120 can then send the requested partial media segments and can store them in the content/data cache 107 or other local storage.
In some examples, the length or duration of a partial media segment representing a first portion of a candidate third party content item can be the same or a certain amount greater (e.g., 100 milliseconds (msec), 200 msec, 500 msec, or other suitable amount of time) than the latency required to communicate a request to the third party content server 120 and to then receive the remainder of the third party content item (the rest of the third party content item in addition to the first portion) that is to be substituted based on detection of a trigger (e.g., decoded information from a watermark of a detected video segment detected by the watermark decoder 105). In one illustrative example, a worst case third party content item substitution system latency can be 90 msec from the time of detection of a video watermark by decoder 105 to a request from the DCI media processor 106 to the third party content server 120 and then to the start of reception of the remainder of the third party content item to be substituted. In such an example, the length of a partial media segment representing a first portion of a candidate third party content item that will be stored in advance of detecting the watermark can include 200 msec of the candidate third party content item. By storing an amount of the candidate third party content items equal to at least the duration of the latency noted above, the media system 101 can begin playing the partial media segment of a third party content item without delay, while obtaining the rest of the third party content item before the partial media segment is finished playing.
In some examples, a media substitution event (e.g., based on a detected watermark or other trigger) can result in the need for a third party content item that is not currently stored in the content/data cache 107 or other local storage. In such examples, DCI media processor 106 or the DCI client 102 can send a request to the third party content server 120 send a specific third party content item. The third party content server 120 can send the third party content item to the DCI media processor 106 or the DCI client 102, which can instruct the video processor and display 103 to display the third party content item once it is received and decoded.
In some examples, the third party content items that are candidates for substitution can be provided to the media system 101 from multiple, disparate third party content servers (in addition to the third party content server 120). Regardless of the number of third party content servers that can provide third party content, the dynamic third party content controller host 122 can provide a network management service as well as load balancing so that any one third party content server does not become overwhelmed by potentially too many simultaneous requests.
In some examples, the media system 101 can accommodate for the relatively high latency of the electronic circuits of modern media devices (e.g., televisions, set-top boxes, among others) that causes delays in the ability of the media devices to rapidly switch between video signals. For example, the media system 101 can utilize a picture-in-picture (PIP) video path. A PIP video path allows media content (e.g., a television channel, a movie, or other media) to be displayed in a PIP window at the same time (e.g., as an overlay, an inset window, or the like) as other media content that is being displayed in a full screen mode. In preparation for displaying one or more substitute third party content items, the media system 101 can establish a PIP frame with a dimension of zero-by-zero (0x0) so that a PIP window is not visible when displayed at the same time as a full screen media stream is displayed. A third party content item to be substituted can be cued to play in the PIP window in response to a trigger (e.g., when a watermark is detected). When the trigger is detected, the system can expand the existing PIP window from the 0x0 dimension to a larger dimension and/or to full-screen. When displayed in full-screen, the third party content item in the PIP window is displayed in substitution of the original video segment.
In some examples, stored third party content items in the content/data cache 107 (or other storage) can be synchronized with the media content 108 by the DCI media processor 106 or the DCI client 102. As noted above, the stored third party content items can include partial media segments including first portions of various third party content items, with remaining portions of the third party content items being provided by third party content server 120 upon request by the DCI media processor 106 or the DCI client 102 (e.g., in response to detecting a watermark in the media content 108). To synchronize a third party content items with the media content 108, the DCI media processor 106 or the DCI client 102 can decode the third party content item into uncompressed video (and in some cases uncompressed audio) and can place the decoded data in an audio-video buffer (e.g., the buffer of the audio-video decoder 503 shown in
In some examples, to provide an uninterrupted play out of a substitute third party content item, the DCI media processor 106 or the DCI client 102 can encode the third party content item so that it is in a video format (e.g., according to MPEG2, H.264, H.265, or other video specification) or other media format used by the media system 101. For example, the program stream containing the received third party content item from the third party content server 120 can be encoded such that the resulting third party content item is made up of the remainder of the stored third party content item (a partial media segment of the third party content item, such as 200 msec, stored in content/data cache 107) followed by the remainder of the third party content item remotely accessed from the third party content server 120 to form a valid program stream in the media format used by the media system 101.
In some cases, as noted above, an entire item of third party content (e.g., an entire advertisement, an entire substitute movie, or other item of third party content) can be stored locally in the media system 101 (e.g., in the content/data cache 107 or other storage). In one illustrative example, a substitute third party content item can be stored in unused random access memory (RAM). In another illustrative example, in another embodiment, a substitute third party content item can be stored in flash memory and/or other semi-permanent storage of the media system 101.
User viewing patterns can follow a Parrado curve, also known as the 80/20 law, meaning that 80% of the households will be turned to the same ten television channels and, hence, DCI clients in different media devices may be requesting the same block of substitute third party content items throughout various dayparts and especially during primetime viewing times. In some cases, certain likely substitution third party content items can be stored (e.g., cached or otherwise stored) in higher performance storage of the media system 101, such as in the content/data cache 107.
As noted above, a watermark can be used to trigger insertion of a third party content item. The watermark decoder 105 can detect watermark data (also referred to as a watermark trigger) in the media content 108 (e.g., in a frame of video provided in a channel of television programming). In some cases, the watermark decoder 105 can implement one or more software applications that are programmed to detect a watermark in one or more video frames of a program of the media content 108 currently being displayed. An example of the structure of a video watermark is described in the Advanced Television Standards Committer (ATSC) A/335 Video Watermarking Standard.
Upon detection of a watermark in the media content 108, the DCI media processor 106 can retrieve a third party content item from the content/data cache 107 (or other storage), which can be displayed in place of an original third party content item. In some cases, if only a partial media segment of a third party content item (e.g., corresponding to the first few seconds or other time period of the third party content item) is stored in the content/data cache 107 (or other storage), the watermark may contain an address (e.g., a uniform resource locator (URL)), which the media system 101 can b use to fetch the remainder of the third party content item from the third party content server 120. The DCI media processor 106 or DCI client 102 can instruct the video source switcher 104 to switch (e.g., via a media synchronization and multiplexer 501) to the third party content item stored in internal content/data cache 107. In response to detection of the watermark and switching to the third party content item, the video source switcher (PIP controller) 104 expands the existing PIP window (that has been maintained at a dimension of 0x) to full-screen, causing the PIP window to overlay the original video segment, as shown in
At the end of the third party content item (e.g., 30 seconds later), the video source switcher 104 can be instructed by the DCI media processor 106 or the DCI client 102 to switch back to the original program video. In some cases, if the media system 101 is instructed to switch media content (e.g., switch television channels), for example based on user input, during the period during which a substitute third party content item is being displayed, the overlay video (e.g., the PIP window) can be removed and the original program video can be displayed. In some cases, control logic can be provided in the DCI media processor 106 or the DCI client 102, which can cause the DCI media processor 106 or the DCI client 102 to stop displaying the substitute third party content item and switch back to the original programming when a channel change is detected. The control logic can also be used to serve third party content items and/or other information to devices (referred to as “second-screen” devices) that are on a same network (e.g., a wired or wireless local area network) as the media system 101. For example, third party content items sent by the media system 101 to second-screen devices can mirror the third party content item currently being displayed by the media system 101 or can be a related item of third party content and/or can include additional information about the third party content item being displaying (e.g., an opportunity to purchase a product presented in the third party content item).
In some examples, as an alternative to detecting a watermark, an upcoming media segment substitution (e.g., an advertisement break) can be determined by monitoring a television channel (or other media) that is upstream of the MVPD distribution to homes. Such monitoring can provide one, two, three, or more seconds of advanced detection of a video segment of interest, which is enough time to send a command to the appropriate media system (e.g., media system 101) to replace media content with a substitute third party content item. For example, a command from a network controller of an MVPD distribution system to the media system 101 can be expressed in plain English as “In 3.4 seconds, replace the third party content item on channel 4 with third party content number 11 in your local cache,” which can be performed by the media system 101 based on the media system 101 previously fetching and storing the replacement third party content item (with number 11) in the content/data cache 107 or other storage.
In some cases, the system 100 can produce additional helpful information to determine media system actions, such as knowledge of how long a media system has been continuously tuned to current media content (e.g., a current television channel), the type of program currently being viewed, prior viewing habits of devices or users in a same home as the media system 101 (e.g., by day and by daypart), user level behavioral characteristics collected by the system 100, any combination thereof, and/or other information. This information can enable the system 100 to react more responsively to a user's viewing habits and to support a multiple-screen environment that many viewers may utilize.
An example of video frames 801 can be seen in
In some examples, the system 100 can attempt to anticipate third party content item substitution opportunities based on a location of the media system 101, based on a current daypart (e.g., morning, afternoon, primetime, late night, among others), based on the channel and/or program stream that the media system 101 is currently displaying, any combination thereof, and/or other factors.
As noted above, the media system 101 can configure a PIP frame with a dimension of zero-by-zero (0x0) pixels in preparation for display of a third party content item in the PIP window, which is illustrated in
One illustrative example of a task performed by the DCI client 102 is the scheduling (in combination with the DCI media processor 106) of available dynamic third party content substitution events for every TV channel participating in the DCI. The DCI client 102 can accommodate any number of service providers offering dynamic third party content (e.g., dynamic advertisement media). In some cases, the dynamic advertising market will utilize common content standards such as the Interactive Advertising Bureau (IAB) promoted identification protocols called the Video Ad Serving Template (VAST). Even for those dynamic ad providers with proprietary protocols and metadata, the system can still accommodate all providers. In some cases, it can be assumed that the dynamic advertising market for television programming will adopt industry standards such as the ATSC Video Watermark as specified in document A/335. Proprietary protocols can be managed for those providers not conforming to any particular standard.
As described above, the techniques described herein provide an optimal platform for the most efficient deployment of a dynamic third party content insertion system. The requirement for low latency and rapid switching of the video signals needed to perform a visually acceptable dynamic third party content insertion system has been met by combining the use of a PIP video channel with optional local processing of the third party content. Such systems and techniques can maintain an optimally responsive service with third party content substitution and/or redirection performed in a timely and unobtrusive manner.
As noted above,
At operation 1104, the process includes requesting (e.g., by the DCI client 102 or the DCI media processor 106) media for upcoming media substitutions from the third party content server 120. For example, the process can request third party content items for the upcoming media substitutions. At an optional operation 1105 (the optionality of operation 1105 is illustrated using the dashed outline of the box), the process includes, if the internal storage of the TV is unavailable or insufficient to store full third party content items, requesting and receiving from the third party content server 120 partial media segments of the candidate third party content item. As noted above, the partial media segments can include a first portion of each candidate third party content item.
At operation 1106, the process includes enabling (e.g., by the DCI client 102 or the DCI media processor 106) a picture-in-picture (PIP) video frame with size of zero-by-zero pixels (as illustrated in
At operation 1108, the process includes monitoring (e.g., continuously monitoring or periodically monitoring) currently the displayed video for a watermark and decoding information in the watermark when a watermark is detected. At operation 1109, in response to detecting a watermark, the process includes instructing (e.g., by the DCI client 102 or the DCI media processor 106) the TV PIP processor (e.g., video source switcher (PIP controller) 104) to start playing the third party content item and to enlarge the PIP to full-screen display (as illustrated in
At operation 1110, the process determines whether a complete third party content item is in local memory (e.g., content/data cache 107). At operation 1111, the process includes sending (e.g., by the DCI client 102 or the DCI media processor 106) a request to the third party content server 120 for the remainder (second portion) of the third party content item to be played, and receiving the second portion of the third party content item. At operation 1111b, the process includes requesting the full length third party content item when a partial media segment of the third party content item is not stored in the local memory.
At operation 1112, if the partial media segment (including the first portion) of the third party content item is stored in the local memory, the process includes combining (e.g., by the DCI client 102 or the DCI media processor 106) the second portion of the third party content item with the first portion (in the partial media segment), and continuing the playout of the third party content item. At operation 1112b, the process combines media streams into frame accurate video (and audio in some cases), as described above.
At operation 1113, the process includes monitoring (e.g., by the DCI client 102 or the DCI media processor 106) for new media content, for the end of playout of the third party content item, or for the end of a third party content timeslot. At operation 1115, based on the detection new media content, detection of the end of playout of the third party content item, or detection of the end of the third party content timeslot, the PIP window is reduced back down to the 0x0 size and the primary video content is displayed (in some cases audio can also be switched from PIP to primary).
At operation 1205, the process includes receiving a first portion (e.g., partial media segments) of a multiplicity of third party content items sent by third party content server 120 and storing the first portion in local storage (e.g., in the third party content item Cache 107). As noted above, the first portion can include a certain time period of the third party content items (e.g., the first 200 msec of the third party content items).
At operation 1206, the process includes preparing the third party content item for playout and establishing a picture-in-picture (PIP) frame for display (e.g., with a dimension of 0x0).
At operation 1207, the process includes detecting a valid watermark. At operation 1208, when a substitute third party content item (or first portion thereof) is stored in local storage, the process includes retrieving a substitute third party content item (or first portion thereof) from local storage. At operation 1209, when the substitute third party content item is not stored in local storage, the process includes requesting the entire substitute third party content item from the third party content server 120.
At operation 1210, the process includes enabling the PIP to display the substitute third party content item over the primary video (in some cases, audio is also switched to PIP). At operation 1211, the process includes monitoring for new media content, for the end of third party content item, or for the end of third party content timeslot. At operation 1212, upon detection of new media content, the end of third party content item, or the end of third party content timeslot, the process includes removing the PIP window from display (and in some cases switching audio back to primary).
In some examples, the process 1300 can include receiving, on the computing device, at least a portion of one or more substitute video segments corresponding to the candidate portions of the video channel, and storing, in storage of the computing device, at least the portion of the one or more substitute video segments. In such examples, the process 1300 can include obtaining, from the storage of the computing device, at least the portion of the substitute video segment corresponding to the candidate portion of the video channel. In some examples, the process 1300 can include obtaining, from a server, at least the portion of the substitute video segment corresponding to the candidate portion of the video channel. In some cases, the process 1300 can include obtaining the entire substitute video segment. In some cases, the process 1300 can include obtaining the portion of (less than the entire) the substitute video segment.
At operation 1310, the process 1300 includes displaying at least the portion of the substitute video segment in the picture-in-picture window. By displaying at least the portion of the substitute video segment, the substitute video segment replaces a video segment included in the video channel with the substitute video segment.
In some examples, at least the portion of the one or more substitute video segments includes partial media segments of the one or more substitute video segments. For instance, the partial media segments of the one or more substitute video segments can include a portion of the one or more substitute video segments. In some examples, at least the portion of the substitute video segment includes a partial media segment of the substitute video segment. For instance, the partial media segment of the substitute video segment can include a portion of the substitute video segment. In such examples, the process 1300 can include requesting a remaining portion of the substitute video segment (the portion of the substitute video segment other than the portion of the video segment), receiving the remaining portion of the substitute video segment, and displaying the remaining portion of the substitute video segment in the picture-in-picture window after at least the portion of the substitute video segment is displayed.
In some examples, the process 1300 can include decoding at least the portion of the substitute video segment and providing the decoded at least the portion of the substitute video segment to a picture-in-picture controller before displaying at least the portion of the substitute video segment in the picture-in-picture window. For instance, as described above, a cueing process can be performed to prepare the third party content item to be displayed in the picture-in-picture window. The cueing process can include decoding the media of the third party content item (e.g., a partial media segment representing a first portion of the third party content item or the entire third party content item) into uncompressed video (and in some cases audio frames), such as using an audio-video decoder 503 shown in
In some examples, the process 1300 can include synchronizing at least the portion of the substitute video segment with content of the video channel. For instance, as described above, the DCI media processor 106 or the DCI client 102 can decode the third party content item into uncompressed video (and in some cases uncompressed audio) and can place the decoded data in an audio-video buffer (e.g., the buffer of the audio-video decoder 503 shown in
In some examples, the process 1300 can include detecting a channel change from the video channel to a different video channel, and stopping display of the substitute video segment in response to detecting the channel change.
In some examples, the processes described herein (e.g., process 1100, process 1200, process 1300, and/or other process described herein) may be performed by a computing device or apparatus, such as a computing device having the computing device architecture 1400 shown in
The components of the computing device can be implemented in circuitry. For example, the components can include and/or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and/or other suitable electronic circuits), and/or can include and/or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. The computing device may further include a display (as an example of the output device or in addition to the output device), a network interface configured to communicate and/or receive the data, any combination thereof, and/or other component(s). The network interface may be configured to communicate and/or receive Internet Protocol (IP) based data or other type of data.
The processes 1100, 1200, and 1300 are illustrated as logical flow diagrams, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes.
Additionally, the processes 1100, 1200, and 1300, and/or any other processes described herein, may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.
Computing device architecture 1400 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 1410. Computing device architecture 1400 can copy data from memory 1415 and/or the storage device 1430 to cache 1412 for quick access by processor 1410. In this way, the cache can provide a performance boost that avoids processor 1410 delays while waiting for data. These and other modules can control or be configured to control processor 1410 to perform various actions. Other computing device memory 1415 may be available for use as well. Memory 1415 can include multiple different types of memory with different performance characteristics. Processor 1410 can include any general purpose processor and a hardware or software service, such as service 11432, service 21434, and service 31436 stored in storage device 1430, configured to control processor 1410 as well as a special-purpose processor where software instructions are incorporated into the processor design. Processor 1410 may be a self-contained system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.
To enable user interaction with the computing device architecture 1400, input device 1445 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. Output device 1435 can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display, projector, television, speaker device, etc. In some instances, multimodal computing devices can enable a user to provide multiple types of input to communicate with computing device architecture 1400. Communications interface 1440 can generally govern and manage the user input and computing device output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.
Storage device 1430 is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) 1425, read only memory (ROM) 1420, and hybrids thereof. Storage device 1430 can include services 1432, 1434, 1436 for controlling processor 1410. Other hardware or software modules are contemplated. Storage device 1430 can be connected to the computing device connection 1405. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 1410, connection 1405, output device 1435, and so forth, to carry out the function.
The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and/or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and/or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, memory or memory devices. A computer-readable medium may have stored thereon code and/or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.
In some embodiments the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.
Specific details are provided in the description above to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and/or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
Individual embodiments may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.
Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.
Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.
The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.
In the foregoing description, aspects of the application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative embodiments of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate embodiments, the methods may be performed in a different order than that described.
One of ordinary skill will appreciate that the less than (“<”) and greater than (“>”) symbols or terminology used herein can be replaced with less than or equal to (“≤”) and greater than or equal to (“≥”) symbols, respectively, without departing from the scope of this description.
Where components are described as being “configured to” perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.
The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and/or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and/or other suitable communication interface) either directly or indirectly.
Claim language or other language reciting “at least one of” a set and/or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one of” a set and/or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.
The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as propagated signals or waves.
The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.
This application is a continuation of U.S. patent application Ser. No. 17/068,540 filed Oct. 12, 2020, which claims the benefit of and priority to U.S. Provisional Patent Application No. 62/914,775, filed on Oct. 14, 2019, the contents of which are hereby expressly incorporated by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62914775 | Oct 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17068540 | Oct 2020 | US |
| Child | 18095786 | US |
