Patent Grant
11895361
Certain embodiments of the disclosure relate to a content recognition system in a broadcast chain. More specifically, certain embodiments of the disclosure relate to a method and system for automatic content recognition and verification in a broadcast chain.
Advancements in the television and broadcasting sector have led to the development of various technologies and broadcasting platforms that are changing the way consumers access and view media content. Broadcasting platforms usually refer to the types of networks that are used to deliver the television signal to the viewer. Currently, all the broadcasting platforms, such as analog terrestrial broadcast, digital terrestrial broadcast, direct-to-home satellite broadcast, cable, Internet Protocol (IP), and over-the-top television (OTT), compete and strive to increase their appeal in order to gain wider audience. The television viewing market is no longer exclusively concerned with whether media content services are viewed on a dedicated television or another device capable of playing media content. As a result of penetration of new technologies in the television and broadcasting sector, it is evident that the success of television broadcasting will be dependent on the ability of the network provider to continuously provide enhanced viewer experience with accurate content delivery.
Traditionally, a network provider (network operator) usually decides what shows to air in the upcoming broadcast year and when to air them. A programming schedule defines what to broadcast and when, ensuring an adequate or maximum utilization of airtime. Accordingly, the media content is packaged into brands, or organized along with graphics into channels to be delivered to consumers. Typically, prior to the delivery of the media content to the consumers, the media content is monitored to verify the placement of various content. Currently, such monitoring is done manually for a network channel, thus practically making unfeasible to monitor a large number of channels at the same time. Thus, an advanced system may be required that provides more feasible and practical way of monitoring the media content prior to multicasting of channels that reduces inefficiency in the various operations of a broadcast chain, and ensures accuracy of content that is to be delivered to provide enhanced viewer experience.
Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with some aspects of the present disclosure as set forth in the remainder of the present application with reference to the drawings.
Systems and/or methods are provided for automatic content recognition and verification in a broadcast chain, substantially as shown in and/or described in connection with at least one of the figures, as set forth more completely in the claims.
These and other advantages, aspects and novel features of the present disclosure, as well as details of an illustrated embodiment thereof, will be more fully understood from the following description and drawings.
Certain embodiments of the disclosure may be found in a method and system for automatic content recognition and verification in a broadcast chain. Various embodiments of the disclosure provide a method and system that simplifies the content recognition and verification process, and provides the network provider with the capability to automatically monitor placement of various graphics, such as channel logos, full screen graphics, partial screen graphics such as lower one-third graphics, symbol, and the like, on actual content of one or more channels in a broadcast chain. Prior to multicasting of the channel, an on-screen view of the channel is verified in a fail-safe manner to ensure accuracy of the overlaid plurality of graphics on actual content of the channel.
In accordance with various embodiments of the disclosure, a graphics verification system is provided that handles content of a channel in a broadcast chain. Content or media content may comprise video, audio and/or ancillary data that is associated with any portion of a particular content. The graphics verification system may be configured to partition an on-screen view of the channel into a plurality of sections based on an overlay of content, for example, a plurality of graphics on the on-screen view of the channel. The plurality of graphics may comprise a channel logo, a sign, a symbol, full screen graphic, partial screen graphic such as lower one-third graphics, a rating key, a text crawl, and/or a ticker. A placement of each of the overlaid plurality of graphics in the plurality of sections is determined with respect to associated content on the on-screen view of the channel. The placement of each of the plurality of graphics in the plurality of sections with respect to the associated content is determined based on a concurrent look-up of each of the plurality of graphics in the plurality of sections and the associated content. The concurrent look-up is executed to identify correct placement of the plurality of graphics intended for a specific content on the on-screen view of the channel. One or more error parameters are detected on the on-screen view of the channel based on the determined placement of each of the overlaid plurality of graphics in the plurality of sections with respect to the associated content. The on-screen view of the channel is calibrated prior to multicasting of the channel, based on the detected one or more error parameters.
In accordance with an embodiment, the concurrent look-up may be executed based on a graphics identifier (ID) of each of the plurality of graphics in the plurality of sections and a content ID of the associated content of each of the plurality of graphics. The graphics ID may correspond to fingerprints of each of the plurality of graphics. The content ID may correspond to the fingerprints of the associated content.
In accordance with an embodiment, graphics fingerprints, which correspond to the fingerprints of the plurality of graphics, may be generated using a graphics engine. The graphics fingerprints may be generated based on one or more visual features in the plurality of graphics. The graphics fingerprints may be generated for a plurality of coordinates that indicates position of the plurality of graphics on the on-screen view of the channel. The generated graphics fingerprints may be stored in a graphics fingerprint database. In accordance with an embodiment, raster fingerprints may be generated using an ingest engine. The raster fingerprints may be generated based on one or more visual features of the associated content. The generated raster fingerprints may be stored in a content fingerprint database.
In accordance with an embodiment, the fingerprints of the overlaid plurality of graphics detected on the on-screen view of the channel may be compared with the graphics fingerprints stored in the graphics fingerprint database. The comparison may be executed to detect the one or more error parameters of the overlaid plurality of graphics. Based on the detected one or more error parameters of the overlaid plurality of graphics, the overlaid plurality of graphics on the on-screen view of the channel may be automatically corrected at the graphics server.
In accordance with various embodiments of the disclosure, fingerprints of the associated content on which the plurality of graphics are overlaid on the on-screen view of the channel may be compared with the raster fingerprints stored in the content fingerprint database. The comparison may be executed to detect the one or more error parameters related to the associated content.
In accordance with various embodiments of the disclosure, the graphics verification system may utilize a time-based parameter and/or a region-based parameter of a programming schedule to verify each of the plurality of graphics in the plurality of sections and the associated content on the on-screen view. The programming schedule may be determined by a broadcast automation system. The time-based parameter may be indicative of timecodes or timing identifier at which the plurality of graphics and the associated content are required to be overlaid on the on-screen view of the channel during multicasting of the channel. The region-based parameter may be indicative of regions on the on-screen view of the channel, where the plurality of graphics and the associated content are required to be overlaid on the on-screen view of the channel during multicasting of the channel. The regions on the on-screen view of the channel may correspond to the plurality of sections.
In accordance with an embodiment, the graphics verification system may further utilize a plurality of content recognition based inline monitoring devices such as automatic content recognition (ACR)-based inline-monitoring devices placed at a plurality of locations in the broadcast chain, to concurrently detect fingerprints of each of the plurality of graphics in the plurality of sections and the associated content on which the plurality of graphics are overlaid.
The graphics verification system 102 may comprise suitable logic, circuitry, and interfaces that may be configured to execute code that handles content of a channel in a broadcast chain. In this regard, the graphics verification system 102 may be configured to automatically recognize the content of the channel in the broadcast chain, verify the recognized content, and thereafter, may trigger an alarm and/or notification that may require human intervention or correct the verified content in the broadcast chain based on any inaccuracy identified during the verification. The graphics verification system 102 may be operated by a broadcasting company, which may be referred to as a broadcast provider (or broadcast operator), or a network provider (or network operator). The broadcast provider may handle a single channel, a plurality of channels, or one or more networks. The broadcast provider may be operable to multicast content via one or more channels, for example, traditional over-the-air broadcast channels, cable television networks, satellite communication networks, the Internet, and/or other content delivery networks (CDNs).
Each of the published data sources 104a, . . . , 104n may be coupled to one or more television networks and may comprise suitable logic, circuitry, and interfaces that may be configured to execute code that provides actual audiences for programs that were aired. As illustrated in
The source device 106 may comprise suitable logic, circuitry, and interfaces that may be configured to communicate content (for example, a video feed) to the graphics verification system 102. In a broadcast chain, in the content production environment, video and audio data are usually transported using a video serial digital interface (SDI), which is defined by the Society of Motion Picture and Television Engineers (SMPTE). The video SDI supports uncompressed or raw content with superior visual quality. In other embodiments, the video and audio can be transported using SDI or RTP transport mechanisms as defined by the SMPTE ST-2110 family of documents. The source device 106 may be configured to transport (or provide) content to the graphics server 114. The source device 106 may be operated by a broadcasting company, which may be referred to as a broadcast provider (or broadcast operator), or a network provider (or network operator). In accordance with an embodiment, the source device 106 may also be communicatively coupled to the network 110.
The broadcast automation system 108 may comprise suitable logic, circuitry, and interfaces that may be configured to determine a programming schedule. The programming schedule defines what to broadcast and when. Alternatively stated, the programming schedule of a channel defines what media content should be scheduled, the ordering of the media content during playout, and when to multicast. The content that is multicast may include both the programming content, such as TV programs, and the non-programming content, such as advertisements or promotional media. In accordance with an exemplary embodiment, the programming schedule may be a text file or an XML file, that comprise a reference mapping of different media content items, including main content and graphics. The programming schedule may indicate content identifier (ID), graphics ID, and may specify information as to which graphics should be playing with which content, when, and where on an on-screen view of the channel.
The network 110 may be any kind of network, or a combination of various networks, and it is shown illustrating the communication that may occur between the graphics verification system 102, the published data sources 104a, . . . , 104n, the source device 106, the broadcast automation system 108, and the user-computing devices 112a, . . . , 112n. For example, the network 110 may comprise one or more of a cable television network, the Internet, a satellite communication network, a wide area network (WAN), a medium area network (MAN), a local area network (LAN), an analogue terrestrial network, a digital terrestrial network, a direct-to-home satellite network, cable, an Internet Protocol (IP), and an over-the-top television (OTT) network.
Each of the user-computing devices 112a, . . . , 112n may refer to an end-user device where the content (for example, the programming content and/or the non-programming content along with the graphics and the associated content) may be streamed to be viewed by a viewer. The number of impressions of a media item, such as an advertisement and/or promotional media, on the user-computing devices 112a, . . . , 112n may determine the advertising impact or promotion impact and number of actual audiences achieved during campaigns. Examples of the user-computing devices 112a, . . . , 112n may include, but are not limited to, connected TVs, connected TV with paired devices (e.g., tablets), and second screen devices, such as smartphones and tablets.
The graphics server 114 may comprise suitable logic, circuitry, and interfaces that may be configured to receive content from one or more content source devices, such as the source device 106, via one or more communication mediums, such as the network 110. The received content may correspond to a clean version of one or more programs, commercials, promotional media (promos), and/or the like in a certain media format. For example, the received content may be in a material exchange format (MXF) operational pattern 1a (OP1a). In accordance with an embodiment, the received content may further comprise graphics content, closed captions, and/or the like along with the one or more programs, commercials, and/or promotional media (promos). The graphics server 114 be configured to generate fingerprints/watermarks of the received content. The generated fingerprints may then be stored in the ACR database 116. Throughout the disclosure, the term fingerprints and watermarks may be used alternatively. The operations applicable for fingerprints may also be applicable to watermarks without limiting the scope of the disclosure.
The ACR database 116 may comprise suitable logic, circuitry, and interfaces that may be configured to store the fingerprints received from the graphics server 114. The fingerprints may correspond to content fingerprints and graphics fingerprints. The ACR database 116 may be further configured to store an association between the generated fingerprints, content IDs of actual content for which fingerprints (or watermarks) are generated, and/or graphics identifiers (IDs) of graphics for which fingerprints (or watermarks) are generated. The ACR database 116 may be implemented as a distributed and fault-tolerant database that is configured to support persistent storage of fingerprints. The ACR database 116 may be configured to store large amounts of content and/or graphics fingerprints. For example and without limitation, the ACR database 116 may store up to, for example, 25,000 hours of content fingerprints and/or graphics fingerprints. The number of hours of content fingerprints and/or graphics fingerprints that can be stored may vary based on the availability of storage capacity in the ACR database 116.
The broadcast chain system 118 may comprise suitable logic, circuitry, and interfaces that may be configured to generate an output stream of at least the graphics and the associated content that are overlaid on the on-screen view of the channel prior to multicasting of the channel to be viewed on the user-computing devices 112a, . . . , 112n. The broadcast chain system 118 may include a plurality of broadcast chain elements, for example and without limitation, the broadcast chain elements may include a playout server, a graphics server, a closed caption (CC) server, a Nielsen system, an Open Authentication Technology Committee (OATC) and Society of Cable Telecommunications Engineers (SCTE) system, and an encoding server. In some embodiments of the disclosure, the broadcast chain system 118 may be distinct and may not form a part of the graphics verification system 102.
The verification server 120 may comprise suitable logic, circuitry, and interfaces that may be configured to ensure a correct placement of each of the graphics and the associated content that are overlaid in one or more regions (or sections) of the on-screen view of the channel, prior to multicasting of the channel. The verification server 120 may determine a placement of each of the graphics and the associated content in the one or more regions of the on-screen view of the channel. The verification server 120 may further execute a concurrent look-up of each of the graphics and the associated content based on at least the fingerprints of each of the graphics and the associated content stored in the ACR database 116. For example, a check or verification may be performed to determine whether the fingerprints of the overlaid graphics (and the associated content) on the on-screen view of the channel matches with graphics fingerprints (and raster fingerprints) stored in the ACR database 116. The verification server 120 may be further configured to detect one or more error parameters of the overlaid graphics (and the associated content) based on at least the matching. Thereafter, the verification server 120 may automatically trigger and alarm or other notification that may require human intervention or interaction and/or correct the placement of the graphics and/or the associated content in the one or more regions of the on-screen view of the channel, based on the detected one or more error parameters.
The ingest engine 122 may be included in the graphics server 114. The ingest engine 122 may comprise suitable logic, circuitry, and interfaces that may be configured to generate fingerprints of the full raster content received by the graphics server 114, and subsequently, may store the generated fingerprints for archival in a content fingerprint database 128 (described in
The graphics engine 124 may be included in the graphics server 114. The graphics engine 124 may comprise suitable logic, circuitry, and interfaces that may be configured to use a fingerprint computation or algorithm to generate fingerprints of the graphics using included in the content received by the graphics server 114 from the source device 106, and subsequently, may store the generated fingerprints for archival in graphics fingerprint database 130 (described in
In operation, the graphics verification system 102 may be configured to handle content of a channel in a broadcast chain. The content in the broadcast chain may comprise programming content (such as main program segments of TV shows, TV programs, video-on-demand (VOD) content, and/or the like), non-programming content (such as advertisements, promotional media, or other commercials), graphics, and/or graphics-related content. The graphics server 114 may be configured to receive content (content ingestion) from the source device 106. The received content may correspond to uncompressed SDI format. A playout server in the broadcast chain system 118 may generate a series of on-screen views, for example, a linear stream of the content in an uncompressed SDI format. The series of on-screen views may be generated based on the programming schedule provided by the broadcast automation system 108. The programming schedule may indicate a time-based parameter and/or a region-based parameter. For example, the programming schedule may indicate a start time, an end time of content in the series of on-screen views. The time-based parameter of the programming schedule may be indicative of timecodes or timing identifier at which the content and/or graphics are required to be overlaid on the on-screen view of the channel during multicasting of the channel. In the broadcast chain system 118, a plurality of graphics, closed captions, and/or Nielsen watermarks, may be overlaid (or inserted) in the received content that is in uncompressed SDI format.
In certain locations in the broadcast chain system 118, an ACR-based monitoring may be done. For example, ACR-based inline monitoring devices may be placed at those locations for content monitoring purpose (an example is shown and described in
In some embodiments of the disclosure, the content fingerprint database 128 and the graphics fingerprint database 130 may be integrated to form an integrated database, for example, the ACR database 116. In some embodiments of the disclosure, as shown, the ingest engine 122, the graphics engine 124, the ACR database 116, the broadcast chain system 118, and the verification server 120 may be integrated to form an integrated system, for example, the graphics verification system 102. In some embodiments of the disclosure, the broadcast chain system 118 may be distinct from the graphics verification system 102. In some embodiments of the disclosure, as shown, the playout server 132, the CC server 136, the Nielsen system 138, the OATC/SCTE system 140, and the encoder server 142 may be integrated to form an integrated system, such as the broadcast chain system 118. In some embodiments of the disclosure, each of the playout server 132, the graphics server 134, the CC server 136, the Nielsen system 138, the OATC/SCTE system 140, and the encoder server 142 may be distinct or provided as an integrated system in the broadcast chain system 118. The functions and/or operations performed by the graphics verification system 102, and its various components, are further described, in
The content database 126 may comprise suitable logic, circuitry, and interfaces that may be configured to store the content (i.e., the full raster) and a plurality of graphics. The content database 126 may further store other associated metadata (e.g., program data, frame number, and content timeline, such as time code) that corresponds to the content in the full raster. In accordance with an embodiment, the content database 126 may include the content in uncompressed serial digital interface (SDI) format. The content database 126 may be configured to transmit the content to the broadcast chain system 18 based on the programming schedule.
The content fingerprint database 128 may comprise suitable logic, circuitry, and interfaces that may be configured to store the raster fingerprints (i.e., the fingerprints of the content associated with the actual content). Each content item may be associated with a specific content ID. The content fingerprint database 128 may be further configured to store an association between the fingerprints (or watermarks) and the corresponding content ID. For example, the association between the fingerprints and the corresponding content ID may be stored in the form of a look-up table or a structured data format.
The graphics fingerprint database 130 may comprise suitable logic, circuitry, and interfaces that may be configured to store the graphics fingerprints of a plurality of graphics. The plurality of graphics may correspond to at least a channel logo, a sign, a symbol, lower one-third graphics, a rating key, and/or a ticker. The plurality of graphics may correspond to dynamic graphics (for example, graphics that exhibit motion) and/or static graphics. Each graphics in the plurality of graphics may be associated with a graphics ID. The graphics fingerprint database 130 may be further configured to store an association between the graphic fingerprints and the corresponding graphics ID. For example, the association between the fingerprints and the corresponding graphics ID may be stored in the form of a look-up table or a structured data format.
The playout server 132 may comprise suitable logic, circuitry, and interfaces that may be configured to receive content from one or more content source devices, such as the source device 106 or the content database 126, via one or more communication mediums, such as the network 110. The received content may correspond to one or more programs, commercials, promotions, graphics-related content, and/or the like. The playout server 132 may further generate a series of on-screen views, for example, a linear stream of the content in an uncompressed SDI format. Each on-screen view may include the content without Nielsen watermarks. The series of on-screen views may be generated based on the programming schedule provided by the broadcast automation system 108. The programming schedule may indicate a time-based parameter and/or a region-based parameter. For example, the programming schedule may indicate a start time, an end time of content in the series of on-screen views. The time-based parameter of the programming schedule may be indicative of timecodes or timing identifier at which the content and/or graphics are required to be overlaid on the on-screen view of the channel during multicasting of the channel. The region-based parameter of the programming schedule may be indicative of specific regions or sections on the on-screen view of the channel where the content and/or graphics are required to be overlaid. The playout server 132 may further communicate the on-screen view to the graphics server 134.
The graphics server 134 may comprise suitable logic, circuitry, and interfaces that may be configured to receive the on-screen view comprising the content from the playout server 132. The graphics server 134 may be configured to insert the plurality of graphics on the on-screen view of the channel. The plurality of graphics may be inserted on the on-screen view based on the time-based parameter and/or the region-based parameter in the programming schedule. For example, the plurality of graphics may be overlaid on the on-screen view of the channel during multicasting of the channel at the timecodes or timing identifier indicated by the time-based parameter in the programming schedule. The plurality of graphics may be overlaid at specific regions or sections on the on-screen view of the channel indicated by the region-based parameter of the programming schedule. The plurality of graphics to be overlaid on the on-screen view may be extracted from the content database 126. In another embodiment, the plurality of graphics may be retrieved from the graphics engine 124 (or from specified locations as indicated by a file path or a Uniform Resource Locator (URL) in the programming schedule). Based on the insertion of the plurality of graphics on the on-screen view of the channel, the graphics server 134 may communicate the on-screen view to the CC server 136.
The CC server 136 may comprise suitable logic, circuitry, and interfaces that may be configured to receive the on-screen view comprising the content and the plurality of graphics from the graphics server 134. The CC server 136 may be configured to insert closed caption, for example, sub-title, on the on-screen view of the channel. The closed caption may be inserted based on the content associated with the on-screen view and/or the programming schedule. Further, the closed caption may be determined in real-time or non-real time based on at least the content associated with the on-screen view, by the CC server 136. The CC server 136 may further communicate the on-screen view to the Nielsen system 138.
The Nielsen system 138 may comprise suitable logic, circuitry, and interfaces that may be configured to receive the on-screen view of the channel from the CC server 136. The Nielsen system 138 may be configured to apply (or embed) Nielsen watermarks in the content associated with the on-screen view of the channel. For example, in linear broadcast chain, the Nielsen system 138 may append watermarks to audio content and/or video content associated with the on-screen view. Such Nielsen watermarks may be used to track the linear broadcast's audience. The Nielsen system 138 may further communicate the on-screen view to the OATC/SCTE system 140.
The OATC/SCTE system 140 may comprise suitable logic, circuitry, and interfaces that may be configured to execute code that handles insertion of triggers, such as SCTE-104-based digital ad-marker, in the content overlaid on the on-screen view. In this regard, the OATC/SCTE system 140 may be configured to insert the triggers as per society of motion picture and television engineers (SMPTE) 2010 standards. The triggers may include a pre-roll value, which indicates splice points related to a plurality of interstitial content items (e.g., advertisements and/or promotional media). The splice points may refer to points of transition from one interstitial content item to another interstitial content item or from main program segments to an interstitial content item. The OATC/SCTE system 140 may further communicate the on-screen view to the encoder server 142.
The encoder server 142 may comprise suitable logic, circuitry, and interfaces that may be configured to execute code that handles encoding of the content on the on-screen view of the channel. The encoder server 142 may encode the content by varying bitrates. The encoder server 142 may further convert the SCTE-104-based digital ad-marker to an SCTE-35-based digital ad-marker during encoding. The encoder server 142 further conditions the encoded content on the on-screen view to ensure correct splicing in the content. The encoded content may be multiplexed and communicated (or stored), via the network 110, for the playout, and finally delivered (multicast) to be viewed on the user-computing devices 112a, . . . , 112n.
The plurality of IMDs, such as the first IMD 144A and the second IMD 144B, may comprise suitable logic, circuitry, and interfaces that may be configured to execute code to read the inserted/embedded fingerprints (and/or watermarks), and communicate the information to the verification server 120. The plurality of IMDs, such as the first IMD 144A and the second IMD 144B in association with the verification server 120 ensures a correct overlay (i.e., placement) of each of the plurality of graphics and the associated content in the various sections of the on-screen view of the channel. The verification server 120 may be configured to execute a concurrent look-up based on at least the fingerprints of each of the plurality of graphics and the associated content in the various sections of the on-screen view, by use of the plurality of IMDs. The plurality of IMDs may be placed at a plurality of locations in the broadcast chain system 118, for example, between the Nielsen system 138 and the OATC/SCTE system 140 or the OATC/SCTE system 140 and the encoder server 142, as shown. It should be understood that the plurality of IMDs may be placed at other locations, for example, after the encoder server 142, for content and graphics monitoring, without limiting the scope of the disclosure.
The user terminal 146 may comprise a keypad, display, speaker, microphone, a pointing or navigation device and/or capability to enable a user, such as the monitoring user 148 to interact with the graphics server 114 and/or one or more elements of the broadcast chain system 118. The user terminal 146 may be configured to present a user interface that enables the monitoring user 148, such as a graphic inserter to configure and interact with components such as the graphics server 114 and/or the graphics server 134. The functions and/or operations performed by the graphics verification system 102, and its various components, are further described, in
The first section 202 may represent a main portion of the on-screen view. The first section 202 may comprise main content, for example, a programming and/or non-programming content without any overlaid graphics, for example. The second section 204 and the third section 206 may overlap the seventh section 214, as shown, and may comprise, for example, overlays, logos, graphics, and/or other elements or related content. The fourth section 208 and the fifth section 210 may be located at the top of the main portion of the on-screen view, as shown, and may usually comprise, for example, overlays, logos, graphics, symbol, sign, and/or other elements or related content. The sixth section 212 may overlap the seventh section 214, as shown, and may comprise content, for example, tickers or text crawl. The seventh section 214 may comprise the lower one-third (⅓) graphics. The lower one-third graphics is a graphics overlay that may be placed in the title-safe lower area of the on-screen view, and may include other sections, as shown. In its simplest form, the lower one-third graphics may correspond to text content overlying a video stream. The lower one-third graphics may further comprise graphical elements, such as boxes, images or shading. Some lower one-third graphics may include animated backgrounds. The eighth section 216 and the ninth section 218 may overlap the seventh section 214, as shown, and may comprise, for example, headlines, overlays, logos, graphics, and/or other elements or related content.
At 302, content may be received from the source device 106 over a communication network, such as the network 110. The content may be ingested to the graphics server 114 using the ingest engine 122. The received content may be uncompressed raw content, such as a SDI video. The received content may comprise one or more programs, commercials and promotional media (promos). Typically, the received content may correspond to a clean version of content in a certain media format. For example, the received content may be in a material exchange format (MXF) operational pattern 1a (OP1a) or other format that supports high quality video (such as SDI video). The received content may correspond to a full raster. The clean version of actual content refers to actual content devoid of any graphics, such as overlaid channel logo, lower third graphics, closed captions, and the like. Alternatively, in certain scenarios, in addition to the actual clean version of content, the received content may also comprise certain graphics, along with the one or more programs, commercials, and/or promotional media (promos).
At 304, fingerprints of the content in the full raster may be generated. The ingest engine 122 may be configured to generate raster fingerprints, which correspond to the fingerprints of the actual content in the full raster. The raster fingerprints may be generated based on one or more content characteristics, for example, one or more visual features, of the content. The ingest engine 122 may communicate the generated raster fingerprints to be stored in the content fingerprint database 128. Alternatively, in certain scenarios, the received content may include pre-embedded watermarks/fingerprints. In such cases, the ingest engine 122 at the graphics server 114 may be configured to detect the watermarks/fingerprints by use of the ACR-module in the ingest engine 122, and communicate the detected raster fingerprints/watermarks to the content fingerprint database 128. In accordance with an embodiment, a plurality of graphics may also be ingested to the graphics server 114 (or another server dedicated for graphics) by use of the graphics engine 124. Alternatively, such plurality of graphics may be pre-stored in the content database 126 or other storage units communicatively coupled to the content database 126 or the graphics server 114. In accordance with an embodiment, a graphics operator may create graphics on the user terminal 146, and ingest into the graphics server 114 or the other servers dedicated for graphics storage, which may then be further communicated and stored in the content database 126.
At 306, the fingerprints of each of the plurality of graphics may be generated. The graphics engine 124 may be configured to generate graphics fingerprints, which correspond to the fingerprints of the plurality of graphics that is to be overlaid on the on-screen view of the channel in the broadcast chain system 118. The graphics fingerprints may be generated based on one or more graphics characteristics, for example, one or more visual features, associated with the plurality of graphics. The coordinates, for example, x-coordinate and y-coordinate, that indicates position of the graphics on the on-screen view, graphic type, such as a channel logo, or a symbol, and the generated fingerprints of a particular graphics item, may be communicated to the graphics fingerprint database 130 by the graphics engine 124. The graphics fingerprint database 130 may be configured to store the received information of X,Y-coordinates of the graphic item, such as a channel logo, on the full raster, graphic type, and the generated graphics fingerprints in association with each other in the graphics fingerprint database 130.
At 308, the received content may be communicated to the broadcast chain system 118. The ingest engine 122 may be configured to communicate the received content to the broadcast chain system 118. In an embodiment, the ingest engine 122 may store the received content in the content database 126. Thereafter, the stored content may be retrieved by the playout server 132 based on the programming schedule received from the broadcast automation system 108. The playout server 132 may be configured to receive content from the content database 126 in the uncompressed SDI video format. The playout server 132 may further generate a series of on-screen views, for example, a linear stream of the content in an uncompressed SDI format. The series of on-screen views may be generated based on the programming schedule provided by the broadcast automation system 108. The programming schedule may indicate a time-based parameter and/or a region-based parameter. For example, the programming schedule may indicate a start time, an end time of content in the series of on-screen views. The time-based parameter of the programming schedule may be indicative of timecodes or timing identifiers at which the content and/or graphics are required to be overlaid on the on-screen view of the channel during multicasting of the channel. The region-based parameter of the programming schedule may be indicative of specific regions or sections on the on-screen view of the channel where the content and/or graphics are required to be overlaid. The playout server 132 may further communicate the on-screen view to the graphics server 134, the CC server 136, and the Nielsen system 138 for insertion of various, closed captions, and Nielsen watermarks on the on-screen views, in accordance with the programming schedule provided by the broadcast automation system 108. Thus, in the broadcast chain system 118, a plurality of graphics, closed captions, and/or Nielsen watermarks, may be overlaid (or inserted) in the received content that is in uncompressed SDI format. The graphics verification system 102 may employ a plurality of IMDs, such as the first IMD 144A and the second IMD 114B, for the content and graphics monitoring at one or more locations in a broadcast chain. The plurality of IMDs, such as the first IMD 144A and the second IMD 114B, may be placed at a plurality of locations in the broadcast chain system 118, for example, between the Nielsen system 138 and the OATC/SCTE system 140 and the OATC/SCTE system 140 and the encoder server 142, as shown in
At 310, an on-screen view of a channel may be partitioned into a plurality of sections. The verification server 120 may be configured to partition the on-screen view of the channel based on an overlay of a plurality of graphics on the on-screen view of the channel in the broadcast chain system 118. The plurality of graphics may comprise, for example, a channel logo, a sign, a symbol, lower one-third graphics, a rating key, and/or a ticker. The on-screen of the channel may further comprise content associated with the plurality of graphics. An example of the partitioning of the on-screen view of a channel into a plurality of sections, such as the sections 202 to 218, has been shown, and described in
At 312, a placement of each of the overlaid plurality of graphics in the plurality of sections with respect to the associated content, may be determined. The verification server 120 may be configured to determine the placement of each of the overlaid plurality of graphics in the plurality of sections with respect to the associated content on the on-screen view of the channel. The placement of each of the plurality of graphics in the plurality of sections may be determined based on a concurrent look-up of each of the plurality of graphics in the plurality of sections and the associated content overlaid on the on-screen view of the channel. The concurrent look-up may be executed by the verification server 120, based on a graphics ID of each of the plurality of graphics in the plurality of sections and a content ID of the associated content of each of the plurality of graphics. The graphics ID may be associated with the fingerprints of each of the plurality of graphics. The content ID may be associated with the fingerprints of the associated content of each of the plurality of graphics. The plurality of IMDs, such as the first IMD 144A and the second IMD 144B, may be configured to execute code to continually or intermittently read and detect the inserted/embedded fingerprints (and/or watermarks), and communicate the information to the verification server 120. As the ACR database 116 stores information and associations among the fingerprints and corresponding IDs, (such as a graphic ID and a corresponding graphic fingerprint in the graphics fingerprint database 130 or a content ID and a corresponding raster fingerprint in the content fingerprint database 128), the verification server 120 may be configured to verify each of the plurality of graphics and the associated content in the various sections of the on-screen view, by use of the first IMD 144A.
Firstly, the verification server 120 may be configured to compare the detected fingerprints of the overlaid plurality of graphics on the on-screen view of the channel with the graphics fingerprints stored in the graphics fingerprint database 130. Thereafter, the verification server 120 may extract graphic IDs of those graphics fingerprints from the graphics fingerprint database 130 that matches with detected fingerprints of overlaid plurality of graphics on the on-screen view of the channel. The extracted graphic IDs represents graphic IDs of the overlaid plurality of graphics on the on-screen view of the channel. Each of the on-screen view of the channel may be the full raster view, which may be associated with a unique timing identifiers, such as a time code (timecode). An example of the time code may be Vertical Interval Timecode (VITC). Similarly, the verification server 120 may also be configured to compare the detected fingerprints of the associated content (actual content) on which the plurality of graphics are overlaid on the on-screen view of the channel with the raster fingerprints stored in the content fingerprint database 128. Thereafter, the verification server 120 may extract content ID from the content fingerprint database 128 that matches with detected fingerprints of the associated content on the on-screen view of the channel. The extracted content ID represents the content ID of the associated content (actual content) on the on-screen view of the channel. Once the graphic IDs of the overlaid plurality of graphics and the content ID of the associated content for the on-screen view of the channel are extracted and available, the verification server 120 may be configured to verify whether the graphic IDs of the overlaid plurality of graphics matches with the graphic IDs specified in the programming schedule for a particular timing identifier, such as the unique time code associated with the full raster (i.e. the on-screen view of the channel). Further, the verification server 120 may be configured to verify whether the content ID of the associated content for the on-screen view of the channel matches with the content ID specified in the programming schedule for the particular timecodes, such as the unique time code associated with the full raster (i.e. the on-screen view of the channel). Thus, the time-based parameter of the programming schedule that is indicative of timecodes at which the content and/or graphics are required to be overlaid on the on-screen view of the channel during multicasting of the channel, is utilized. Further, as X,Y-coordinates and graphic type information is also associated with each of the graphics IDs in the graphics fingerprint database 130, thus the verification server 120 may also verify such information with the metadata in the programming schedule for the matched graphics ID for the particular time code, such as the unique time code associated with the full raster. This is how the region-based parameter of the programming schedule that is indicative of specific regions or sections on the on-screen view of the channel where the content and/or graphics are required to be overlaid, is utilized. Thus, the placement of each of the overlaid plurality of graphics in the plurality of sections with respect to associated content on the on-screen view of the channel, is determined.
At 314, one or more error parameters may be detected on the on-screen view. The one or more error parameters may be detected to identify a correct placement of each of the plurality of graphics intended for a specific content on the on-screen view of the channel. The verification server 120 may be configured to detect the one or more error parameters on the on-screen view based on the determined placement of each of the overlaid plurality of graphics in the plurality of sections with respect to the associated content. For example, the one or more error parameters may be detected based on a mismatch in the fingerprints of the overlaid plurality of graphics and the associated content with respect to the graphics fingerprints and the raster fingerprints, respectively. The mismatch may be determined based on the concurrent look-up of the overlaid plurality of graphics and the associated content using the fingerprints of the overlaid plurality of graphics and the associated content. The one or more error parameters may also correspond to errors related to positions of the plurality of graphics in the plurality of sections on the on-screen view. The verification server 120 may further utilize the time-based parameter and/or the region-based parameter of the programming schedule that corresponds to the plurality of graphics and the content in the full raster, to execute the concurrent look-up.
At 316, the on-screen view of the channel may be calibrated based on the detected one or more error parameters. The verification server 120 may be configured to calibrate the on-screen view of the channel, prior to multicasting of the channel. For example, the on-screen view of the channel may be calibrated based on the detected one or more error parameters of the overlaid plurality of graphics. The overlaid plurality of graphics on the on-screen view of the channel may be automatically calibrated based on the detected one or more error parameters. For example, the verification server 120 may replace the incorrect placement of an overlaid graphics of the overlaid plurality of graphics with a correct graphics retrieved from the content database 126 or other server dedicated for graphics storage as specified in the programming schedule provided by the broadcast automation system 108. Similarly, the on-screen view of the channel may be calibrated based on the detected one or more error parameters related to the associated content on which the plurality of graphics are overlaid. For example, the verification server 120 may replace the incorrect content item currently associated with the overlaid plurality of graphics with a correct content item retrieved from the content database 126. The correct content item refers to a content item that is expected to be played at a specific time instant with respect to the overlaid plurality of graphics on the on-screen view as specified in the programming schedule.
At 318, the on-screen view of the channel may be re-calibrated at a plurality of locations in the broadcast chain. For example, another IMD, such as the second IMD 144B, may be placed between the OATC/SCTE system 140 and the encoder server 142, as shown in
In accordance with the exemplary scenario 400A, one or more IMDs, such as the first IMD 144A (as shown in
The verification server 120 may be configured to verify whether the graphic IDs of the overlaid plurality of graphics 404 to 414 matches with the graphic IDs specified in the programming schedule for the on-screen view of the channel “CXX” at the unique time code associated with the full raster (i.e. the current on-screen view). The verification server 120 may be configured to verify whether the content ID of the weather content 402 on-screen view of the channel “CXX” matches with the content ID specified in the programming schedule for the particular timing identifier, such as the unique time code associated with the full raster (i.e. the current on-screen view of the channel “CXX”). Thus, the verification server 120 may be configured to determine that all the plurality of graphics 406 to 414 (except the LIVE symbol 404 in the fifth section 210) are accurately placed in the plurality of sections 202 to 218 with respect to associated content, such as the weather content 402, on the on-screen view of the channel “CXX”. The verification server 120 may be configured to detect an error parameter of incorrect placement of the LIVE symbol 404 on the on-screen view of the channel “CXX”. The graphic ID of the LIVE symbol 404 may not be found in the programming schedule for the unique time code, associated with the full raster (i.e. the current on-screen view of the channel “CXX”) for the weather content 402. In other words, it may be ascertained that the LIVE symbol 404 was not meant for the weather content 402 and was supposed to be overlaid on subsequent content item, for example, a live coverage of an event, and should have been appeared after certain number of video frames (such as after 3 frames) as specified in the programming schedule. In accordance with an embodiment, the verification server 120 may be configured to calibrate the on-screen view of the channel “CXX”. For example, the LIVE symbol 404, may be removed from the channel “CXX.”
Referring to
In this example, the graphics verification system 102 employing the processing system 502 may be implemented with bus architecture, represented generally by bus 508. The bus 508 may include any number of interconnecting buses and bridges depending on the specific implementation of the graphics verification system 102 and the overall design constraints. The bus 508 links together various circuits including the one or more processors, represented generally by the processor 504, the non-transitory computer-readable media, represented generally by the computer-readable medium 506, the hardware graphics server 114, the hardware ACR database 116, the hardware broadcast chain system 118, and the hardware verification server 120 which may be configured to perform one or more operations or methods described herein. A bus interface 510 provides an interface between the bus 508 and a transceiver 512. The transceiver 512 provides a means for communicating via the network 110 with various other apparatus, such as the published data sources 104a, . . . , 104n, the source device 106, the hardware content database 126, the broadcast automation system 108, and the user-computing devices 112a, . . . , 112n.
The processor 504 may be configured to manage the bus 508 and general processing, including the execution of a set of instructions stored on the computer-readable medium 506. The set of instructions, when executed by the processor 504, causes the graphics verification system 102 to execute the various functions described herein for any particular apparatus. The computer-readable medium 506 may also be used for storing data that is manipulated by the processor 504 when executing the set of instructions. The computer-readable medium 506 may also be configured to store data for one or more of the ingest engine 122, the graphics engine 124, the hardware ACR database 116, the content database 126, the hardware broadcast chain system 118, and the hardware verification server 120.
In an aspect of the disclosure, the processor 504, the computer-readable medium 506, or a combination of both may be configured or otherwise specially programmed to execute the operations or functionality of the hardware ACR database 116, the hardware broadcast chain system 118, and the hardware verification server 120, or various other components described herein. For example, the processor 504, the computer-readable medium 506, or a combination of both may be configured or otherwise specially programmed to perform the operations and functionality of the hardware first IMD 144A, the hardware second IMD 144B, the hardware ACR database 116, and the hardware verification server 120, as described with respect to
Various embodiments of the disclosure comprise a graphics verification system 102 that may be configured to handle automatic content recognition and verification in a broadcast chain. The graphics verification system 102 may comprise, for example, the ingest engine 122, the graphics engine 124, the ACR database 116, the verification server 120, and a plurality of IMDs, such as the first IMD 144A and the second IMD 144B. The ACR database 116 may comprise the content fingerprint database 128 and the graphics fingerprint database 130. The broadcast chain system 118 may comprise the playout server 132, the graphics server 134, the CC server 136, the Nielsen system 138, the OATC/SCTE system 140, and the encoder server 142. The plurality of IMDs, such as the first IMD 144A and the second IMD 144B, may be placed in the broadcast chain system 118 for content monitoring. The verification server 120 may be configured to partition an on-screen view of the channel into a plurality of sections based on an overlay of a plurality of graphics on the on-screen view of the channel. The verification server 120 may be configured to determine a placement of each of a plurality of graphics overlaid in a plurality of sections of an on-screen view of a channel. The placement of each of the plurality of graphics in the plurality of sections may be determined with respect to the associated content of the plurality of graphics, based on a concurrent look-up of each of the plurality of graphics in the plurality of sections and the associated content overlaid on the on-screen view of the channel. The concurrent look-up may be executed based on fingerprints of each of the plurality of graphics in the plurality of sections and the associated content overlaid on the on-screen view of the channel. Based on the determined placement of each of the overlaid plurality of graphics in the plurality of sections with respect to the associated content, one or more error parameters are detected on the on-screen view of the channel. The verification server 120 may further calibrate the on-screen view of the channel based on the detected one or more error parameters, prior to multicasting of the channel. For example, any inaccuracy detected in the placement of the plurality of graphics and the associated content on the on-screen view of the channel causes the verification server 120 to replace the incorrect placement with the correct placement of the plurality of graphics and the associated content.
The verification server 120 in association with the plurality of IMDs, such as the first IMD 144A and the second IMD 144B, provides significant productivity and efficiency improvements since the process of monitoring a plurality of sections at the same time on the on-screen views has been simplified and a real time or near-real time in a broadcast chain. Further, the verification server 120 and the plurality of IMDs provides an ability to the graphics verification system 102 to automatically detect any inaccuracy in the placement of the plurality of graphics and the associated content of hundreds of channels in the broadcast chain prior to encoding and multicasting of the channels. Furthermore, based on the detected inaccuracy in the placement of the plurality of graphics and the associated content, the verification server 120 may automatically correct the inaccurate placement. Thus, the broadcast chain in which the content is automatically monitored, verified, and corrected in case of inaccurate placement, may not be required to pass manual observations that may be time consuming and costly, when hundreds of channels are required to monitored at the same time. Furthermore, such automation may result in much faster processing time (faster computation time without sacrificing accuracy), and avoid human errors.
Various embodiments of the disclosure may provide a non-transitory computer-readable medium having stored thereon, computer implemented instruction that when executed by a processor causes the graphics verification system 102 to execute operations to handle content of a channel in a broadcast chain. The content may comprise video, audio and/or ancillary data that is associated with any portion of a particular content. The graphics verification system 102 may execute operations comprising partitioning an on-screen view of the channel into a plurality of sections based on an overlay of a plurality of graphics on the on-screen view of the channel. A placement of each of the overlaid plurality of graphics in the plurality of sections is determined with respect to associated content on the on-screen view of the channel. The placement of each of the plurality of graphics in the plurality of sections with respect to the associated content is determined based on a concurrent look-up of each of the plurality of graphics in the plurality of sections and the associated content. One or more error parameters are detected on the on-screen view of the channel based on the determined placement of each of the overlaid plurality of graphics in the plurality of sections with respect to the associated content. The on-screen view of the channel is calibrated prior to multicasting of the channel, based on the detected one or more error parameters.
As utilized herein the terms “circuits” and “circuitry” refer to physical electronic components (i.e. hardware) and any software and/or firmware (“code”) which may configure the hardware, be executed by the hardware, and/or otherwise be associated with the hardware. As used herein, for example, a particular processor and memory may comprise a first “circuit” when executing a first one or more lines of code and may comprise a second “circuit” when executing a second one or more lines of code. As utilized herein, “and/or” means any one or more of the items in the list joined by “and/or”. As an example, “x and/or y” means any element of the three-element set {(x), (y), (x, y)}. As another example, “x, y, and/or z” means any element of the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. As utilized herein, the term “exemplary” means serving as a non-limiting example, instance, or illustration. As utilized herein, the terms “e.g.,” and “for example” set off lists of one or more non-limiting examples, instances, or illustrations. As utilized herein, circuitry is “operable” to perform a function whenever the circuitry comprises the necessary hardware and code (if any is necessary) to perform the function, regardless of whether performance of the function is disabled, or not enabled, by some user-configurable setting.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising,”, “includes” and/or “including”, when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Further, many embodiments are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that various actions described herein can be performed by specific circuits (e.g., application specific integrated circuits (ASICs)), by program instructions being executed by one or more processors, or by a combination of both. Additionally, these sequence of actions described herein can be considered to be embodied entirely within any non-transitory form of computer readable storage medium having stored therein a corresponding set of computer instructions that upon execution would cause an associated processor to perform the functionality described herein. Thus, the various aspects of the disclosure may be embodied in a number of different forms, all of which have been contemplated to be within the scope of the claimed subject matter. In addition, for each of the embodiments described herein, the corresponding form of any such embodiments may be described herein as, for example, “logic configured to” perform the described action.
Another embodiment of the disclosure may provide a non-transitory machine and/or computer readable storage and/or media, having stored thereon, a machine code and/or a computer program having at least one code section executable by a machine and/or a computer, thereby causing the machine and/or computer to perform the steps as described herein to handle automatic content recognition and verification in a broadcast chain.
The present disclosure may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which when loaded in a computer system is able to carry out these methods. Computer program in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code or notation; b) reproduction in a different material form.
Further, those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, algorithm, and/or steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The methods, sequences and/or algorithms described in connection with the embodiments disclosed herein may be embodied directly in firmware, hardware, in a software module executed by a processor, or in a combination thereof. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor.
While the present disclosure has been described with reference to certain embodiments, it will be noted understood by, for example, those skilled in the art that various changes and modification could be made and equivalents may be substituted without departing from the scope of the present disclosure as defined, for example, in the appended claims. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present disclosure without departing from its scope. The functions, steps and/or actions of the method claims in accordance with the embodiments of the disclosure described herein need not be performed in any particular order. Furthermore, although elements of the disclosure may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated. Therefore, it is intended that the present disclosure not be limited to the particular embodiment disclosed, but that the present disclosure will include all embodiments falling within the scope of the appended claims.
This patent application makes reference to, claims priority to, claims the benefit of, and is a Continuation Application of U.S. patent application Ser. No. 15/396,596, filed Dec. 31, 2016. This application also makes reference to: U.S. Pat. No. 9,172,994, application Ser. No. 13/730,352; andU.S. Pat. No. 8,776,105, application Ser. No. 13/730,656. Each of the above referenced patents is hereby incorporated herein by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5305464 | Frett | Apr 1994 | A |
| 5410326 | Goldstein | Apr 1995 | A |
| 5437050 | Lamb et al. | Jul 1995 | A |
| 5712995 | Cohn | Jan 1998 | A |
| 5874686 | Ghias et al. | Feb 1999 | A |
| 5918223 | Blum et al. | Jun 1999 | A |
| 5956683 | Jacobs et al. | Sep 1999 | A |
| 5991737 | Chen | Nov 1999 | A |
| 6121530 | Sonoda | Sep 2000 | A |
| 6184877 | Dodson et al. | Feb 2001 | B1 |
| 6201176 | Yourlo | Mar 2001 | B1 |
| 6314577 | Pocock | Nov 2001 | B1 |
| 6408272 | White et al. | Jun 2002 | B1 |
| 6504089 | Nagasawa et al. | Jan 2003 | B1 |
| 6556218 | Alcorn | Apr 2003 | B1 |
| 6760720 | De Bellis | Jul 2004 | B1 |
| 6765595 | Lee et al. | Jul 2004 | B2 |
| 6834308 | Ikezoye et al. | Dec 2004 | B1 |
| 6968337 | Wold | Nov 2005 | B2 |
| 6990453 | Wang et al. | Jan 2006 | B2 |
| 7080325 | Treibach-Heck et al. | Jul 2006 | B2 |
| 7095871 | Jones | Aug 2006 | B2 |
| 7134132 | Ngo | Nov 2006 | B1 |
| 7174293 | Kenyon et al. | Feb 2007 | B2 |
| 7190971 | Kawamoto | Mar 2007 | B1 |
| 7194752 | Kenyon et al. | Mar 2007 | B1 |
| 7266343 | Yli-Juuti et al. | Sep 2007 | B1 |
| 7363278 | Schmelzer et al. | Apr 2008 | B2 |
| 7444353 | Chen et al. | Oct 2008 | B1 |
| 7500007 | Ikezoye et al. | Mar 2009 | B2 |
| 7529659 | Wold | May 2009 | B2 |
| 7562012 | Wold et al. | Jul 2009 | B1 |
| 7562392 | Rhoads et al. | Jul 2009 | B1 |
| 7565327 | Schmelzer | Jul 2009 | B2 |
| 7624416 | Vandermolen | Nov 2009 | B1 |
| 7703114 | Thukral | Apr 2010 | B2 |
| 7707088 | Schmelzer | Apr 2010 | B2 |
| 7711652 | Schmelzer | May 2010 | B2 |
| 7730509 | Boulet et al. | Jun 2010 | B2 |
| 7747749 | Erikson et al. | Jun 2010 | B1 |
| 7783489 | Kenyon et al. | Aug 2010 | B2 |
| 7797249 | Schmelzer et al. | Sep 2010 | B2 |
| 7813954 | Price | Oct 2010 | B1 |
| 7870574 | Kenyon et al. | Jan 2011 | B2 |
| 7877290 | Arsenault | Jan 2011 | B1 |
| 7877438 | Schrempp et al. | Jan 2011 | B2 |
| 7900228 | Stark et al. | Mar 2011 | B2 |
| 7917645 | Ikezoye et al. | Mar 2011 | B2 |
| 8006274 | Scott, III | Aug 2011 | B2 |
| 8046803 | Lee | Oct 2011 | B1 |
| 8117635 | Hendricks et al. | Feb 2012 | B2 |
| 8249422 | Narahara et al. | Aug 2012 | B2 |
| 8281339 | Walker | Oct 2012 | B1 |
| 8335833 | Parkinson | Dec 2012 | B1 |
| 8407750 | Vorbau | Mar 2013 | B2 |
| 8413206 | Wyatt et al. | Apr 2013 | B1 |
| 8418206 | Bryant et al. | Apr 2013 | B2 |
| 8553148 | Ramaswamy et al. | Oct 2013 | B2 |
| 8572650 | Townsend et al. | Oct 2013 | B2 |
| 8732473 | Bisso et al. | May 2014 | B2 |
| 8732739 | Sillerman | May 2014 | B2 |
| 8793274 | Yu et al. | Jul 2014 | B2 |
| 8810512 | Andersson | Aug 2014 | B2 |
| 9094618 | Hundemer | Jul 2015 | B1 |
| 20020032906 | Grossman | Mar 2002 | A1 |
| 20020069100 | Arberman | Jun 2002 | A1 |
| 20020073419 | Yen et al. | Jun 2002 | A1 |
| 20020078441 | Drake et al. | Jun 2002 | A1 |
| 20020083060 | Wang et al. | Jun 2002 | A1 |
| 20020120925 | Logan | Aug 2002 | A1 |
| 20020133817 | Markel | Sep 2002 | A1 |
| 20020178447 | Plotnick et al. | Nov 2002 | A1 |
| 20030002638 | Kaars | Jan 2003 | A1 |
| 20030035075 | Butler et al. | Feb 2003 | A1 |
| 20030055699 | O'connor | Mar 2003 | A1 |
| 20030056101 | Epstein | Mar 2003 | A1 |
| 20030101451 | Bentolila et al. | May 2003 | A1 |
| 20030149975 | Eldering et al. | Aug 2003 | A1 |
| 20030154475 | Rodriguez et al. | Aug 2003 | A1 |
| 20030172381 | Janevski | Sep 2003 | A1 |
| 20040003397 | Boston et al. | Jan 2004 | A1 |
| 20040003406 | Billmaier | Jan 2004 | A1 |
| 20040031046 | Weinblatt et al. | Feb 2004 | A1 |
| 20040034874 | Hord et al. | Feb 2004 | A1 |
| 20040068737 | Itoh et al. | Apr 2004 | A1 |
| 20040143349 | Roberts et al. | Jul 2004 | A1 |
| 20040143845 | Lin et al. | Jul 2004 | A1 |
| 20040189710 | Goulden et al. | Sep 2004 | A1 |
| 20040205810 | Matheny et al. | Oct 2004 | A1 |
| 20040210847 | Berson et al. | Oct 2004 | A1 |
| 20040255322 | Meadows et al. | Dec 2004 | A1 |
| 20050022211 | Veselov et al. | Jan 2005 | A1 |
| 20050078223 | Liu et al. | Apr 2005 | A1 |
| 20050097599 | Plotnick et al. | May 2005 | A1 |
| 20050149968 | Konig et al. | Jul 2005 | A1 |
| 20050235307 | Relan et al. | Oct 2005 | A1 |
| 20050278731 | Cameron et al. | Dec 2005 | A1 |
| 20060015923 | Chuah et al. | Jan 2006 | A1 |
| 20060026636 | Stark et al. | Feb 2006 | A1 |
| 20060031684 | Sharma et al. | Feb 2006 | A1 |
| 20060187358 | Lienhart et al. | Aug 2006 | A1 |
| 20060195860 | Eldering et al. | Aug 2006 | A1 |
| 20060282847 | Gupte | Dec 2006 | A1 |
| 20070052854 | Yeh et al. | Mar 2007 | A1 |
| 20070078009 | Lockton et al. | Apr 2007 | A1 |
| 20070192784 | Postrel | Aug 2007 | A1 |
| 20070250901 | McIntire | Oct 2007 | A1 |
| 20080064490 | Ellis | Mar 2008 | A1 |
| 20080104634 | Gajdos et al. | May 2008 | A1 |
| 20080127253 | Zhang | May 2008 | A1 |
| 20080222690 | Kim | Sep 2008 | A1 |
| 20080227436 | Gantman | Sep 2008 | A1 |
| 20080244640 | Belleguie | Oct 2008 | A1 |
| 20080250447 | Rowe et al. | Oct 2008 | A1 |
| 20080305815 | McDonough | Dec 2008 | A1 |
| 20080310731 | Stojancic et al. | Dec 2008 | A1 |
| 20080313140 | Pereira et al. | Dec 2008 | A1 |
| 20090009532 | Hallberg | Jan 2009 | A1 |
| 20090064261 | Jung | Mar 2009 | A1 |
| 20090077046 | Narahara et al. | Mar 2009 | A1 |
| 20090077489 | Homma | Mar 2009 | A1 |
| 20090094637 | Lemmons | Apr 2009 | A1 |
| 20090106104 | Upendran et al. | Apr 2009 | A1 |
| 20090119723 | Tinsman | May 2009 | A1 |
| 20090133049 | Bradley | May 2009 | A1 |
| 20090150930 | Sherwin et al. | Jun 2009 | A1 |
| 20090222754 | Phillips et al. | Sep 2009 | A1 |
| 20090235317 | Igarashi | Sep 2009 | A1 |
| 20090259690 | Bogdanov | Oct 2009 | A1 |
| 20090288113 | Skinner | Nov 2009 | A1 |
| 20090317053 | Morley | Dec 2009 | A1 |
| 20090318114 | Bertoni | Dec 2009 | A1 |
| 20090319522 | Karstens | Dec 2009 | A1 |
| 20090327894 | Rakib et al. | Dec 2009 | A1 |
| 20100007797 | Stojancic | Jan 2010 | A1 |
| 20100043040 | Olsen | Feb 2010 | A1 |
| 20100095337 | Dua | Apr 2010 | A1 |
| 20100121691 | Shifflett et al. | May 2010 | A1 |
| 20100125498 | Jaramillo | May 2010 | A1 |
| 20100158391 | Cunningham | Jun 2010 | A1 |
| 20100162344 | Casagrande et al. | Jun 2010 | A1 |
| 20100169906 | Takahashi | Jul 2010 | A1 |
| 20100169911 | Zhang | Jul 2010 | A1 |
| 20100175078 | Knudson | Jul 2010 | A1 |
| 20100205627 | Cesmedziev | Aug 2010 | A1 |
| 20100205628 | Davis et al. | Aug 2010 | A1 |
| 20100242065 | Murakami | Sep 2010 | A1 |
| 20100245681 | Harris | Sep 2010 | A1 |
| 20100269128 | Gordon | Oct 2010 | A1 |
| 20100303338 | Stojancic et al. | Dec 2010 | A1 |
| 20100306193 | Pereira et al. | Dec 2010 | A1 |
| 20100318515 | Ramanathan et al. | Dec 2010 | A1 |
| 20100325135 | Chen et al. | Dec 2010 | A1 |
| 20110067046 | Cox et al. | Mar 2011 | A1 |
| 20110067066 | Barton | Mar 2011 | A1 |
| 20110078729 | Lajoie et al. | Mar 2011 | A1 |
| 20110078733 | Lee | Mar 2011 | A1 |
| 20110088063 | Ben-Romdhane et al. | Apr 2011 | A1 |
| 20110107363 | Sanghavi | May 2011 | A1 |
| 20110122255 | Haritaoglu | May 2011 | A1 |
| 20110177775 | Gupta et al. | Jul 2011 | A1 |
| 20110191806 | Knudson et al. | Aug 2011 | A1 |
| 20110219419 | Reisman | Sep 2011 | A1 |
| 20110280434 | Mamidwar et al. | Nov 2011 | A1 |
| 20110283322 | Hamano | Nov 2011 | A1 |
| 20110283327 | Zhu | Nov 2011 | A1 |
| 20110289098 | Oztaskent et al. | Nov 2011 | A1 |
| 20110289114 | Yu et al. | Nov 2011 | A1 |
| 20110289524 | Toner | Nov 2011 | A1 |
| 20110307931 | Shuster | Dec 2011 | A1 |
| 20110311095 | Archer | Dec 2011 | A1 |
| 20110313856 | Cohen et al. | Dec 2011 | A1 |
| 20110314491 | Delidais et al. | Dec 2011 | A1 |
| 20110314495 | Zenor | Dec 2011 | A1 |
| 20120017236 | Stafford et al. | Jan 2012 | A1 |
| 20120042334 | Choi | Feb 2012 | A1 |
| 20120054848 | Salowey et al. | Mar 2012 | A1 |
| 20120079521 | Garg et al. | Mar 2012 | A1 |
| 20120096491 | Shkedi | Apr 2012 | A1 |
| 20120117584 | Gordon | May 2012 | A1 |
| 20120124625 | Foote | May 2012 | A1 |
| 20120144416 | Wetzer | Jun 2012 | A1 |
| 20120167133 | Carroll | Jun 2012 | A1 |
| 20120174157 | Stinson | Jul 2012 | A1 |
| 20120185895 | Wong et al. | Jul 2012 | A1 |
| 20120185905 | Kelley | Jul 2012 | A1 |
| 20120192227 | Fleischman | Jul 2012 | A1 |
| 20120215789 | Ramanathan | Aug 2012 | A1 |
| 20120246693 | Iqbal | Sep 2012 | A1 |
| 20120303442 | Patwa | Nov 2012 | A1 |
| 20120311618 | Blaxland | Dec 2012 | A1 |
| 20120317240 | Wang | Dec 2012 | A1 |
| 20120331496 | Copertino et al. | Dec 2012 | A1 |
| 20130014146 | Bhatia et al. | Jan 2013 | A1 |
| 20130019262 | Bhatia et al. | Jan 2013 | A1 |
| 20130047178 | Moon | Feb 2013 | A1 |
| 20130047180 | Moon et al. | Feb 2013 | A1 |
| 20130125163 | Chen et al. | May 2013 | A1 |
| 20130132999 | Pandey | May 2013 | A1 |
| 20130162902 | Musser | Jun 2013 | A1 |
| 20130163957 | Ikizyan | Jun 2013 | A1 |
| 20130167171 | Kitazato et al. | Jun 2013 | A1 |
| 20130174191 | Thompson, Jr. et al. | Jul 2013 | A1 |
| 20130198768 | Kitazato | Aug 2013 | A1 |
| 20130205315 | Sinha et al. | Aug 2013 | A1 |
| 20130205316 | Sinha et al. | Aug 2013 | A1 |
| 20130205318 | Sinha et al. | Aug 2013 | A1 |
| 20130205319 | Sinha et al. | Aug 2013 | A1 |
| 20130205335 | Sinha et al. | Aug 2013 | A1 |
| 20130208942 | Davis | Aug 2013 | A1 |
| 20130222597 | Brink | Aug 2013 | A1 |
| 20130239163 | Kim et al. | Sep 2013 | A1 |
| 20130305335 | Syed et al. | Nov 2013 | A1 |
| 20130332951 | Gharaat | Dec 2013 | A1 |
| 20130339999 | Sangiovanni | Dec 2013 | A1 |
| 20140058732 | Labsky et al. | Feb 2014 | A1 |
| 20140082659 | Fife | Mar 2014 | A1 |
| 20140089815 | Gildfind et al. | Mar 2014 | A1 |
| 20140150022 | Oh | May 2014 | A1 |
| 20140164994 | Myslinski | Jun 2014 | A1 |
| 20140201774 | Neumeier | Jul 2014 | A1 |
| 20140337880 | Sorbel | Nov 2014 | A1 |
| 20150070587 | Emeott | Mar 2015 | A1 |
| 20160073155 | Subramaniam et al. | Mar 2016 | A1 |
| 20160261894 | Li | Sep 2016 | A1 |
| 20170134822 | Nguyen | May 2017 | A1 |
| 20170180795 | Cremer et al. | Jun 2017 | A1 |
| 20190261030 | Grover | Aug 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 1354276 | Dec 2007 | EP |
| 1485815 | Oct 2009 | EP |
| 2011055201 | Mar 2011 | JP |
| 9517746 | Jun 1995 | WO |
| Entry |
|---|
| Advisory Action for U.S. Appl. No. 15/396,596 dated Jul. 31, 2019. |
| Blackbum, Steven G., “Content Based Retrieval and Navigation of Music,” University of Southampton Faculty of Engineering and Applied Science, Department of Electronics and Computer Science, Mar. 10, 1999. |
| Blackburn, Steven G., “Search by Humming,” University of Southampton Faculty of Engineering, Department of Electronics and Computer Science, May 8, 1997. |
| Final Office Action in U.S. Appl. No. 15/396,596 dated Sep. 12, 2018. |
| Final Office Action in U.S. Appl. No. 15/396,596 dated May 22, 2019. |
| Ghias, Asif et al., “Query by Humming—Musical Information Retrieval in an Audio Database,” ACM Multimedia 95—Electronic Proceedings, Nov. 5-9, 1995, San Francisco, CA. |
| Non-Final Office Action in U.S. Appl. No. 15/396,596 dated Apr. 13, 2018. |
| Non-Final Office Action in U.S. Appl. No. 15/396,596 dated Jan. 29, 2019. |
| Notice of Allowance for U.S. Appl. No. 15/396,596 dated Mar. 2, 2020. |
| Notice of Allowance in U.S. Appl. No. 15/396,596 dated Sep. 18, 2019. |
| Proquest, PR Newswire, New York, “Mobile Music: Comcast Cellular First in U.S. to Trial Breakthrough Interactive Music Service Call CD,” Feb. 11, 1999, p. 1. |
| Taylor, Chuck, “Company Lets Listeners Dial for CDs StarCD Turns Cell Phones Into Radio-Music Storefronts,” Billboard: Jun. 26, 1999; 111, 26; General Interest Module, p. 86. |
| Tseng, Yuen-Hsien, “Content-Based Retrieval for Music Collections,” SIGIR99 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA, Aug. 15-19, 1999. |
| Whittle, Robin, “Future Developments in the Music Market,” Contemporary Music Summit in Canberra Apr. 27, 1995, Apr. 11, 1995. |
| Number | Date | Country | |
|---|---|---|---|
| 20200120382 A1 | Apr 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15396596 | Dec 2016 | US |
| Child | 16709151 | US |
