Patent Application
20170334234
The present invention relates to digital watermarks, and more particularly to a system and method for identifying the source of counterfeit copies of multimedia works using layered simple digital watermarks.
A wide variety of techniques for embedding watermarks into multimedia works have been in use for many years. However, these older methods were susceptible to discovery or intentional removal by individuals attempting to circumvent the copy protection. In other some cases, such embedded watermarks were also susceptible to accidental removal.
Watermarks are based on modulating some aspect of a media transmission in such a way that the modulation can be later recovered and decoded. For steganographic watermarks, these changes must be small so that they can go unnoticed by the viewer. However, given the limited number of bits used to represent broadcast media programs the range of tolerable adjustments may be limited, for example, to one or two LSB steps, particularly for such basic parameters as Y, Cb, and Cr values. In sonic cases, such as over a large area of solid color, even a single bit of variation can become visible to the viewer.
One approach to avoiding easily-detected variations is to include a schedule feature to allow the mixture of simple watermarks to be periodically changed to prevent persons making and/or distributing unauthorized copies of multimedia works from knowing for sure exactly what the watermarking pattern is. Another approach is to adjust the intensity of the watermark based on the context. This context adjustment feature may be based on a prior analysis of the content being broadcast or a specified category of a live broadcast that is not available for prior analysis. The analysis or category specifies the appropriate level of watermark intensity for a particular section of content and is added to the watermark mode variation schedule applied to the watermarked content, so that the watermark intensity can be adjusted at the multimedia output player without the need for content analysis at the player. However, for on-demand content there is no equivalent approach available because there is no meaningful schedule (i.e., the nature of live-streaming is such that the multimedia works are not readily available for prior analysis). As a result, it may be the case that intensity and watermark mode settings can only be incorporated as part of a conditional access authorization process prior to a consumer receiving the multimedia content.
It is also the case that many of the existing approaches for watermarking digital content are primarily focused on embedding a generic copyright notice into the work, or identifying the copyright owner, or enabling automated searches for illegal copies. Many of these approaches require complex Fourier transform and psychoacoustic or psychovisual analysis of the content to improve either the robustness of the watermark or its degree of concealment. Most of these more advanced methods require specialized hardware to support the complexity of their methods.
Virtually all known watermarking methodologies combine, at the multimedia production and/or distribution sources, either the analog or digital representation of a multimedia work with the representation of the watermark which may be variously described as modulation, signal, noise, or auxiliary data. As such, existing approaches generally fail to anticipate the need for applying a watermark from within a generic rendering device, such as a set top box or handheld device. Trends in the distribution of copyrighted multimedia works are such that a need has arisen for watermarking material from within these platforms. However, these platforms generally lack the means to support any but the most basic of watermarking methods. Thus, while much of the existing watermarking approaches are focused on either identifying the copyright holder or marking the work as being copyrighted, there also exists a need to identify the endpoint of the distribution system (i.e. the particular user and display device) that generated a counterfeit copy so that the endpoint can be terminated and any future copying avoided.
The present invention addresses the above shortcomings through a system and method of layered simple watermarks. In general, the present invention is based on the realization that, while a single simple watermark may be easy to locate and circumvent, layers of watermarks are much more difficult to identify. Moreover, removing some but not all of the watermarks will only serve to show an intent to defraud that is more easily prosecuted than an allegedly accidental publication of a copy made for personal use. The basic concept of any watermark is to deter unlawful use of copyrighted material by making it possible to demonstrate that the copy is unlawful. This deterrent is more effective when the watermark can be used to identify a particular device that made the copy or even a particular user. If, in addition, there are multiple layers of watermarks that can be selected, such as on-the-fly remotely by the distributor of the copyrighted material, then it becomes impossible for a potential pirate to have complete confidence that all of the watermarks have been located and safely disabled.
The present invention is also based on the enhancement of media security by dividing a media work such as a video into enhanced security segments that may be searched on an example basis, including by retrieval of corresponding security encoded segments, decoding of the segments, and comparison with a segment of the media of interest to identify changes and/or anomalies in the media segment indicative of unauthorized alteration and/or copying.
In accordance with the principles of the invention, the selection of simple watermarks is derived from those parameters of audio, video, text and other aspects of a multimedia work that can be altered slightly such that the change can be detected by instrumentation, but not by casual observation. These parameters include, but are not limited to, image brightness, audio phase relative to video, audio tone or equalization, image horizontal and vertical placement, video color temperature, anamorphic distortions, image black level, image hue, background noise floor, and text characteristics such as font size, boldness and tilt. These changes can be distributed spatially, temporally, or in other domains such as frequency or wavelet.
Since these changes are intended to impart a watermark that contains information, in the context of the present invention these changes are considered to be modulations of the associated parameter. The modulation strategy can include redundancy, bit scrambling and error correction coding to improve the ease with which the watermark can be read. These modulations of basic parameters are for the most part simple to apply using commonly available infrastructure elements in most display devices.
As a further measure to confound attempts to analyze the currently applied layers of watermarking, the invention may allow one or more of the layers to he without content such that there is nothing to decode. The intent of such layer(s) is to further increase the doubt in a potential pirate's mind that all watermarks have been located and disabled.
Architecturally, the application of watermarks must be integral to the decryption and decoding of a protected multimedia work with the goal e end user cannot access any copies of the presentation that lack watermarks.
in one preferred embodiment of the present invention, that of a set top box, the above items are sufficient to form a robust deterrent to the unlawful copying of copyrighted works. Since a set top box is a secure environment and set top box manufacturers must regularly certify that their design is resistant to tampering, it is a widely accepted standard of security in the business of distributing copyrighted works.
In the case of more generic handheld devices, however, security is less predictable. To that end, in an alternate preferred embodiment of the present invention, that of a handheld device, the decrypt-decode-watermark process is made more tamper resistant by adding one or more of the following features. The process is based on code that periodically or continuously sends tamper metrics back to the server as part of an encrypted handshake routine to confirm that an unaltered copy of the code remains in control of the process. Alternately, the process may be based on code that is incomplete, as stored in the handheld device, and relies on code (or pseudocode) embedded in the transmission of the multimedia work that is never stored on the handheld device, thus eliminating the possibility of fully reverse engineering the locally stored code. Further, the process is based on code that is sensitive to real-time delays, or other indications that the code has been halted even momentarily by a debug or dynamic code analysis tool, or that the multimedia stream has been stored and played back at a later time. Should any of these anti-tamper methods detect an issue, the system can be configured to take remedial action, such as triggering an automatic update or disabling/erasing the process altogether.
For a watermark such as in the present invention to be useful for forensic purposes, it must be practical to recover the watermark from illegally recorded video. As the decode process can be technically challenging, it is sometimes best for a user of the present invention to engage watermark decoding as an external service rather than an internal activity.
The present invention further provides for increased efficiency and security in watermark decoding. As the analysis and decoding of the layers of simple watermarks described herein are most readily accomplished as a comparison between the original work without watermark and the copy with watermark. The simplest way to reveal the watermark for analysis and decoding is to subtract out the original content leaving the watermark as a residual. However, in the context of a broadcast system there is the problem that original archive copies and illegal copies discovered on the Internet are not easily related, e.g., it is not clear where to find the original based on having a copy from the Internet. Moreover, in a system that is widely implemented, for each watermark decoding event this difficult matching process must be repeated an enormous number of times.
Access to the original content is also a problem, as broadcasters are concerned with the security risk associated with releasing an unsecured copy of original content from their secured archives for use in a watermark investigation. At a minimum, broadcasters may require editing the original content down to a shortened clip, so that even if that portion of the content was inadvertently released somewhere along the watermark application and/or decoding process, the broadcaster does not suffer the loss of a full copy of the content. However, because such content control requires manual processing of the content, this is not an efficient or economically attractive approach.
The need for security of the information being sent out and the need for efficiently locating the matching original and adding security features to the content sent for analysis are addressed in the present invention in part by the use a searchable archive of multimedia works. This approach reduces the process of locating the matching original to a simple database search, and the security issues raised by the need to send out an original from a secure facility may he addressed by structuring the archive in a manner that does riot make full copies of original content vulnerable to inadvertent release at any point in the watermarking system. For example, files may be stored at reduced resolution as compared to the original, the data may be segmented such that no one file contains a complete copy of the content (such as a complete movie or a complete television program), and/or by not sending out all of the segments of a program from the searchable archive to accomplish a decode. As a further security measure, all files sent from the archive may be encrypted.
One embodiment of the present invention is in the form of an added subsystem referred to in the industry as a “proof-of-play” system. A typical proof-of-play system makes a somewhat low resolution recording of the final broadcast stream including all inserted commercials. Its intended use is to validate to paying advertising customers that their commercials have been delivered in their intended program slots. Added to this system is a segmentation process that breaks the recording up into shortened portions (for example, 10 minutes each) that also align with the boundaries between programs based on the electronic program guide data. Each of those shortened portions would receive a hashing code, and the segmented portions would then be encrypted to further protect the content. Similarly, the subsystem is configured to segment a suspect copy and search for matching hash codes within the database of previously recorded programs. The combination in this embodiment of lower resolution, lack of any one file with a full length program, file encryption on top of secure HTTPS communications and constructing the decode service center in a secure facility address the security concerns, while simultaneously providing an indexed database of previously broadcast programs that may be readily used with an automatic search process to retrieve files needed for watermark decoding analyses.
It should be noted that while the process above is described in terms of recording a conventional broadcast, it is common knowledge that the broadcast industry is shifting heavily to streaming and on-demand services in which there really is no common broadcast, i.e., where each viewer individually retrieves files from a media server. One skilled in the art will recognize that the same the same kind of segmentation and indexing process can be employed in an Internet-based media distribution model, for example, based on a conventional broadcast stream being fed into the an Internet-based system or by directly interrogating the media files loaded into such a system.
One weakness of simple watermarks that is often remarked in the literature is that their effectiveness can be heavily dependent on the content of the audio and video being watermarked. To address these issues, in the present invention analysis or categorization of the multimedia programs being broadcast may be made to determine the optimal intensity and method for watermarking that program or category of programs. This information may be incorporated into the watermark variety schedule and passed to the endpoint display devices, thus relieving the endpoint display devices of the need to do that analysis on the fly.
The present invention may also be configured to address the problem arising from the variety of video contexts that simple watermarks may either wash out or become too visible if not adjusted to suit the context. This is addressed by pre-analyzing the source video to optimize the watermark application to specific a video and/or audio construct, and make the resulting data available to the watermark system that schedules the variety in the watermark. The adjustment of the watermark intensity may then be incorporated by the watermark systems as another parameter to be included in a watermarking schedule. As a practical matter, the majority of the target output players in the market lack the hardware resources to be able to analyze and implement such a watermarking scheme in real time.
A further aspect of the present invention includes addressing security concerns in the context of a web browser. It is well known in the industry that web browsers are aggressively probed for weaknesses by “hackers” seeking to defeat security protections, and such weaknesses are commonly found. For a watermark to be effective, it must be applied in a protected environment that is resistant to efforts to extract a clear copy of the multimedia work prior to watermarking. A web browser typically is not such an environment.
The present invention may provide enhanced security in a web browser environment by providing an additional background application that serves as a secure execution environment. Many interfaces between the background application and the web browser are possible. In a preferred embodiment, the background application is constructed as a nano-scale proxy server which may be accessed like any other web site from the browser. The Javascript content of the remote web site may also be constructed to make this routing invisible to the user.
This approach is similar in intent to the Encrypted Media Extensions of HTML5, but has the advantage of being more extensible and browser independent. A further advantage is that it is a specialty application not subject to as much attention from the hacking community as are the main standardized aspects of HTML. While the application must be made native to each intended execution environment, this is not unlike the current accepted practice for mobile applications, and therefore this disadvantage is not a major deterrent to the approach.
The present invention's advantages include: (i) the ability to be implemented using circuits that are commonly available in all modern multimedia-capable processors because the watermarks on which the method of the present invention are relatively simple, i.e., there is a significant system cost reduction because no special hardware is required within the STBs and smart devices; (ii) the ability to entirely implement the present invention in software, including software that substantially increases the utility and functionality of the associated electronic devices, i.e., the system may be readily implemented across a broad spectrum of the multimedia distribution marketplace, requiring only access to the STB firmware and a set of player apps for common smart devices; and (iii) enabling future improvements in multimedia security to be readily incorporated because the security of the present invention is based on variety rather than complexity, and therefore the range of watermarking strategies may be readily expanded as additional hardware features friendly to watermarking are released in future hardware. While these and other benefits potentially require the present invention to use more complex decoding processes, this is off-set to some extent by the fact that the number of decode points needed are very few as compared to the number of display endpoints. Thus, even with this trade off the present invention provides a system with significant economic advantages compared to with prior approaches.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings. It should be recognized that the one or more examples in the disclosure are non-limiting examples and that the present invention is intended to encompass variations and equivalents of these examples. The disclosure is written for those skilled in the art. Although the disclosure uses terminology and acronyms that may not be familiar to the layperson, those skilled in the art will be familiar with the terminology and acronyms used herein.
The present invention is described in the context of broadcast systems, such as cable, satellite, IPTV and streaming broadcast systems, and content output devices, such as a set top box and a handheld display device, in which a plurality of watermarking methods are available to identify the device and/or the associated user. These watermarking methods may be selectively enabled or disabled according to a watermarking strategy transmitted from a watermarking server at the broadcast headend to each set top box and/or handheld display device. The present invention is not limited to only the foregoing, and includes other systems and content output devices, including those used for scheduled streaming, on-demand streaming, on-demand file transfer, and social media.
The watermarking system, in the example of use with a set top box, includes the following components:
The watermark server may include a watermarking database containing:
The above described watermark server may be configured to transmit watermarking codes and/or strategies to set top boxes and other display devices. In addition, watermarking algorithms may be transmitted to set top boxes and other display devices either as static code or as on-the-fly code or pseudocode embedded in the program transport stream. Moreover, the watermark server may be configured to transmit status query messages to set top boxes and other display devices and for receiving and logging responses received from the set top boxes and other display devices.
With respect to a set top box configured in accordance with the principles of the invention, such a device may be configured to render multimedia content on a display device. Such a set top box may also be configured to receive watermarking codes and strategies from the aforementioned watermark server.
The set top box may also be configured to receive the watermarking algorithms from the above watermark server either as static code or as on-the-fly code or pseudocode embedded in the program transport stream, as well as to receive status query messages from the watermark server and to transmit any required responses back to the server.
Another aspect of the invention relates to methods for applying watermarking algorithms to modulate the watermarking code onto one or more program parameters in accordance with a currently active watermarking strategy.
Still another aspect of the invention relates to retransmitting a program stream to a handheld display device with the watermarking already embedded in the program content or with watermarking instructions tightly bound to the program stream.
It should be appreciated that the present invention may incorporate methods for applying watermarking algorithms in, at a minimum. One or more of the following forms: modulating image brightness, modulating audio phase relative to video, modulating audio tone or equalization, modulating image horizontal and vertical placement, modulating video color temperature, modulating image anamorphic distortions, modulating image black level, modulating image hue, modulating background noise floor, modulating text characteristics such as font size, boldness and tilt, and modulating any other available characteristic of the multimedia program.
All of the modulation methods above may be based on changes that may be distributed spatially, temporally, or in other domains such as frequency or wavelet.
In one embodiment, a system configured in accordance with the principles of the invention may be configured to populate watermarking server database tables in a watermark server, provide connectivity from the watermark server to the transport stream and the Internet, and to roll out watermark programming and/or related watermark data to watermark capable set top boxes and apps for other display devices. The system may also be configured to either automatically vary the watermarking strategy or o expect changes to the watermarking strategy to be a part of the daily schedule. The system may establish and enter either a daily schedule or a recurring schedule for the control of watermark strategy timing if enabled above.
In certain embodiments, the system operates by having a watermark server maintain a dialog with each connected endpoint device to confirm that each is authorized, has assigned watermark codes and shows no evidence of tampering. It should be appreciated that, in systems that lack a return path to the head end, an authorization/watermark code may be transmitted on demand and retained by the device.
As part of the system, the watermark server may transmit the watermark strategy schedule to each endpoint. Each connected display device may correspondingly receive and store its watermark codes and the watermark strategy schedule. As a multimedia program is displayed on a connected display device, the program is rendered while embedding whichever watermarks are required by the current watermark strategy schedule.
For enhanced security and efficiency of operation, optionally the watermarking system may further include a searchable archive of previously broadcast multimedia programs. Such an archive may further be configured to:
For enhanced concealment and readability of the watermark, optionally the watermarking system a further be configured for watermark optimization by specifying both the tensity of the watermark and the watermarking strategy that is most appropriate for a particular multimedia work or for a category of multimedia works. Alternatively, watermark optimization may be configured as a function of a determination if the media's general category.
The embodiment of the control aspects of the present invention will vary in accordance with the nature of the multimedia program delivery system into which it is incorporated. In the case of a conventional DBV-S or similar one-way broadcast system, control of the watermark is embedded into the transport stream as ancillary data, and each endpoint device monitors the stream extracting whatever control data is appropriate for its use. In the case of an IPTV system, much the same control strategy can be employed but there is the added benefit of a return path from each endpoint that can be used for gathering endpoint status and reports of tampering with the security of the watermark system. In the case of an Internet live streaming or on-demand program delivery system, the customary audio/video delivery mechanism may make no allowances for ancillary data, forcing the watermark system to rely instead on a client-server relationship between the endpoint display devices and the broadcast studio for control of the system. This implies that control data is requested by each endpoint individually rather than being pushed from the server en masse as in the previous cases. One skilled in the art will recognize that communications between the distribution point for the multimedia programs and the end point display devices must exist for the present invention to function, but its precise nature is not central to the invention.
Embodiments of the present invention are now described in conjunction with the Figures.
The processes for managing the watermark in this embodiment are coded as at least three of these four threads running concurrently. A first thread manages broadcast data handling decryption, channel selection and watermark control: the ASI monitor thread is used for unidirectional DVB systems and the Ethernet monitor thread is used for bidirectional IPTV systems. A second thread manages the watermark variety schedule, and a third thread manages the actual insertion of the watermark, particularly those that are applied in a temporal sequence. One skilled in the art would see that a set top box could also be managed by other methods such as the one described for handheld devices below.
These processes are described in more detail as follows.
ASI Monitor Thread. The ASI monitor thread 201 manages data received from a broadcast stream, in this embodiment a DVB broadcast stream. The thread begins at step 2.00 (identified in
If at step 2.04 it is determined that the packet is part of an active stream, the packet is tested at step 2.05 to determine whether the packet contains video and/or audio data. If video and/or audio data is present, the packet is forwarded to the set top box's video and/or audio decoder for decoding at step 2.06, and control reverts back to step 2.00 to await receipt of another packet from the distribution medium 104.
If at step 2.05 it is determined that the packet not include video and/or audio data, audio/video data, control proceeds to step 2.07. At step 2.07 the packet is tested to determine whether it contains an update to the watermark ID code for the set top box. If a watermark ID code update is present, at step 2.08 the watermark ID code is saved in the set top box for later use.
If at step 2.07 the system determines that the packet does not include a watermark ID code, the packet is tested at step 2.09 to determine whether it contains an update to the watermark strategy schedule. If a watermark strategy schedule update is present, at step 2.10 the watermark strategy schedule update is saved in the set top box for later use. In either case, control reverts back from step 2.09 or step 2.10 to step 2.00 to await receipt of another packet from the distribution medium 104.
The present invention is not limited to the specific ASI processing described above. For example, the system may be configured to determine whether a received packet contains mix of information type (for example, video and/or audio and watermark data), and parse the content of the packet accordingly.
Ethernet Monitor Thread. The Ethernet monitor thread manages data received as part of an IPTV broadcast stream or the set top box feedback portion of an ASI-based broadcast system, starting at step 2.20 (identified in
If it is determined in step 2.24 that the packet does not contain video and/or audio data, at step 2.27 the packet is tested to determine whether it contains an update to the watermark ID code for the set top box. If a watermark ID code update is present, at step 2.28 the watermark ID code is saved in the set top box for later use.
If at step 2.27 it is determined that the packet does not contain an update to the watermark ID code for the set top box, then at step 2.29 the packet is tested to determine whether it contains an update to the watermark strategy, such as an update to the watermark schedule. If a watermark strategy update is present, at step 2.30 the watermark strategy change ID is saved in the set top box for later use.
If it is determined in step 2.29 that the packet does not a schedule update, the packet is tested at step 2.31 to determine whether the packet includes a request from the server for status from the set top box. If such a request is present the request is forwarded for processing at step 2.32 to send the requested set top box status information back to the server. In either case, control reverts back from step 2.31 or step 2.32 to step 2.00 to await receipt of another packet from the distribution medium 104.
Timed Event Thread. The Timed Event Thread controls events that need to happen at a scheduled time. Beginning at step 2.40 (identified in
Regardless of whether a watermark strategy change is implemented, control passes to step 2.43, where the system compares the current time to the time for the next scheduled tamper check. If the current time has not reached the scheduled time, control reverts back to step 2.40 to re-start the thread
If the current time has reached the scheduled time, at step 2.44 the set top box executes a tamper check to determine whether there have been any attacks on the security of the system. At step 2.45, if the system determines that the result of the tamper check was satisfactory, control reverts back to step 2.40. On the other hand, if the system determines that the tamper check was failed, control moves to step 2.46 to cause action to block tampering, for example, by reloading software, disabling the set top box or other remedial action. Control then reverts back to step 2.40 to re-start the thread.
Watermark Thread. The Watermark Thread manages the rendering of watermarks on the set top box output. In some set top boxes it may be possible to identify the user, in addition to being able to identify the individual set top box. Beginning at step 2.60 (identified in
At step 2.64, the system begins to execute a loop that runs for each type of watermark. Specifically, at step 2.65 the system first compares the current watermark type to the current watermark strategy to determine whether the current watermark type is part of the current watermark strategy. If not, control reverts to step 2.64 for the next watermark type.
If the current watermark type is part of the current watermark strategy, at step 2.66 the system determines whether the current watermark type is a temporal sequential watermark, and if so, whether it is time for the next step in the temporal sequence. If it is time for the next step in the sequence, at step 2.67 the sequence for the current watermark is advance to the next step. At step 2.68 the current state of the current watermark is rendered to be presented in the set top box output.
At step 2.69 the system checks whether there are more watermark types to loop through, and either shifts control back to the beginning of the loop at step 2.64, or reverts back to step 2.60 to await receipt of another packet from the distribution medium 104.
Ethernet Monitor Thread. The Ethernet monitor thread manages data received as part of an Internet video stream, as well and other control data received via Ethernet. The thread 302 begins at step 3.00 (identified in
If it is determined in step 3.04 that the packet does not contain video and/or audio data, at step 3.07 the packet is tested to determine whether it contains an update to the watermark ID code for the handheld device. If a ark ID code update is present, at step 3.08 the watermark ID code is saved in the handheld device for later use.
If at step 3.07 it is determined that the packet does not contain an update to the watermark ID code for the handheld device, then at step 3.09 the packet is tested to determine whether it contains an update to the watermark strategy, such as an update to the watermark schedule. If a watermark strategy update is present, at step 310 the watermark strategy change ID is saved in the handheld device for later use.
If it is determined in step 3.09 that the packet does not a schedule update, the packet is tested at step 3.11 to determine Whether the packet includes a request from the server for status from the handheld device. If such a request is present the request is forwarded for processing at step 3.12 to send the requested handheld device status information back to the server.
At step 3.13 the system determines whether the packet is part of a block of a dynamically sent watermark procedure. If so, it is cached at step 3.14 for later execution. In either case, control reverts back from step 3.13 or step 3.14 to step 3.00 to await receipt of another packet from the distribution medium 104.
Timed Event Thread. The Timed Event Thread controls events that need to happen at a scheduled time. Beginning at step 3.20 (identified in
Whether or not a watermark strategy change is implemented, control passes to step 3.23, where the system compares the current time to the time for the next scheduled tamper check. If the current time has not reached the scheduled time, control reverts back to step 3.20 to re-start the thread.
If the current time has reached the scheduled time, at step 3.24 the handheld device executes a tamper check to determine whether there have been any attacks on the security of the system. At step 3.25, if the system determines that the result of the tamper check was satisfactory, control continues to step 3.27. On the other hand, if the system determines that the tamper check was failed, control moves to step 3.26 to cause action to block tampering, for example, by reloading software, disabling the handheld device or other remedial action. Control then continues to step 3.27.
At step 3.27, the system determines whether the current time has reached a schedule time for the next data update from the watermark server. If the current time has reached the scheduled time, at step 3.28 the handheld device sends a request for an update to the watermark server.
Watermark Thread. The Watermark Thread manages the rendering of watermarks on the handheld device output. In some handheld device it may be possible to identify the user, in addition to being able to identify the individual handheld device. Beginning at step 3.40 (identified in
At step 3.44, the system begins to execute a loop that runs for each type of watermark. Specifically, at step 3.45 the system first compares the current watermark type to the current watermark strategy to determine whether the current watermark type is part of the current watermark strategy. If not, control reverts to step 3.44 for the next watermark type.
If the current watermark type is part of the current watermark strategy, at step 3.46 the system determines whether the current watermark type is a temporal sequential watermark, and if so, whether it is time for the next step in the temporal sequence. If it is time for the next step in the sequence, at step 3.47 the sequence for the current watermark is advance to the next step. At step 3.48 the current state of the current watermark is rendered to be presented in the handheld device output.
At step 3.49 the system checks whether there are more watermark types to loop through, and either shifts control back to the beginning of the loop at step 3.44, or reverts hack to step 3.40 to await receipt of another packet from the distribution medium 104.
Ethernet Monitor Thread. The Ethernet monitor thread manages data received via Ethernet. The thread 402 begins at step 4.00 (identified in
If at step 4.02 it is determined that the packet does not contain status information, at step 4.04 the packet is checked to determine whether watermark schedule information is present that needs to be imported into the watermark variety schedule. If watermark schedule data is determined by the system to be present, the watermark variety schedule is updated in the database in step 4.05.
At step 4.06 the system determines whether the packet includes a data request from an endpoint device. If such a request is present, the system in step 4.07 forwards the requested data to the requesting endpoint device, followed by reversion of control back to step 4.01 to wait for the next packet.
If it is determined in step 4.06 that the packet does not include a data request from an endpoint device, at step 4.08 the system determines whether it contains registration data that will be used to identify the endpoint device and/or user with a watermark ID. If present, the registration data is logged to the database at step 4.09.
At step 4.10 the packet is checked to determine whether it contains a tamper warning from an endpoint device, and any such event is logged to the database in step 4.11. Whether or not tamper warning information is present in the packet, control reverts back from step 4.10 or step 4.11 to step 4.00 to await receipt of another packet.
Timed Event Thread, The Timed Event Thread controls events at the watermark server that need to happen at a scheduled time. Beginning at step 4.20 (identified in
Console Thread. The Console Thread manages action requests from the system operator via the server console. Beginning at step 4.40 (identified in
The system at step 4.43 next determines whether the operator input is an endpoint device refresh request, and in response to such a request at step 4.44 identifies the endpoint device as active in the system, sends watermark ID and schedule data to stream-based devices, such as handheld devices.
At step 4.45 the system determines whether the operator input is a request for a report. If a report request is present, the system generates the requested report at step 4.42, and control reverts to step 4.40.
Similarly,
The decoding of a watermark is based on the idea that where watermarks are modulations of a particular parameter of a multimedia work, that modulation may be measured and converted to a digitized waveform. The digital waveform may then be analyzed using the full range of digital signal processing techniques that are available for other aspects of audio/video production. An example embodiment of a process of decoding a watermark is described with the aid of
At the beginning of decoding thread 701 the user at step 7.01 selects files representing the multimedia work being examined in both watermarked and not-watermarked form. These files can be the original multimedia production files or can be previous reductions of those files to parameter hashing data. At step 7.02 the frame rates of the files are normalized to allow frame-by-frame comparison that is useful for some forms of sequential watermarks. Next, at step 7.03, the parameters of the original multimedia work that have been modulated with the watermark are reduced to hashing statistics, either from the original production file or from the results of some previous analysis. These hashing statistics are then converted in step 7.04 into a digital waveform representation of the particular watermark being analyzed.
At step 7.05, in a manner known to one skilled in the art, various noise reduction filters and techniques are applied to the generated waveform to extract the watermark and reject all other signals that are part of the original measurement. The noise-reduced waveform is then aligned at step 7.06 with a conversion template to convert the waveform into a binary representation of the watermark.
This embodiment provides an optional opportunity at step 7.07 for the user to inspect the waveform and the result of the template conversion, take advantage of the fact that the human eye may sometimes see patterns that elude automated algorithms, and apply appropriate corrections to the binary representation of the watermark.
Next, at step 7,08 the binary representation of the watermark is converted into a human-readable representation and is presented to the user for review.
As a part of this process, in this embodiment at step 7.09, error correction codes inherent in the watermark may be tested to validate the results of the decode process.
In the embodiment illustrated in
Ethernet Monitor Thread. In the Ethernet monitor thread 802, beginning at step 8.00 (identified in
At step 8.04 the packet is checked to determine if it is part of a monitored transport stream. Depending on the determination result, in this embodiment one of two different sequences is executed:
If the packet is part of a monitored transport stream, at step 8.05 the system determines whether the packet requires decryption, and if so, decrypts the data. The unencrypted packet is then de-multiplexed from the transport stream in step 8.06 and assigned to a particular program stream or ancillary data stream.
Next, at step 8.07 any Electronic Program Guide data is extracted from the packet and saved for inclusion as metadata in the resulting program segment file. The packet content is then analyzed in step 8.08 for a matching signature and other parameters useful for later decoding. This content is also decimated (i.e., the number of bits used reduced to encode each frame, thereby reducing the quality of the playback and making it less attractive to a potential copyright infringer) to a specified content bitrate if required before adding it to the current program segment file.
At step 8.09, the completed file segments are combined with metadata and analysis results, and stored in a database. Control then reverts back to step 8.00.
If on the other hand at step 8.04 the packet is determined to not be part of a monitored transport stream, at step 8.11 the system determines whether the packet is associated with a live stream. If not, control reverts back to step 8.00. If, however, the packet is associated with a live stream, the system at step 8.12 determines whether the packet requires decryption, and if so, decrypts the data, and then the packet content is analyzed at step 8.13 for a matching signature and other parameters useful for later decoding. This content is also decimated to specified content bitrate if required before adding it to the current program segment file.
After the processing in step 8.13, control passes to step 8.14, where completed file segments are combined with metadata and analysis results and stored in the database, and then control reverts back to step 8.00.
Console Thread. In the console thread beginning at step 8.20 (identified in
If at step 8.21 the received system operator input is determined to not be a match request, control proceeds directly to step 8.25, where the system then checks the operator input to see if it is a media file analysis request. If so, at step 8.26 the system executes a file analysis and segmentation process on the selected file, and the result of the analysis and segmentation process is stored in the database at step 8.27.
If at step 8.25 the received system operator input is determined to not be a media file analysis request, control proceeds to step 8.28 to determine whether the system operator input is a system configuration change request. If not, control reverts have to step 8.20 to await the next system operator input.
If the operator input is a system configuration change request, then global configuration items are identified. In this embodiment, video is to be delivered by multicast to the whole network, while this application only needs to receive data from the particular URL to be monitored. Thus, at step 8.29 a URL for each stream that is to be monitored, analyzed and segmented is identified, at step 8.30 conditional access credentials are identified, at step 8.31 the PID of TS-based electronic program guide (“EPG”) data and the URI, of stream-based EPG data are identified, at step 8.32 the target file size and target bit rate for the recorded file segments are identified, and at step 8.33 the URL of the Watermark Decode system to which the results of match requests need to be sent is identified. Control then reverts have to step 8.20 to await the next system operator input.
The database 907 may also receive and respond to copy-to-original match requests 909 received by a search facility 910 which attempts to locate relevant original content (complete or in segments) for comparison to a corresponding copy of the content. If the database 907 responds with the requested comparison content, and if the conditional access process 902 allows, the content is encrypted in an encryption process 911 and forwarded via link 912 to a watermark decoding system (which may be local or remote) for subsequent extraction of the watermark data based on the information retrieved from the database 907.
For enhanced security in a web browser environment, optionally the watermarking system may be configured such that the endpoint display device previously described as a STB or handheld device is instead a web browser. The web browser may be supplemented with an independent security application. In one embodiment, the security application takes the form of a small-scale local server in which all requests to the remote video server site are redirected through the local server, which in turn requests the video data from the remote video web server. In this case, “local” refers to means an application running on the same computer as the web browser. The web browser thus accesses the local application through the local loopback address, such as 127.0.0.1, at which the local application functions as a web server. Data requests intended for a compatible remote video server are instead passed to the local server, which then reflects the requests to the remote video server. When the local server receives the response from the remote video server, it applies decryption and/or watermarking to the content before passing the data on to the web browser. This redirection to the local server can be made unobtrusive to the user by encapsulating the references to the local URL in Javascript sent from the remote web server. Of course, one skilled in the art will recognize that there are a number of other equivalent embodiments which might use other methods for communicating with the browser, such as the plug-in method or the device driver method.
The local server provides a more secure environment in which to execute such operations as decryption and watermark application. It further serves as a platform for monitoring attempts to bypass or tamper with the security features or to interface with physical security devices such as USB dongles or accessory cards.
At step 1201 the local server 1104 receives a data request from web browser 1103 that is routed to the local server 1104 instead of to the Internet 1102. The local server 1104 at step 1202 redirects the request through Internet 1102 to the remote video server 1101.
The response from the remote server 1101 is received at the local server 1104 in step 1203, and if decryption is required at step 1204 the local server 1104 decrypts the response the remote server 1101.
At step 1205 the local server 1104 applies a watermark to the contents of the response received from remote server 1101, and if required encrypts the content with the watermark at step 1206. The local server 1104 then at step 1207 forwards the watermarked content to web browser 1103.
The local server 1104 in this embodiment also executes tests to determine whether its security is intact and whether there are efforts to tamper with or bypass the security. At step 1208 a security check in the environment is performed, and at step 1209 the local server 1104 transmits relevant tamper metrics to remote server 1101 for further remote processing.
The foregoing disclosure has been set forth merely to illustrate the invention and is not intended to be limiting. Since modifications of the disclosed embodiments incorporating the spirit and substance of the invention may occur to persons skilled in the art, the invention should be construed to include everything within the scope of the appended claims and equivalents thereof.
This application claims priority under 35 U.S.C. §119(e) to U.S. provisional application Ser. No. 62/338,670, filed May 19, 2016, the entire disclosure of which is incorporated by this reference into the present U.S. patent application.
| Number | Date | Country | |
|---|---|---|---|
| 62338670 | May 2016 | US |
