Patent Application
20180091860
The invention relates to a system and method for enhancing a media recording. The invention further relates to a sender device or receiver device for use in the system. The invention further relates to a computer program product comprising instructions for causing a processor system to perform the method.
Due to the ubiquity of digital cameras and screens, it may frequently occur that a camera recording of a scene includes a screen displaying visual content as part of the camera recording. This may take place coincidentally. For example, when recording a home video in someone's living room with a digital video camera, there may be a television playing out a television show in the background. As such, the home video may include a camera recording of the television and the visual content playing on the television at the time of recording.
Media recordings may also more structurally include camera recordings of screens displaying visual content. Here and in the following, the term ‘screen’ refers to displays such as those included in televisions, monitors, tablet devices, smartphones, etc., including two-dimensional, three-dimensional, light field and holographic displays, but also to projection screens and other types of surfaces on which visual content may be rendered, as well as to other types of visual rendering of visual content.
A non-limiting example of the more structural recording of screens displaying visual content can be found in the field of videoconferencing systems and mobile video communication applications (e.g. Skype, Lync, WebRTC, FaceTime), which allow remotely located people to have real-time conversations by recording audio via a microphone and video via a camera and transmitting the resulting media recording to the parties involved. Initially, videoconferencing systems focused on recording only the people involved in the conversation as people typically will sit in front of the camera. Advancements in camera recording techniques, such as increased resolution and a larger angle of view, have made it possible to record much more than just the person; the camera can also record his/her environment such as the living room or office cubicle, including any screens that may be present, such as a television screen which is showing television content, or a tablet device which is showing visual media. Furthermore, videoconferencing technology is increasingly used for non-mediated shared experiences, where participants share their activities and environment using videoconferencing, for others to see and join. For example, in social television experiences, participants will share their experience of watching a television content item, enabling others to see their room and their television screen. As another example, users can also deliberately record their television screen in order to comment on what is being displayed and share the resulting recording with other users.
As such, camera recordings nowadays frequently include screens displaying visual content. A clear disadvantage, however, is that in such a camera recording, the visual content displayed on the screen is typically represented poorly in the recording; other parts of the scene typically look better, or even much better.
There may be a variety of reasons for this, including but not limited to:
To improve the quality of the visual content in the camera recording, one could opt to increase the quality of the camera recording, e.g., by increasing the recording resolution, framerate and/or video quality. Disadvantageously, this may lead to a larger size camera recording. This may be undesirable or impossible due to bandwidth or storage constraints, and may not be possible when using generally available current-day recording devices such as smartphones or tablets, which do not contain such high quality camera functions. Moreover, even when feasible, an increase in recording quality does not address problems such as dynamic range problems, etc.
It would be advantageous to obtain a system or method for enhancing a media recording which comprises a camera recording of a scene, with the scene including a screen displaying visual content, to obtain an enhanced media recording.
The following aspects of the invention involving replacing, in the camera recording, the visual content shown on the screen with a version which is originally recorded or generated. As such, a ‘digital-to-light-to-digital’ conversion step may be avoided, being at least one reason for the visual content having a poor quality in the camera recording. Namely, in the camera recording, the visual content is shown after having been converted, by way of being displayed, from the digital domain to the light domain and then, by way of the camera recording, back into the digital domain.
In accordance with a first aspect of the invention, a method may be provided for enhancing a media recording, which may comprise:
In accordance with another aspect of the invention, a computer program may be provided for causing a processor system to perform the method.
In accordance with another aspect of the invention, a system may be provided for enhancing a media recording, which may comprise:
In accordance with other aspects of the invention, a sender device and a receiver device may be provided for use in the system.
The above measures involve accessing a media recording which comprises at least a camera recording of a scene. For example, a media stream may be accessed, representing an encoded version of a media recording. Another example is that a still image made by a camera may be accessed. The camera recording is of a scene which includes a screen displaying visual content. As such, the camera recording may at least intermittently show the screen displaying the visual content, or part thereof, e.g., if the screen is only partially included in the recording frame of the camera recording, or if part of the screen is covered by another object in the scene.
The camera recording may be analysed to determine a location of the screen in the camera recording. The location may be expressed as coordinates. For example, in case of a rectangular screen, the coordinates may represent one or more corners of the screen. The coordinates may take any suitable form, such as image grid coordinates (column number, row number) or normalized image coordinates.
An original version of the visual content may then be accessed. Here, the term “original version” refers to a version which is not obtained by the indirection of a camera recording of a screen displaying the visual content. Rather, an original version represents a version which is originally recorded or generated. A non-limiting example is that, if the visual content shown on the screen is obtained by play-out of a media stream, the same media stream is accessed. Another example is that a television may show a specific television channel, and a TV signal containing that same television channel, or a recorded version of the television channel, may be accessed as an original version of the visual content. Yet another example is that if the visual content shown on the screen represents a slide from a presentation, a computer file of the presentation is accessed. Compared to the camera recording of the visual content, the original version of the content may be of a higher quality in that one or more of the reasons for the visual content having a poor quality in the media recording, as enumerated in the background section, may be avoided. In particular, the original version may avoid the ‘digital-to-light-to-digital’ conversion step of the visual content having been converted, by way of being displayed, from the digital domain to the light domain and then, by way of the camera recording, back into the digital domain.
The visual content displayed on the screen may then be replaced in the camera recording with the original version of the visual content. For that purpose, use may be made of the coordinates of the screen. For example, the original version of the visual content may be overlaid over the screen in the camera recording, thereby replacing the recorded version of the visual content in the camera recording. Since the original version of the visual content may be better in quality than the visual content shown in the camera recording, an enhanced media recording may be obtained.
The inventors have recognized that, with the ongoing increase in digitalization, when a camera recording is obtained of a screen displaying visual content, that an original version of the visual content is normally available in digital form and may be accessed. Such an original version may be used to replace the visual content as shown on the screen in the camera recording. By replacing such camera-recorded visual content with the original version of the visual content, the quality of the visual content may be improved. A further advantage is that it may not be needed to otherwise increase the quality of the camera recording so as to better capture the visual content shown on the screen. Yet a further advantage of replacing the visual content in the camera recording is that it may not be needed to display the original version in a separate window, e.g., as an inserted picture-in-picture or side-by-side with the camera recording, which may otherwise affect the composition of the scene. For example, if the camera recording shows a presenter pointing at the visual content, such pointing is preserved and would otherwise be lost if the visual content were to be separately shown. Yet another advantage may be that one or more, or even all, of the problems associated with recording a screen, as enumerated in the background section, may be avoided.
In an embodiment, accessing the original version of the visual content may comprise:
Although several possibilities exist for accessing the original version of the visual content, it may at times be needed or desired to identify the visual content displayed on the screen in order to access the original version of the visual content. For example, if there are multiple media streams available at a resource location, with each representing different visual content, the appropriate media stream may be retrieved after the visual content displayed on the screen has been identified. Accordingly, the visual content may first be identified, and based thereon, a resource location may identified which comprises the original version of the visual content. Here, the term ‘resource’ may refer to a server, storage medium, broadcast channel, etc., whereas the ‘resource location’ may represent information which allows the resource to be accessed, such as an internet address, for example an Universal Resource Locator (URL) address.
In an embodiment, identifying the visual content displayed on the screen may comprise:
The visual content may be identified by applying an automatic content recognition technique to the media recording. Such automatic content recognition is known per se. An advantage of using automatic content recognition may be that it may not be needed to obtain further information from the recording location, such as play-out information from a media device playing-out the visual content on the screen, to identify the visual content. Effectively, no additional information may be needed from such a media device. It is noted that the automatic content recognition may still involve information exchange with other entities, such as a content recognition database.
In an embodiment, the automatic content recognition technique may comprise determining at least one of: an audio watermark, a video watermark, or a fingerprint, of the content data. The automatic content recognition technique, e.g., when using a video watermark, may be applied only on the area of the screen as shown in the camera recording, for example using the coordinates of the screen. Any suitable automatic content recognition technique may be used as known per se from the field of automatic content recognition, including those based on watermarking and/or finger printing. It is noted that the content recognition may take additional or other information into account besides visual data. For example, the visual content may be associated with audio content which may be identifiable by making use of an audio watermark embedded in the audio content.
In an embodiment, the visual content displayed on the screen may represent a play-out by a media device, and identifying the visual content displayed on the screen may comprise obtaining play-out information from the media device which is indicative of said visual content. The visual content displayed on the screen may represent a play-out by a media device, such as a connected media player. As such, said visual content may be identified with the aid of the media device. In particular, play-out information may be used which is generated by the media device and which is indicative of the visual content. For example, the play-out information may identify a media stream including the resource location at which the media stream is available. Another example is that the play-out information may identify a program title.
In an embodiment, obtaining the play-out information may comprise:
With the ubiquity of connected media devices, it has become possible to obtain the play-out information from such a media device via a (local) network. For example, the media device may broadcast or otherwise send their current activity, e.g., using multicast DNS, DLNA, DIAL or other media protocols. The media device may be queried for the play-out information, e.g., using the same or similar protocols.
In an embodiment, the replacing of the visual content in the camera recording of the scene may comprise adjusting one or more visual properties of the original version of the visual content. The original version of the visual content may have an appearance which differs from the visual content in the camera recording of the scene, and in general may mismatch the appearance of the overall camera recording. As such, one or more visual properties of the original version of the visual content may be adjusted prior to, or when inserting it into the camera recording. This may provide a more pleasing, natural experience to a viewer of the media recording.
In an embodiment, the one or more visual properties may include one or more of: contrast, brightness, white balance, dynamic range, frame rate, spatial resolution, geometry, focus, 3D angle, 3D depth. The geometry of the visual content in the camera recording of the scene may be non-rectangular, e.g., due to camera distortions, the camera being misaligned with respect to the screen (e.g., not recording the screen directly face-on), etc. As such, the geometry of the original version of the visual content may be adjusted prior to, or when inserting it into the camera recording. Similarly, other visual properties may be adjusted to better match the appearance of the overall camera recording. In case the camera recording is a three-dimensional (3D) recording, also 3D parameters such as 3D angle or 3D depth may be adjusted.
In an embodiment, the media recording may be obtained by a sender device for transmission to a receiver device, the replacing of the visual content in the camera recording of the scene may be performed by the receiver device, and the method may further comprise:
Rather than being performed by a single device, the method may also be performed using several devices, such as those of a sender/receiver system in which the media recording may be obtained by a sender device for transmission to a receiver device, with the receiver device then replacing the visual content in the camera recording of the scene with the original version of the visual content. An example of such a system is a videoconferencing system. In this particular example, each videoconferencing client may act both as a sender device for the transmission of a locally recorded media stream, and as a receiver device for the reception of remotely recorded media stream(s). However, there may also be a unilateral transmission of a media recording from a sender device to a receiver device. In general, several possibilities exist for the receiver device being enabled to retrieve the original version of the visual content from the resource location. For example, the sender device may retrieve and subsequently transmit the original version of the visual content to the receiver device, or may transmit metadata to the receiver device which is indicative of a resource location from which the original version of the visual content is accessible. In general, the receiver device may be a play-out device for playing out the enhanced media recording. However, the receiver device may also be an intermediate device further transmitting the enhanced media recording to one or more play-out devices.
In an embodiment, the sender device may comprise:
In an embodiment, the receiver device may comprise:
In an embodiment, the method may further comprise the sender device including in the metadata the coordinates of the screen in the camera recording. As such, it may not be needed anymore for the receiver device to determine coordinates of the screen in the camera recording, as such coordinates may have been determined and made available by the sender device. Metadata to this effect may be provided.
In an embodiment, the receiver device may, in addition to the second input interface and the replacement subsystem, further comprise:
As such, the receiver device may carry out all claimed operations. For example, the receiver device may use an automated content recognition technique to identify the visual content that is to be replaced, retrieve an original version of the visual content, and insert this original version into the camera recording.
It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and/or aspects of the invention may be combined in any way deemed useful.
Modifications and variations of the method and/or the computer program product, which correspond to the described modifications and variations of the system, can be carried out by a person skilled in the art on the basis of the present description.
These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. In the drawings,
It should be noted that items which have the same reference numbers in different Figures, have the same structural features and the same functions, or are the same signals. Where the function and/or structure of such an item has been explained, there is no necessity for repeated explanation thereof in the detailed description.
The following list of reference numbers is provided for facilitating the interpretation of the drawings and shall not be construed as limiting the claims.
The following embodiments of a system and method involving replacing, in the camera recording, the visual content shown on the screen with a version which is originally recorded or generated. As such, an (much) improved quality of the visual content in the camera recording may be obtained. A general explanation is provided with reference to
It will be appreciated that a method according to the invention may be implemented in the form of a computer program which comprises instructions for causing a processor system to perform the method. The method may also be implemented in dedicated hardware, or as a combination of the above.
The computer program may be stored in a non-transitory manner on a computer readable medium. Said non-transitory storing may comprise providing a series of machine readable physical marks and/or a series of elements having different electrical, e.g., magnetic, or optical properties or values.
It is noted that the camera recording may be part of an overall media recording, which may comprise additional components, such as, e.g., subtitle overlays, additional audio tracks, various metadata, etc. However, the media recording may also solely consist of the camera recording. As such, both terms may be used interchangeably where appropriate. It is further noted that the camera recording may be video, but may equally comprise, or be comprised of, one or more still images.
The system 100 is shown to comprise a first input interface 110 for accessing the media recording 030. The first input interface 110 may take any suitable form, such as network interface to a local or wide area network, a storage interface to an internal or external data storage, etc. The media recording 030 may be pre-recorded, but may also be a real-time, ‘live’ stream. As also shown in
The system 100 is further shown to comprise an analysis subsystem 120 for analysing the camera recording. Such analysis may involve determining coordinates of the screen in the camera recording. However, as will be elucidated in following paragraphs, the analysis subsystem 120 may also have other, e.g., additional, functionality. The coordinates may be determined through image analysis techniques, as known per se from the field of image analysis. Examples of such techniques are described in following paragraphs with reference to the tracking of screens.
The system 100 is further shown to comprise a second input interface 130 for accessing an original version of the visual content. Like the first input interface 110, the second input interface 130 may be of any suitable type, such as a network interface to a local or wide area network, a storage interface to an internal or external data storage, etc. Said original version 060 may be pre-recorded, but may also be a real-time, ‘live’ stream. As also shown in
The system 100 is further shown to comprise a replacement subsystem 140 for replacing, in the camera recording 032 and using the coordinates of the screen, the visual content displayed on the screen by the original version 060 of the visual content, thereby obtaining an enhanced camera recording 042 and thus an enhanced media recording 040. For that purpose, the replacement subsystem is shown to receive the original version 060 of the visual content from the second input interface 130 and the media recording 030 from the first input interface 110. However, as will be shown with reference to
General Aspects
In general, embodiments of the system and method may comprise:
When relating to the analysis of the camera recording, such functions may be performed by the analysis subsystem, and otherwise by the replacement subsystem. For example, the analysis subsystem may detect a media device which is assumed to be rendering the visual content on the screen. It is noted that in some cases, the screen may comprise the media device or vice versa, such as in the case of a television having integrated media player functionality. However, in other cases, the media device may be directly or indirectly connected to the screen. Examples of media devices include, but are not limited to, televisions, monitors, projectors, media players and recorders, set-top boxes, smartphones, cameras, PCs, laptops, tablet devices, smart watches, smart glasses, professional video equipment, etc.
Detecting the Media Device
Detecting the media device playing-out the visual content may comprise one or more of:
Identifying the Visual Content
Identifying the visual content played-out by the media device may comprise one or more of:
It is noted that the visual content may be described as metadata, for instance using a Television Domain Name System (TV-DNS) system (http://www.w3.org/TR/TVWeb-URI-Requirements, http://tools.ietf.org/html/rfc2838), and may thus be announced, signalled or stored in the form of such metadata.
In cases the camera recording is a video recording rather than, e.g., a still image, the analysis subsystem may track the screen in the video recording, or may track the media device in the video recording, e.g., in case the screen is comprised in the media device. Here, the term tracking may refer to one or more coordinates of the screen being identified over time, e.g., in different image frames. Such tracking may enable the spatially accurate replacement of the visual content shown on the screen. Namely, the camera and the screen may mutually move over time, causing the screen to be located at different image coordinates. To track the screen, image and/or object tracking techniques may be used, as are well known in the art and widely available. For example, the CDVS standard, ISO/IEC FDIS 15938-13 (the most recent published version as of the time of invention), provides the means of extracting visual features from images (key points and their coordinates) and compressing them in a compact bit-stream. The tracking data may be stored as associated metadata to the recording. The metadata may also contain device motion information, timing information (e.g. for synchronization purposes), occlusion information. The annotations pertaining to the video may be expressed using the MPEG-7 standard ISO/IEC 15938-3 which allows spatio-temporal annotations. For example, this standards allows to express the coordinates of a region, e.g. an object, over multiple frames, i.e. from time t1 to time t2 of the video, which may be used for tracking the screen in the video recording.
Accessing an Original Version of the Visual Content
Accessing an original version of the visual content may involve the media device itself providing said original version of the visual content, for example, by streaming a media stream in the form of an MPEG-DASH stream. Alternatively or additionally, a resource location may be identified which comprises said original version. For example, metadata made available to the replacement subsystem may contain a brief identification of the TV Channel which is being played-out on the screen in the camera recording, e.g., the identifier “BBC 1”. The replacement subsystem may then identify and access the channel “BBC 1”, e.g., via an Internet Protocol Television (IPTV) service, from which a media stream of the visual content may be accessed.
Replacing the Visual Content
Having obtained access to the original version of the visual content, the visual content displayed on the screen may be replaced by the original version of the visual content, thereby obtaining an enhanced media recording. Such replacement may be, but does not need to be, performed in real-time and in a synchronized manner, so that the visual content in the enhanced media recording is synchronized, to at least a certain degree, with the visual content previously shown in the media recording. Said synchronization aspects will be further elucidated with reference to ‘time alignment’.
The replacing of the visual content displayed on the screen by the original version of the visual content may be performed in a number of ways. For example, the replacement subsystem may overlay or otherwise insert the original version of the visual content into the camera recording. It is noted that such replacing may not need to be pixel accurate, nor does it need to fully replace the visual content displayed on the screen. For example, the original version of the visual content may be alpha-blended into the camera recording, with a residual of the camera-recorded visual content (e.g., a 1-alpha weighted residual) thus remaining in the camera recording.
It is noted that if the visual content is obtained from play-out of a particular version of the visual content, e.g., a particular media stream, the replacement is not limited to the replacement by the particular version being played-out, but may rather involve a different version. For example, the replacement may be by a processed version having been sampled-down or having a lower bitrate. Such a processed version may not affect the perceived quality or may even enhance the perceived quality, as will be further elucidated with reference to ‘Video conferencing aspects’.
The replacement may be performed at various stages. For example, the replacement may be already performed in the recording device itself, such that an encoded version of the media recording contains the original version. Another way is to have a receiver device access both the media recording and the original version of the visual content, and insert the original version into the media recording. This aspect will be further elucidated in the following paragraphs. The replacement may also be performed during play-out of the media recording. As such, the enhanced media recording may not be separately stored but rather may be generated ‘on the fly’.
System Partitioning
It will be appreciated that the analysis subsystem and the replacement subsystem may be part of a single device. However, both subsystems may also be part of different devices, or may be implemented in a distributed manner. A non-limiting example is that of a sender/receiver system in which, at a sender side, the media recording may be obtained by a sender device for transmission to a receiver device, with, at a receiver side, the receiver device then replacing the visual content in the camera recording of the scene with the original version of the visual content. Here, the sender device may comprise the first input interface and the analysis subsystem, and the receiver device may comprise the second input interface and the replacement subsystem. A non-limiting example of such a system is a videoconferencing system.
Generally speaking, the sender device, the screen and the recording device may be co-located, e.g., in a same room, same building, same outside area. However, this is not a requirement, in that the sender device 300 may be located at the sender side, e.g., at a ‘sending’ location, whereas the screen may be located and recorded by the recording device elsewhere, e.g., at a third location, i.e., a ‘recording’ location.
The sender device 300, and particularly its analysis subsystem, may analyse the camera recording comprised in, or represented by the media recording 030, to determine coordinates of the screen in the camera recording. For that purpose, the earlier described tracking techniques may be used. The sender device 300 may then format and make available these coordinates as metadata 052. Specific examples of such metadata will be given in following paragraphs. As part of the metadata 052, the sender device 300 may include the resource location information 064.
Discovery
Actively polling the network can be based on various protocols. One example is the UPnP protocol. Here, M-SEARCH is used to first discover devices in the local network, either directly or through a UPnP server. An example of a discovery message is shown below. This is a general discovery message for discovering all UPnP devices. Instead of searching for all devices with ssdp:all, also discovery messages can be sent for specific devices, e.g., for media renderers. A display device, e.g. a television, in UPnP would typically be a media renderer.
An M-SEARCH is multicasted on the local network, specifying what is looked for, in this case all devices. In
M-SEARCH*HTTP/1.1
HOST: 239.255.255.250:1900
MAN: “ssdp:discover”
MX: 2 (seconds to delay response)
ST: ssdp:all (search for all devices)
USER-AGENT: Android/4.3 UPnP/1.1 Smartphone/3.0 (example values)
The response may be a 200 OK message containing information on the device that responds, in this case the media device 012 being a MediaRenderer.
HTTP/1.1 200 OK
CACHE-CONTROL: max-age=1800
DATE: Sun, 22 Mar. 2015 08:49:37 GMT
EXT:
LOCATION: http://192.168.1.5/description
SERVER: android/4.3 UPnP/1.1 television/1.0
ST: ssdp:all
USN: uuid:2fac1234-31f8-11b4-a222-08002b34c003:: urn:schemas-upnp-org:service:MediaRenderer:1
BOOTID.UPNP.ORG: 1426860725
CONFIGID.UPNP.ORG: 123456
SEARCHPORT.UPNP.ORG: 49152
Alternatively or additionally, as shown in
NOTIFY*HTTP/1.1
HOST: 239.255.255.250:1900
CACHE-CONTROL: max-age=1800
LOCATION: http://192.168.1.5/description
NT: urn:schemas-upnp-org:service:MediaRenderer:1
NTS: ssdp:alive
SERVER: android/4.3 UPnP/1.1 television/1.0
USN: uuid:2fac1234-31f8-11b4-a222-08002b34c003:: urn:schemas-upnp-org:service:MediaRenderer:1
BOOTID.UPNP.ORG: 1426860725
CONFIGID.UPNP.ORG: 123456
SEARCHPORT.UPNP.ORG: 49152
Note that the examples of
Signalling Screen Coordinates
With further reference to the analysis subsystem detecting the coordinates of the screen in the camera recording, these coordinates may be signalled to others, such as the replacement subsystem. This signalling may involve the analysis subsystem formatting and making available the coordinates in the form of metadata. Such metadata may be generated by encoding the detected screen in X and Y coordinates. However, even though a screen is usually rectangular, a screen may be also recorded at an angle. In such a case, the coordinates may represent all four corners of the screen. Also, the information about the visual content may be detected and signalled to others. Below is an example of such metadata in XML.
It is noted that the above XML-based metadata is shown to indicate the coordinates of a rectangular screen. For other types of screens, more or less metadata may need to be supplied. For example, smartwatches may have round displays which may appear oval when captured from an angle. In such a case, a coordinate for the center may be detected and signalled, as well as parameters describing the circle or oval. For curved screens, the top and bottom of the screen may not be straight lines. As such, in addition to coordinates of the corners, parameters may be detected and signalled describing the curvature. For holographic projections or light field displays, 3D coordinates may be used to describe the area where the 3D images are displayed. The screen may also be partially occluded in the camera recording, or only be partially shown in the field of view of the recording device. As such, the coordinates may also describe a polygon representing the non-occluded, visible part of the screen.
It is noted that for formatting and making available the coordinates in the form of metadata, the ISO/IEC standard 23001-10 titled ‘Carriage of Timed Metadata Metrics of Media in ISO Base Media File Format’ may be used. Although at the time of writing this standard only contains timed metadata relative to MPEG Green standard (see ISO/IEC 23001-11) and visual quality metrics such as PSNR, MPEG have started the process to amend 23001-10 to add the carrying of 2D coordinates as well.
Time Alignment
When replacing the visual content displayed on the screen by the original version of the visual content, the replacement may use the detected coordinates of the screen as the place to insert the original version. But, such replacing may also have a temporal aspect, as videos change over time. Accordingly, the insertion of the original version may be synchronized with the displayed visual content in the camera recording, in that exactly same content may be shown after replacement as before. This may involve identifying a playout point in the camera recording, and identify this same playout point in the original version, and use this during replacement. For that purpose, any known technique from media synchronization may be used, including buffering and seeking ahead in a video. It is noted that in some cases, for example where a presenter interacts with the visual content shown on the screen, it may be desired to synchronize the original version to a relatively high degree with the camera recording, e.g., having a remaining difference in the magnitude of tens or hundreds of millisecond. However, in many cases, the exact timing is of lesser importance, and the insertion of the original version may shift somewhat in time compared to the displayed visual content in the recording. As an example, the screen may show a TV channel, e.g., channel ‘NPO1’. If this TV channel is accessed for replacement, the currently available play-out may be used. This may be different in play-out timing from the displayed visual content in the camera recording, as TV channels' play-out may vary at various locations, depending on TV provider, distribution technology used, transcoding during distribution, etc. Such differences are usually in the order of magnitude of several seconds, and may be as large as a minute. As such, the enhanced version of the media recording may differ somewhat in timing of the visual content shown on the screen in the scene.
Adjustment of Visual Properties
With further reference to the adjustment of one or more visual properties of the original version 060 of the visual content, as earlier described with reference to
Efficiently Encoding the Media Recording
The visual content shown on screen in the media recording is to be replaced by an original version of the visual content. As such, when encoding the media recording before said replacement occurs, e.g., for transmission or storage, the media recording may be encoded in an optimized manner to obtain a higher coding efficiency. The following describes two possible actions, which may also be combined.
A first action is the pre-processing of the media recording, which may involve making the area representing the displayed visual content easy to encode. This way, the area will account for fewer bits in the encoded bit stream. A possible way of doing so is to substitute all pixel values in this area of the captured video frames by a same pixel value, e.g., ‘zero’ or black. Namely, uniform areas are efficiently encodeable for encoders when leveraging intra prediction or block matching mechanisms.
A second action is a so-termed region-of-non-interest coding. Numerous encoders, regardless of the video coding standard, offer the possibility to define regions in the video frame for which more or less quality (more or less bits) should be allocated. Within the present context, it may be beneficial to assign a poor quality to the area representing the displayed visual content. Generally, the quality of this region is tuned via the Quantization Parameter (QP). The higher the QP, the lower the quality of the encoded stream. By locally applying higher QPs to this region, one can achieve this region of ‘non-interest’ coding of the visual content displayed on screen.
A third action may constitute an alternative to the second action, which may require a modified encoder. Namely, one may consider not encoding what it is not needed. In this case, the coordinate of the region to discard, i.e., the area representing the displayed visual content, may be used directly by the encoder to leave them out when encoding the video stream. Effectively, the output bit stream may then contain frames with “holes”. Such discarding of regions may be involve use of High Efficiency Video Coding (HEVC) tiles. For example, assuming there is only one screen shown in the camera recording, the recording device may define a tiling grid for the HEVC encoder in such a way that the tiles representing the screen may be discarded during the encoding process. The tiling grid might adjusted dynamically based on the position of the screen. Alternatively, the tiling grid might be static and the tiles that contain only the pixels from the visual content displayed on screen may be discarded.
Video Conferencing Aspects
It is noted that, in a video conferencing scenario, it may not be needed to use the same stream that user A sees for user B; if the screen presenting the recording for user B is small, or is provided in a low resolution, it might suffice to retrieve a low bitrate version of the visual content to be displayed in the view of user B. Here and in the following, a reference to user A is understood to be a reference to his/her sender device, and a reference to user B is understood to be a reference to his/her receiver device. Example: User A watches a full HD TV channel (1920×1080 pixels) on his/her large-screen TV, involving a bit rate of 10 Mbit/s. User B only sees a scaled down version of the TV of user A in his recorded view so a lower resolution version (SD) may suffice to get an acceptable result. It is noted that this may also in general apply to the visual content being played-out from a media stream, in that it may not be needed to retrieve the same media stream in order to replace the visual content shown in the screen in the camera recording. Rather, a different, e.g., lower bitrate version may be retrieved. Still a higher quality may be obtained, e.g., by avoiding the digital-to-light-to-digital conversion step. With further reference to a video conferencing scenario, user A and user B may access the same media stream by said media stream being efficiently distributed amongst them, e.g., by distribution via multicast or peer-to-peer (P2P). The system may also detect or resolve that the resource user A is watching is also available for user B, but via a different route. Example: User A is watching the TV channel ‘NPO1’ via a subscription of TV provider A; the system may then detect that user B can access a media stream of said TV channel via a subscription of IPTV provider B, so that it is not needed to transfer the media stream from user A to user B.
Other General Aspects
It is noted that if camera recording shows a screen from a PC, tablet or smartphone or other type of computing device, the screen capture functionality of said computing device may be used as a media source for the original version of the visual content, in that screen capture(s) may be accessed and used in replacing the visual content displayed on the screen in the camera recording.
It is noted that the analysis subsystem and/or the replacement subsystem may be embodied as, or in, a single device or apparatus, such as the recording device or another user device. The device or apparatus may comprise one or more microprocessors which execute appropriate software. The software may have been downloaded and/or stored in a corresponding memory, e.g., a volatile memory such as RAM or a non-volatile memory such as Flash. Alternatively, the analysis subsystem and/or the replacement subsystem may be implemented in the device or apparatus in the form of programmable logic, e.g., as a Field-Programmable Gate Array (FPGA). In general, each functional unit of the system may be implemented in the form of a circuit. It is noted that the analysis subsystem and/or the replacement subsystem may also be implemented in a distributed manner, e.g., involving different devices or apparatuses. For example, the analysis subsystem and/or the replacement subsystem may be implemented as a software-based function being performed by entities within a media distribution network, such as servers.
In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
| Number | Date | Country | Kind |
|---|---|---|---|
| 15165075.1 | Apr 2015 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2016/059031 | 4/22/2016 | WO | 00 |
