Patent Application
20200382826
This invention relates to systems and methods for enhancing background content in discriminatively encoded video.
For a given bit rate limit, the quality of compressed video largely depends on the dynamicity of the scene. In general, the more dynamic the scenes are, the poorer the quality that a user will see. In many categories of video content, there are often one or more points-of-interest (POIs) in a frame. For example, in a music concert video, the performers would likely be the major points-of-interest, while the stage background is likely of less interest. In a video of a sports game, the players are likely the major points-of-interest, while the background (e.g., the playing field and/or audience area) is of less interest. In such types of video content, a user's eye is typically drawn to the points-of-interest and less drawn to the background.
Nevertheless, current video encoders typically do not discriminate between point-of-interest content and background content (i.e., non-point-of-interest content) in videos. The entire frame is typically compressed in the same way, thereby degrading the quality of the point-of-interest content in the same way as the background content. This also inefficiently utilizes bandwidth when streaming videos over a network having a given bit rate limit. When background content has unnecessary spatial/temporal dynamicity, a conventional video encoder typically allocates more information to the background content to deal with the dynamicity. Similarly, even though point-of-interest content is more important to the user, the point-of-interest content may be allocated less information due to the complex background content. This behavior results in various visual artifacts around and/or within the points-of-interest, thereby degrading a user's viewing experience.
In view of the foregoing, what are needed are systems and methods to enable video encoders to discriminate between point-of-interest content and background content. This will ideally enable the video encoders to allocate more information to point-of-interest content and less information to background content. In such systems and methods, techniques are also needed to enhance the visual quality of background content that has been allocated less information.
The invention has been developed in response to the present state of the art and, in particular, in response to the problems and needs in the art that have not yet been fully solved by currently available systems and methods. Accordingly, systems and methods have been developed to enhance background content in discriminatively encoded video. The features and advantages of the invention will become more fully apparent from the following description and appended claims, or may be learned by practice of the invention as set forth hereinafter.
Consistent with the foregoing, a method for enhancing background content in discriminatively encoded video is disclosed. In one embodiment, such a method includes providing video content in the form of point-of-interest content and background content. The background content is of lower visual quality than the point-of-interest content. The method provides an enhancement model that may be applied to the background content to enhance the visual quality of the background content and thereby yield enhanced background content. The method further enables the enhanced background content to be combined with the point-of-interest content to yield combined content that may be viewed by a user.
A corresponding system and computer program product are also disclosed and claimed herein.
In order that the advantages of the invention will be readily understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, the embodiments of the invention will be described and explained with additional specificity and detail through use of the accompanying drawings, in which:
It will be readily understood that the components of the present invention, as generally described and illustrated in the Figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the invention, as represented in the Figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of certain examples of presently contemplated embodiments in accordance with the invention. The presently described embodiments will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout.
The present invention may be embodied as a system, method, and/or computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
The computer readable program instructions may execute entirely on a user's computer, partly on a user's computer, as a stand-alone software package, partly on a user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, a remote computer may be connected to a user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
Referring to
As shown, the computing system 100 includes at least one processor 102 and may include more than one processor 102. The processor 102 may be operably connected to a memory 104. The memory 104 may include one or more non-volatile storage devices such as hard drives 104a, solid state drives 104a, CD-ROM drives 104a, DVD-ROM drives 104a, tape drives 104a, or the like. The memory 104 may also include non-volatile memory such as a read-only memory 104b (e.g., ROM, EPROM, EEPROM, and/or Flash ROM) or volatile memory such as a random access memory 104c (RAM or operational memory). A bus 106, or plurality of buses 106, may interconnect the processor 102, memory devices 104, and other devices to enable data and/or instructions to pass therebetween.
To enable communication with external systems or devices, the computing system 100 may include one or more ports 108. Such ports 108 may be embodied as wired ports 108 (e.g., USB ports, serial ports, Firewire ports, SCSI ports, parallel ports, etc.) or wireless ports 108 (e.g., Bluetooth, IrDA, etc.). The ports 108 may enable communication with one or more input devices 110 (e.g., keyboards, mice, touchscreens, cameras, microphones, scanners, storage devices, etc.) and output devices 112 (e.g., displays, monitors, speakers, printers, storage devices, etc.). The ports 108 may also enable communication with other computing systems 100.
In certain embodiments, the computing system 100 includes a wired or wireless network adapter 114 to connect the computing system 100 to a network 116, such as a local area network (LAN), wide area network (WAN), storage area network (SAN), or the Internet. Such a network 116 may enable the computing system 100 to connect to or communicate with one or more servers 118, workstations 120, personal computers 120, mobile computing devices, storage devices, or other devices. The network 116 may also enable the computing system 100 to connect to or communicate with another network by way of a router 122 or other device 122. Such a router 122 may allow the computing system 100 to communicate with servers, workstations, personal computers, storage devices, or other devices located on different networks.
Referring to
Unfortunately, many current video encoders do not discriminate between point-of-interest content and background content (i.e., non-point-of-interest content). The entire frame is typically compressed in the same way, thereby degrading the quality of the point-of-interest content in the same way as the background content. This inefficiently utilizes bandwidth when streaming videos over a network that can support a given bit rate. When background content has unnecessary spatial/temporal dynamicity, a conventional video encoder typically allocates more information to the background content to deal with the dynamicity. Similarly, even though point-of-interest content is of greater importance to a user, the point-of-interest content may be allocated less information due to the complex background content. This treatment of the point-of-interest and background content may result in various visual artifacts around and within the points-of-interest, thereby degrading a user's viewing experience.
Thus, systems and methods are needed to enable video encoders to discriminate between point-of-interest content and background content. Such systems and methods will ideally allocate more information to point-of-interest content than to background content. In such systems and methods, techniques are also needed to enhance the visual quality of background content that has been allocated less information.
The method 200 then analyzes 204 frames 300 of the video content to identify the specified points-of-interest 302 therein. This analysis step 204 may include using one or more types of frame analysis. For example, in one approach, the frames 300 of the video data are analyzed by a type of object detector, as shown in
Once the points-of-interest 302 are identified in the frames 300 of the video content, the method 200 may adjust 206 quality parameters for portions (e.g., pixels) of the video content that contain the points-of-interest 302. This may include, for example, providing finer quantization values, an increased number of P-frames, and/or finer motion estimation for portions 304 of the video content that contain points-of-interest 302. In general, the step 206 may allocate more information to portions 304 of the video content that contain points-of-interest 302 in order to provide more visual quality to these portions 304.
On the other hand, the method 200 may counter-adjust 208 quality parameters of the portions 306 of the video content that do not contain points-of-interest (i.e., the background 306) in order to stay within a bit rate limit when transmitting the encoded video content over a network. This may include, for example, providing coarser quantization values, a decreased number of P-frames, and/or coarser motion estimation for portions of the video content that do not contain points-of-interest. In general, the step 208 may allocate less information to portions 306 of the video content that do not contain points-of-interest 302, thereby reducing its visual quality while enabling the video content as a whole to achieve a desired bit rate goal.
Once the method 200 has adjusted the quality parameters of both the points-of-interest 302 and background 306, the method 200 may output 210 a compressed video stream having the adjusted quality parameters. This video stream may include both the high quality points-of-interest as well as the reduced quality background. One embodiment of a system and method for discriminatively encoding video in the manner described in
Referring to
Other examples of enhancement models 502 that may be used to enhance the visual quality of background content 306 are disclosed in articles entitled “Accurate Image Super-Resolution Using Very Deep Convolutional Networks” published on Nov. 14, 2015 and authored by Kim et al., and “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network” published on Sep. 15, 2016 and authored by Ledig et al.
Referring to
The server 118 may also transmit, to the client device 100, an enhancement model 502 configured to enhance the visual quality of the background content 306. The enhancement model 502 may be, for example, an enhancement model 502 described in association with
The timing for transmitting the enhancement model 502 from the server 118 to the client device 100 may differ in different embodiments. In certain embodiments, the enhancement model 502 is transmitted to the client device 100 in advance of the discriminatively encoded video. This will allow the client device 100 to apply the enhancement model 502 to the background content 606 as it is streamed from the server 118. In other embodiments, video content is played on the client device 100 immediately upon reception with or without the enhancement model 502. The client device 100 may begin enhancing the background content 606 of the discriminatively encoded video once the enhancement model 502 is fully received. This may allow unenhanced background content 606 to be played on the client device 100 up until the time the enhancement model 502 is received and applied to the background content 606.
In yet other embodiments, the client device 100 may determine the dominance of point-of-interest content 604 in discriminatively encoded video as it is being played or decoded. The transmission rate of the enhancement model 502 may be adjusted based on the prevalence of point-of-interest content 604 in the discriminatively encoded video. As more point-of-interest content 604 is detected in the discriminatively encoded video, the transmission rate (e.g., download rate) of the enhancement model 502 may be increased so that it is received more expeditiously at the client device 100 and applied to the background content 606.
The flowcharts and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. Other implementations may not require all of the disclosed steps to achieve the desired functionality. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
