Patent Application
20140044197
The streaming of multimedia over networks continues to grow at a tremendous rate. In some aspects, the continued growth of multimedia streaming may be attributed to its increasing presence and/or importance in new media and entertainment applications, as well as gains in its use in educational, business, travel, and other contexts. In some instances, the networks used for streaming multimedia may be wired or wireless and may include the Internet, television broadcast, satellite, cellular, and WiFi networks. Important to a video experience is the quality of video received for viewing by a user. In some aspects, increasing service capacity and enhancing end-user quality of experience (QoE) may be facilitated by different optimization techniques.
A number of adaptive video streaming techniques have been proposed in an effort to increase service capacity and enhance end-user QoE. Some such techniques address streaming capacity and quality problems by encoding a video source into short segments at different pre-determined bitrates. The encoded short segments of video are then delivered over a network based on the available network bandwidth and processing conditions.
While techniques considering available network bandwidth and processing conditions may or may not address some broad video quality issues to an extent, such techniques are not typically adaptive to, responsive to, or even aware of the variety of the types of video transmitted.
Aspects of the present disclosure herein are illustrated by way of example and not by way of limitation in the accompanying figures. For purposes related to simplicity and clarity of illustration rather than limitation, aspects illustrated in the figures are not necessarily drawn to scale. Further, where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements.
The following description describes a method and system that may support processes and operations to improve a quality and an efficiency of a video transmission by providing a content-aware video adaption technique. As will be explained in greater detail below, the present disclosure herein provides some embodiments of a technique or mechanism that adaptively selects coding parameters and allocates resources based on the content of a video sequence being encoded for transmission over a network. The technique(s) disclosed herein may, in some embodiments, operate to minimize bitrate consumption and/or improve the quality of the encoded video transmitted over the network.
In some regards, the present disclosure includes specific details regarding method(s) and system(s) for implementing the processes and systems herein. However, it will be appreciated by one skilled in the art(s) related hereto that embodiments of the present disclosure may be practiced without such specific details. Thus, in some instances aspects such as control mechanisms and full software instruction sequences have not been shown in detail in order not to obscure other aspects of the present disclosure. Those of ordinary skill in the art will be able to implement appropriate functionality without undue experimentation given the included descriptions herein.
References in the present disclosure to “one embodiment”, “some embodiments”, “an embodiment”, “an example embodiment”, “an instance”, “some instances” indicate that the embodiment described may include a particular feature, structure, or characteristic, but that every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
Some embodiments herein may be implemented in hardware, firmware, software, or any combinations thereof. Embodiments may also be implemented as executable instructions stored on a machine-readable medium that may be read and executed by one or more processors. A machine-readable storage medium may include any tangible non-transitory mechanism for storing information in a form readable by a machine (e.g., a computing device). In some aspects, a machine-readable storage medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; and electrical and optical forms of signals. While firmware, software, routines, and instructions may be described herein as performing certain actions, it should be appreciated that such descriptions are merely for convenience and that such actions are in fact result from computing devices, processors, controllers, and other devices executing the firmware, software, routines, and instructions.
Accordingly, graph 100 demonstrates that a video encoding and transmission method that uses fixed (en)coding parameter(s) for all video content may result in either a waste of bandwidth or a degradation in video quality.
At operation 205, incoming video content may be classified into a variety of video content categories. The video received at operation 205 may come from any source, including live feeds and being retrieved from a storage location. The video received at operation 205 may be classified based on one or more characteristics of the video itself (i.e., the content of the video). In some embodiments, a motion intensity characteristic of the received video may be evaluated and the video may be categorized into one of three categories—low motion, intermediate motion, or high motion.
At operation 210, one or more video coding profiles may be adaptively generated for the video content based on, at least, the plurality of video content categories determined at operation 205. As illustrated in
The coding profiles adaptively generated at operation 210 based at least on the determined plurality of video content categories may be stored or output in a record or file, used as an input for further processing and transmission of the video content, and for other processes.
Referring to
As further illustrated in
Video content is provided by or received from video source 405. Video source 405 may be any type of mechanism for providing the video content, including a live or re-broadcast data stream and a file or record including a video sequence retrieved from a storage facility (i.e., memory). The video content from video source 405 is fed to a video content analyzer 410. Video content analyzer 410 may operate to analyze the content characteristics of the video from video source 405. In some embodiments, video content analyzer 410 may include video feature extraction mechanisms or techniques to identify different characteristics of the content of the video. Video content analyzer 410 may further classify the video content into different categories based on the categorized video content (e.g., operations 205 and 305).
An indication of the different video categories associated with the video content analyzed by video content analyzer 410 is provided to a content-aware coding profile generator 415. Content-aware coding profile generator 415 may gather information from multiple sources to adaptively generate optimized coding profiles for different types of video content. In some embodiments, the different types of video content corresponds to the different categories of the video content. In some aspects, the input information to content-aware coding profile generator 415 may include, at least, the video content categories from video content analyzer 410. Additional input information to content-aware coding profile generator 415 may include, for example, video quality scores calculated at the server 400 by a video quality assessment tool 430 and network condition and other user requirement feedback 420.
Coding profile generator 415 may operate to generate one or more content-optimized coding profiles by adaptively selecting a target bitrate, an encoding resolution, an encoding frame rate, a rate control algorithm, a frame structure, a group of picture (GOP) size, a number of a specific type of frame (e.g., bi-directional of “B” frames), and other coding parameters, alone and in combinations thereof. It will be appreciated that the present disclosure encompasses these and other coding parameters, whether specifically enumerated herein.
Coding profile generator 415 may provide the one or more content-optimized coding profiles generated thereby to a multimedia streaming codec 425. Codec 425 may use the content-optimized coding profiles to encode the video content from video source 405 with the appropriate coding profiles generated by video coding profile generator 415. The appropriate coding profile(s) may optimally match the type of content in the video.
The encoded video output by codec 425 is provided, in part, to video quality assessment (VQA) tool 430. VQA tool 430 may calculate video quality or VQA score(s) for the encoded video. The VQA score(s) may be passed to content-aware coding profile generator 415. Upon receipt of the VQA scores, content-aware coding profile generator 415 may recursively adjust the coding parameters used therein and generate optimized coding profiles based on, at least, the video content and the VQA scores.
In some embodiments, reference-based VQA metrics such as MS-SSIM may be used since the video source is available at the server side.
Applicant has realized the effectiveness of the processes disclosed herein by determining a bitrate minimization using the content-aware video adaption processes disclosed herein and comparing them to baseline coding schemes that use a fixed coding profile for all video sequences. The video sequences used in the evaluation and the following tables include the publically available “Aspen”, “ControlledBurn”, “RedKayak”, “SpeedBag”, “TouchdownPass”, and “WestWindEasy” video sequences under different bitrates.
Table 1 below shows the gains observed for the content-aware video adaptation method(s) herein compared to baseline schemes in which a fixed coding profile is applied to all of the input video sequences. In the example of Table 1, it is assumed that users are satisfied when an average PSNR (Peak Signal to Noise Ratio) that is greater than 34 dB. The baseline schemes relating to Table 1 use fixed quantization parameters (QPs) to encode the video sequences while the content-aware (i.e., optimized) method adaptively selects the coding parameters based on the different types of video content characteristics detected in the input video sequence. As seen, the results listed in the Table 1 show that in order to satisfy users for all video sequences, an average bitrate saving of 3.55 Mbps is achieved using the content-aware video adaptation process disclosed herein.
Table 2 below provides, as an example, a listing of the coding parameter settings for each video sequence of Table 1.
Processor 605 communicates with a storage device 630. Storage device 630 may comprise any appropriate information storage device, including combinations of magnetic storage devices (e.g., a hard disk drive), optical storage devices, and/or semiconductor of solid state memory devices. In some embodiments, storage device may comprise a database system.
Storage device 630 stores a program code 635 that may provide computer executable instructions for processing requests from, for example, client devices in accordance with processes herein. Processor 605 may perform the instructions of the program 635 to thereby operate in accordance with any of the embodiments described herein. Program code 635 may be stored in a compressed, uncompiled and/or encrypted format. Program code 635 may furthermore include other program elements, such as an operating system, a database management system, and/or device drivers used by the processor 605 to interface with, for example, peripheral devices. Storage device 630 may also include data 645 such as a video sequence and/or user preferences or settings. Data 645, in conjunction with context-aware coding profile generator 640, may be used by system 600, in some aspects, in performing the processes herein, such as processes 200 and 300.
All systems and processes discussed herein may be embodied in program code stored on one or more computer-readable media. Such media may include, for example, a floppy disk, a CD-ROM, a DVD-ROM, one or more types of “discs”, magnetic tape, a memory card, a flash drive, a solid state drive, and solid state Random Access Memory (RAM), Read Only Memory (ROM) storage units, and other non-transitory media. Furthermore, the systems and apparatuses disclosed or referenced herein may comprise hardware, software, and firmware, including general purpose, dedicated, and distributed computing devices, processors, processing cores, and microprocessors. In some aspects, the processes and methods disclosed herein may be delivered and provided as a service. Embodiments are therefore not limited to any specific combination of hardware and software.
Embodiments have been described herein solely for the purpose of illustration. Persons skilled in the art will recognize from this description that embodiments are not limited to those described, but may be practiced with modifications and alterations limited only by the spirit and scope of the appended claims.
