Patent Application
20090268821
The following description relates generally to digital video coding, and more particularly to techniques for motion estimation.
The evolution of computers and networking technologies from high-cost, low performance data processing systems to low cost, high-performance communication, problem solving, and entertainment systems has increased the need and desire for digitally storing and transmitting audio and video signals on computers or other electronic devices. For example, everyday computer users can play/record audio and video on personal computers. To facilitate this technology, audio/video signals can be encoded into one or more digital formats. Personal computers can be used to digitally encode signals from audio/video capture devices, such as video cameras, digital cameras, audio recorders, and the like. Additionally or alternatively, the devices themselves can encode the signals for storage on a digital medium. Digitally stored and encoded signals can be decoded for playback on the computer or other electronic device. Encoders/decoders can use a variety of formats to achieve digital archival, editing, and playback, including the Moving Picture Experts Group (MPEG) formats (MPEG-1, MPEG-2, MPEG-4, etc.), and the like.
Additionally, using these formats, the digital signals can be transmitted between devices over a computer network. For example, utilizing a computer and high-speed network, such as digital subscriber line (DSL), cable, T1/T3, etc., computer users can access and/or stream digital video content on systems across the world. Since the available bandwidth for such streaming is typically not as large as locally accessing the media within a computer, and because processing power is ever-increasing at low costs, encoders/decoders often aim to require more processing during the encoding/decoding steps to decrease the amount of bandwidth required to transmit the signals.
Accordingly, encoding/decoding methods have been developed, such as motion estimation, to provide block (e.g., pixel or region) prediction based on a previous reference frame, thus reducing the amount of block information that should be transmitted across the bandwidth as only the prediction need be encoded and not necessarily the entire block. For example, motion vector prediction and early termination are used in some implementations to achieve fast motion estimation. These methods, however, can introduce peak signal to noise ratio loss. Moreover, the methods for motion estimation and video coding are usually computationally expensive, and introduce recurrent dependency among adjacent blocks during encoding.
The following presents a simplified summary in order to provide a basic understanding of some aspects described herein. This summary is not an extensive overview nor is intended to identify key/critical elements or to delineate the scope of the various aspects described herein. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
Efficient inter-frame motion estimation is provided that mitigates adjacent block (e.g., pixel or regions of pixels) dependency in video frames by rearranging block encoding order and utilizes a fast motion estimation algorithm for determining motion vectors. Additionally, at least a portion of the motion estimation can be performed on a graphics processing unit (GPU) to achieve high-degree parallelism. Thus, selecting a block encoding order that removes adjacent block dependency can allow the parallel architecture of the GPU to synchronously encode a number of blocks in the video frame increasing encoding efficiency. Moreover, a fast motion estimation algorithm can be performed for encoding the blocks by leveraging the GPU.
For example, an encoding determination for a block in motion estimation can require motion vector information with respect to adjacent blocks of a video frame, such as calculating a motion vector predictor as a median of a number of adjacent block motion vectors. Therefore, ordering encoding of the blocks such that blocks independent of each other can be concurrently encoded following encoding of required adjacent blocks allows for advantageous utilization of parallel processing, which can be performed via a GPU parallel architecture, for example. Additionally, in one example, a multiple step search algorithm can be performed to locate an optimal motion vector for the motion estimation using the GPU to concurrently search for potentially matched blocks, or pixels thereof, between a current block and a reference block.
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings. These aspects are indicative of various ways which can be practiced, all of which are intended to be covered herein. Other advantages and novel features may become apparent from the following detailed description when considered in conjunction with the drawings.
Parallel block video encoding using fast motion estimation is provided where independent blocks of pixels or regions can be concurrently encoded based at least in part on adjacent previously encoded blocks using motion estimation and/or motion vector prediction. In one example, parallel processing functionality of a graphics processing unit (GPU) can be leveraged to effectuate the concurrent encoding. Moreover, fast motion estimation algorithms, such as a multiple-step search algorithm, can be utilized for efficient motion vector determination of given blocks. In addition, the multiple-step search algorithm can be performed using the GPU for parallel processing thereof, in one example.
For example, the blocks of a video frame, which can be one or more pixels or regions of pixels of varying size, can be ordered for encoding such that the order ensures requisite adjacent blocks for calculating a motion vector predictor (the median or mean average motion vector based on a number of adjacent blocks) have been encoded. Moreover, the blocks ordered with the same number are independent of each other for encoding purposes allowing the similarly ordered blocks to be encoded concurrently. Furthermore, the motion estimation encoding process can utilize a three step search (TSS) type of algorithm to determine the motion vector based on comparison with a number of reference blocks. It is to be appreciated that a modified TSS algorithm can be used in addition or alternative, such as a five-step search (FSS), six-step search (SSS), etc. A cost can be computed as to decoding the motion vector or a residue between the motion vector and the predictor, and the video block can be accordingly encoded.
Various aspects of the subject disclosure are now described with reference to the annexed drawings, wherein like numerals refer to like or corresponding elements throughout. It should be understood, however, that the drawings and detailed description relating thereto are not intended to limit the claimed subject matter to the particular form disclosed. Rather, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the claimed subject matter.
Now turning to the figures,
In one example, a video frame can be separated into a number of video blocks by the video coding component 104 (or motion estimation component 102), as mentioned, for encoding the frame using motion estimation. Moreover, the blocks can be ordered by the video coding component 104 for encoding such that the encoding can be concurrently performed for given independent blocks. In this regard, a parallel processor can be utilized by the motion estimation component 102 to search video blocks for determining motion vectors based on a reference block in parallel increasing efficiency in the prediction and therefore the encoding. For example, a graphics processing unit (GPU) can have a parallel architecture, and thus, can be utilized for general purpose computing (GPGPU) in this way. It is to be appreciated that substantially any motion estimation algorithm can be utilized by the motion estimation component 102 to determine motion vectors, including but not limited to step searches, as shown by way of example below, full searches, and/or the like.
Moreover, motion vectors of surrounding blocks can be utilized to create a motion vector predictor and estimate cost of encoding residue between the predictor and the motion vector for the current block. Thus, the video coding component 104 can take this into account when ordering the blocks to ensure the requisite blocks for computing the motion vector predictor are encoded before the appropriate block. Additionally, the video coding component 104 can encode the video block according to the motion vector based on the reference block in parallel by utilizing the GPU. Parallelizing these steps of motion estimation can significantly decrease processing time for encoding video according to a motion estimation algorithm. It is to be appreciated that the motion estimation component 102 and/or video coding component 104 can leverage, or be implemented within, a GPU or other processor, in separate processors, and/or the like, in one example.
In addition, the motion estimation component 102, video coding component 104, the functionalities thereof, and/or processors implementing the functionalities, can be integrated in devices utilized in video editing and/or playback. Such devices can be utilized, in an example, in signal broadcasting technologies, storage technologies, conversational services (such as networking technologies, etc.), media streaming and/or messaging services, and the like, to provide efficient encoding/decoding of video to minimize bandwidth required for transmission. Thus, more emphasis can be placed on local processing power (e.g., one or more central processing units (CPU) or GPUs) to accommodate lower bandwidth capabilities, in one example, and appropriate processors can be utilized, such as GPGPU, to efficiently encode video.
Referring to
For example, the step search component 202 can be utilized to determine motion vectors for video blocks based on one or more reference blocks. The step search component 202 can perform a multiple-step search for a given video block to be encoded by evaluating the block with respect to a set of reference blocks of a previous video reference frame. For example, the block to be encoded can be compared to a similarly positioned block of the reference image as well as additional surrounding blocks. In typical step searches, for example, blocks at eight substantially equidistant positions from the similarly positioned block in the reference frame can be evaluated as well; typically, the positions are at the four corners and midpoints at the four edges of the search window. One or more of the nine total blocks with a computed minimum cost for coding the motion vector can become a next focal point where eight surrounding, but nearer in proximity, video blocks can be evaluated moving-in on the block with a lowest cost until the video block is evaluated with respect to immediately surrounding blocks. Thus, the range chosen for the step search algorithm can affect the number of steps necessary to arrive at a minimum cost video block utilized to determine the appropriate motion vector. For example, FSS can allow up to a 16 block search window from each direction from the video block, and SSS can allow up to a 32 block search; thus, for a given number of steps n, a 2n pixel search window can be utilized. It is to be appreciated that the search can be performed over variable lengths or sizes of blocks, or pixels thereof, for example. Furthermore, a similar step search of substantially any degree can be utilized in this regard, or a completely different fast motion estimation algorithm, such as a full search for example, can be used by the step search component 202. This is just one of many possible example searches to be utilized.
In addition, the block ordering component 204 can order the video blocks of a frame being encoded such that a number of blocks can be encoded in parallel, for instance where the blocks do not depend from the other blocks that are encoded at the same time. For example, the video coding component 104 can evaluate motion vectors of surrounding blocks to calculate a motion vector predictor for the current block being encoded and can estimate a cost of coding a motion vector residue between the determined motion vector and the motion vector predictor. Thus, the blocks can be ordered by the block ordering component 204 such that requisite blocks for calculating the motion vector predictor for a given block are encoded by the video coding component 104 first. Additionally, blocks that are independent of one another can be encoded by the video coding component 104 at the same time.
In one example, the cost of coding the residue can be calculated using the following Lagrangian cost function,
J(m,λ)=D(C,P(m))+λ(R(m−p)),
where C is the original video signal, P is the reference video signal, m is the current motion vector, p is the motion vector predictor for the current block (e.g., a median of surrounding motion vectors), and λ is the Lagrange multiplier, which can be quantization parameter (QP) independent. Moreover, R(x) represents bits used to encode motion information; D(x) can be a sum of absolute differences (SAD) between the original video signal and the reference video signal or SAD of Hadamard-transformed coefficients (SATD). A motion vector can be selected by the video coding component 104 such to minimize the cost computed by the foregoing function. It is to be appreciated that other cost functions can be used as well. Additionally, the cost of coding the motion vector can be compared with a cost of encoding a residue motion vector related to the difference in a predicted motion vector and the actual motion vector, in one example; the resulting encoding can depend on the calculated cost. Further, it is to be appreciated that the fast motion estimation algorithm chosen by the step search component 202 can be different for given video blocks in one example. Moreover, as described, the functionalities provided by the step search component 202 and/or the block ordering component 204, as well as predicting motion vectors, can leverage, or be implemented within, a GPU having parallel architecture to provide further efficiency.
Turning now to
In one example, some coding standards, such as H.264/AVC utilize the block immediately left of the current block as well as the block immediately above the current block and the block to the upper right of the current block to predict the motion vector for the current block. Thus, for the blocks numbered 7, the block labeled 5 as well as the two blocks labeled 6 can be utilized to predict a motion vector for block 7. Because the blocks are lower in number, they are already encoded as motion vectors and can be averaged to produce the predicted motion vector for a given block 7. The blocks of the example video frame portion 300 can be encoded from top left to bottom right in this regard, and a parallel processor, such as a GPU or other processor, can be utilized to concurrently encode like-numbered blocks rendering the encoding more efficient than where all blocks depend from one another.
It is to be appreciated that the blocks can be ordered in substantially any way according to the algorithm being utilized. For example, the aforementioned ordering can be reversed starting at the bottom right and working to the top left, etc. Moreover, it is to be appreciated that portions of a video frame can be encoded in parallel by one or more GPUs or other processors as well. Thus, the video frame portion 300 can be one of many portions, or macro blocks, of a larger video frame, which can be encoded using the mechanisms explained above in parallel with other portions, for example. Furthermore, as described, the encoding for each video block can be performed using substantially any fast motion estimation algorithm, such as a multiple-step search (e.g., TSS, FSS, SSS, or substantially any number of steps), a full search, and/or the like to estimate a best motion vector for the given video block. Subsequently, the cost of encoding the motion vector or a residue between the motion vector and the predicted motion vector can be weighed in deciding which to encode, in one example.
Referring now to
In one example, the video coding component 104 can utilize the variable block size selection component 402 to separate a given video frame into one or more video blocks. As described above, the blocks can be square or can have a different number of pixels in given rows or columns of the block; additionally, the blocks can be single pixels or portions thereof, for example. Moreover, the blocks can be of varying size throughout the video frame. In one example, the video blocks are 4 pixels by 4 pixels. Additionally, the blocks can be grouped into sets of macro blocks, in one example. The inference component 404 can be utilized by the variable block size selection component 402 to determine an optimal size for one or more blocks or macro blocks of the video frame. The inference can be made based at least in part on previous encodings (within the same or different video), CPU/GPU ability, bandwidth requirements, video size, etc.
In addition, the video blocks can be ordered by the block ordering component 204. As described, the ordering can relate to preserving ability to encode one or more video blocks in parallel. Again, the inference component 404 can infer such an order based at least in part on a desired encoding scheme or direction (e.g., top left to bottom right, etc.), type of processor being utilized, resources available to the processor, bandwidth requirements, video size, previous orderings, and/or the like. Furthermore, the step search component 202 can leverage the inference component 404 to select a fast motion estimation algorithm to utilize for determining one or more motion vectors related to a give video block. For example, the inference can be made as described above, depending on a previous algorithm, processing ability or requirements, time requirements, size requirements, bandwidth available, etc. Additionally, the inference component 404 can make inferences based on factors such as encoding format/application, suspected decoding device or capabilities thereof, storage format and location, available resources, etc. for the above-mentioned components. The inference component 404 can also be utilized in determining location or other metrics regarding a motion vector, and the like.
The aforementioned systems, architectures and the like have been described with respect to interaction between several components. It should be appreciated that such systems and components can include those components or sub-components specified therein, some of the specified components or sub-components, and/or additional components. Sub-components could also be implemented as components communicatively coupled to other components rather than included within parent components. Further yet, one or more components and/or sub-components may be combined into a single component to provide aggregate functionality. Communication between systems, components and/or sub-components can be accomplished in accordance with either a push and/or pull model. The components may also interact with one or more other components not specifically described herein for the sake of brevity, but known by those of skill in the art.
Furthermore, as will be appreciated, various portions of the disclosed systems and methods may include or consist of artificial intelligence, machine learning, or knowledge or rule based components, sub-components, processes, means, methodologies, or mechanisms (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, classifiers . . . ). Such components, inter alia, can automate certain mechanisms or processes performed thereby to make portions of the systems and methods more adaptive as well as efficient and intelligent, for instance by inferring actions based on contextual information. By way of example and not limitation, such mechanism can be employed with respect to generation of materialized views and the like.
In view of the exemplary systems described supra, methodologies that may be implemented in accordance with the disclosed subject matter will be better appreciated with reference to the flow charts of
At 506, the blocks can be ordered to allow parallel encoding thereof. As described, depending on a motion estimation algorithm, blocks utilized for estimating or predicting motion vectors for a current block can be encoded before the current block. However, the blocks can be ordered such that blocks independent of each other for encoding purposes can be encoded in parallel as shown supra. It is to be appreciated that the blocks can be ordered in substantially any manner to achieve this end; the examples shown above are for the purpose of illustrating of possible schemes. At 508, a portion of the blocks can be concurrently encoded according to the imposed order. This can be performed via a GPU in one example.
As used herein, the terms “component,” “system” and the like are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an instance, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
The word “exemplary” is used herein to mean serving as an example, instance or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Furthermore, examples are provided solely for purposes of clarity and understanding and are not meant to limit the subject innovation or relevant portion thereof in any manner. It is to be appreciated that a myriad of additional or alternate examples could have been presented, but have been omitted for purposes of brevity.
Furthermore, all or portions of the subject innovation may be implemented as a method, apparatus or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed innovation. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device or media. For example, computer readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips . . . ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ), smart cards, and flash memory devices (e.g., card, stick, key drive . . . ). Additionally, it should be appreciated that a carrier wave can be employed to carry computer-readable electronic data such as those used in transmitting and receiving electronic mail or in accessing a network such as the Internet or a local area network (LAN). Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.
In order to provide a context for the various aspects of the disclosed subject matter,
With reference to
The system memory 816 includes volatile and nonvolatile memory. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer 812, such as during start-up, is stored in nonvolatile memory. By way of illustration, and not limitation, nonvolatile memory can include read only memory (ROM). Volatile memory includes random access memory (RAM), which can act as external cache memory to facilitate processing.
Computer 812 also includes removable/non-removable, volatile/non-volatile computer storage media.
The computer 812 also includes one or more interface components 826 that are communicatively coupled to the bus 818 and facilitate interaction with the computer 812. By way of example, the interface component 826 can be a port (e.g., serial, parallel, PCMCIA, USB, FireWire . . . ) or an interface card (e.g., sound, video, network . . . ) or the like. The interface component 826 can receive input and provide output (wired or wirelessly). For instance, input can be received from devices including but not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, other computer and the like. Output can also be supplied by the computer 812 to output device(s) via interface component 826. Output devices can include displays (e.g., CRT, LCD, plasma . . . ), speakers, printers and other computers, among other things. Moreover, the interface component 826 can have an independent processor, such as a GPU on a graphics card, which can be utilized to perform functionalities described herein as shown supra.
The system 900 includes a communication framework 950 that can be employed to facilitate communications between the client(s) 910 and the server(s) 930. Here, the client(s) 910 can correspond to program application components and the server(s) 930 can provide the functionality of the interface and optionally the storage system, as previously described. The client(s) 910 are operatively connected to one or more client data store(s) 960 that can be employed to store information local to the client(s) 910. Similarly, the server(s) 930 are operatively connected to one or more server data store(s) 940 that can be employed to store information local to the servers 930.
By way of example, one or more clients 910 can request media content, which can be a video for example, from the one or more servers 930 via communication framework 950. The servers 930 can encode the video using the functionalities described herein, such as block parallel fast motion estimation, encode blocks of the video as related to a reference frame, and store the encoded content in server data store(s) 940. Subsequently, the server(s) 930 can transmit the data to the client(s) 910 utilizing the communication framework 950, for example. The client(s) 910 can decode the data according to one or more formats, such as H.264/AVC or other MPEG level decoding, utilizing the encoded motion vector or residue information to decode frames of the media. Alternatively or additionally, the client(s) 910 can store a portion of the received content within client data store(s) 960.
What has been described above includes examples of aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the disclosed subject matter are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the terms “includes,” “has” or “having” or variations in form thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising” as comprising is interpreted when employed as a transitional word in a claim.
