Patent Grant
8654842
Multimedia content production and distribution operations typically include video encoding. Video encoding processes are typically very data and computationally intensive. As a result, video encoding processes can be very time consuming. For example, it may take several tens-of hours for a software encoder to encode a high-quality high definition movie. Since quality and speed of video encoding processes are significant factors for successful multimedia content production and distribution pipelines, systems and techniques to increase the speed at which high quality video content can be encoded would be useful.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In view of the above, a video encoding acceleration service to increase one or more of the speed and quality of video encoding is described. The service acts as an intermediary between an arbitrary video encoder computer program application and arbitrary video acceleration hardware. The service receives one or more queries from the video encoder to identify implementation specifics of the video acceleration hardware. The service interfaces with the video acceleration hardware to obtain the implementation specifics. The service communicates the implementation specifics to the video encoder. The implementation specifics enable the video encoder to: (a) determine whether one or more of speed and quality of software encoding operations associated with the video encoder can be increased with implementation of a pipeline of one or more supported encoding pipeline configurations and capabilities, and (b) implement the pipeline by interfacing with the service.
In the Figures, the left-most digit of a component reference number identifies the particular Figure in which the component first appears.
Overview
Systems and methods for accelerated video encoding provide a video encoding acceleration service. This service allows an arbitrary video encoder application to interface, in a device independent manner, with arbitrary video acceleration hardware to define and implement a substantially optimal video encoding pipeline. To accomplish this, the service exposes video acceleration (VA) application program interfaces (APIs). These APIs encapsulate a model of the video encoding process. To define an encoding pipeline, the video encoder application uses the VA APIs to query implementation specifics (e.g., capabilities, etc.) of available video (graphics) acceleration hardware. The video encoder evaluates these specifics in view of the application's particular video encoding architecture (software-implemented) to identify any encoding operations that could benefit (e.g., speed and/or quality benefits) from being accelerated in hardware. Such operations include, for example, motion estimation, transform, and quantization operations and inverse operations such as Motion compensation, inverse transforms and inverse quantization. The API also allows the video encoder to design an encoding pipeline that substantially minimizes dataflow transitions across buses and processors associated with the host computing device and the acceleration hardware, and thereby, further increase encoding speeds. The API also allows the acceleration hardware to influence the location of the data to improve local caching (e.g. the video acceleration hardware may functional more efficiently on memory local to the video hardware).
Based on these evaluations the video encoder designs a customized video encoding pipeline that performs some number of encoding operations in software and some number of encoding operations using the acceleration hardware (i.e., at least a subset of the operations that could benefit from being hardware accelerated). The encoder application then uses the API to create the pipeline and encode video content. This customized pipeline is substantially optimized as compared to a completely software-implemented pipeline because certain encoding operations are accelerated and data transitions between the host and the acceleration hardware are minimized. Additionally, processing time freed up by accelerating certain aspects of the encoding process and minimizing data transitions allow the host processor(s) to perform higher-quality encoding operations with freed-up processing cycles. The API is also designed to allow components to operate in parallel so that computational resource usage can be maximized.
These and other aspects of the systems and methods for accelerated video encoding are now described in greater detail.
An Exemplary System
Although not required, the systems and methods for accelerated video encoding are described in the general context of computer-executable instructions (program modules) being executed by a computing device such as a personal computer and graphics (video) encoding acceleration hardware. Program modules generally include routines, programs, objects, components data structures, etc., that perform particular tasks or implement particular abstract data types.
In this implementation, video encoder 116 is an arbitrary video encoder. This means that the particular architecture, operation, data formats, etc, implemented and/or utilized by video encoder 116 are arbitrary. For example, video encoder 116 may be distributed by a third party, an OEM, etc. Additionally, although
Video processing modules 112 receive compressed or uncompressed input video data 122. When input video data 122 is compressed (already encoded), video processing modules 112 decode the input video data 122 to produce decoded source video data. Such decoding operations are performs by a decoder module. In another implementation, partially decoded data could also be retained to farther assist the encoding process. For purposes of exemplary illustration, such a decoder module is shown as a respective portion of “other video processing modules” 120. Thus, decoded source video data is represented either by input video data 122 that was received in a decoded state, or represented with results of decoding input video data 122 that was received in an encoded state. Decoded source video data is shown as a respective portion of “other program data” 124.
To design and implement a customized video encoding pipeline that can be used to encode decoded source video data into encoded video data 126, video encoder 116 interfaces with video encoding acceleration service 118 via video acceleration (VA) APIs 128. One exemplary implementation of multiple possible implementations of VA APIs 128 is described in the Appendix. To define an encoding pipeline, the video encoder application uses respective ones of the VA API 128 (e.g., please see the Appendix, §3.4, IVideoEncoderService) to obtain implementation specifics of available acceleration hardware 130. Such implementation specifics include, for example:
Responsive to receiving such requests from the video encoder 116, video encoding acceleration service 118 queries the video acceleration hardware 130 for the requested implementation specifics and returns information associated with the corresponding responses from the acceleration hardware 130 to the video encoder 116. Video encoding acceleration service 118 interfaces with the video acceleration hardware 130 using a corresponding device driver. Such a device driver is shown as respective portion of “other program modules” 114.
Video encoder 116 evaluates the implementation specifics supported by acceleration hardware 130 in view of the application's particular video encoding architecture (software-implemented) to identify any encoding operations that could benefit (e.g., speed and/or quality benefits) from being accelerated in hardware, select a search profile to encapsulate a trade-off between video encoding quality and speed, minimize data transitions across buses and between processors, etc. Exemplary operations that may benefit from hardware acceleration include, for example, motion estimation, transform, and quantization. For example, one reason to perform quantization in hardware is to minimize dataflow between pipeline stages.
In this example implementation, video encoder 116 (
Such operations may include converting uncompressed (YUV) video data to compressed MPEG-2, or it may include transcoding video data from MPEG-2 data format to WMV data format. For purposes of exemplary illustration, assume that the transcoding operations include a full or partial decompression stage followed by an encode stage (there are more efficient models which by-pass decompression and work purely in the transform (DCT) space). A number of video compression formats make use of motion estimation, transform and quantization to achieve compression. Of the compression stages, motion estimation is typically the slowest step, including a massive search operation where an encoder (e.g., video encoder 116) attempts to find the closest matching reference macroblock for macroblocks in a given image.
Once the optimal motion vectors are determined (e.g., via block 206) for each of the macroblocks, the encoder 116 computes the differential residues (e.g., via block 208) based on the previously coded image and the optimal motion vector. The motion vector, along with the differential residue is a compact representation of the current image. The motion vector data is further represented differentially. The host encoder can optionally request the re-evaluation of motion vectors by the video acceleration hardware to find a macroblock with a smaller combined motion vector and/or residual. The resulting differential motion vector data, and the residual data are compacted (e.g., via block 218), for example, using techniques like run-length encoding (RLE) and differential coding (e.g.: Huffman and Arithmetic coding) to generate the final coded stream of bits (encoded video data 126) to communicate to a destination (block 218) for presentation to a user. In this example, the operations of blocks 206, 208 and 214 through 218 (e.g., operations such as motion estimation (206), mode decision, motion vector (MV) selection and rate control (208), prediction formation (210), transform and quantization operations (214), quantizer inversion and transform and version (216) and entropy coding (218)) are well-known in the art and are thus not described further herein.
Referring again to
With respect to selecting a search profile, the quality of motion vectors refers to a bitrate of a stream generated by the use of the motion vectors. High quality motion vectors are associated with low bitrate streams. The quality is determined by the completeness of the block search, the quality of the algorithm, the distance metric used, etc. High quality motion vectors should be used to perform high quality video encode operations. To address this, video encoding acceleration service 118 provides a generic construct called a search profile to encapsulate a trade-off between quality and time. The search profile also includes meta-data to identify the search algorithm used by the acceleration hardware 130, etc. Video encoder 116 chooses a particular search profile based on the particular requirements of the encoder's implementation.
With respect to minimizing data transitions across buses and between processors, an encode process implemented by a video encoding pipeline configuration will typically include several processing stages, each of which may or may not be accelerated via acceleration hardware 130. In cases where video encoder 116 determines to utilize hardware acceleration in successive stages of the encode pipeline, it may not be necessary to move data from acceleration hardware 130 based memory 132 to the system memory 106 associated with the host computing device 102, and then back to acceleration hardware based memory 132 for the next stage, and so on.
More particularly, while pointers to various types of video and motion vector data may be transferred back and forth between the host computing device 102 and the acceleration hardware 130, in one implementation, actual data is copied to system memory 106 only when the data pointer (a D3D9 Surface pointer) is explicitly locked using, for example, IDirect3DSurface9::LockRect. Exemplary interfaces for locking a surface are known (e.g., the well-known IDirect3DSurface9::LockRect.interface). Thus, in cases where two encoding pipeline stages follow one another, and host computing device 102 does not need to do perform any intermediate processing, host computing device 102 can decide not to “Lock” the allocated buffer between the processing stages. This will prevent a redundant memory copy of data, and thereby, avoid unnecessary data movement/transfers. In this manner, video encoder 116 design a video encoding pipeline that substantially minimizes data transfers across buses and between processors, and thereby, further increase video encoding speeds.
At this point, video encoder 116 has evaluated the implementation specifics supported by acceleration hardware 130 in view of the application's particular video encoding architecture (software-implemented) to identify any encoding operations that could benefit from being accelerated in hardware, selected a search profile, minimized data transitions across buses and between processors, and/or so on. Based on these determinations, video encoder 116 selects a particular pipeline configuration to encode decoded source video data, and thereby, generate encoded video data 126. Next, video encoder 116 interfaces with video encoding acceleration service 118 to create an encoder object to implement the selected pipeline (please see the Appendix, CreateVideoEncoder API, §3.4.6). In this implementation, an encoder object (e.g., a regular COM object) is created by identifying the selected pipeline configuration and one or more of the following: a format for the output encoded bitstream, the number of input and output data streams associated with the pipeline configuration, static configuration properties, a suggested number of buffers (surfaces) for association with the different I/O streams based on the selected pipeline configuration, and a driver specified allocator queue size based on resources a graphics device driver is able to gather, and other parameters. (Queue size and the number of data buffers are essentially referring to the same thing; one is “suggested”, the other is “actual”).
Next, video encoder 116 uses the created encoder object to interface with the video encoding acceleration service 118 to encode the decoded source video data. To this end, the encoder object submits execute requests to acceleration hardware 130 (please see the Appendix, IVideoEncode:Execute API, §3.2.3).
In view of the above, system 100 allows arbitrary implementations of video encoder applications 116 to define and create video encoding pipeline configurations during runtime to take full advantage of available video encoding acceleration hardware to increase encoding speed and quality. As part of these runtime configuration operations, the video encoder 116 can use VA APIs 128 to specify that the encoding pipeline is to implement iterative directed searching (multiple search passes of increasing refinement), define and use generically selectable search strategies (e.g., selecting a search algorithm based on quality metrics independent of any knowledge of details about the actual algorithm been employed), utilize format independent methodologies (e.g., where a video encoder 116 is unaware of the particular image format of input video data 122 and the acceleration hardware 130 is unaware of the compressed output format for the encoded video data 126) to control searching, adapt data sizes (e.g., where the video encoder 116 selects a macro block size based on a search algorithm), and so on.
An Exemplary Procedure
At block 302, video encoder 116 (
At block 310, video encoder 116 creates an encoding object that implements an encoding pipeline configured to execute the identified video encoding operations that may benefit from hardware acceleration in acceleration hardware 130, implement the speed/quality tradeoffs (e.g., via a selected search profile), and minimize data flow transitions. At block 312, video encoder uses the created encoder object to encode the decoded source video data according to the sequence of operations and encoding architecture delineated by the customized video encoding pipeline generated at block 310. These encoding operations of block 312 generate encoded video data 126 (
Although the systems and methods for accelerated video encoding have been described in language specific to structural features and/or methodological operations or actions, it is understood that the implementations defined in the appended claims are not necessarily limited to the specific features or actions described.
For example, although API's 128 of
In another example with respect to image stabilization, in one implementation video encoder 116 computes motion vectors for all macroblocks and decoded source data. Video encoder 116 then determines whether there is global motion in the image. This is accomplished by correlating all motion vector values and determining whether the correlated values are similar. If so, then video encoder 116 concludes that there is global motion. Alternatively the video encoder 116 utilizes a large macroblock size and determines if there is overall motion of the large macroblock. After determining whether global motion is present, if video encoder 116 also finds that the global motion vector tends to be jerky across frames, video encoder 116 concludes that there is camera jerkiness and compensates for this before starting noise filtering and encoding operations.
Accordingly, the specific features and operations of system 100 are disclosed as exemplary forms of implementing the claimed subject matter.
This Appendix describes various exemplary aspects of an exemplary implementation of the video encoding acceleration APIs 128 (
2 Exemplary Design
2.1 Encoder Layout
Block 402 represents a number n image surfaces of input data. In one implementation, each image surface represents a respective D3D surface. Operations of block 404 implement noise filtering operations. In this implementation, the noise filtering operations are implemented to by the known IDirectXVideoProcessor interface, although other interfaces could also be used. Block 406 represents noise filtered image surface data. At block 408, motion estimation operations are implemented, possibly via acceleration hardware 130 as described above. The motion vectors are represented in buffer 410. Operations of block 412 implement motion compensation. The resulting coded image surfaces are represented at block 414. Residual data computation operations of block 418 operate on image data from buffers 406 and 410, resulting in residual data in buffer 420. In view of the residual data, operations of function 422 perform DCT and quantization processes to generate the residual data of buffer 424. Entropy coding operations of function 416 and inverse quantization and IDCT operations of function 426 operate on this residual data (transformed and quantized data). Function 426 generates residual data that is quantized and inverse quantized as shown in buffer 428. In view of the data in buffer 428, function 412 implements motion compensation operations.
Significantly, the various data in respective ones of the buffer 402, 406, 414, 420, 424 and 428 are well-known in the art. Additionally, operations associated with respective ones of the functions 404, 408, 412, 416, 418, 422 and 426 are also well-known in the art. Thus it is not the respective data and details of the operations of the video encoder that are novel. In contrast to conventional encoders, encoder 116 (
2.2 Pipeline or Mode Configurations
Acceleration hardware 130 (
The design allows for split (or multi-stage) pipelines where data goes back and forth between the host PC 102 and the hardware 130, before the final output is obtained. The following described pipeline configurations represent non-split, single stage pipelines.
2.2.1 VA2_EncodePipe_Full
In this implementation, the number of streams (NumStreams) is five, although other numbers of streams could also be used. The actual StreamIds are shown in the diagram in parentheses.
(This is an example of a single stage, non-split pipeline, and hence the Stage parameter of Execute is not used).
Stream Descriptions
Note that StreamType_* is an abbreviation for VA2_Encode_StreamType_*.
Input Image
Reference Images
Motion Vectors
Residues
Decoded Image
NumStreams for this pipeline configuration is three. The StreamIds for the various streams are show in the diagram in paranthesis.
This is a single stage, non-split pipeline and the Stage parameter of Execute does not apply.
3 Exemplary API
3.1 Interface Definition
3.1.1 IVideoEncoder
3.1.2 IVideoEncoderService
3.2 Methods: IVideoEncoder
3.2.1 GetBuffer
This function returns buffers (encode surfaces) for use in the exemplary Execute call described below. The buffers are released promptly after use by calling ReleaseBuffer to avoid stalling the pipeline.
Parameters
StreamId
StreamType
Blocking
pBuffer
S_OK
E_NOTAVAILABLE
E_INVALIDPARAMETER
VFW_E_UNSUPPORTED_STREAM
E_FAIL
Normally, this function returns control very soon as the buffers are already present in the allocator queue. The only conditions under which this function should block (or return E_NOTAVAILABLE) are when all buffers from the allocator queue have been submitted to the device, or being consumed by the application, and hence not released.
3.2.2 ReleaseBuffer
This function is used to release a surface back into the allocator queue for reuse via GetBuffer.
Parameters
StreamId
StreamType
pBuffer
S_OK
E_FAIL
This function is used to submit requests to acceleration hardware 130. It provides input and output data buffers obtained via GetBuffer, as well as some configuration information. The function is asynchronous and its completion is indicated by the event being signaled. The completion status is indicated using the pStatus parameter which is allocated on the heap, and checked only after the event has been signaled.
The buffers supplied as parameters to this function are not be read from (eg: via LockRect), or written to by the encoding application until the function has truly completed. In this implementation, true completion is implied by an error value being returned by the function, or if this function returns success, then by the signaling of hEvent (parameter to this function). When the same buffer is input to several instances of the Execute call, it is not be accessed until all associated Execute calls have completed. The pointer to a surface in use by Execute may still be supplied as a parameter to VA functions like Execute since this doesn't require the data to be locked. This last rule explains how the same input image may be used in multiple Execute calls at the same time.
The buffers supplied to this call obey the allocator semantics negotiated at creation time. If an external allocator is used when GetBuffer is expected to be used, this function will return E_FAIL.
Parameters
Stage
NumInputDataParameters
pInputData
NumOutputDataPararmeters
pOutputData
NumConfigurationParameters
pConfiguration
hEvent
pStatus
S_OK
E_FAIL
If the event handle gets signaled, it means that LockRect should complete instantly when called on any of the output surfaces since they are ready. In particular, the LockRect call is expected to not block for any length of time by waiting on any event handles. Nor is it allowed to waste CPU time through busy spins.
3.3 Data Structures: Execute
The Execute call has data parameters and configuration parameters. Specific data parameters can be thought of as deriving from VA2_Encode_ExecuteDataParameter base class (or structure) and specific configuration parameters can be thought of as deriving from VA2_Encode_ExecuteConfigurationParameter base class (or structure).
3.3.1 VA2_Encode_ExecuteDataParameter
Members
Length
StreamId
Members
Length
StreamId
ConfigurationType
This structure acts as a base type for more specialized configuration information. The base type is typecast to a more specialized type based on the ConfigurationType parameter. The mapping between ConfigurationType and the specialized structures is described in the table below.
3.3.3 DataParameter_MotionVectors
Members
Length
StreamId
pMVSurface
Members
Length
StreamId
pResidueSurfaceY
pResidueSurfaceCb
pResidueSurfaceCr
Members
Length
StreamId
pImageData
Members
Length
StreamId
DataType
NumReferenceImages
pReferenceImages
Members
Length
StreamId
DataType
pYCbCrImage
Members
pSurface
Field
Interlaced
Window
Members
Length
StreamId
ConfigurationType
pMEParams
3.3.10 VA2_Encode_SearchResolution
Description
FullPixel
HalfPixel
QuarterPixel
In computing sub-pixel motion vector values, the encoder estimates luma and chroma values using interpolation. The specific interpolation scheme is format dependent, and the following GUIDs (part of static configuration) control the interpolation scheme.
3.3.11 VA2_Encode_SearchProfile
Members
QualityLevel
Time Taken
SearchTechnique
SubpixelInterpolation
A GUID indicating the subpixel interpolation scheme used.
Remarks
There is no universally accepted definition of absolute quality, so we are settling for a relative measure. The values indicated against TimeTaken should follow a strict proportion rule. If profile 1 takes 10 ms and profile 2 takes 20 ms, the TimeTaken values should be in the ratio 20/10=2.
3.3.12 VA2_Encode_MBDescription
Members
ConstantMBSize
MBWidth
MBHeight
MBCount
pMBRectangles
Members
DetectSceneChange
MaxDistanceInMetric
TimeLimit
Search WindowX
SearchWindowY
Members
SearchProfileIndex
DistanceMetric
HintX
HintY
Members
Length
StreamId
ConfigurationType
StepSize
The methods in this interface allow an application to query the hardware for its capabilities and create an encoder object with a given configuration.
3.4.1 GetPipelineConfigurations
Parameters
pCount
pGuids
S_OK
E_OUTOFMEMORY
E_FAIL
Parameters
pCount
pGuids
Parameters
pCount
pGuids
S_OK
E_OUTOFMEMORY
E_FAIL
Parameters
pCount
pSearchProfiles
S_OK
E_OUTOFMEMORY
E_FAIL
Parameters
pMECaps
S_OK
E_FAIL
This function creates an instance of IVideoEncoder.
Parameters
PipelineGuid
FormatGuid
NumStreams
pConfiguration
pStreamDescription
SuggestedAllocatorProperties
pActualAllocatorProperties
ppEncode
S_OK
E_FAIL
Members
VariableMBSizeSupported
MotionVectorHintSupported
MaxSearchWindowX
MaxSearchWindowY
MaxImageWidth
MaxImageHeight
MaxMBSizeX
MaxMBSizeY
Members
TransformOperator
PixelInterpolation
Quantization
NumMotionVectorsPerMB
MVLayout
ResidueLayout
Members
ExternalAllocator
NumSurfaces
Members
Length
StreamType
This base structure is typecast to a derived type on the StreamType field. The typecasts are described in the documentation for VA2_Encode_StreamType.
3.5.5 VA2_Encode_StreamType
Type Descriptions
VA2_Encode_StreamType_Video
VA2_Encode_StreamType_MV
VA2_Encode_StreamType_Residues
Members
Length
StreamType
VideoDescription
Members
Length
StreamType
MVType
MVLayout
Members
Length
StreamType
SamplingType
LumaWidth
LumaHeight
ChromaCbWidth
ChromaCbHeight
ChromaCrWidth
ChromaCrHeight
3.6.2 New D3D Formats
Motion Vectors Surfaces and Residue Surfaces are associated with the above new D3D Format types which indicate the size of individual Motion Vectors and Residues. This size information is used by the driver when the application creates surfaces using one of the surface or resource creation APIs provided by. The resource flag associated with encode surfaces is VA2_EncodeBuffer.
3.6.3 VA2_Encode_MVSurface
This structure is effectively derived from IDirect3DSurface9, and carries state information that allows one to interpret the contents of the embedded D3D surface.
Members
pMVSurface
MVType
MVLayout
DistanceMetric
This enumeration value is used to decode the contents of the Motion Vector D3D9 Surface. Depending on the type of Motion Vector, one of several different Motion Vector structures is used to interpret the contents of the surface.
Description
VA2_Encode_MVType_Simple8
VA2_Encode_MVType_Simple16
VA2_Encode_MVType_Extended8
VA2_Encode_MVType_Extended16
Description
Type A
Type B
Type C
3.6.6 VA2_Encode_MotionVector8
Members
x
y
Members
x
y
Members
x
y
ImageIndex
Distance
Members
x
y
ImageIndex
Distance
In this implementation, a residue surface is an array of signed integer values that are two bytes long—e.g., of type INT16. This scheme is practical. For example, MPEG-2 deals with 9 bit residue values and H.264 deals with 12 bit residues. Also, if the original data was YUY2, the luma values occupy one byte each, and hence the residues use 9 bits (0−255=−255). Further, applying a DCT-type transform increases the data requirement to 11 bits per residue value. All of these cases are adequately addressed by using 2 byte long signed residue values.
The width of a residue surface is the number of residue values in a line. For example, a 640×480 progressive image with 4:2:2 sampling has 640 luma values and 320 chroma values per line. The size of associated the luma surface is 640×480×2 and that of the chroma surface is 320×480×2 bytes.
Residue Surfaces are created using the D3DFMT_RESIDUE16 format flag and VA2_EncodeBuffer resource type.
3.7.1 Luma Plane
Plane=VA2_Encode_Residue_Y
Sampling=VA2_Encode_SamplingType_*
3.7.2 Chroma 4:2:2
Plane=VA2_Encode_Residue_U or VA_Encode_Residue_V
Sampling=VA2_Encode_SamplingType—422
3.7.3 Chroma 4:2:0
Plane=VA2_Encode_Residue_U or VA_Encode_Residue_V
Sampling=VA2_Encode_SamplingType—420
4 Exemplary DDI Documentation
Extension Devices are a pass-through mechanism provided by the VA Interfaces to add new functionality in addition to Video Decoder and Video Processor functions. For example, in one implementation, such a mechanism is used to support a new Video Encoder function.
Extension Devices are analogous to an untyped funnel through which the application can send/receive data to/from the driver. The meaning of the data is unknown to the VA stack, and is interpreted by the driver based on the pGuid parameter of the CreateExtensionDevice call, and the Function parameter of ExtensionExecute.
VA Encode uses the following GUID value (same as the uuid of IVideoEncoder):
4.1 Enumeration and Capabilities
Extension Devices are enumerated using the FND3DDDI_GETCAPS with the type parameter being set to GETEXTENSIONGUIDCOUNT or GETEXTENSIONGUIDS. The codec application looks for VA_Encoder_Extension in the list of extension guids returned by GETEXTENSIONGUIDS to determine whether VA Encode support is available.
4.1.1 FND3DDDI_GETCAPS
When querying for capabilities of the extension device (the Encoder device), the GETEXTENSIONCAPS is used with the following structure as pInfo in the D3DDDIARG_GETCAPS structure.
4.1.2 VADDI_QUERYEXTENSIONCAPSINPUT
The pGuid parameter of VADDI_QUERYEXTENSIONCAPSINPUT is set to VA_Encoder_Extension.
The output of GETEXTENSIONCAPS is encapsulated in the pData parameter of D3DDDIARG_GETCAPS. The pData parameter is interpreted as follows:
The actual creation happens via a D3DDDI_CREATEEXTENSIONDEVICE call, whose primary argument is shown below:
4.2 Encode Functionality
The actual extension unit functions are invoked via a D3DDDI_EXTENSIONEXECUTE call. The instance of the Extension Unit is already associated with a GUID, so the type of the extension unit is already known when the execute call is made. The only additional parameter is Function which indicates the particular operation to perform. For example an Extension Device of type Encoder, may support MotionEstimation as one of its functions. Typically, the Extension Device will have a GetCaps function of its own that enumerates the capabilities of the Extension Device.
The pBuffers parameter is not used by VA Encode, and should be considered a reserved parameter. The Function parameter takes the following values for VA Encode:
#define VADDI_Encode_Function_Execute 1
The pPrivateInput and pPrivateOutput parameters of D3DDDIARG_EXTENSIONEXECUTE are used to encapsulate the parameters of the Execute API call.
4.2.1 VADDI_Encode_Function_Execute_Input
This parameter contains the input parameters to the Execute API call.
4.2.2 VADDI_Encode_Function_Execute_Output
This structure encapsulates the output data from the Execute call.
4.3 Extension Device Structures
The following sections describe various structures and function callbacks associated with the VA Extension mechanism.
4.3.1 VADDI_PRIVATEBUFFER
4.3.2 D3DDDIARG_EXTENSIONEXECUTE
The hDevice parameter refers to a D3D9 device, and it is created using a call to D3DDDI_CREATEDEVICE.
4.3.3 FND3DDDI_DESTROYEXTENSIONDEVICE
4.3.4 FND3DDDI_EXTENSIONEXECUTE
4.3.5 D3DDDI_DEVICEFUNCS
4.4 D3D9 Structures and Functions
The following D3D structures and callback represent a generic D3D mechanism to obtain the capabilities of an extension device.
5 Exemplary Programming Model
5.1 Pipeline Efficiency
In one implementation, to achieve maximum efficiency, the encoder application 116 is structured to full utilize processor(s) 104 and graphics hardware 130. In one example, while Motion Estimation is in progress for a certain frame, the Quantization Step may be executed on a different frame.
Obtaining full hardware utilization is facilitated with a multi-threaded encoder.
5.1.1 Example: Single Motion Vector (Pipeline Full)
The following 2-threaded application (in pseudo-code) illustrates one way for the encoder 116 to implement a 2-stage software pipeline, and offers some examples of how to use the VA Encode interfaces 128 effectively. This particular implementation enforces a buffering of k=AllocatorSize as seen in the software thread. This accounts that there is asynchrony in the submission of a hardware request: the hardware thread submits requests while the software thread picks up the results after a while and processes them.
ProcessInput above may be considered a wrapper around Execute and GetBuffer, while ProcessOutput may be considered a wrapper around a Wait on the execute event, followed up with appropriate ReleaseBuffer calls.
Parameter k represents the buffer between the pipeline stages. It denotes the allocator size, and as a starting point, we could use the same value used in the allocator negotiation between the Codec and the VA Encoder object (the queue length). If k is larger than the allocator size, then the ProcessInput call is likely to block anyway even before the k buffers get used.
The goal of the application should be to maximize time spent in SoftwareThread without blocking on ProcessOutput. In other words the application should be working on the VLE( ) and Bitstream( ) functions most of the time. If the hardware is very slow, then ProcessOutput( ) will block despite the allocator size of “k”. Software will always be “ahead”. The above pipeline is efficient only to the extent that the hardware takes about as much time to process a buffer as software takes to run VLE and Bitstream. All that the buffering of “k” achieves is to pad for jitters.
The following code fragment shows an exemplary pseudocode implementation of GetBuffer and ReleaseBuffer.
The following sketches out the codec's implementation of ProcessInput and ProcessOutput:
Here is an alternate implementation of Codec::ProcessInput that is non-blocking as is the norm.
5.1.2 Example: Multiple Motion Vectors
An exemplary complex pipeline is now described, where the encoder 116 requests multiple motion vectors from hardware and chooses one based on various parameters and resubmits them for processing. The following code naively continues to use a 2-stage pipeline as before, requests multiple motion vectors and resubmits the best one. There is inherent serialization involved in this.
In the above example, software operatios are blocked on ProcessOutput and ProcessOutput2 half of the time, negatively effecting pipeline efficiency. On the other hand CPU utilization will be quite low, and the overall throughput is still higher than non-accelerated encode.
A 3-stage pipeline based on 3 threads will solve the serialization problem as follows:
Since there are 3 pipeline stages, additional buffer is added to pad between the two hardware stages. Hence the two values k1 and k2.
This application is a continuation-in-part of co-pending U.S. patent application Ser. No. 11/276,336 filed on Feb. 24, 2006, titled “Accelerated Video Encoding”, and hereby incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5805228 | Proctor et al. | Sep 1998 | A |
| 5926226 | Proctor et al. | Jul 1999 | A |
| 5990958 | Bheda et al. | Nov 1999 | A |
| 6044408 | Engstrom et al. | Mar 2000 | A |
| 6101276 | Adiletta et al. | Aug 2000 | A |
| 6128026 | Brothers, III | Oct 2000 | A |
| 6252905 | Pokrinchak et al. | Jun 2001 | B1 |
| 6434196 | Sethuraman et al. | Aug 2002 | B1 |
| 6721359 | Bist et al. | Apr 2004 | B1 |
| 6748019 | Lin et al. | Jun 2004 | B1 |
| 7634776 | Parameswaran et al. | Dec 2009 | B2 |
| 20020065952 | Sullivan et al. | May 2002 | A1 |
| 20020114394 | Ma | Aug 2002 | A1 |
| 20030052909 | Mo et al. | Mar 2003 | A1 |
| 20030118103 | Guevorkian et al. | Jun 2003 | A1 |
| 20030156643 | Song | Aug 2003 | A1 |
| 20030195998 | Estrop | Oct 2003 | A1 |
| 20040228405 | Yu et al. | Nov 2004 | A1 |
| 20040268397 | Dunbar et al. | Dec 2004 | A1 |
| 20050024363 | Estrop | Feb 2005 | A1 |
| 20050025241 | Sullivan et al. | Feb 2005 | A1 |
| 20050063461 | Lee et al. | Mar 2005 | A1 |
| 20050078119 | Chiaruzzi et al. | Apr 2005 | A1 |
| 20050094730 | Chang et al. | May 2005 | A1 |
| 20050123057 | MacInnis et al. | Jun 2005 | A1 |
| 20060056708 | Shen et al. | Mar 2006 | A1 |
| 20060197765 | Dutta et al. | Sep 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 1347650 | Sep 2003 | EP |
| WO0207446 | Jan 2001 | WO |
| WO0219095 | Mar 2002 | WO |
| Entry |
|---|
| Rossi, “Multicore Signal Processing Platform with Heterogeneous Configurable Hardware Accelerators”, 2013, IEEE, p. 1-14. |
| “CinePlayer DVD Decoder Pack for Windows XP; Summary”, http://www.sonic.com/products/Consumer/CinePlayer/Technology/acceleration.aspx. 14 pages. |
| http://www.hisdigital.com/html/9000series—brochure.pdf. |
| http://msdn.microsoft.com/library/default.asp?url=/library/en-us/graphics/hh/graphics/dxvaguide—d531ae3a-1132-44e9-9931-d0bdd76e676f.xm.asp. |
| http://www.nforcershq.com/article-print-891.html. |
| “OC Workbench”, http://www.ocworkbench.com/hardware/Leadtek/winfast%20gts/winfast2.htm, 2 pages. |
| “Sage TV; Why does SageTV “look” better than it's competition? (picture quality)”, http://htpcnew.com/main/php? id=interview1, 3 pages. |
| “Why MetaVR Uses Microsoft DirectX”, http://www.metavr.com/technology/directx.html, 3 pages. |
| “Windows Media; Enabling DirectX Video Acceleration”, http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wmform95/htm/enablingdirectxvideoacceleration.asp, 3 pages. |
| “CinePlayer DVD Decoder Pack for Windows XP; Summary”, http://www.sonic.com/products/Consumer/CinePlayer/Technology/acceleration.aspx. Feb. 23, 2006, 14 pages. |
| Excalibur RADEON, http://www.hisdigital.com/html/9000series—brochure.pdf downloaded at least as early as Jun. 15, 2005. |
| http://msdn.microsoft.com/library/default.asp?url=/library/en-us/graphics/hh/graphics/dxvaguide—d531ae3a-1132-44e9-9931-d0bdd76e676f.xm.asp downloaded at least as early as Jun. 15, 2005. |
| http://www.nforcershq.com/article-print-891.html downloaded at least as early as Jun. 15, 2005. |
| “OC Workbench”, http://www.ocworkbench.com/hardware/Leadtek/winfast%20gts/winfast2.htm, Feb. 23, 2006, 2 pages. |
| “Sage TV; Why does SageTV “look” better than it's competition? (picture quality)”, http://htpcnew.com/main/php?id=interview1, Feb. 23, 2006, 3 pages. |
| “Why MetaVR Uses Microsoft DirectX”, http://www.metavr.com/technology/directx.html, Feb. 23, 2006, 3 pages. |
| “Windows Media; Enabling DirectX Video Acceleration”, http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wmform95/htm/enablingdirectxvideoacceleration.asp, Feb. 23, 2006, 3 pages. |
| Barjatya, “Block Matching Algorithms for Motion Estimation”, Digital Image Processing in Natural Sciences and Medicine, No. class ece 6620, Apr. 26, 2004, pp. 1-6. |
| Extended European Search Report mailed Jul. 5, 2012 for European patent application No. 07751404.0, 16 pages. |
| Harrand, et al., “A Single-Chip CIF 30-Hz, H261, H263, and H263+ Video Encoder/Decoder with Embedded Display Controller”, IEEE Journal of Solid-State Circuits, IEEE Service Center, Piscataway, NJ, USA, vol. 34, No. 11, Nov. 1, 1999. |
| Kelly, et al., “Fast Image Interpolation for Motion Estimation using Graphics Hardware”, Real-Time Imaging VIII, Jan. 20-22, 2004, San Jose, CA, USA, Proceedings of Electronic Imaging, Science and Technology, Jan. 20, 2004, pp. 184-194. |
| Korean Office Action mailed Apr. 24, 2012 for Korean patent application No. 10-2008-7020009, a counterpart foreign application of US patent No. 7,929,599, 3 pages. |
| Pastrnak, et al., “Hierarchical QoS concept for Multiprocessor System-on-chip” Workshop on Resource Management for Media Processing in Networked Embedded Systems, Mar. 31, 2005, pp. 139-142. |
| Sullivan, et al., “Windows Platform Design Notes: Designing Hardward for the Microsoft Family of Operating Systems—Microsoft DirectX VA: Video Acceleration API/DDI” Jan. 23, 2001, pp. 1-88, retrieved from the internet at http://download.microsoft.com/download/1/6/1/161ba512-40e2-4cc9-843a-923143f3456c/. |
| Yang, et al., “Search Speed and Power Driven Integraged Software and Hardware Optimizations for Motion Estimation Algorithms”, 2004 IEEE International Conference on Multimedia and Expo, Jun. 27-30, 2004, Taipei, Taiwan, IEE Operations Center, Piscataway, NJ, vol. 1, pp. 707-710. |
| JP Office Action mailed Dec. 25, 2012 for JP patent application No. 2008-556422 a counterpart foreign application of US patent No. 7,929,599, 8 pages. |
| Shen et al., “Accelerate Video Decoding With Generic GPU”, IEEE Transaction on Circuits and Systems for Video Technology, IEEE, May 2005, vol. 15, No. 5, pp. 685-pp. 693. |
| Cheng et al., “Fast Blocking Matching Algorithms for Motion Estimation”, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processings Proceedsings, Alanta, May 7-10, 1996, pp. 2311-2314. |
| European Office Action mailed May 25, 2013 for European Patent Application No. 07751404.0, a counterpart foreign application of US Patent No. 7,929,599, 7 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20070201562 A1 | Aug 2007 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11276336 | Feb 2006 | US |
| Child | 11673423 | US |
