Patent Application
20080130742
Video information is ubiquitous in today's world. Children learn from television shows and educational lesions prepared for video. Adults use video for entertainment and to keep informed with current events. Digital versatile disks (DVDs), digital cable, and satellite television use digital video data, in contrast to the older analog mechanisms for recording and distributing video information. Digital video data is becoming more and more prevalent in today's home and office environment.
The amount of numeric computation involved in processing digital video data requires an enormous amount of computational power. Generating digital video data of typical quality for one second's worth of video requires performing between tens of millions and a billion arithmetic computations.
Hardware can be used to speed up video computations, compared with software encoders, decoders, and transcoders for digital video data. However, typical approaches to hardware design operate only with video data in one particular format at one particular resolution. Thus there is a need for hardware that works with video data of different resolutions, standards, formats, etc.
Various embodiments of the invention provide systems and methods for processing video data. In a preferred embodiment, a video processor is configured at run time so as to operate on video data having various attributes. These streams can have various combinations of attributes, including format, standardization, resolution, encoding, compression, or other attributes. The video processor operates on the various streams of video data according to a dynamic control mechanism including, but not limited to, a program or dynamically configurable values held in a register.
Some embodiments of the invention provide a video processor that can be dynamically configured via a sequence of instructions, where the instructions include information on the attributes of the current video data. This information configures the video processor to receive video data with specified attributes, to generate video data with specified attributes, or both.
The operation of the video processor is controlled by at least one sequence of instructions. Multiple sequences may be employed concurrently, thereby enabling the processor to concurrently generate and/or receive video streams of different attributes.
In some embodiments of the invention, one or more control processors provide the instruction sequences to the video processor. The control processors and the video processor are adapted to operate together as coprocessors. Alternatively or additionally, the instruction sequences may be stored in one or more instruction memories or queues associated with the video processor.
Some embodiments of the invention provide an adder array that can be dynamically configured via control mechanisms including but not limited to instructions, register values, control signals, or programmable links. As the processor sends video data through the adder array, the array generates the numerical, logical, or sequential computational results required to process the video data.
Adder arrays are used, in some embodiments of the invention, to compute difference functions and thereby generate error vectors. Error vectors may be used to detect motion among sets of video data that are temporally related. Error vectors may also be used to compute residual data that is encoded in the output video.
Adder arrays are also used, in some embodiments of the invention, to compute filtering functions that are applied to the video data. Such filtering functions include, but are not limited to, interpolation or decimation of the incoming video prior to motion detection.
Decimated or interpolated PELs may be used to generate output video having different attributes than the input video. Alternatively or additionally, decimated or interpolated PELs that persist only during an encoding or compressing process may be used to increase the accuracy or perceived quality of the output video. Such uses include but are not limited to: hierarchical techniques to increase the performance of motion detection based on reducing the resolution of the input video for an initial top-level motion scan; or interpolated techniques for increasing the accuracy of motion detection.
Some embodiments of the invention provide dynamically configurable hardware search operations. Such operations can be used when encoding video for various purposes, including but not limited to, detecting motion among sets of video data that are temporarily related. A single hardware operation compares a reference block within one set of video data to a set of regions within the same or a different set of video data. In addition to the attributes of the video data being dynamically configurable, the number of regions within the set and the relative offset among the various regions is dynamically configurable.
Some embodiments of the invention are implemented using adaptive computing machines (ACMs). An ACM includes a plurality of heterogeneous computational elements, each coupled to an interconnection network.
Objects, features, and advantages of various embodiments of the invention will become apparent from the descriptions and discussions herein, when read in conjunction with the drawings. Technologies related to the invention, example embodiments of the invention, and example uses of the invention are illustrated in the following figures:
FIG. A1 shows a high level functional block diagram of a video processor according to an embodiment of the invention.
FIG. A2 shows a domain video node (DVN) according to an embodiment of the invention.
FIG. A3 shows how the center search position is defined according to an embodiment of the invention.
FIG. A4 shows a search region within a 48×48 reference block according to an embodiment of the invention.
FIG. A5 shows a process of searching the center 8×5 positions concurrently according to an embodiment of the invention.
FIG. A6 shows a process of searching the center nine sets of 8×5 positions according to an embodiment of the invention.
FIG. A7 shows a process of half-PEL bilinear interpolation according to an embodiment of the invention.
FIG. A8 shows a process of full-PEL to half-PEL bilinear interpolation according to an embodiment of the invention.
FIG. A9 shows a half-PEL numbering convention according to an embodiment of the invention.
FIG. A10 shows a process of h.264 six-tap-filter-based half PEL interpolation according to an embodiment of the invention.
FIG. A11 shows a process of h.264 half PEL interpolation based on a six tap filter according to an embodiment of the invention.
FIG. A12 shows some of the elements of an adaptive computing machine (ACM) node according to an embodiment of the invention.
FIG. A13 shows some of the hardware elements within the domain video node (DVN) programming model according to an embodiment of the invention.
FIG. A14 shows a state transition table for a finite state machine (FSM) according to an embodiment of the invention.
FIG. A15 shows a task parameter list (TPL) for a DVN according to an embodiment of the invention.
FIG. A16 shows a format for locations 0x00 to 0x03 within a TPL according to an embodiment of the invention.
FIG. A17 shows a format for locations 0x04 and 0x05 within a TPL according to an embodiment of the invention.
FIG. A18 shows a format for acknowledge+test parameters within a TPL according to an embodiment of the invention.
FIG. A19 shows a format for setup and continue registers according to an embodiment of the invention.
FIG. A20 shows a format for setup and continue byte codes according to an embodiment of the invention.
FIG. A22 shows formats for “typical” task TPL entries and byte codes according to an embodiment of the invention.
FIG. A23 shows a memory map, according to an embodiment of the invention, for a DVN memory organized as eight banks having 512 words of 32 bits.
FIG. A24 shows formats for command buffer data structures according to an embodiment of the invention.
FIG. A25 shows formats for “to controller” status words according to an embodiment of the invention.
FIG. A26 shows formats for reference block longwords according to memory and clock cycles for M-by-N 5×5 search ranges according to an embodiment of the invention.
FIG. A27 shows formats for fractional PEL cost functions and tables according to an embodiment of the invention.
FIG. A28 shows a memory layout for a reference block having M=50 columns and N=50 rows according to an embodiment of the invention.
FIG. A29 shows a memory layout for a current block of size 16 PELs by 16 PELs according to an embodiment of the invention.
FIG. A30 shows a search region within a 50 by 50 reference block according to an embodiment of the invention.
FIG. A31 shows a process of searching the center 5 by 5 positions concurrently according to an embodiment of the invention.
FIG. A32 shows a process of searching the center 9 sets of 5 by 5 positions concurrently according to an embodiment of the invention.
FIG. A33 shows a process of exhaustive search of forty nine sets of 5 by 5 positions according to an embodiment of the invention.
FIG. A34 shows reference block sizes for plus/minus 17 search, without reloading of the pre-interpolation buffer, according to an embodiment of the invention.
FIG. A35 shows full PEL search areas that require pre-interpolation buffer reloading according to an embodiment of the invention.
FIG. A36 shows fill PEL to half PEL bilinear interpolation according to an embodiment of the invention.
FIG. A37 shows a half-PEL bilinear interpolated array according to an embodiment of the invention.
FIG. A38 shows a 6-tap-filter-based half PEL interpolation for H.264 video according to an embodiment of the invention.
FIG. A39 shows a summary of a half PEL interpolation finite impulse response (FIR) filter according to an embodiment of the invention.
FIG. A40 shows a half PEL numbering convention according to an embodiment of the invention.
FIG. A41 shows a summary of a sub-PEL interpolation FIR filter for Microsoft Windows® media video (WMV) according to an embodiment of the invention.
FIG. A42 shows a quarter PEL numbering convention according to an embodiment of the invention.
FIG. A43 shows an execution unit for a DVN according to an embodiment of the invention.
The descriptions, discussions and figures herein illustrate technologies related to the invention and show examples of the invention and of using the invention. Known methods, procedures, systems, circuits, or elements may be illustrated and described without giving details so as to avoid obscuring the principles of the invention. On the other hand, details of specific embodiments of the invention are described, even though such details may not apply to other embodiments of the invention.
Some descriptions and discussions herein use abstract or general terms including but not limited to receive, present, prompt, generate, yes, or no. Those skilled in the art use such terms as a convenient nomenclature for components, data, or operations within a computer, digital device, or electromechanical system. Such components, data, and operations are embodied in physical properties of actual objects including but not limited to electronic voltage, magnetic field, and optical reflectivity. Similarly, perceptive or mental terms including but not limited to compare, determine, calculate, and control may also be used to refer to such components, data, or operations, or to such physical manipulations.
FIG. A1 is a high level functional block diagram of a video processor according to an embodiment of the invention. Video processor 100 includes one or more video data devices 110 that hold or provide network access to video data, one or more video computation engines 120, one or more devices 130 that hold current status information, one or more control modules 140, and one or more control & configuration data devices 150 that hold or provide network access to control & configuration data.
Video data 110 includes, but need not be not limited to, input video data 112, output video data 114, and intermediate results 116. In various embodiments of the invention, video data 110 is held in one or more of: memory modules within the same integrated circuit as the rest of video processor 100; memory devices external to that IC; register files; or other circuitry that holds data.
Video computation engine 120 includes, but need not be limited to the following: input multiplexer 122; filter function circuit 124; motion search circuit 126; error, e.g. sum of absolute differences (SAD) computation circuit 127; and output formatter circuit 128. Motion estimation engine 120 receives input video data 112 and intermediate results 116, and generates intermediate results 116 and output video 114.
The processing functions and operations that video computation engine 120 applies to the data received in order to generate results 116 and 114 is controlled by control signals 146. Under the control of signals 146, various embodiments of engine 120 perform various functions including but not limited to encoding, decoding, transcoding, and motion estimation.
The attributes of video input 112, video output 114, and intermediate results 116 are specified by control signals 146. These attributes include, but need not be limited to, format, standardization, resolution, encoding, or degree of compression.
Control signals 146 are generated by control module 140 based on control & configuration data 150, and current status data 130. Control module 140 includes, but need not be limited to, a hardware task manager 142, finite state machine 144, and a command decoder 146.
Current status 130 includes, but need not be not limited to, indications that various items within video data 110 are ready to be processed, and indications that various data items within control and configuration data 150 are ready to control the processing operations that are applied to video data 110. Current status 130 implements a data flow model of computation, in that processing occurs when the video data and the control & configuration data is ready.
Control and configuration data 150 includes, but need not be not limited to, command sequences 152, task parameter lists 154, and addresses and pointers 156. Addresses and pointers 156 refer to various data items within video data 110, control and configuration data 150, or both. In various embodiments of the invention, video data 110 is held in one or more of: memory modules within the same integrated circuit as the rest of video processor 100; memory devices external to that IC; register files; or other circuits or devices that hold data.
The various components of video data 110 have various attributes, including but not limited to format, standardization, and resolution. A component within video data 110 may have a format that includes at least one of a version of MPEG-2, MPEG-4, a version of Windows media video (WMV), or a version of the International Telecommunications Union (ITU-T) standard X.264. X.264 is also known as joint video team (JVT) after the group who is leading the development of the standard. X.264 is also know as the International Standards Organization/International Electrotechical Commission (ISO/IEC) standard Motion Picture Experts Group Layer 4 (MPEG-4) part 10. A component within video data 110 may have a resolution that includes any resolution between about one quarter CEL (as would be used, for example by a still picture taken by cell phone) and about a resolution defined by a version of high definition television (HDTV).
An adaptive computing machine (ACM) includes heterogeneous nodes that are connected by a homogeneous network. Such heterogeneous nodes include, but are not limited to domain video nodes (DVNs). Such a homogeneous network includes, but is not limited to a matrix interconnect network (MIN). Further description of some aspects of some embodiments of the invention, such the MIN and other aspects on adaptive computing machines (ACMs) are described in the patents and patent applications cited above as related applications.
The domain video node (DVN) can be implemented as a node type in an ACM, or it can be implemented in any suitable hardware and/or software design approach. For example, functions of embodiments of the invention can be designed as discrete circuitry, integrated circuits including custom, semi-custom, or other; programmable gate arrays, application-specific integrated circuits (ASICs), etc. Functions may be performed by hardware, software or a combination of both, as desired. Various of the functions described herein may be used advantageously alone, or in combination or subcombination of other functions, including functions known in the prior art or yet to be developed. The DVN is a reconfigurable digital video processor that achieves a power-area-performance metric comparable to an ASIC.
In one embodiment, the DVN is included in an ACM with the additional benefit of reconfigurability, enabling it to execute a number of commonly used motion estimation algorithms that might require multiple ASICs to achieve the same functionality. However, other embodiments can implement the DVN, or parts or functions of the DVN, in any suitable manner including in a non-configurable, or partially configurable design.
Video processing includes a motion estimation step that requires considerable computational power, especially for higher frame resolutions. The Domain Video Node (DVN) is a type of an ACM node that provides for performing the motion estimation function and other video processing functions.
A simplified block diagram of the DVN is shown in FIG. A2. Motion estimation consists of comparing a block of picture elements (i.e. PELs) of a current frame with a block from a previous frame, a subsequent frame, or both to locate the block(s) with the lowest distortion. Such comparing can, for example, include the commonly used ‘sum of absolute differences’ (SAD) metric for the selection criterion.
The DVN includes eight 2 KB node memories that allow for sixteen SAD operations each clock cycle. This also allows double buffering for overlapped data input and computation/results output for 16×16 current blocks (macroblocks) and 48×48 reference blocks for search areas of +/−16 PEL in each dimension.
DVN operations include: 1) Full PEL search; 2) 18×18 full PEL to 33×33 half PEL bilinear interpolation; 3) Half PEL search; 4) Output best metric and origin of the associated 16×16 macroblock to the specified node, and output 16×16 macroblock itself to a decoder motion compensation engine.
Full PEL Search: The DVN supports two modes of operation for full PEL search. One mode evaluates each 16×16 position one at a time, performing 16 SAD operations every clock period; the second mode evaluates forty contiguous positions concurrently, performing 160 SAD operations every clock period. For both modes, evaluation consists of calculating the ‘sum of absolute differences’ (SAD) metric between the current 16×16 macroblock and one-of-1089 16×16 macroblocks from the 48×48 reference block, always saving the best metric and the origin of the associated 16×16 macroblock.
For each mode, the search pattern can be a number of hardwired patterns or a number of positions specified by a list of x,y coordinates (origins) generated by the node and stored in a queue located in the DVN. Various queue data structures can be used and various early completion strategies can be supported.
There are 1089 candidate search positions within the 48×48 reference block. The ‘center’ position, with relative coordinates H(0), V(0), is shown in FIG. A3. Five search positions are highlighted in FIG. A4: the center position, H(0), V(0); the upper-leftmost position, H(−16), V(−16); the upper rightmost position, H(+16), V(−16); the lower leftmost position, H(−16), V(+16); and the lower rightmost position, H(+16), V(+16).
An example for searching 40 positions concurrently—the 8×5 positions at the center of the 48×48 reference block—is shown in FIG. A5.
An example for searching nine sets of 40 positions at the center of the reference block is shown in FIG. A6. The horizontal search range is −12 to +11, the vertical search range is −7 to +7, and the total number of positions searched is 360.
Half PEL Bilinear Interpolation: After full PEL search for the 16×16 macroblock with the best distortion metric, bilinear interpolation is performed on the surrounding 18×18 PEL array, yielding a 33×33 half PEL array, as shown in FIG. A7. Calculations used for half PEL bilinear interpolation are summarized in FIG. A8.
Half PEL Search: After performing bilinear interpolation, the DVN calculates a SAD metric for each of the nine candidate half PEL positions. A numbering convention for the half PEL data is shown in FIG. A9. Note from the figure that the metric for the full PEL position (position ‘5’) has already been calculated. Therefore, half PEL search consists of generating the SAD metrics for each of the other eight half PEL positions to determine which of the nine candidates has the best metric.
Output the Results: Output the best metric and the origin of the associated 16×16 macroblock within the 33×33 half PEL array to the specified node. Output the 16×16 macroblock itself to the decoder's motion compensation engine.
DVN Summary of Operation: The DVN includes the QST Node Wrapper, eight 2 KB node memories (each organized as 512 words by 32 bits), and a motion estimation engine (ME). The ME performs full PEL search, bilinear interpolation, and half PEL search as outlined above. It outputs the best metric and the origin of the associated 16×16 macroblock to the specified node; and it outputs the 16×16 macroblock itself to the decoder's motion compensation engine.
Each of eight node memories is partitioned into a ping pong buffer pair to allow fully overlapped data input and computation/results output. After one data set has been transferred from the specified memory to the eight DVN memories, the DVN ME performs its calculations and outputs its results. During this time, the next data set is transferred from another system memory to the eight DVN memories. And so on.
Each data set consists of a 48×48 PEL array (reference block) and a 16×16 PEL array (current block). These data are distributed among the eight node memories in a manner that optimizes the ME's access to the data. Generally, this includes access to four bytes of data from each of eight node memories every clock period. This allows sixteen SAD operations to be performed each clock period during ‘fast search’ mode, where one 16×16 search position is evaluated at a time. Additionally, for ‘forty concurrent search positions’ mode, it allows access to data from four rows of the reference block and data from four rows of the current block every clock period.
Node Memory Access Summary: Each data set is transferred from the specified memory to the eight DVN node memories via the matrix interconnect network (MIN) of the Adaptive Computing Machine (ACM) architecture, controlled by the specified node. For the 48×48 PEL reference block: Rows 0, 4, 8, . . . , 44 are written into sequential locations within node memory one; Rows 1, 5, 9, . . . , 45 are written into sequential locations within node memory two; Rows 2, 6, 10, . . . , 46 are written into sequential locations within node memory three; and Rows 3, 7, 11, . . . , 47 are written into sequential locations within node memory four. For the 16×16 PEL current block: Rows 0, 4, 8, and 12 are written into node memories one and five; Rows 1, 5, 9, and 13 are written into node memories two and six; Rows 2, 6, 10, and 14 are written into node memories three and seven; and Rows 3, 7, 11, and 15 are written into node memories four and eight.
During full PEL search, the ME reads reference block data from memories one through four, and it reads current block data from memories five through eight. During bilinear interpolation, the ME reads full PEL reference block data from memories one through four, and it writes reference block half PEL interpolated data into memories five through eight. During half PEL search, the ME reads reference block half PEL interpolated data from memories five through eight, and it reads current block data from memories one through four. Finally, to output the 16×16 macroblock to the decoder's motion compensation engine, the ME reads from memories five through eight and writes to the MIN.
Node Memory Organization (during data input to the node memories): For node memories one through four, reference block data will be stored sequentially at locations 0x00 to 0x8F; current block data will be stored sequentially at locations 0xC0 to 0xCF. For node memories five through eight, current block data will be stored sequentially at locations 0xC0 to 0xCF.
Node Memory Organization (during bilinear interpolation): For node memories five through eight, half PEL data for candidate position 1(7) will be stored sequentially at locations 0x00 to 0x10. Half PEL data for candidate position 2 (8) will be stored sequentially at locations 0x20 to 0x30. Half PEL data for candidate position 3 (9) will be stored sequentially at locations 0x40 to 0x50. Half PEL data for candidate position 4 will be stored sequentially at locations 0x80 to 0x8F. Half PEL data for candidate position 5 will be stored sequentially at locations 0x90 to 0x9F. Half PEL data for candidate position 6 will be stored sequentially at locations 0xA0 to 0xAF.
DVN Programming Model: All configuration and control information for the DVN will reside in its TPL. Conditional operation will employ a suitable number of the wrapper's 64 counters and a suitable data structure for the search queue. A suitable number and nature of the hardwired search patterns for both full PEL search modes can be used.
Future DVN capabilities can, for example, include but are not limited to: increased SAD strength for faster full PEL search; ¼ PEL processing; and bicubic interpolation.
The adaptive computing machine (ACM) includes heterogeneous nodes that are connected by a homogeneous network. Each node includes three elements: wrapper, memory, and execution unit as shown in FIG. A12.
The domain video node (DVN) is a reconfigurable digital video processor that achieves a power-area-performance metric comparable to application specific integrated circuits (ASIC), with the additional benefit of reconfigurability, enabling it to execute a number of commonly used motion estimation algorithms that might require multiple ASICs to achieve the same functionality.
A block diagram that depicts the DVN programming environment is shown in FIG. A13. High level control of the DVN is provided by the hardware task manager (HTM) in the DVN node wrapper and a finite state machine (FSM) in the DVN EU. This interface includes the ACM-standard EU_RUN, EU_CONTINUE, and EU_TEARDOWN signals from the wrapper to the EU, and the ACK+TEST and EU_DONE responses from the EU to the wrapper.
The transition table for the FSM is shown in FIG. A14. The FSM assumes one of six states: Idle; Setup; Execute; ACK+TEST; Wait; or Continue. When the node is not enabled or the execution unit (EU) is not enabled, the FSM will enter, and remain in, its idle state. The FSM will also transition to its idle state whenever the node wrapper asserts the ‘eu_abort’ signal. Additionally, the FSM transitions to the idle state when it exits the wait state in response to the node wrapper's assertion of the ‘eu_teardown’ signal.
A typical operational sequence for the FSM will include the following transitions:
The above sequence returns control of the FSM to the node wrapper's HTM after ACK+TEST. The DVN also can be programmed to retain control after ‘ack-ing’. Options include conditional/unconditional state transitions to continue or idle (after asserting ‘eu_done’). Test conditions include the sign (msb) of any of the node wrapper's 64 counters, the sign (msb) of certain TPL entries, and the sign (msb) of a byte code controlled 5-bit down counter.
All of the information required for task execution is contained in a ‘task parameter list’ (TPL) located in node memory. During setup, DVN hardware retrieves from its node memory a pointer to the TPL. Pointers for up to 31 tasks are stored in 31 consecutive words in node memory at a location specified by the node wrapper signals {tpt_br[8:0], active_task[4:0]}.
Each DVN TPL is assigned 64 longwords of node memory. The first eight entries of the TPL are reserved for specific information. The remaining 56 longwords are available to store parameters for setup, ACK+TEST, and continue.
TPL location 0x00 contains two parameters for setup: the starting address in node memory for its byte code and the offset from the TPL pointer to its parameters. TPL location 0x01 contains two parameters for ACK+TEST: the starting address in node memory for its byte code and the offset from the TPL pointer to its parameters. TPL location 0x02 contains one parameter for teardown: the starting address in node memory for its byte code. (Currently, there is nothing to ‘teardown’, so this is a placeholder in the event that there is a requirement later on). TPL location 0x03 contains three parameters for continue: the starting address in node memory for its byte code and the offsets from the TPL pointer to its two sets of parameters. TPL locations 0x04 and 0x05 are used for a pair of semaphores that the byte code can manipulate; and locations 0x06 and 0x07 are reserved. TPL locations 0x08 through 0x3F are available for storing all other task parameters.
The layout for the DVN task parameter list is shown in FIG. A15. The TPL stores sets of parameters for each of three states: setup, ACK+TEST, and continue. For the continue state, there can be two sets of parameters, with each set having the same number of parameters but different TPL pointer offsets. This allows for the conditional selection between one-of-two sets of global parameters, such as consumer node(s) destination, fractional PEL interpolation filters, and so on. The set of parameters for a specific state must be stored in consecutive locations in the TPL. As byte code LOAD instructions execute, the DVN hardware selects the next parameter in the set. The offset from the TPL pointer to the location of the first parameter for that state must be in the range: 0x08 to 0x3F.
DVN TPL Memory/Register Formats: The format for TPL locations 0x00 through 0x03 that contain byte code starting addresses and TPL pointer offsets is shown in FIG. A16. The format for the two semaphores at TPL locations 0x04 and 0x05 is shown in FIG. A17. The format for the ACK+TEST parameters beginning at TPL location (TPL pointer plus ACK+TEST offset) is shown in FIG. A18. The formats for the registers that are loaded with TPL parameters during setup and continue are shown in FIG. A19.
Control and Status Register (CSR): The DVN CSR includes fields which specify which interpolation filters to use, how to interpolate at reference block edges, which of 64 flags indicates ‘command queue redirection’, which ping/pong buffer pairs to use, and, for the reference block, the number of longwords per row.
Tables 1 and 2 describe bits [6:5] and bits [4:0].
Bits [13:8] are the command queue redirection flag. Always while processing a command queue, the DVN monitors continuously one of 64 wrapper counter sign bits indicated by bits [13:8]. When the counter sign bit is set to 1 ′b1, The DVN will clear the bit, send the ‘switch’ indication to the controller, suspend processing from command queue (A/B), and resume processing from command queue (B/A).
Bits [20:16] are buffer selectors. Each of the five bits selects between a pair of ping/pong buffers. Bit16 is the ‘Reference Block Buffer Selection,’ where 0 selects reference block A; and 1 selects reference Block B. Bit17 is the ‘Current Block Buffer Selection,’ where 0 selects current Block A; and 1 selects current block B. Bit18 is the ‘Command Queue Buffer Selection,’ where 0 selects command queue A; and 1 selects command queue B. Bit19 is the ‘Horizontal Cost Function Buffer Selection’ where 0 selects horizontal cost function table A; and 1 selects horizontal cost function table B. Bit20 is the ‘Vertical Cost Function Buffer Selection,’ where 0 selects vertical cost function table A; and 1 selects vertical cost function table B.
Bits [28:24] are the ‘Reference Block Longwords per Row.’ The number of longwords comprising a reference block row is required by the node memory addressing unit to support strides from row (k) to row (k+4).
Controller Node Locator Register (CNLR): The DVN communicates with its controller using the PTP protocol, which requires a port number (i.e. bits [5:0]) and a node id (i.e. bits [15:8]).
Destination Nodes Locator Register (DNLR): When processing command queue instructions that include output to destination (consumer) nodes, the DVN will use its one bit logical output port field to select one of two destination nodes: When the bit is set to 1 ′b0, its output will be directed to node id (i.e. bits(15:8]), port number (i.e. bits [5:0]); and when the bit is set to ″b1, its output will be directed to node id (i.e. bits [31:24]), port number (i.e. bits [21:16]).
DVN Byte Codes: The DVN overhead processor (OHP) interprets two distinct sets of byte codes: one for setup and continue and another for ACK+TEST. The FSM state indicates which set should be used by the byte code interpreter. For setup and continue, there is only one byte code: LOAD as shown in FIG. A20. There is a more comprehensive set of byte codes for ACK+TEST. These are summarized in
For the DVN, one of the programming steps requires writing byte code and generating task parameter lists. Byte code is always executed ‘in line’, and byte code must start on longword boundaries. Byte code execution is ‘little endian’; that is, the first byte code that will be executed is byte[7:0] of the first longword, then byte[15:8], then byte[23:16], then byte[31:24], then byte[7:0] of the second longword, and so on.
TPL/Byte Code Programming Example: The task parameter list and byte code for a ‘typical’ task are shown in FIG. A22. The TPL consists of 21 longwords, and there are 5 longwords of byte code. Setup requires three clock cycles to load registers from the TPL; DONE is indicated in the last of three cycles. ACK+TEST requires a minimum of ten clock cycles: two for producer acknowledgements, two for consumer acknowledgements, two (or more) to test (and wait, if necessary) for input buffer ready, two (or more) to test (and wait, if necessary) for output buffer ready, and two to test a counter sign to select one of two sets of configuration registers. Continue requires three clock cycles to update the CSR, CNLR, and DNLR configuration registers.
Command Queue Overview: The Programming Model for the DVN includes the creation of sequences of instructions that are stored in command queues in DVN node memory. These instructions control the motion estimation operations of full PEL search, fractional PEL interpolation, fractional PEL search, (perhaps macroblock signed differences) and the outputting of results.
The DVN memory map is shown in FIG. A23. For the processing of each macroblock, the DVN will be directed by an entry in its task parameter list (TPL) to fetch its instructions from one of two buffers in its node memory: Command Queue A; or Command Queue B. The data structures for the two will be identical. The intent is to store a fixed set of instructions in one (or both) static command queue(s), or to construct an interactive sequence of commands in one (or both) dynamic command queue(s), and to select between the two buffers on a macroblock-by-macroblock basis. Including a second command queue allows for terminating one sequence of instructions and continuing with a new sequence of instructions, while preserving the entire previous sequence of instructions for subsequent reuse.
Each of the two command queues will be transferred to DVN node memory using the standard ACM PTP (or DMA) protocols (consuming one or two of the available 32 input ports). Command Queue A will consist of 64 longwords (32 bits per longword) in node memory six, locations 0x00 to 0x3F, and Command Queue B will consist of 64 longwords in node memory seven, locations 0x00 to 0x3F.
The DVN controller node initializes Command Queue A (CQA) by writing 0x6C00 into the DVN wrapper port translation table (PTT) at the location corresponding to the assigned DVN input port number. This sets the CQA buffer to a size of 256 bytes (64 longwords) with a starting address of ‘memory=6, physical address=0x000’. The initialization value for Command Queue B (CQB)=0x6E00, ‘memory=7, physical address=0x000’. The controller loads CQA and CQB, 64-longword circular buffers, using PTP writes. It re-initializes the buffers by rewriting 0x6C00/0x6E00 into the appropriate locations in the DVN PTT. The DVN fetches CQA/CQB instructions using a 6-bit counter (initialized to zero).
The data structure for the command queues is shown in FIG. A24. Bit 31 is used to indicate a ‘valid’ instruction, and it must be set to 1 ′b1 in every instruction that the controller writes into the buffer. When the DVN reads a queue entry with bit 31 set to 1′ b0, it will stall until that bit has been set to 1 ′b1 or until it is redirected.
While processing an instruction sequence fetched from CQA/CQB, the controller can command the DVN to: 1) terminate its processing of instructions from one queue; 2) clear its pointer to the other queue; or 3) resume its processing from location zero of the other queue. This ‘switch’ command from the controller to the DVN utilizes one of 64 counter signs in the DVN node wrapper. The controller writes the appropriate PCT/CCT counter with the msb set to 1 ′b1. (We assume here that the bit had previously been initialized to 1 ′b0). The DVN continuously monitors this bit while processing. At DVN task setup, a six bit register is loaded from the TPL to select the appropriate one-of-64 counter signs to monitor. As part of its ‘switch’ routine, the DVN will clear this bit by sending the ‘self-ack, initialize counter’ MIN word, and it will send the ‘switch complete’ message to the controller (See FIG. A25).
Description of the DVN Command Queue Data Structures: The data structures used in the command queue of the DVN are described below with reference to FIG. A24.
Evaluate the Metric for a Single 16×16 Macroblock—Command 0x0: Determine metric for one position (X, Y), where 0,0 indicates the upper left PEL of the reference block; X is a positive integer in the range of 0 to 79, and Y is a positive integer in the range of 0 to 49. These positive integers indicate the displacements from the upper left PEL. The DVN performs the evaluation of the metric: SAD256 plus horizontal cost function plus vertical cost function. The cost functions are stored in node memory.
Determine Best Metric for One or More Sets of 5×5 Positions—Command 0x1: Determine the best metric from m(H)-by-n(V) 5×5 sets of positions starting at position (X, Y). X is a positive integer in the range of 0 to 79, and Y is a positive integer in the range of 0 to 49. m is a positive integer in the range of 1 through 16; n is a positive integer in the range of 1 through 10 (See FIG. A26).
Half PEL Interpolation—Command 0x2: For a 16×16 macroblock, perform full PEL to half PEL interpolation. Bit [24] is ‘Evaluate’: 0 means interpolate only; and 1 means interpolate and evaluate eight (nine) half PEL 16×16 macroblocks. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector to select macroblock; and 1 means use (X, Y) reference, i.e. bits [14:8] and bits [5:0], to select the macroblock.
Quarter PEL Interpolation—Command 0x3: Perform quarter PEL interpolation. Bit
is ‘Evaluate’: 0 means interpolate only; and 1 means interpolate and evaluate eight (nine) quarter PEL 16×16 macroblocks. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector from half PEL search to select the macroblock; and 1 means use half PEL buffer location, i.e. bits [11:0], to select the macroblock.
Full PEL Signed Difference—Command 0x4: Perform the signed difference between the 16×16 current block and a 16×16 full PEL macroblock selected from the reference block. Output to the selected destination the 256 values as 16-bit 2's complement integers in the range −255 to 255 packed in 128 longwords. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector from full PEL search to select the macroblock; and 1 means use (X, Y) reference, i.e. bits [14:8] and bits [5:0], to select macroblock. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each port, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Half PEL Signed Difference—Command 0x5: Perform the signed difference between the 16×16 current block and a 16×16 half PEL macroblock selected from the half PEL buffer. Output to the selected destination the 256 values as 16-bit 2's complement integers in the range −255 to 255 packed in 128 longwords. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector from half PEL search to select the macroblock; and 1 means use the half PEL buffer location, i.e. bits [11:0] to select the macroblock. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each one, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Quarter PEL Signed Difference—Command 0x6: Perform the signed difference between the 16×16 current block and a 16×16 quarter PEL macroblock selected from the quarter PEL buffer. Output to the selected destination the 256 values as 16-bit 2's complement integers in the range −255 to 255 packed in 128 longwords. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector from quarter PEL search to select the macroblock; and 1 means use the half PEL buffer location, i.e. bits [11:0] to select the macroblock. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each one, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Output a Full PEL 16×16 Macroblock—Command 0x7: Transfer the selected full PEL 16×16 macroblock from the reference block in DVN node memory to the indicated destination. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector from full PEL search to select the macroblock; and 1 means use the (X,Y) reference, i.e. bits [14:8] and bits [5:0], to select the macroblock. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each one, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Output a Half PEL 16×16 Macroblock—Command 0x8: Transfer the selected half PEL 16×16 macroblock from the half PEL buffer in DVN node memory to the indicated destination. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector from the half PEL search to select the macroblock; and 1 means use the half PEL buffer location, i.e. bits [11:0] to select the macroblock. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each one, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Output a Quarter PEL 16×16 Macroblock—Command 0x9: Transfer the selected quarter PEL 16×16 macroblock from the quarter PEL buffer in DVN node memory to the indicated destination. Bit [25] is ‘Mode’: 0 means use ‘best metric’ vector from the quarter PEL search to select the macroblock; and 1 means use the quarter PEL buffer location, i.e. bits [11:0] to select the macroblock. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each one, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Output Best Metric—Command 0xA: Transfer the best metric (saturated unsigned 16 bit integer) to the indicated destination(s). Bit [25] is ‘Mode’: 0 means s end to controller only; and 1 means send to controller and send to indicated destination. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each one, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Output Motion Vector—Command 0xB: Transfer the motion vector, with quarter PEL resolution, to the indicated destination(s). Bit [25] is ‘Mode’: 0 means s end to controller only; and 1 means send to controller and send to indicated destination. Bit [26] is ‘Destination,’ which selects from one of two logical output ports. For each one, the TPL contains the associated routing, input port number, and memory mode indication for the selected destination.
Done—Command 0xC: The DONE command indicates to the DVN that there is no additional processing to be performed for the ‘current block’. After reading this command from the queue, the DVN will send the DONE status word to the controller; and the DVN FSM will transition to the ACK+TEST state.
Echo—Command 0xF: The ECHO command instructs the DVN to return a status word that is the command itself. This is intended to be a diagnostics aid.
DVN to Controller Status Word: After the DVN executes each command, it will send a status word to the controller, using the ACM PTP protocol. The DVN TPL will contain the appropriate routing field, input port number, and memory mode indication that is required to support this operation. This information is transferred to the DVN CNLR during ‘setup’ and ‘continue’ operations. A summary of the status word is shown in FIG. A25.
For Commands 0x0 and 0x1, the status word includes the best full PEL metric thus far for this particular current block. It also includes the corresponding vector, in full PEL units with a range of 0 to 79 for the horizontal dimension and 0 to 49 for the vertical dimension. For Command 0x2, full PEL to half PEL interpolation, when bit [24] of the command is set to 1 ′b1) (interpolate and evaluate), status word bits [15:0] indicate the value for the best half PEL metric, and bits [19:16] indicate with which of nine half PEL candidates the best metric is associated. (The half PEL numbering convention shown in FIG. A41 is used to code bits [19:16]). When bit [24] of the command is set to 1 ′b0 (interpolate only), bits [15:0] indicate the value for the previously evaluated best full PEL metric, and bits [19:16] are set to 0xF to indicate interpolation ‘done’.
For Command 0x3, half PEL to quarter PEL interpolation, when bit [24] of the command is set to 1 ′b1 (interpolate and evaluate), status word bits [15:0] indicate the value for the best quarter PEL metric, and bits [19:16] indicate with which of nine quarter PEL candidates the best metric is associated. (The quarter PEL numbering convention shown in FIG. A43 is used to code bits [19:16]). When bit [24] of the command is set to 1 ′b0 (interpolate only), bits [15:0] indicate the value for the previously evaluated best metric (either full PEL or half PEL if the latter has been evaluated), and bits [19:16] are set to 0xF to indicate interpolation ‘done’.
For Commands 0x4, 0x5, 0x6, 0x7, 0x8, and 0x9, the status word will be, respectively, 0xFF010000, 0×FF020000, 0×FF040000, 0×FF080000, 0×FF100000, and 0xFF200000 to indicate ‘done’. For Command 0xA, the status word will be the best metric, a saturated, unsigned 16 bit integer. For Command 0xB, the status word will be the motion vector, with quarter PEL resolution. For Command 0xC, the status word will be 0xFF400000 to indicate ‘done’. For Command 0xF, the status word will echo the command itself. This is intended to be a diagnostics aid. In response to the ‘command queue redirection’ flag, the status word will be 0xFF800000 to indicate ‘done’.
DVN Node Memory: The DVN includes eight 2 KB physical memories, each organized as 512 words-by 32 bits per word, for a total node memory capacity of 16 KB. This allows the reading of 32 bytes of video data and the processing of sixteen ‘sum of absolute differences’ of these data each clock cycle. The 16 KB capacity also allows double buffering for fully overlapped data input/output and computation.
Any ACM can write into DVN node memory using the PTT in the DVN wrapper and any of the three services: PTP, DMA, or RTI. Additionally, the Knode can PEEK/POKE DVN node memory.
The DVN node memory map is shown in FIG. A23. The allocated buffers are described in Table 3:
This assumes that reference blocks and current blocks are transferred from the ‘Data Mover’ to the eight DVN node memories via the MIN. This also assumes that the ‘Data Mover’ is controlled by a PSN node. Reference blocks are organized M-PEL per row by N-PEL per column. M and N are integer multiples of 5, consistent with the 5×5 positions-at-a-time search strategy. M has a range of 20 to 95 (20, 25, 30, . . . , 90, 95); N has a range of 20 to 65 (20, 25, 30, . . . , 60, 65).
Reference Block Options: A summary of allowable combinations for M and N that do not exceed the 3840 byte reference block capacity, along with the attendant number of exhaustive search processing cycles, is shown in FIG. A26.
For each M-PEL by N-PEL reference block transfer from the Data Mover to DVN node memory as follows: Rows 0, 4, 8, . . . , [M−(M)mod 4] are written into sequential locations in node memory zero; Rows 1, 5, 9, . . . , [M−(M−1)mod 4] are written into sequential locations in node memory one; Rows 2, 6, 10, . . . , [M−(M−2)mod 4] are written into sequential locations in node memory two; and Rows 3, 7, 11, . . . , [M−(M−3)mod 4] are written into sequential locations in node memory three.
For each 16-PEL by 16-PEL current block transfer from the Data Mover to DVN node memory: Rows 0, 4, 8, and 12 are written into node memories zero and four; Rows 1, 5, 9, and 13 are written into node memories one and five; Rows 2, 6, 10, and 14 are written into node memories two and six; and Rows 3, 7, 11, and 15 are written into node memories three and seven.
During full PEL search, the DVN motion estimation engine (ME) reads reference block data from memories zero through three, and it reads current block data from memories four through seven. During half PEL interpolation, the ME reads full PEL reference block data from memories zero through three, and it writes half PEL interpolated data into memories four through seven. When half PEL search is performed concurrently with half PEL interpolation, the ME reads current block data from memories four through seven. During quarter PEL interpolation, the ME reads half PEL interpolated data from memories four through seven, and it writes quarter PEL interpolated data into memories four through seven. When quarter PEL search is performed concurrently with quarter PEL interpolation, the ME reads current block data from memories zero through three. During output, the ME reads full PEL macroblocks from memories zero through three, fractional PEL macroblocks from memories four through seven, and for macroblock signed differences, current blocks from memories zero through three/four through seven for full PEL/fractional PEL macroblocks, respectively.
Cost Function Tables: For each full PEL search position, the DVN evaluates the metric: SAD256 plus horizontal cost function plus vertical cost function. The cost functions are stored in node memory. The cost function is an unsigned 16 bit integer.
For an M-column by N-row reference block, there are M horizontal cost functions and N vertical cost functions. In each horizontal cost function table, two 16 bit integers, a pair of horizontal cost functions, are packed in each 32 bit longword in the table. The cost functions for columns 0 and 1 are stored at table location 0, the cost functions for columns 2 and 3 are stored at table location 1, and so on. Cost functions for even numbered columns are stored in bits [15:0] of the longword; cost functions for odd numbered columns are stored in bits [31:16] of the longword.
In each vertical cost function table, two 16 bit integers, a pair of vertical cost functions, are packed in each 32 bit longword in the table. The cost functions for rows 0 and 1 are stored at table location 0, the cost functions for rows 2 and 3 are stored at table location 1, and so on. Cost functions for even numbered rows are stored in bits [15:0] of the longword; cost functions for odd numbered rows are stored in bits [31:16] of the longword. Fractional PEL cost functions and Cost Function Tables A and B are shown in FIG. A27.
Reference Block Overview: As shown in FIG. A23, two 3840 byte reference block buffers are allocated in DVN node memory. Each buffer is stored in four physical memories to allow access to 16 PEL each clock period. We will employ a numbering convention that is referenced to the upper left PEL, whose (H) horizontal and (V) vertical coordinates will be designated (H:0, V:0). The upper left PEL of reference block A will be stored in the high order byte of memory location 0x000; and the upper left PEL of reference block B will be stored in the high order byte of memory location 0x100. Rows 0, 4, 8, . . . of the reference block will be stored in memory 0; rows 1, 5, 9, . . . will be stored in memory 1; rows 2, 6, 10, . . . will be stored in memory 2; and rows 3, 7, 11, . . . will be stored in memory 3.
An M=50 column by N=50 row reference block will be stored in node memories 0 through 3 as shown in FIG. A28. Each 16 PEL by 16 PEL current block will be stored in node memories 0 through 3, and also in node memories 4 through 7, as shown in FIG. A29.
The 50 PEL by 50 PEL reference block allows for a search range of +/−17 PEL in each dimension (referenced to the center search position at ([H:17, V:17]) as shown in FIG. A30. There are 1225 candidate search positions within a 50 PEL by 50 PEL reference block. The ‘center’ position, with relative coordinates H(17), V(17), is shown in FIG. A30. Five search positions are highlighted in FIG. A30: the center position, H(17), V(17); the upper-leftmost position, H(0), V(0); the upper rightmost position, H(34), V(0); the lower leftmost position, H(0), V(34); and the lower rightmost position, H(34), V(34).
Searching the center 5×5 positions currently within such a reference block is shown in FIG. A31. In this case, the total number of positions searched is 25, and the search range is +/−2 PEL in each dimension. Searching the center nine sets of 5×5 positions within such a reference block is shown in FIG. A33. In this case, the total number of positions searched is 9×5×5=225 and the search range is: +/−7 PEL in each dimension. Exhaustively searching all 49 sets of 5×5 positions within such a reference block is shown in FIG. A34. In this case, the total number of positions searched is 49×5×5=1225, and the search range is: +/−17 PEL in each dimension.
When fractional PEL processing is included, the programmer may select between two strategies:
Full PEL Search: The DVN supports two modes of operation for full PEL search. One mode evaluates an arbitrary 16×16 position within the reference block, performing 16 SAD operations every clock period. The second mode evaluates m-by-n sets of (5×5=25) positions concurrently, performing 200 maximum (160 average) SAD operations every clock period. For both modes, ‘evaluation’ consists of calculating the ‘sum of absolute differences’ (SAD) metric between the current 16×16 macroblock and any 16×16 macroblock within the reference block plus a horizontal cost function plus a vertical cost function, always saving the best metric and the origin of the associated 16×16 macroblock. The search pattern is governed by instruction sequences that are written by a controlling PSN node into one of two command queues in DVN memory. Unique command codes for each queue entry indicate whether a single position or m-by-n 5×5 positions should be evaluated for a given (X,Y) origin.
Half PEL Interpolation: After full PEL search for the 16×16 macroblock with the best distortion metric, half PEL interpolation is performed on the surrounding ([16+n]×[16+n]) PEL array, where n is the filter order. For bilinear interpolation, WMV, H.264, and MPEG4, n=2, 4, 6, and 8, respectively. The calculations required for half PEL bilinear interpolation are summarized in FIG. A37. The 33×33 half PEL array that results from bilinear interpolation of the 18×18 full PEL array associated with the ‘best metric’ 16×16 macroblock is shown in FIG. A38. The 33×33 half PEL array that results from H.264 6-tap filter interpolation is shown in FIG. A39. A summary of supported half PEL interpolation FIR filters is shown in FIG. A40. For the half PEL positions of ½ PEL-horizontal-shift and ½ PEL-vertical-shift, the same filters are used on the full PEL, ½ PEL interpolated values, maintaining the full precision of these values.
Half PEL Search: After half PEL interpolation using one of the four filtering options, the half PEL metrics are calculated for each of nine candidate half PEL positions. The metric is the sum of the SAD256 metric plus the half PEL cost function. A numbering convention for the nine candidate half PEL positions is shown in FIG. A41. Note from the figure that the full PEL metric for the full PEL position (position ‘5’) has been calculated previously. However, the half PEL metric for position ‘5’ will be the its full PEL metric, minus its full PEL horizontal and vertical cost functions, plus its half PEL cost function for this position (see FIG. A27). Half PEL search consists of generating the metric of SAD+half PEL cost function for each of the nine candidates and selecting the one that has the best metric.
Quarter PEL Interpolation: After half PEL search for the 16×16 macroblock with the best distortion metric, quarter PEL interpolation is performed on the surrounding ([16+n]×[16+n]) PEL array, where n is the filter order. For WMV, n=4; for H.264, and MPEG4, which use bilinear interpolation, n=2. The WMV sub-PEL four tap FIR filters for the ¼ PEL position, the ½ PEL position, and the ¾ PEL position are summarized in FIG. A42.
Quarter PEL Search: After quarter PEL interpolation, the quarter PEL metrics are calculated for each of nine candidate quarter PEL positions. The metric is the sum of the SAD256 metric plus the quarter PEL cost function. A numbering convention for the nine candidate quarter PEL positions is shown in FIG. A43. Note from the figure that only nine-of-16 quarter PEL positions must be calculated. Note, too, that one metric for the nine candidate quarter PEL positions (position ‘5’) has been calculated previously. However, the quarter PEL metric for position ‘5’ will be its half PEL metric, minus its half PEL cost function, plus its quarter PEL cost function for this position (see FIG. A27). Quarter PEL search consists of generating the metric of SAD+quarter PEL cost function for each of the nine candidates and selecting the one that has the best metric.
Macroblock Signed Differences: The DVN also will be capable of calculating and outputting a 256 element array of 9-bit signed differences between any 16 PEL by 16 PEL macroblock in DVN node memory (full PEL, half PEL, or quarter PEL) and either 16 PEL by 16 PEL current block in DVN node memory. The array is output to a destination node as 128 longwords, where each longword is a packed pair of 16-bit signed integers representing the sign-extended 9-bit signed differences of the 16×16 macroblocks.
Summary of Programming Issues Related to the DVN: Referring to FIG. A13, the following are programming requirements for the DVN: construct and load the task parameter list (TPL) into DVN node memory; construct and load the byte code into DVN node memory; configure the DVN hardware task manager (HTM); direct the Data Mover to construct and transfer to DVN node memory the requisite reference blocks, current blocks, and cost function tables, along with the appropriate data flow acknowledgements; construct and transfer the requisite command queue instruction sequences to DVN node memory; configure the HTM in the DVN's controller node to support its processing of messages from the DVN; and configure DVN destination nodes (consumers) to receive output from the DVN and to send to the DVN the appropriate data flow acknowledgements.
DVN Hardware Overview: A simplified block diagram of the DVN execution unit is shown in FIG. A44. The DVN includes the QST Node Wrapper, eight 2 KB node memories (each organized as 512 words by 32 bits), and an execution unit (EU), consisting of an overhead processor (OHP) and a motion estimation engine (ME). The ME performs full PEL/half PEL/quarter PEL search, half PEL and quarter PEL interpolation, and macroblock signed differences. It outputs the best metric and the origin of the associated 16×16 macroblock to the controller node and a destination node; it outputs the associated 16×16 macroblock to a destination node performing the decoder motion compensation function; and it can output macroblock signed differences to a destination node.
Features of the DVN include: separate control units for the motion estimation engine and the overhead processor for maximum efficiency; compact byte code and efficient TPL load/store operation to minimize overhead for setup, ACK+TEST, and continue; finite state machine (FSM) based PLA control of motion estimation engine for superior power/area metrics compared with sequencer+control memory-based implementations; innovative algorithm implementations for enhanced performance and reduced power; eight physical memory block architecture to allow massive parallelism; and high-performance memory interfaces with ‘same write address/read address’ resolution.
The motion estimation engine includes several functional elements: control program unit (CPU); memory address generator unit (AGU); data path unit (DPU); sum of absolute differences (SAD), signed differences; multi-format sub-PEL interpolation; wrapper interface unit (WIU); and memory interface unit (MIU)
The DVN performs the following operations: 1) full PEL search over (up to) 80 PEL in the horizontal dimension and (up to) 50 PEL in the vertical dimension; 2) full PEL to 35×35 half PEL interpolation using one of four (perhaps more) filters; 3) half PEL search; 4) 35×35 half PEL to nine x16×16 quarter PEL interpolation; 5) quarter PEL search; and 6) output best metric and origin of the associated 16×16 macroblock to the controlling PSN node, and output 16×16 macroblock itself to the node performing decoder motion compensation; 7) calculate and output the 256 signed differences between a selected 16×16 macroblock (full PEL, half PEL, or quarter PEL) and the current 16×16 macroblock.
Each of eight node memories is partitioned into a ping pong buffer pair to allow fully overlapped data input and computation/results output. After one data set has been transferred from the Data Mover to the eight DVN memories, the DVN ME performs its calculations and outputs its results. During this time, the next data set is transferred from the Data Mover to the eight DVN memories. And so on.
Each data set includes a variably sized reference block, a 16×16 PEL array (current block) and fractional PEL cost function tables. These data are distributed among the eight node memories in a manner that optimizes the ME's access to the data. Generally, this includes access to four bytes of data from each of eight node memories every clock period. This allows sixteen SAD operations to be performed each clock period when one 16×16 search position is evaluated at a time. Additionally, for ‘twenty five concurrent search positions’ mode, it allows access to data from four rows of the reference block and data from four rows of the current block every clock period.
An adaptive computing machine (ACM) includes heterogeneous nodes that are connected by a homogeneous network. Each node includes three elements: wrapper, memory, and execution unit as shown in
The Domain Video Node (DVN) can be implemented as a node type in an ACM, or it can be implemented in any suitable hardware and/or software design approach. For example, functions of embodiments of the invention can be designed as discrete circuitry, integrated circuits including custom, semi-custom, or other; programmable gate arrays, application-specific integrated circuits (ASICs), etc. Functions may be performed by hardware, software or a combination of both, as desired. Various of the functions described herein may be used advantageously alone, or in combination or subcombination of other functions, including functions known in the prior art or yet to be developed. reconfigurable digital video processor that achieves a power-area-performance metric comparable to application specific integrated circuits (ASIC),
In a preferred embodiment, the DVN is included in an ACM with the additional benefit of reconfigurability, enabling it to execute a number of commonly used motion estimation algorithms that might require multiple ASICs to achieve the same functionality. However, other embodiments can implement the DVN, or parts or functions of the DVN, in any suitable manner including in a non-configurable, or partially configurable design.
Motion estimation creates a model of the current frame based on available data in one or more previously encoded frames. The technique used in the DVN (and in most video codecs) is block matching motion estimation as depicted in
A simplified block diagram of the DVN motion estimation engine is shown in
The DVN includes the QST Node Wrapper, eight 2 KB node memories (each organized as 512 words by 32 bits), and an execution unit (EU), consisting of an overhead processor (OHP) and a motion estimation engine (ME). The ME performs double pixel/full pixel/half pixel/quarter pixel search, half pixel and quarter pixel interpolation, and macroblock signed differences. It outputs the best distortion metric (motion vector) and the origin of the associated 16×16 macroblock (8×8 block) to the controller node and a destination node; it outputs the associated 16×16 macroblock (8×8 block) to a destination node performing the decoder motion compensation function; and it can output 16×16 macroblock (8×8 block) signed differences to a destination node.
Features of the DVN include:
The motion estimation engine includes five major functional elements:
The DVN performs the following operations:
Each of eight node memories is partitioned into a ping pong buffer pair to allow fully overlapped data input and computation/results output. After one data set has been transferred from the Data Mover to the eight DVN memories, the DVN ME performs its calculations and outputs its results. During this time, the next data set can be transferred from the Data Mover to the eight DVN memories. And so on.
Each data set includes a variably sized reference block, a 16×16 pixel (nominal) array (current block) and fractional pixel cost function tables. These data are distributed among the eight node memories in a manner that optimizes the ME's access to the data. Generally, this includes access to four bytes of data from each of eight node memories every clock period. This allows sixteen SAD operations to be performed each clock period when one 16×16 search position is evaluated at a time.
Additionally, for ‘twenty five concurrent search positions’ mode, it allows access to data from four rows of the reference block and data from four rows of the current block every clock period.
The DVN includes eight 2 KB physical memories, each organized as 512 words-by 32 bits per word, for a total node memory capacity of 16 KB. This allows the reading of 32 bytes of video data and the processing of sixteen ‘sum of absolute differences’ of these data each clock cycle. The 16 KB capacity also allows double buffering for fully overlapped data input/output and computation.
Any ACM resource can write into DVN node memory using the PTT in the DVN node wrapper and any one of the three services: PTP, DMA, or RTI. Additionally, the Knode can PEEK/POKE DVN node memory.
The DVN node memory map is shown in
The following buffers have been allocated within the available 16 KB of node memory:
We assume that reference blocks and current blocks are transferred from the ‘Data Mover’ to the eight DVN node memories via the MIN. We also assume that the ‘Data Mover’ is controlled by a TBD node.
As shown in
Reference blocks are organized M-pixels per row by N-pixels per column. Both M and N have a range of 29 to 100. A summary of allowable row/column combinations that do not exceed the 4032 byte reference block capacity is shown in
The DVN supports a number of motion estimation search strategies, and each strategy suggests different organizations of the reference blocks. For all three search strategies, ‘searching’ consists of calculating the ‘sum of absolute differences’ (SAD) metric between the current 16×16 macroblock (or 8×8 block) and any 16×16 macroblock (or 8×8 block) within the reference block plus a horizontal cost function plus a vertical cost function, always saving the best metric and the origin of the associated 16×16 macroblock (8×8 block).
The search pattern is governed by instruction sequences that are written by a controller node into the command queues in DVN memory. Unique command codes for each queue entry indicate the desired search mode.
The most general strategy allows for any one-position-at-a-time SAD256 operation on a 16 pixel by 16 pixel current block and any 16 pixel by 16 pixel macroblock within the reference block. For this general case, any combination of reference block rows and columns that does not exceed the 4032 byte reference block capacity is permitted.
A second search strategy evaluates at one time twenty five 16-by-16 pixel macroblocks within the reference block, using a 5-row by 5-column kernel. For this strategy, M (pixels per row) and N (pixels per column) should be integer multiples of 5.
Specifically, for a search range of m×5 by n×5 with an (B=16 pixel) by (B=16 pixel) macroblock:
M=m×5+[(B−1)=15], and
N=n×5+[(B−1)=15].
Allowable combinations for M and N that satisfy these equations and the 4032 byte buffer capacity constraint are shown in
The third search strategy, hierarchical search, down-samples an m-by-n portion (any or all) of the reference block, and stores the resulting m/2-by-n/2 array in the 1024 byte down-sampled reference block (also referred to as the ‘double pixel array’). The 16×16 current block also is down-sampled and the resulting 8-by-8 array is stored in the down-sampled current block buffer. The down-sampled reference block is exhaustively searched with the 8×8 down-sampled current block. The motion vector obtained from this step is used to search a small, 4-by-4 area in the reference block.
The relationship between the M-by-N reference block organization and the down-sampled reference block organization reflects the filter order, t, used in the down-sampling operation and the desired number of m×5 by n×5 searches in the double pixel array using the down-sampled (B=8) by (B=8) current block:
For t=4, 6, 8, the size of the reference block should be:
M=2×{m×5+[(B−1)=7]+(t−2)}, and
N=2×{m×5+[(B−1)=7]+(t−2)}
For resulting double pixel arrays of size:
N
H
=m×5+[(B−1)=7], and
N
V=n×5+[(B−1)=7].
Allowable combinations for M, N, m, n and t that satisfy these equations and the 4032 byte buffer capacity constraint are shown in
An M=50 column by N=50 row reference block will be stored in node memories 0 through 3 as shown in
The 50 pixel by 50 pixel reference block allows for a search range of +/−17 pixels in each dimension as shown in
Searching the center 5×5 positions concurrently within such a reference block is shown in
Searching the center nine sets of 5×5 positions within such a reference block is shown in
Exhaustively searching all 49 sets of 5×5 positions within such a reference block is shown in
A down-sampled 27×27 double pixel array is shown in
When the DVN is directed to generate a double pixel array from either of the 4032 byte reference blocks, it will read reference block data from memories 0 through 3, apply the specified filter, and write the double pixel array into the 1024 byte down-sampled reference block buffer in memories 4 through 7. For the 27-by-27 example of
Exhaustive search of the 27×27 double pixel array with a 5×5 kernel and 8×8 blocks requires 4×4=16 iterations to evaluate 16×25=400 positions. This is illustrated in
There are several buffers allocated for the ‘Current Block’. (See the DVN memory map,
When bilinear interpolation is used to down-sample the reference blocks and the current blocks, each current block buffer will contain a 16 pixel by 16 pixel macroblock, or 256 bytes.
When a 4-tap filter is used for down-sampling, each current block will contain a (16+2=18) pixel by (16+2=18) pixel array, or 324 bytes.
When a 6-tap filter is used for down-sampling, each current block will contain a (16+4=20) pixel by (16+4=20) pixel array, or 400 bytes.
When an 8-tap filter is used for down-sampling, each current block will contain a (16+6=22) pixel by (16+6=22) pixel array, or 484 bytes.
Each current block buffer is stored in four physical memories to allow access to 16 pixels each clock period. For the 16 pixel by 16 pixel current block array:
This is illustrated in
For the 18 pixel by 18 pixel current block array:
This is illustrated in
For the 20 pixel by 20 pixel current block array:
This is illustrated in
For the 22 pixel by 22 pixel current block array:
This is illustrated in
The down-sampling of these arrays to generate the 8×8 block used for double pixel search is illustrated in
The DVN programmer may chose bilinear interpolation or one of the even order filters shown in
When the DVN is directed to generate an 8×8 down-sampled current block from either of the 576 byte current block buffers, it will read current block data from memories 4 through 7, apply the specified filter, and write the 64 byte down-sampled current block buffer in memories 0 through 3 as shown in
For hierarchical search, the 64 byte down-sampled current block data in memories 0 through 3 is used to search the double pixel array stored in memories 4 through 7. The unpadded 256 byte Current Block data stored in memories 4 through 7 is used to search the reference block data stored in memories 0 through 3. In both cases, this memory organization allows access to 32 bytes of data each clock cycle.
When the DVN programmer favors exhaustive full pixel search over hierarchical search, several of the allocated buffers in the DVN memory map become available for other purposes. When hierarchical search is not employed, the following are not required:
After determining a motion vector from hierarchical search or exhaustive search, the DVN programmer may choose to improve the motion vector estimate with half pixel search. Additionally, the programmer may choose to further improve the motion vector estimate with quarter pixel search. When fractional pixel processing is performed, the programmer must select between two strategies:
1) Load an ‘oversized’ reference block with a sufficient number of pixels to support interpolation for any search outcome. The size of such an ‘oversized’ reference block can be determined by the size of the current macroblock (m), the search range (h, v) and the number of taps (n) for the half pixel interpolation filter. Specifically, the size will be [m+h+n−1] horizontal pixels by [m+v+n−1] vertical pixels. For example, for a 16×16 pixel macroblock, +/−17 pixel search range horizontally and vertically, and an H.264 6-tap interpolation filter, the reference block must be 56×56 pixels, (3136 bytes) as shown in
or
2) Load a reference block with the minimum number of pixels consistent with the desired search range. The size of such a reference block can be determined by the size of the current macroblock (m) and the search range (h, v). Specifically, the size will be [m+h−1] horizontal pixels by [m+v−1] vertical pixels. For example, for a 16×16 pixel macroblock and a +/−17 pixel search range horizontally and vertically, the reference block will be 50×50 pixels (2500 bytes), considerably smaller than the 3136 bytes required for strategy 1), above. Then, if the search outcome requires, reload the reference block buffer with [m+n] horizontal pixels by [m+n] vertical pixels before proceeding with the interpolation step. This is summarized in
Note that when the ‘four vector’ mode is a component of the search strategy, then the size of the reference block must be increased to accommodate the additional search range. For example, for 8×8 searches with +2, −2 search range, the H.264-based 56×56 reference block-depicted in
After hierarchical search or exhaustive search to estimate the full pixel motion vector, half pixel interpolation is performed on the surrounding ([16+n]×[16+n]) pixel array, where n is the filter order. For bilinear interpolation, WMV, H.264, and MPEG4, n=2, 4, 6, and 8, respectively.
The calculations required for half pixel bilinear interpolation are summarized in
The 35×35 half pixel array that results from bilinear interpolation of the 18×18 full pixel array associated with the ‘best metric’ 16×16 macroblock is shown in
The 19×19 half pixel array that results from bilinear interpolation of the 10×10 full pixel array associated with an 8×8 block is shown in
The half pixel array that results from H.264 6-tap filter interpolation is shown in
The half pixel array that results from H.264 6-tap filter interpolation for an 8×8 block is shown in
The half pixel array that results from MPEG4 8-tap filter interpolation is shown in
The half pixel array that results from MPEG4 8-tap filter interpolation for an 8×8 block is shown in
If one wishes to defer the 1MV/4MV decision until half pixel search has been performed, a 43×43 half pixel array referenced to the full pixel 16×16 motion vector allows one 16×16 search and four 8×8 searches. This is shown in
Note that the 43×43 array corresponds to full pixel 8×8 search ranges of +2, −2 pixels and half pixel search ranges of +½, −½ pixels for both 16×16 and 8×8 searches).
A summary of supported half pixel interpolation FIR filters is shown in
After half pixel interpolation using one of the four filtering options, the half pixel metrics are calculated for each of nine candidate half pixel positions. The metric is the sum of the SAD256 (SAD64 for 8×8 blocks) metric plus the half pixel cost function. A numbering convention for the nine candidate half pixel positions is shown in
Half pixel search consists of generating the metric of SAD+half pixel cost function for each of the nine candidates and selecting the one that has the best metric.
The memory map for the 43×43 half pixel array that allows both one 16×16 search and four 8×8 searches is shown in
The memory map for the 35×35 half pixel array generated from a 16 pixel×16 pixel macroblock is shown in
The memory map for four buffers for 19×19 half pixel arrays that will be generated from 8 pixel×8 pixel blocks is shown in
After half pixel search for the 16×16 macroblock (8×8 blocks) with the best distortion metric, quarter pixel interpolation is performed on the surrounding ([16+n]×[16+n]) pixel array, where n is the filter order. For WMV, n=4; for H.264, and MPEG4, which use bilinear interpolation, n=2.
The WMV sub-pixel four tap FIR filters for the ¼ pixel position, the ½ pixel position, and the ¾ pixel position are summarized in
After quarter pixel interpolation, the quarter pixel metrics are calculated for each of nine candidate quarter pixel positions. The metric is the sum of the SAD256 (SAD64) metric plus the quarter pixel cost function. A numbering convention for the nine candidate quarter pixel positions is shown in
Quarter pixel search consists of generating the metric of SAD+quarter pixel cost function for each of the nine candidates and selecting the one that has the best metric.
The memory map for the 2048 bytes of quarter pixel data generated from the half pixel array associated with a 16 pixel×16 pixel macroblock is shown in
Because of buffer size limitations, the 256 bytes associated with the ninth (center) position [‘candidate (5)’ shown in
For half pixel position 2, the 16×16 macroblock is stored in the sixteen columns 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, and 32 of the sixteen rows 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, and 31.
For half pixel position 3, the 16×16 macroblock is stored in the sixteen columns 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, and 33 of the sixteen rows 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, and 31.
For half pixel position 4, the 16×16 macroblock is stored in the sixteen columns 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, and 31 of the sixteen rows 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, and 32.
For half pixel position 5, the 16×16 macroblock is stored in the sixteen columns 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, and 32 of the sixteen rows 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, and 32.
For half pixel position 6, the 16×16 macroblock is stored in the sixteen columns 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, and 33 of the sixteen rows 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, and 32.
For half pixel position 7, the 16×16 macroblock is stored in the sixteen columns 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, and 31 of the sixteen rows 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, and 33.
For half pixel position 8, the 16×16 macroblock is stored in the sixteen columns 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, and 32 of the sixteen rows 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, and 33.
For half pixel position 9, the 16×16 macroblock is stored in the sixteen columns 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, and 33 of the sixteen rows 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, and 33.
The byte maps for the nine possibilities are shown in
For fractional pixel processing of 8 pixel×8 pixel blocks, the 2048 byte quarter pixel buffer is organized as four 512 byte quarter pixel buffers. A memory map for these four buffers is shown in
Because of buffer size limitations, the 64 bytes associated with the ninth (center) position [‘candidate (5)’ shown in
For half pixel position 1, the 8×8 block is stored in the eight columns 1, 3, 5, 7, 9, 11, 13, and 15 of the eight rows 1, 3, 5, 7, 9, 11, 13, and 15.
For half pixel position 2, the 8×8 block is stored in the eight columns 2, 4, 6, 8, 10, 12, 14, and 16 of the eight rows 1, 3, 5, 7, 9, 11, 13, and 15.
For half pixel position 3, the 8×8 block is stored in the eight columns 3, 5, 7, 9, 11, 13, 15, and 17 of the eight rows 1, 3, 5, 7, 9, 11, 13, and 15.
For half pixel position 4, the 8×8 block is stored in the eight columns 1, 3, 5, 7, 9, 11, 13, and 15 of the eight rows 2, 4, 6, 8, 10, 12, 14, and 16.
For half pixel position 5, the 8×8 block is stored in the eight columns 2, 4, 6, 8, 10, 12, 14, and 16 of the eight rows 2, 4, 6, 8, 10, 12, 14, and 16.
For half pixel position 6, the 8×8 block is stored in the eight columns 3, 5, 7, 9, 11, 13, 15, and 17 of the eight rows 2, 4, 6, 8, 10, 12, 14, and 16.
For half pixel position 7, the 8×8 block is stored in the eight columns 1, 3, 5, 7, 9, 11, 13, and 15 of the eight rows 3, 5, 7, 9, 11, 13, 15, and 17.
For half pixel position 8, the 8×8 block is stored in the eight columns 2, 4, 6, 8, 10, 12, 14, and 16 of the eight rows 3, 5, 7, 9, 11, 13, 15, and 17.
For half pixel position 9, the 8×8 block is stored in the eight columns 3, 5, 7, 9, 11, 13, 15, and 17 of the eight rows 3, 5, 7, 9, 11, 13, 15, and 17.
The byte maps for the nine origins are shown in
When (in the 1MV/4MV decision is deferred until results are compared for half pixel searches, then locating the center of nine candidate positions in the 43×43 half pixel array for subsequent quarter pixel searches becomes more challenging (See
For each of four 8×8 searches, there are twenty five possible motion vectors for half pixel search, and for each of the 25, nine possible motion vectors for subsequent quarter pixel search! One (and only one) example:-)
For the lower right 8×8 block, for a full pixel vector (relative to the 16×16 motion vector) of H=10, V=10 and a half pixel vector (relative to this 8×8 full pixel vector) of H=−½, V=−½, the upper leftmost pixel of the center of nine possibilities for the quarter pixel search is located in column 25 of row 25 of the half pixel array (columns and rows numbered from 0 through 42). The physical location in memory for this pixel is: memory 5, address 0x110, bits[15:8].
The cost function tables are stored in node memory. The cost function is an unsigned 16 bit integer. For an M-column by N-row reference block, there are (M−7) horizontal cost functions and (N−7) vertical cost functions. In each cost function table, two 16 bit integer cost functions are packed into each 32 bit longword in the table.
Cost Function Tables A/B will be stored in node memories 4 and 5 as shown in
The fractional pixel cost functions are shown in
For each full pixel search position, the DVN evaluates the metric:
For an M-column by N-row reference block, there are (M−15) horizontal cost functions and (N−15) vertical cost functions.
For each double pixel search position, the DVN evaluates the metric:
For hierarchical search, for the final four sets of 8×8 block evaluations, there are four pairs of horizontal/vertical cost functions. The DVN evaluates the metric:
For fractional pixel searches with a 16×16 macroblock, the DVN evaluates the metric:
For fractional pixel searches with an 8×8 block, the DVN evaluates the metric:
During double pixel search, the DVN motion estimation engine (ME) reads down-sampled reference block data from memories four through seven, and it reads down-sampled current block data from memories zero through three.
During full pixel search, the DVN motion estimation engine (ME) reads reference block data from memories zero through three, and it reads current block data from memories four through seven.
During half pixel interpolation, the ME reads full pixel reference block data from memories zero through three, and it writes half pixel interpolated data into memories four through seven. When half pixel search is performed concurrently with half pixel interpolation, the ME reads current block data from memories four through seven.
During quarter pixel interpolation, the ME reads half pixel interpolated data from memories four through seven, and it writes quarter pixel interpolated data into memories four through seven. When quarter pixel search is performed concurrently with quarter pixel interpolation, the ME reads current block data from memories four through seven.
During output, the ME reads full pixel macroblocks from memories zero through three, fractional pixel macroblocks from memories four through seven; and for macroblock signed differences, current blocks from memories four through seven.
The DVN will be capable of calculating and outputting a 256 (64) element array of 9-bit signed differences between any 16 pixel by 16 pixel macroblock (8 pixel by 8 pixel block) in DVN node memory (full pixel, half pixel, or quarter pixel) and either 16 pixel by 16 pixel current block (8 pixel by 8 pixel block) in DVN node memory. The array is output to a destination node as 128 (32) longwords, where each longword is a packed pair of 16-bit signed integers representing the sign-extended 9-bit signed differences of the 16×16 macroblocks (8×8 blocks).
The first element of the array is packed into bits[15:0] of the first longword, the second element of the array is packed into bits[31:16] of the first longword, the third element of the array is packed into bits[15:0] of the second longword, and so on.
A block diagram that depicts the DVN programming environment is shown in
High level control of the DVN is provided by the hardware task manager (HTM) in the DVN node wrapper and a finite state machine (FSM) in the DVN EU. This interface includes the ACM-standard EU_RUN, EU_CONTINUE, and EU_TEARDOWN signals from the wrapper to the EU, and the ACK+TEST and EU_DONE responses from the EU to the wrapper.
The transition table for the FSM is shown in
The FSM assumes one of six states:
When the node is not enabled or the eu is not enabled, the FSM will enter, and remain in, its idle state. The FSM will also transition to its idle state whenever the node wrapper asserts the ‘eu_abort’ signal. Additionally, the FSM transitions to the idle state when it exits the wait state in response to the node wrapper's assertion of the ‘eu_teardown’ signal.
A typical operational sequence for the FSM will include the following transitions:
The above sequence returns control of the FSM to the node wrapper's HTM after ack+test. The DVN also can be programmed to retain control after ‘ack-ing’. Options include conditional/unconditional state transitions to continue or idle (after asserting ‘eu_done’). Test conditions include the sign (msb) of any of the node wrapper's 64 counters, the sign (msb) of certain TPL entries, and the sign (msb) of a byte code controlled 5-bit down counter.
All of the information required for task execution is contained in a ‘task parameter list’ (TPL) located in node memory. During setup, DVN hardware retrieves from its node memory a pointer to the TPL. Pointers for up to 31 tasks are stored in 31 consecutive words in node memory at a location specified by the node wrapper signals: {tpt_br[8:0], active_task[4:0]}.
Each DVN TPL is assigned 64 longwords of node memory. The first eight entries of the TPL are reserved for specific information. The remaining 56 longwords are available to store parameters for setup, ack+test, and continue.
TPL location 0x00 contains two parameters for setup: the starting address in node memory for its byte code and the offset from the TPL pointer to its parameters.
TPL location 0x01 contains two parameters for ack+test: the starting address in node memory for its byte code and the offset from the TPL pointer to its parameters.
TPL location 0x02 contains one parameter for teardown: the starting address in node memory for its byte code. (Currently, there is nothing to ‘teardown’, so this is a placeholder in the event that there is a requirement later on).
TPL location 0x03 contains three parameters for continue: the starting address in node memory for its byte code and the offsets from the TPL pointer to its two sets of parameters.
TPL locations 0x04 and 0x05 are used for a pair of semaphores that the byte code can manipulate; and locations 0x06 and 0x07 are reserved for tbd.
TPL locations 0x08 through 0x3F are available for storing all other task parameters.
The layout for the DVN task parameter list is shown in
The TPL stores sets of parameters for each of three states: setup, ack+test, and continue. For the continue state, there can be two sets of parameters, with each set having the same number of parameters but different TPL pointer offsets. This allows for the conditional selection between one-of-two sets of global parameters, such as consumer node(s) destination, fractional pixel interpolation filters, and so on.
The set of parameters for a specific state must be stored in consecutive locations in the TPL. As byte code LOAD instructions execute, the DVN hardware selects the next parameter in the set.
The offset from the TPL pointer to the location of the first parameter for that state must be in the range: 0x08 to 0x3F.
The format for TPL locations 0x00 through 0x03 that contain byte code starting addresses and TPL pointer offsets is shown in
The format for the two semaphores at TPL locations 0x04 and 0x05 is shown in
The format for the ack+test parameters beginning at TPL location (TPL pointer plus ack+test offset) is shown in
The formats for the registers that are loaded with TPL parameters during setup and continue are shown in
The DVN CSR includes fields which specify which interpolation filters to use, how to interpolate at reference block edges, offsets for the horizontal/vertical cost function tables, the number of longwords per row in the selected reference block buffer, and which ping/pong buffer pairs to use.
Fractional Pixel Interpolation Filter Selection Bits[4:0]
Interpolation Method at Reference Block Edges Bits[6:5]
WMV Version 9.0 Frame-Level Round Control Value Bit[7]
Bits[14:8] represent an unsigned 7-bit integer that is added to the vertical displacement of a macroblock to form the address of the vertical cost function table. This allows for possible cost function table reuse for more than one current block.
Bits[22:16] represent an unsigned 7-bit integer that is added to the horizontal displacement of a macroblock to form the address of the horizontal cost function table. This allows for possible cost function table reuse for more than one current block.
The number of longwords comprising a reference block row is required by the node memory addressing unit to support strides from row (k) to row (k+4). The range for bits[28:24] is 8 through 25. (See
Each of three bits selects between a pair of ping/pong buffers:
The DVN communicates with its controller using the PTP protocol, which requires a port number (bits[5:0]) and a node id (bits[15:8]). Additionally, this register indicates which of the node wrapper's 64 flags will be used for Command Queue Redirection,
While processing a command queue, the DVN will monitor one of 64 node wrapper counter sign bits indicated by bits[31:16] after it completes the execution of each instruction. When the counter sign bit is set to 1 ′b1, the DVN will clear the bit, send the Command Queue Redirection Acknowledgement to the controller, load the queue 7-bit program counter with the value in the controller-programmable 7-bit redirection register, and resume processing at this location.
When processing command queue instructions that include output to destination (consumer) nodes, the DVN will use its one bit logical output port field to select one of two destination nodes: When the bit is set to 1 ′b0, its output will be directed to node id (bits(15:8), port number (bits[5:0]); and when the bit is set to ″b1, its output will be directed to node id (bits[31:24]), port number (bits[21:16]).
The DVN overhead processor (OHP) interprets two distinct sets of byte codes: one for setup and continue and another for ack+test. The FSM state indicates which set should be used by the byte code interpreter.
For setup and continue, there is only one byte code: LOAD as shown in
There is a more comprehensive set of byte codes for ack+test. These are summarized in
At this time, ack+test byte codes 0x80 through 0xFF are reserved, and the current implementation treats these codes as NOPs, or ‘no operation’.
For the DVN, one of the programming steps requires writing byte code and generating task parameter lists. Byte code is always executed ‘in line’, and byte code must start on longword boundaries. Byte code execution is ‘little endian’; that is, the first byte code that will be executed is byte[7:0] of the first longword, then byte[15:8], then byte[23:16], then byte[31:24], then byte[7:0] of the second longword, and so on.
The task parameter list and byte code for a ‘typical’ task are shown in
Setup requires three clock cycles to load registers from the TPL; DONE is indicated in the last of three cycles.
ack+test requires a minimum of ten clock cycles: two for producer acknowledgements, two for consumer acknowledgements, two (or more) to test (and wait, if necessary) for input buffer ready, two (or more) to test (and wait, if necessary) for output buffer ready, and two to test a counter sign to select one of two sets of configuration registers.
continue requires three clock cycles to, update the CSR, CNLR, and DNLR configuration registers.
Programming requirements for the DVN include the creation of sequences of instructions that are stored in the command queue in DVN node memory (the DVN memory map is shown in
In the memory map, the 512 byte command queue is located in node memory 7 at physical addresses 0x140 to 0x1BF. The programmer could view this resource as two distinct queues that store a fixed set of instructions in one (or both) static command queue(s); or the programmer could construct an interactive sequence of commands in one (or both) dynamic command queue(s), and select between the two queues on a macroblock-by-macroblock basis. Including a second command queue allows for terminating one sequence of instructions and continuing with a new sequence of instructions, while preserving the entire previous sequence of instructions for subsequent reuse.
In general, the programmer can manage the 128 longword command queue resource in any number of ways. The programmer may view this resource as one 128-longword queue, two 64-longword queues, one larger queue that holds the ‘most likely’ sequences of instructions plus a smaller queue used as a ‘scratch’ for ‘if’ sequences of instructions, and so on.
The DVN controller node writes into the command queue using the PTP protocol after it has initialized the appropriate entry in the DVN wrapper port translation table (PTT).
To initialize the PTT, the programmer will use Service Code 0x6:
Poke Address+Data(D16+A12),EU=0.
A12={7 ′b0100000, p[4:0]}, where p[4:0] is the desired port number.
D16={9 ′b011110101, a[6:0]}, where a[6:0] is the one of 128 locations in the command queue where the first entry will be written.
It is important to note that after each PTP write operation, the node wrapper PTT address generator post increments by one (modulo buffer size, which is 128 in this example), so that the command queue addressing sequence will be 0, 1, . . . , 126, 127, 0, 1 and so on.
The DVN fetches command queue instructions using a queue 7-bit counter. When the counter is initialized to zero, it will be pointing to location 0 of the command queue. After each command queue instruction has been processed, the counter will be incremented by one. When the counter state is 0x7F, corresponding to the last location in the queue, its next state will be 0x00, the first location in the queue.
There is a poke-able 7-bit ‘redirection’ register in the DVN that allows the programmer to alter the contents of the queue counter.
To write this register, the programmer will use Service Code 0x6: Poke Address+Data (D16+A12), EU=1.
A12=12 ′b000000000000; and
D16={9 ′b000000000, reg_val[6:0]}.
The following events will result in the loading of the 7-bit queue counter with the contents of this 7-bit register.
1) the DVN FSM state: setup+byte code: DONE.
2) the DVN responding to the Command Queue Redirection Indication (QRI).
While the DVN is processing an instruction sequence fetched from the command queue, the controller can redirect the DVN to a different instruction using the QRI. The QRI is simply one of 64 counter signs in the DVN node wrapper. The controller writes the appropriate PCT/CCT counter with the msb set to 1 ′b1. (We assume here that the bit had previously been initialized to 1 ′b0). The DVN will sample this bit after it has completed the processing of an instruction; it will also sample this bit while it is waiting for the next valid instruction to be written into the queue; finally, when there is ‘search early termination’, the DVN command queue interpreter will continuously sample the QRI until it is asserted.
At DVN task setup, a six bit register is loaded from the TPL to select the appropriate one-of-64 counter signs that will be used for the QRI. In response to QRI recognition by the DVN, it will clear this bit by sending the ‘self-ack, initialize counter’ MIN word, and it will send the Command Queue Redirection Acknowledgement to the controller (See below).
The data structure for the command queue is shown in
Command 0x0 is used for the ‘wait’ instruction. When the DVN reads a queue entry: ‘Command 0x0, Wait’, it will stall until some other command is written into that entry or until it is redirected by the QRI.
The ECHO command instructs the DVN to return a status word that is the bit pattern of the command itself. This is intended to be a diagnostics aid.
This command instructs the DVN to load its Search Threshold Register (STR) with Bits[15:0] of the command. During search, the DVN will compare the ‘best metric’ with the STR contents. When the early termination indication is enabled and the ‘best metric’ is less than this value, the ‘threshold bettered’ indication will be asserted in the DVN message to the controller node. The command queue interpreter will stall until it receives its QRI redirection indication from the controller node telling it ‘what's next?’
When Bit[24]=1 ′b0/1′b1, down-sample the reference block/current block using the filter indicated by Bits [17:16] to generate a double pixel array.
The entire reference block will be down-sampled, and its size must be specified by bits[14:8], X pixels, for the number of pixels per row, and by bits[6:0], Y pixels, for the number of pixels per column. Allowable combinations for X and Y which do not exceed the 4032 byte reference block capacity are summarized in
For the current block, the input array must be 16×16=256 pixels, 18×18=324 pixels, 20×20=400 pixels, or 22×22=484 pixels for bilinear interpolation, 4-tap filter, 6-tap filter, and 8-tap filter, respectively.
Evaluate the Sum of Absolute Differences (SAD) plus Horizontal Cost Function plus Vertical Cost Function for a single position or one or more sets of positions for 16×16 macroblocks or 8×8 blocks.
The various search parameters allow for a number of different search strategies:
When bit[27], block size, is set to 1 ′b0, search will be performed with an 8×8 block from the current block or the down-sampled current block. (The specific block is indicated with bits[23:20]). The evaluated metric will be:
When bit[27] is set to 1 ′b1, search will be performed with the 16×16 current block. The evaluated metric will be:
When bit[26] is set to 1 ′b0, the search origin will be X, Y offsets stored in the motion vector register (or the Results Buffer in Node Memory). When bit[26] is set to 1 ′b1, the search origin will be the X,Y locations indicated by bits[14:8] and bits[6:0], respectively, of the search command.
When the X,Y option is selected for double pixel search of the down-sampled reference block (where all motion vectors are at ½ pixel positions), the ‘½’ pixel component of the motion vector is implied.
Note that the upper leftmost motion vector for the down-sampled reference block will be X=Y=1½, X=Y=2½, or X=Y=3½ when the down-sampling was performed with bilinear interpolation or 4-tap filter, 6-tap filter, or 8-tap filter, respectively.
When the search strategy includes 1MV/4MV comparisons at half pixel resolution, bit[25] should be set to 1 ′b1 to indicate that a 43×43 half pixel array exists in node memory. Otherwise, bit[25] should be set to 1 ′b0 to indicate that a 35×35 half pixel array (for 16×16 searches) or four 19×19 half pixel arrays (for 8×8 searches) exists in node memory.
For all search commands, bit[24], early termination, may be set to either 1 ′b0 or 1 ′b1 to disable/enable the early termination function.
When early termination is enabled, any trial that produces a metric less than the value in the STR will result in the DVN terminating the search, sending appropriate status to the controller node, and stalling until it receives a QRI.
When early termination is disabled, the DVN will process the command, send status to the controller node, then fetch the next command from the queue.
For 16×16 fractional pixel searches (One Vector Mode), the programmer must set bits[23:20] to select the buffers 1, that is 0x8 for half pixel buffer 1 and 0xC for quarter pixel buffer 1.
For 8×8 fractional pixel searches (Four Vector Mode), the buffers 1, 2, 3, and 4 are associated with the upper leftmost 8×8 block, the upper rightmost 8×8 block, the lower leftmost 8×8 block, and the lower rightmost 8×8 block, respectively. Therefore:
for the upper leftmost 8×8 block, the programmer must set bits[23:20] to 0x8 for half pixel buffer 1 and 0xC for quarter pixel buffer 1;
for the upper rightmost 8×8 block, the programmer must set bits[23:20] to 0x9 for half pixel buffer 2 and 0xD for quarter pixel buffer 2,
for the lower leftmost 8×8 block, the programmer must set bits[23:20] to 0xA for half pixel buffer 3 and 0xE for quarter pixel buffer 3; and
for the lower rightmost 8×8 block, the programmer must set bits[23:20] to 0xB for half pixel buffer 4 and 0xF for quarter pixel buffer 4.
When bits[23:20], buffer select, are set to 0x0, search will be performed within the down-sampled double pixel reference block using the down-sampled 8×8 current block. (Note: Bit[27] must be set to 1 ′b0 when bits[23:20] are set to 0x0).
The evaluated metric will be:
When bits[23:20] are set to 0x1, search will be performed within the reference block using the 16×16 current block. Bit[27] must be set to 1 ′b1.
The evaluated metric will be:
When bits[23:20] are set to 0x4, 0x5, 0x6, or 0x7, search will be performed within the reference block using the upper left, upper right, lower left, or lower right 8×8 block of the current block, respectively. Bit[27] must be set to 1 ′b0.
The evaluated metric will be:
When bits[23:20] are set to 0x8, full search with half pixel resolution will be performed on the nine candidate positions within half pixel buffer 1 (or the corresponding area of the 43×43 half pixel array for combined 1MV/4MV mode). With bit[27], block size, the programmer can select either 16×16 or 8×8 search mode. For the 8×8 mode, the upper leftmost 8×8 block of the current block will be used.
For half pixel searches with a 16×16 macroblock, the DVN evaluates the metric:
For half pixel searches with an 8×8 block, the DVN evaluates the metric:
When bits[23:20] are set to 0x9, full search with half pixel resolution will be performed on the nine candidate positions within half pixel buffer 2 (or the corresponding area of the 43×43 half pixel array for combined 1MV/4MV mode). Bit[27] must be set to 1 ′b0 to indicate 8×8 search mode, and the upper rightmost 8×8 block of the current block will be used.
When bits[23:20] are set to 0xA, full search with half pixel resolution will be performed on the nine candidate positions within half pixel buffer 3 (or the corresponding area of the 43×43 half pixel array for combined 1MV/4MV mode). Bit[27] must be set to 1 ′b0 to indicate 8×8 search mode, and the lower leftmost 8×8 block of the current block will be used.
When bits[23:20] are set to 0xB, full search with half pixel resolution will be performed on the nine candidate positions within half pixel buffer 3 (or the corresponding area of the 43×43 half pixel array for combined 1MV/4MV mode). Bit[27] must be set to 1 ′b0 to indicate 8×8 search mode, and the lower rightmost 8×8 block of the current block will be used.
When bits[23:20] are set to 0xC, full search with quarter pixel resolution will be performed on eight of the nine candidate positions within quarter pixel buffer 1 plus the center of nine candidate positions within half pixel buffer 1. With bit[27], block size, the programmer can select either 16×16 or 8×8 search mode. For the 8×8 mode, the upper leftmost 8×8 block of the current block will be used.
For quarter pixel searches with a 16×16 macroblock, the DVN evaluates the metric:
For quarter pixel searches with an 8×8 block, the DVN evaluates the metric:
When bits[23:20] are set to 0xD, full search with quarter pixel resolution will be performed on eight of the nine candidate positions within quarter pixel buffer 2 plus the center of nine candidate positions within half pixel buffer 2. Bit[27] must be set to 1 ′b0 to indicate 8×8 search mode, and the upper rightmost 8×8 block of the current block will be used.
When bits[23:20] are set to 0xE, full search with quarter pixel resolution will be performed on eight of the nine candidate positions within quarter pixel buffer 3 plus the center of nine candidate positions within half pixel buffer 3. Bit[27] must be set to 1 ′b0 to indicate 8×8 search mode, and the lower leftmost 8×8 block of the current block will be used.
When bits[23:20] are set to 0xF, full search with quarter pixel resolution will be performed on eight of the nine candidate positions within quarter pixel buffer 4 plus the center of nine candidate positions within half pixel buffer 4. Bit[27] must be set to 1 ′b0 to indicate 8×8 search mode, and the lower rightmost 8×8 block of the current block will be used.
For all search commands, bit[19], initialize registers, may be set to either 1 ′b0 or 1′b1 to disable/enable the initialization of the ‘best metric’ register and the motion vector register. When bit[19] is set to 1 ′b1, the ‘best metric’ register is initialized to 0xFFFF, and the motion vector register is initialized to zero (X=0, Y=0) before the search operation is initiated.
When bit[19] is set to 1 ′b0, the contents of these registers will not be modified before the search operation is initiated. Subsequently, during the search operation, the contents of these registers will be modified if (and only if) an evaluation returns a metric that is less than the current value in the ‘best metric’ register.
Bits[18:16] are used to select the search pattern:
When bits[18:16] are set to 3 ′b000:
When bits[18:16] are set to 3 ′b001:
When bits[18:16] are set to 3 ′b010:
When bits[18:16] are set to 3 ′b011:
When bits[18:16] are set to 3 ′b100:
Note that the programmer can use multiple search instructions with bits[18:16] set to 3 ′b010: ‘Evaluate one set of 5×5 positions’ to realize any arbitrary search order of sets of 5×5 macroblocks.
When bits[18:16] are set to 3 ′b101:
When bit[26], search origin, is set to 1 ′b1, bits[14:8] are used to indicate the offset from the upper leftmost pixel in a row of the reference block. The resolution is one full pixel.
When bit[26] is set to 1 ′b0, bits[14:8] are don't care.
When bit[26], search origin, is set to 1 ′b1, bits[6:0] are used to indicate the offset from the upper leftmost pixel in a column of the reference block. The resolution is one full pixel.
When bit[26] is set to 1 ′b0, bits[6:0] are don't care (
Compare the ‘One Vector’ and ‘Four Vectors’ metrics. Specifically, add the four metrics saved from searching four sets of 5×5 positions with four 8×8 blocks, and compare the result with the metric for a search with the 16×16 current block. Send a message to the controller node indicating which mode produced the better metric, then stall until the QRI is asserted.
This is a major decision point where the controller node determines whether to continue with fraction pixel processing, and, if it does continue, to select ‘One Vector’ mode or ‘Four Vectors’ mode.
The programmer sets Bit[22] to 1 ′b0 for a comparison of the full pixel metrics or to 1 ′b1 for a comparison of the half pixel metrics. All metrics are stored in the Results Buffer in node memory (See
Perform full pixel to half pixel or half pixel to quarter pixel interpolation. Various parameters allow for a number of different interpolation options:
When bit[27], block size, is set to 1 ′b0, full pixel to half pixel interpolation will produce a 19×19 half pixel array; half pixel to quarter pixel interpolation will produce eight 8×8 quarter pixel arrays.
When bit[27] is set to 1 ′b1, full pixel to half pixel interpolation will produce a 35×35 half pixel array; half pixel to quarter pixel interpolation will produce eight 16×16 quarter pixel arrays.
When bit[25]=1 ′b1: Combined 1MV/4MV Half Pixel Buffer Organization:
When bit[18] is set to 1 ′b0 for full pixel to half pixel interpolation, bit[27] must be set to 1 ′b1. A 43×43 half pixel array will be produced.
When bit[18] is set to 1 ′b1 for half pixel to quarter pixel interpolation:
When the search option has been enabled for this command:
When bit[26] is set to 1 ′b0, the search origin will be X, Y offsets stored in the motion vector register (or the Results Buffer in Node Memory). When bit[26] is set to 1 ′b1, the search origin will be the X,Y locations indicated by bits[14:8] and bits[6:0], respectively, of the search command.
When the search option has not been enabled for this command, bit[26] is don't care.
When the search strategy includes 1MV/4MV comparisons at half pixel resolution, bit[25] should be set to 1 ′b1 to indicate that a 43×43 half pixel array will be created for full pixel to half pixel interpolation (bit[18] set to 1 ′b0) or that a previously interpolated 43×43 half pixel array exists in node memory (bit[18] set to 1′b1: half pixel to quarter pixel interpolation).
Otherwise, bit[25] should be set to 1 ′b0 to indicate that a 35×35 half pixel array (for 16×16 macroblocks) or four 19×19 half pixel arrays (for 8×8 blocks) will be created for full pixel to half pixel interpolation (bit[18] set to 1 ′b0) or that a previously interpolated 35×35 half pixel array or four 19×19 half pixel arrays exist in node memory (bit[18] set to 1′b1: half pixel to quarter pixel interpolation).
For all interpolation commands for which the search function is enabled, bit[24], early termination, may be set to either 1 ′b0 or 1 ′b1 to disable/enable the early termination function.
When early termination is enabled and search is enabled (bit[17] is set to 1 ′b1), after all nine candidate positions have been evaluated, the resulting ‘best metric’ will be compared with the value in the STR. When the metric is less than the value in the STR, the DVN will send appropriate status to the controller node, and it will stall until it receives a QRI.
When early termination is disabled or search is disabled (or both are disabled), the DVN will process the command, send status to the controller node, then fetch the next command from the queue.
For Full Pixel to Half Pixel Interpolation with Bit[25]=1 ′b0: Normal Half Pixel Buffer Organization:
When a full pixel 16×16 macroblock is interpolated to produce a 35×35 half pixel array (bit[27]=1′ b1, bit[18]=1 ′b0), that array will be stored in the 1892 byte half pixel buffer in node memory as shown in
When a full pixel 8×8 block is interpolated to produce a 19×19 half pixel array (bit[27]=1 ′b0, bit[18]=1 ′b0), that array will be stored in the appropriate one of four 400 byte half pixel buffers in node memory as shown in
2 ′b00 to select Buffer 1 for the upper leftmost 8×8 block; starting address=0x8C8.
2 ′b01 to select Buffer 2 for the upper rightmost 8×8 block; starting address=0x8E4.
2 ′b10 to select Buffer 3 for the lower leftmost 8×8 block; starting address=0x900.
2 ′b11 to select Buffer 4 for the lower rightmost 8×8 block; starting address=0x91C.
For Full Pixel to Half Pixel Interpolation with Bit[25]=1 ′b1: Combined 1MV/4MV Half Pixel Buffer Organization:
For full pixel to half pixel interpolation in the combined 1MV/4MV mode (bit[25]=1 ′b1), the DVN will produce a 43×43 half pixel array as shown in
For Half Pixel to Quarter Pixel Interpolation
When bit[27] is set to 1′b1, a 1296 byte half pixel buffer will be interpolated to produce eight 16×16 quarter pixel arrays (a total of 2048 bytes), and those results will be stored in the 2048 byte quarter pixel buffer in node memory. For this operation, bits[21:20] should be set to 2 ′b00.
When bit[27] is set to 1 ′b0, a 19×19 half pixel array will be interpolated to produce eight 8×8 quarter pixel arrays (a total of 512 bytes), and those results will be stored in the indicated one of four 512 quarter pixel buffers in node memory (See
2 ′b00 to select Buffer 1 for the upper leftmost 8×8 block.
2 ′b01 to select Buffer 2 for the upper rightmost 8×8 block.
2 ′b10 to select Buffer 3 for the lower leftmost 8×8 block.
2 ′b11 to select Buffer 4 for the lower rightmost 8×8 block.
For all interpolation commands where search is enabled (bit[17]=1 ′b1), bit[19], ‘initialize registers’, may be set to either 1 ′b0 or 1′b1 to disable/enable the initialization of the ‘best metric’ register and the motion vector register. When bit[19] is set to 1 ′b1, the appropriate ‘best metric’ register is initialized to 0xFFFF, and the associated motion vector register is initialized to zero (X=0, Y=0) before the search operation is initiated. When bit[19] is set to 1 ′b0, the contents of these registers are preserved before the search operation is initiated. Subsequently, during the search operation, the contents of these registers will be modified if (and only if) an evaluation returns a metric that is less than the current value in the ‘best metric’ register.
For full pixel to half pixel interpolation, the programmer sets bit[18] to 1 ′b0.
For half pixel to quarter pixel interpolation, the programmer sets bit[18] to 1 ′b1.
The search order is shown in
When the search option has been enabled for this command and bit[26], search origin, has been set to 1 ′b1, bits[14:8] are used to indicate the offset from the upper leftmost pixel in a row of the reference block. The resolution is one full pixel.
When the search option has not been enabled or when bit[26] is set to 1 ′b0 (or both), bits[14:8] are don't care.
When the search option has been enabled for this command and bit[26], search origin, has been set to 1 ′b1, bits[6:0] are used to indicate the offset from the upper leftmost pixel in a column of the reference block. The resolution is one full pixel.
When the search option has not been enabled or when bit[26] is set to 1 ′b0 (or both), bits[6:0] are don't care.
Perform the signed difference between the indicated blocks. The indicated reference block/fractional pixel block will be subtracted from the current block. Various parameters allow for a number of options:
For signed difference between 16×16 blocks, the DVN will output to the selected destination the 256 values as 16-bit 2's complement integers in the range −255 to 255, packed into 128 longwords.
For signed difference between 8×8 blocks, the DVN will output to the selected destination the 64 values as 16-bit 2's complement integers in the range −255 to 255, packed into 32 longwords.
When bit[27], block size, is set to 1 ′b0, the DVN will perform the signed difference between an 8×8 block from the current block and an 8×8 reference block/fractional pixel block.
When bit[27] is set to 1 ′b1, the DVN will perform the signed difference between the 16×16 current block and a 16×16 reference block/fractional pixel block.
When bits[23:20] are set to 0x1 to select the reference block buffer:
When bit[26] is set to 1 ′b0, the reference block buffer origin will be X, Y offsets stored in the motion vector register (or the Results Buffer in Node Memory) associated with the ‘best metric’. When bit[26] is set to 1 ′b1, the origin will be the X,Y locations indicated by bits[14:8] and bits[6:0], respectively, of the ‘Output Signed Differences’ command.
For all other values for bits[23:20], bit[26] must be set to 1 ′b0, and the buffer origin will be indicated by a previously determined motion vector associated with a ‘best metric’.
When the output signed difference is for a fractional pixel array, bit[25] should be set to 1 ′b1 when a 43×43 half pixel array exists in node memory, and it should be set to 1 ′b0 when a 35×35 half pixel array or four 19×19 half pixel arrays exist in node memory.
When the output signed difference is for a full pixel array, bit[25] is ‘don't care’.
Bit[24] allows the programmer to select from one of two logical output ports. For each one, the DNLR will contain the associated routing, input port number, and memory mode indication for the selected destination (See
Bits[23:20] indicate the buffer whose elements will be differenced from the corresponding elements of the 16×16 current block (8×8 block).
When bit[27], block size, is set to 1 ′b0 for an 8×8 block, any of the buffer selection codes except 0x1 can be selected.
When bit[27] is set to 1 ′b1 for a 16×16 block, only codes 0x1 (reference block buffer), 0x8 (half pixel buffer 1), and 0xC (quarter pixel buffer 1) are allowed.
When bit[26], buffer origin, has been set to 1 ′b1, bits[14:8] are used to indicate the offset from the upper leftmost pixel in a row of the reference block. The resolution is one full pixel.
When bit[26] has been set to 1 ′b1, bits[6:0] are used to indicate the offset to the buffer origin from the upper leftmost pixel in a column of the reference block. The resolution is one full pixel.
Output the indicated array to the indicated destination. Various parameters allow for a number of options:
When bit[27], block size, is set to 1 ′b0, the DVN will output an 8×8 block. The 64 values will be 8-bit unsigned integers in the range 0 to 255 packed into 16 longwords.
When bit[27] is set to 1 ′b1, the DVN will output a 16×16 macroblock. The 256 values will be 8-bit unsigned integers in the range 0 to 255 packed into 64 longwords.
When bits[23:20] are set to 0x1 to select the reference block buffer:
When bit[26] is set to 1 ′b0, the reference block buffer origin will be X, Y offsets stored in the motion vector register (or the Results Buffer in Node Memory) associated with the ‘best metric’. When bit[26] is set to 1 ′b1, the origin will be the X,Y locations indicated by bits[14:8] and bits[6:0], respectively, of the command.
For all other values for bits[23:20], bit[26] must be set to 1 ′b0, and the buffer origin will be indicated by the motion vector associated with a previously determined ‘best metric’.
When the output is a fractional pixel array, bit[25] should be set to 1 ′b1 when a 43×43 half pixel array exists in node memory, and it should be set to 1 ′b0 when a 35×35 half pixel array or four 19×19 half pixel arrays exist in node memory.
When the output is for a full pixel array, bit[25] is ‘don't care’.
Bit[24] allows the programmer to select between one of two logical output ports as the indicated destination. For each one, the DNLR will contain the associated routing, input port number, and memory mode indication for the selected destination (See
Bits[23:20] indicate the specific buffer that the DVN should output.
When bit[27], block size, is set to 1 ′b0 for an 8×8 block, any of buffer selection codes except 0x01 can be selected.
When bit[27] is set to 1 ′b1 for a 16×16 block, only codes 0x1 (reference block buffer), 0x8 (half pixel buffer 1), and 0xC (quarter pixel buffer 1) are allowed.
When bit[26], buffer origin, has been set to 1 ′b1, bits[14:8] are used to indicate the offset from the upper leftmost pixel in a row of the reference block to the buffer origin. The resolution is one full pixel.
When bit[26], buffer origin, has been set to 1 ′b1, bits[6:0] are used to indicate the offset from the upper leftmost pixel in a column of the reference block to the buffer origin. The resolution is one full pixel.
Output the indicated result to the indicated destination(s). Various parameters allow for a number of options:
For bit[27] set to 1 ′b0, transfer the indicated best metric (saturated unsigned 16 bit integer) to the selected destination(s).
For bit[27] set to 1 ′b1, transfer the indicated motion vector, with quarter pixel resolution, to the indicated destination(s).
These data are stored in the Results Buffer in node memory as shown in
Bit[24] allows the programmer to select from one of two logical output ports. For each one, the DNLR will contain the associated routing, input port number, and memory mode indication for the selected destination (See
The DONE command indicates to the DVN that there is no additional processing to be performed for the ‘current block’. After reading this command from the queue, the DVN will send the DONE status word to the controller; and the DVN FSM will transition to the ACK+TEST state.
After the DVN executes each command, it will send a status word(s) to the controller, using the ACM PTP protocol. The DVN TPL in node memory will contain the appropriate routing field, input port number, and memory mode indication that is required to support this operation. This information will be transferred from the TPL to the CNLR during ‘setup’ and ‘continue’ operations. For each of these status words, bits[31:28] will echo command bits[31:28].
Bits[31:28] will be set to 0xF for the status word that acknowledges the ‘Command Queue Redirection’ Indication.
The status word summary is shown in
There is no status word associated with Command 0x0: Wait.
For the following commands, the status word simply echoes command bits[31:28]; bits[27:0] are set to zeros. The DVN sends the status word(s) to the controller to indicate that the corresponding command has been executed.
For the following commands, the status word also echoes command bits[31:28], and additionally it includes some result that was produced during the execution of the corresponding command.
Bits[31:0] of this status word will echo command bits[31:0]. This is intended to be a diagnostics aid.
Bits[15:0] of this status word indicate the value stored in the threshold register for early termination of search.
If the search command included the enabling of early termination (Bit[24] set to 1 ′b1) and the best metric produced is less than the value in the STR, Bit[27] will be set to 1 ′b1, indicating that the DVN has stalled, waiting for the QRI; otherwise Bit[27] will be set to 1 ′b0, indicating that DVN processing will continue.
Bits[26:25] of this status word indicate whether it is the only one that will be sent after the completed execution of the last search command, or whether it is the first or second of two status words sent after the completed execution of the last search command:
When the search command instructed the DVN to search one or more sets of 5×5 positions or nine candidate fractional pixel positions, two status words will be sent to the controller following the execution of the command; otherwise, one status word will be sent.
Bits[24:16] will be set to zero.
Bits[15:0] will indicate the ‘best metric’ produced during the execution of the search command.
Bits[24:16] will be set to the Y component of the motion vector associated with the ‘best metric’ produced during the execution of the search command. The resolution will be quarter pixel.
Bits[15:0] will indicate the ‘best metric’ produced during the execution of the command.
Bits[24:16] will be set to the X component of the motion vector associated with the ‘best metric’ produced during the execution of the search command. The resolution will be quarter pixel.
Bits[15:0] will indicate the metric associated with the center position of the 5×5 set that produced the ‘best metric’ during the execution of the command.
A comparison is made between the ‘best metric’ produced for searches with the 16×16 current block and the sum of the four ‘best metrics’ produced by successive searches with each of four 8×8 blocks. These data are stored in the Results Buffer in node memory (See
Bits[15:0] of this status word indicate the better metric; and Bit[16]=1 ′b0/1 ′b1 indicates that the better metric resulted from ‘One Vector’ Mode/‘Four Vector’ Mode), respectively.
If the interpolate command included the enabling of early termination (Bit[24] set to 1 ′b1), the enabling of search function (Bit[17] set to 1 ′b1), and the best metric produced is less than the value in the STR, Bit[27] will be set to 1 ′b1, indicating that the DVN has stalled, waiting for the QRI; otherwise Bit[27] will be set to 1 ′b0, indicating that DVN processing will continue.
Bits[26:25] of this status word indicate whether it is the only one that will be sent after the completed execution of the last interpolate command, or whether it is the first or second of two status words sent after the completed execution of the last interpolate command:
When bit[17] of the interpolate command was set to 1′ b1, enabling the search function, two status words will be sent to the controller following the execution of the command;
When bit[17] was set to 1 ′ b0, only one status word will be sent.
Bits[24:0] will be set to zero.
Bits[24:16] will be set to the Y component of the motion vector associated with the ‘best metric’ produced during the execution of the command. The resolution will be quarter pixel.
Bits[15:0] will indicate the ‘best metric’ produced during the search of the nine fractional pixel candidate positions.
Bits[24:16] will be set to the X component of the motion vector associated with the ‘best metric’ produced during the execution of the command. The resolution will be quarter pixel.
Bits[15:0] will indicate the metric associated with the center position of the nine fractional pixel candidates.
When bit[27] of the command was set to 1 ′b0, bits[15:0] of this status word will be the selected ‘best metric’, a saturated, unsigned 16 bit integer. Bit[27] of the status word will be set to 1 ′b0 to echo bit[27] of the command.
When bit[27] of the command was set to 1 ′b1, bits[24:16] and bits[8:0] of this status word represent, respectively, the X and Y components of the selected motion vector, with quarter pixel resolution. X and Y represent the displacement from the upper leftmost pixel of the reference block, for which (by convention) the motion vector is X=0, Y=0. Bit[27] of the status word will be set to 1 ′b1 to echo bit[27] of the command.
Command Queue Redirection Acknowledgement
Bits[31:28] of this status word will be set to 0xF to indicate the detection of QRI. After the DVN sends this status word to the controller, it will update its 7-bit queue counter with the contents of the programmable 7-bit register and continue processing.
The controller node can read/peek certain DVN registers. This is accomplished by the controller node POKEing the DVN ‘send’ register, and the DVN sending a message to the controller that includes the ‘read/peek’ data.
To write the ‘send’ register, the programmer will use Service Code 0x6: Poke Address+Data (D16+A12), EU=1.
A12=12 ′b0000000000001; and
D16={14 ′b00000000000000, reg_sel[1:0]}.
The DVN will use ‘Service Code 0xA, EU-targeted Messages’ to send the ‘read/peek’ data to the controller node. Aux[5:0] will be set to 6 ′b000000. The data formats are shown in
The following have been identified as programming requirements for the DVN (likely there will be others). There is no implied order for these numerous operations.
Any suitable programming language can be used to implement the routines of the present invention including C, C++, Java, assembly language, etc. Different programming techniques can be employed such as procedural or object oriented. The routines can execute on a single processing device or multiple processors. Although the steps, operations or computations may be presented in a specific order, this order may be changed in different embodiments. In some embodiments, multiple steps shown as sequential in this specification can be performed at the same time. The sequence of operations described herein can be interrupted, suspended, or otherwise controlled by another process, such as an operating system, kernel, etc. The routines can operate in an operating system environment or as stand-alone routines occupying all, or a substantial part, of the system processing. Functions can be performed in hardware, software or a combination of both. Unless otherwise stated, functions may also be performed manually, in whole or in part.
In the description herein, numerous specific details are provided, such as examples of components and/or methods, to provide a thorough understanding of embodiments of the present invention. One skilled in the relevant art will recognize, however, that an embodiment of the invention can be practiced without one or more of the specific details, or with other apparatus, systems, assemblies, methods, components, materials, parts, and/or the like. In other instances, well-known structures, materials, or operations are not specifically shown or described in detail to avoid obscuring aspects of embodiments of the present invention.
A “computer-readable medium” for purposes of embodiments of the present invention may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, system or device. The computer readable medium can be, by way of example only but not by limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, system, device, propagation medium, or computer memory.
A “processor” or “process” includes any human, hardware and/or software system, mechanism or component that processes data, signals or other information. A processor can include a system with a general-purpose central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a geographic location, or have temporal limitations. For example, a processor can perform its functions in “real time,” “offline,” in a “batch mode,” etc. Portions of processing can be performed at different times and at different locations, by different (or the same) processing systems.
Reference throughout this specification to “one embodiment”, “an embodiment”, or “a specific embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention and not necessarily in all embodiments. Thus, respective appearances of the phrases “in one embodiment”, “in an embodiment”, or “in a specific embodiment” in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics of any specific embodiment of the present invention may be combined in any suitable manner with one or more other embodiments. It is to be understood that other variations and modifications of the embodiments of the present invention described and illustrated herein are possible in light of the teachings herein and are to be considered as part of the spirit and scope of the present invention.
Embodiments of the invention may be implemented by using a programmed general purpose digital computer, by using application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum or nano-engineered systems, components and mechanisms may be used. In general, the functions of the present invention can be achieved by any means as is known in the art. Distributed, or networked systems, components and circuits can be used. Communication, or transfer, of data may be wired, wireless, or by any other means.
It will also be appreciated that one or more of the elements depicted in the drawings/figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application. It is also within the spirit and scope of the present invention to implement a program or code that can be stored in a machine-readable medium to permit a computer to perform any of the methods described above.
Additionally, any signal arrows in the drawings/FIGs. should be considered only as exemplary, and not limiting, unless otherwise specifically noted. Furthermore, the term “or” as used herein is generally intended to mean “and/or” unless otherwise indicated. Combinations of components or steps will also be considered as being noted, where terminology is foreseen as rendering the ability to separate or combine is unclear.
As used in the description herein and throughout the claims that follow, “a”, an and “the” includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.
The foregoing description of illustrated embodiments of the present invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed herein. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of the present invention, as those skilled in the relevant art will recognize and appreciate. As indicated, these modifications may be made to the present invention in light of the foregoing description of illustrated embodiments of the present invention and are to be included within the spirit and scope of the present invention.
Thus, while the present invention has been described herein with reference to particular embodiments thereof, a latitude of modification, various changes and substitutions are intended in the foregoing disclosures, and it will be appreciated that in some instances some features of embodiments of the invention will be employed without a corresponding use of other features without departing from the scope and spirit of the invention as set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit of the present invention. It is intended that the invention not be limited to the particular terms used in following claims and/or to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention will include any and all embodiments and equivalents falling within the scope of the appended claims.
This application claims priority from the following: Provisional U.S. Patent Application 60/570,087 “Digital Video Node for An Adaptive Computing Machine”, by Master et al; U.S. patent application Ser. No. ______ [TBD]______ “Processor for Video Data,” filed on May 9, 2005, by Master, et al; U.S. Patent Publication No. 2003/0115553 “Computer Processor Architecture Selectively Using Finite-State-Machine for Control Code Execution” by Master et al; U.S. Patent Application No. 2004/0093601 “Method, System and Program for Developing And Scheduling Adaptive Integrated Circuitry and Corresponding Control or Configuration Information” by Master et al; U.S. Patent Publication No. 2004/0177225 “External Memory Controller Node” by Master et al; U.S. Patent Application Publication No. 2004/0181614 “Input/Output Controller Node in an Adaptable Computing Environment” by Master et al. These applications are each incorporated herein by reference for all purposes, as is U.S. Pat. No. 6,836,839 “Adaptive Integrated Circuitry with Heterogeneous and Reconfigurable Matrices of Diverse and Adaptive Computational Units Having Fixed, Application Specific Computational Elements” by Master et al.
| Number | Date | Country | |
|---|---|---|---|
| 60570087 | May 2004 | US |
