BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing summary of the invention, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention.
FIG. 1 is a block diagram of 3D graphics hardware according to at least some embodiments of the invention.
FIGS. 2A-2B illustrate an example of filtering according to at least some embodiments.
FIG. 3 is a flow chart for an algorithm similar to that described in connection with FIG. 2, and which is in some embodiments performed by an IC such as IC 10 of FIG. 1.
FIG. 4 is an HDR image magnified using conventional point sampling.
FIG. 5 is an HDR image is magnified using conventional floating point bilinear filtering.
FIG. 6 is an HDR image magnified using bilinear interpolation of integer values of floating point intensity values.
FIG. 7 shows two images (at different exposures) of 8×8 texels of very bright blue and green colors filtered using a conventional floating point bilinear filter.
FIG. 8 shows two images (at different exposures) of the same pattern from FIG. 7, but which has been bilinearly filtered by treating intensity values as integers.
DETAILED DESCRIPTION
Embodiments of the invention facilitate processing of graphical data such as high dynamic range (HDR) texture data using simpler hardware than would be required using conventional techniques. As a result, higher quality images can be rendered on a display more quickly. In at least some embodiments, integer bit patterns of floating point data values are filtered. For example, and as described in further detail below, one floating point data format defines the most significant bit of a multi-bit field as a sign bit, a group of the next most significant bits as an exponent, and the least significant bits as a mantissa. When performing a filtering computation using such a multi-bit floating point value, the bits are simply treated as the binary representation of an integer. After filtering operations are performed, the bit patterns of the filtered values are once again treated as floating point values. Although treatment of floating point values as integers is not mathematically equivalent to performing floating point calculations, the results are visually similar to (and in some cases better than) those obtainable from floating point filtering.
FIG. 1 is a block diagram of 3D graphics hardware according to at least some embodiments of the invention. The hardware of FIG. 1 includes one or more integrated circuit (IC) chips which are configured to process graphical data in one or more of the manners described herein. IC 10 includes control logic for performing calculations, for receiving data input (e.g., graphics to be displayed), for performing read/write operations and for performing other tasks associated with displaying graphical images. Random access memory (RAM) 12 stores image data (e.g., texture files) received and/or processed by IC 10, as well as other data. In at least some embodiments, IC 10 is a microprocessor that accesses programming instructions and/or other data stored in a read only memory (ROM) 14. In some such embodiments, ROM 14 stores programming instructions 16 that cause IC 10 to perform operations according to one or more of the methods described herein. In at least some other embodiments, one or more the methods described herein are hardwired into IC 10. In other words, IC 10 is in such cases an application specific integrated circuit (ASIC) having gates 18 and other logic dedicated to the calculations and other operations described herein. In still other embodiments, IC 10 may perform some operations based on execution of programming instructions read from ROM 14 and/or RAM 12, with other operations hardwired into gates and other logic of IC 10. IC 10 outputs image data to a display buffer 20. In particular, buffer 20 stores data specifying the red, green and blue color component intensities for each pixel of display 22. Display 22 may be, e.g., an LCD display.
For simplicity, FIG. 1 shows IC 10, RAM 12, ROM 14 and display buffer 20 as discrete elements. In some embodiments, however, some or all of these elements may reside on a single IC. The groupings shown with boxes 24, 26 and 28 are but three examples of the manners in which the components could be combined onto a single IC. Other combinations are implemented in other embodiments. In still other embodiments, the functions described in connection with any of IC 10, RAM 12, ROM 14 and display buffer 20 are distributed across multiple ICs.
In at least some embodiments, the hardware of FIG. 1 is incorporated into a larger device 30. Device 30 could be a mobile communication device (e.g., a cellular telephone, a mobile telephone having wireless internet connectivity, or another type of wireless mobile terminal) having a speaker, antenna, communication circuitry, a keypad (and/or other input mechanism(s)), etc. Device 30 could alternatively be a PDA, a notebook computer, a desktop computer (e.g., a PC), a video game console, etc.
FIGS. 2A-2B illustrate an example of filtering according to at least some embodiments. Shown in FIG. 2A are four bitmaps 40a, 40b, 40c and 40d for a mip-mapping image pyramid corresponding to an arbitrary texture pattern 42. In the example of FIGS. 2A-2B, texture 42 is to be mapped to a surface having a screen pixel size that is smaller than in bitmap 40a, but that is larger than in bitmap 40b. A sampling point P corresponds to a point on the surface to be rendered. Although point P may be an actual screen pixel of the display, this need not be the case. Also shown in FIG. 2A are four regions (represented in FIG. 2A as squares) in each of bitmaps 40a and 40b that are nearest to sampling point P. In bitmap 40A, those regions (or texels) are numbered 43, 44, 45 and 46. In bitmap 40b, the four nearest texels are numbered 48, 49, 50 and 51. Each of the texels 43-46 and 48-51 (as well as other texels in bitmaps 40a, 40b, 40c and 40d) is stored as a set of three 32-bit data values. Each of the 32-bit data values for each texel represents an intensity of a red, green or blue color component.
In the example of FIG. 2A, bit patterns for texel data values are stored as single precision floating point values according to IEEE standard 754. In particular, the most significant bit (MSB) of each bit pattern represents the sign, with 0 indicating a positive value and 1 indicating a negative value. The next eight most significant bits represent a base-2 exponent biased by 127. The remaining 23 bits represent a mantissa. By way of illustration, and when interpreted as a floating point value, the bit pattern “01000010010111011010011100100010” corresponds to a decimal value of 55.413216. For simplicity, the example of FIGS. 2A-2B assumes that all texel data values for bitmaps 40a-40d are positive. Treatment of negative values is discussed below.
When performing filtering calculations on the bit patterns for the texel values of bitmaps 40a and 40b, those bit patterns are not treated as floating point values. Instead, the filtering calculations are performed with the texel bit patterns interpreted as binary representations of integers. Continuing the illustration from above, the bit pattern “01000010010111011010011100100010” represents a decimal value of 1,113,433,890 when interpreted as an integer. For convenience, interpretation of a bit pattern as an integer will also be referred to as using the integer value of that bit pattern.
As further shown in FIG. 2A, bilinear filtering for point P in bitmap 40a yields three 32-bit patterns representing weighted averages of the red, green and blue components of texels 43-46. Beginning at the right side of FIG. 2A under bitmap 40a, bit patterns 43R, 44R, 45R and 46R are the red color component intensities of texels 43-46, and as mentioned above are stored (e.g., in RAM 12 in FIG. 1) as 32-bit floating point values. Bit patterns 43G-46G and 43B-46B are, respectively, values for the green and blue color component intensities of texels 43-46. As also indicated above, bit patterns 43G-46G and 43B-46B (as well as bit patterns for other texels of bitmaps 40a-40d) are also stored as 32-bit floating point values. A weighted average R′ (also a pattern of 32 bits) is calculated by treating values 43R-46R as integers and averaging their integer values based on the position of point P relative to each of texels 43-46. A 32-bit weighted average G′ is calculated by treating values 43G-46G as integers and averaging their integer values based on the position of point P relative to each of texels 43-46. A 32-bit weighted average B′ is calculated by treating values 43B-46B as integers and averaging their integer values based on the position of point P relative to each of texels 43-46.
Bit patterns 48R-51R are the red color component intensities of texels 48-51. Bit patterns 48G-51G and 48B-51B are, respectively, values for the green and blue color component intensities of texels 48-51. A similar bilinear filtering for point P in bitmap 40b yields three 32-bit values R″, G″ and B″ for the weighted averages of the red, green and blue components of texels 48-51.
Trilinear filtering is then performed by interpolating between the 32-bit bilinearly-filtered color intensity values for bitmaps 40a and 40b, with those values (R′, R″, G′, G″, B′, B″) again treated as integers. The interpolation is based on relative sizes of bitmaps 40a and 40b and of the surface onto which the texture is being mapped. Integer values of the bit patterns R′ and R″ are interpolated to yield a 32-bit value R(P). In a similar fashion, G′ and G″ are interpolated to yield G(P), and B′ and B″ are interpolated to yield B(P). Similar bilinear and trilinear filtering operations are performed for other sampling points and other texels.
The trilinearly-filtered values R(P), G(P) and B(P) (as well as similarly obtained values for other sampling points) may then be subjected to further processing (e.g., pixel shading, anisotropic filtering, etc.). In some embodiments, at least some portions of that processing also treats bit patterns representing color intensity values as integers. In other embodiments, the additional processing treats 32-bit color intensity values as floating point values. Ultimately, and as shown in FIG. 2B, the processed color intensity values are used to control the pixels of a display screen (e.g., display 22 of FIG. 1).
As indicated above, the example of FIGS. 2A-2B assumes that all texel values are positive. For negative texel color intensity values, additional steps are performed. Because a MSB of “1” is used to indicate that a floating point value is negative, that bit can be masked out before interpreting the bits of a negative color intensity value as an integer. Otherwise, the “1” MSB would (for 32-bit floating point values) increase the integer value of the color intensity by 2,147,483,648 (i.e., 231). The sign of the color intensity can be preserved by, e.g., a separately-stored flag. As another alternative, integer values may be transformed, prior to filtering, so that the entire numeric range of intensity values is continuous in the integer domain. For example, and if intensities are stored with 16 bits, the 15 least significant bits could be inverted for non-negative values before and after filtering. In many circumstances, however, provision for negative values is not needed.
In some cases, negative numbers could be obtained from mathematical look-up tables. In some such cases, and if interpolation of arbitrary look-up table values is required, conventional bilinear interpolation and/or pixel shading could be used.
FIG. 3 is a flow chart for an algorithm similar to that described in connection with FIGS. 2A-2B, and which is in some embodiments performed by an IC such as IC 10 of FIG. 1. In block 101, the bit maps of a texture image pyramid appropriate for mip-mapping that texture to particular surface are identified. In step 102, a sampling point P within each of the identified texture bitmaps is selected. In step 103, the texels closest to the sampling point P in the first identified bitmap are located. Each of the texels is represented by three bit patterns storing floating point values: a value for a red color component intensity, a value for a green color component intensity, and a value for a blue color component intensity. In block 108, a weighted average is calculated using the integer values of the bit patterns representing the red color component intensities of the texels located in block 103. Similar weighted averages are calculated for the green and blue intensities in blocks 109 and 110, respectively.
The algorithm then proceeds to block 113, where the four texels of the second identified bitmap closest to sampling point P are located. The algorithm then proceeds through blocks 115-117, where weighted averages are calculated using the integer values of the bit patterns representing the red, green and blue color component intensities of the texels located in block 113. The algorithm next proceeds to block 121. In block 121, the results from blocks 108 and 115 are interpolated. In block 122, the results from blocks 109 and 116 are interpolated. In block 123, the results from blocks 110 and 117 are interpolated. In blocks 126-128, additional processing may be performed on the interpolated values from blocks 121-123. As indicated above, this additional processing could include pixel shading, additional filtering, etc. Blocks 126-128 are shown in broken lines to indicate that some or all of the additional processing may be omitted.
The algorithm then proceeds to block 130 and determines if there are additional sampling points to be processed. If so, the algorithm proceeds on the “yes” branch to block 102 and repeats the above-described operations. If there are no additional sampling points to process, the algorithm proceeds on the “no” branch. As indicated by the ellipsis in FIG. 3, the texel data may then be further processed. In block 131, the texel data (or further processed data based on the texel data) is used to generate an image on a display.
There are numerous variations on the above described processing in other embodiments. For example, the sampling point P could be mapped to other bitmaps of the texture image pyramid (e.g., to bitmaps 40b and 40c or to bitmaps 40c and 40d). The bilinear and trilinear filtering could be performed in the opposite order, and/or additional (and/or different types) of filtering performed. Although the above examples were described using single precision IEEE floating point format, other types of floating point formats could be used.
Because calculations with integers can be performed much faster than and with fewer operations than calculations with floating point values, the processing techniques described above permit higher quality rendering without the additional processing requirements associated with floating point calculations. FIGS. 4-8, which are color images that have been printed in black and white, demonstrate results of image processing using techniques such as are described above. FIG. 4 is a high dynamic range (HDR) image magnified using conventional point sampling. FIG. 5 is an HDR image that is magnified using conventional floating point bilinear filtering. The saturated regions of the image include jagged edges despite the interpolation. FIG. 6 is an HDR image magnified using bilinear interpolation of integer values of floating point intensity values. The non-linearity of the integer value interpolation produces smother highlights while maintaining the appearance in other regions of the image. FIG. 7 shows an 8×8-texel image, containing very bright blue and green pixels, at two different exposures and a very large magnification, filtered using a conventional bilinear filter. Most of the intensity values are saturated to blue or green, and a cyan halo (not present in the original image) is generated. FIG. 8 is otherwise the same as FIG. 7, but the images have been produced by bilinear filtering such that the intensity values are treated as integers. Although a dark halo is generated, the overall result is smoother.
Techniques in accordance with various embodiments of the invention can be implemented in numerous ways. For example, existing graphical processing software can be modified to include such techniques. As one illustration of such a modification, Appendix A contains code which can be added to source code for the “exrdisplay” program (available from Industrial Light & Magic, a division of Lucas Digital Ltd. LLC of California, USA), which program can be used to display images in the OPENEXR format. Persons skilled in the art could insert the code in Appendix A into the proper location of the exrdisplay program without undue experimentation.
Although specific examples of carrying out the invention have been described, those skilled in the art will appreciate that there are numerous variations and permutations of the above-described systems and methods that are contained within the spirit and scope of the invention as set forth in the appended claims. As but one example, bilinear filtering is in some embodiments performed using integer values of bit patterns, but trilinear filtering is performed using floating point values of bit patterns. These and other modifications are within the scope of the invention as set forth in the attached claims. In the claims, various portions are prefaced with letter or number references for convenience. However, use of such references does not imply a temporal relationship not otherwise required by the language of the claims.
APPENDIX A
|
|
bool bFilterUsingFloats = false;
// boolean or integer filtering
|
float ratioX = 0.5f;
// scaling factors
|
float ratioY = 0.5f;
|
Array<Rgba> scaled (w*h);
// filtered image
|
int limitX = w;
|
int limitY = h;
|
if (ratioX >= 1.0f) {
|
limitX = w/ratioX;
|
}
|
if (ratioY >= 1.0f) {
|
limitY = h/ratioY;
|
}
|
float fractionX, fractionY, oneMinusX, oneMinusY;
|
int ceilX, ceilY, floorX, floorY;
|
Rgba c1, c2, c3, c4, result;
|
for (int x = 0; x < limitX; ++x) {
|
for (int y = 0; y < limitY; ++y) {
|
floorX = (int)floor(x*ratioX);
|
floorY = (int)floor(y*ratioY);
|
ceilX = floorX + 1;
|
if (ceilX >= w) {
|
ceilX = floorX;
|
}
|
ceilY = floorY + 1;
|
if (ceilY >= h) {
|
ceilY = floorY;
|
}
|
fractionX = x * ratioX − floorX;
|
fractionY = y * ratioY − floorY;
|
oneMinusX = 1.0 − fractionX;
|
oneMinusY = 1.0 − fractionY;
|
c1 = mainWindow->pixels[(floorY*w)+floorX];
|
c2 = mainWindow->pixels[(floorY*w)+ceilX];
|
c3 = mainWindow->pixels[(ceilY*w)+floorX];
|
c4 = mainWindow->pixels[(ceilY*w)+ceilX];
|
if (bFilterUsingFloats) { // filter using 16-bit halfs
|
half b1, b2;
|
// R
|
b1 = (half)(oneMinusX * c1.r + fractionX * c2.r);
|
b2 = (half)(oneMinusX * c3.r + fractionX * c4.r);
|
result.r = (half)(oneMinusY * float(b1) + fractionY * float(b2));
|
// G
|
b1 = (half)(oneMinusX * c1.g + fractionX * c2.g);
|
b2 = (half)(oneMinusX * c3.g + fractionX * c4.g);
|
result.g = (half)(oneMinusY * float(b1) + fractionY * float(b2));
|
// B
|
b1 = (half)(oneMinusX * c1.b + fractionX * c2.b);
|
b2 = (half)(oneMinusX * c3.b + fractionX * c4.b);
|
result.b = (half)(oneMinusY * float(b1) + fractionY * float(b2));
|
scaled[(y*w)+x] = result;
|
} else { // filter using 16-bit shorts
|
unsigned short temp1, temp2;
|
// R
|
temp1 = (unsigned short)(oneMinusX * c1.r.bits( ) + fractionX *
|
c2.r.bits( ));
|
temp2 = (unsigned short)(oneMinusX * c3.r.bits( ) + fractionX *
|
c4.r.bits( ));
|
result.r.setBits((oneMinusY * temp1) + (fractionY * temp2));
|
// G
|
temp1 = (unsigned short)(oneMinusX * c1.g.bits( ) + fractionX *
|
c2.g.bits( ));
|
temp2 = (unsigned short)(oneMinusX * c3.g.bits( ) + fractionX *
|
c4.g.bits( ));
|
result.g.setBits((oneMinusY * temp1) + (fractionY * temp2));
|
// B
|
temp1 = (unsigned short)(oneMinusX * c1.b.bits( ) + fractionX *
|
c2.b.bits( ));
|
temp2 = (unsigned short)(oneMinusX * c3.b.bits( ) + fractionX *
|
c4.b.bits( ));
|
result.b.setBits((oneMinusY * temp1) + (fractionY * temp2));
|
scaled[(y*w)+x] = result;
|
}
|
}
|
}
|
// replace the original image with the scaled one
|
for (int i = 0; i < w*h; ++i) {
|
mainWindow->pixels[i] = scaled[i];
|
}
|
|
Patent Application
20080001961
