Embodiments of the invention relate generally to the field of data compression and storage and, more particularly, to compression and storage of image data for image processing.
Memory requirements can be a significant consideration in designing any data processing device or system. Such consideration is especially important in the design of system-on-chip (SOC) devices, which squeeze most, if not all, circuit functions on a single chip.
System-on-chip cameras, such as but not limited to cell phone cameras for example, typically integrate both image sensor and image processing hardware on the same semiconductor die. Consequently, the image processing hardware of SOC imager devices must be very area-efficient. Many imaging devices use line buffers to store and then simultaneously read out multiple lines of image data for image processing. As these line buffers can consume significant chip space in a SOC imager device, a reduction in the size of the line buffers is desirable.
In the following description, reference is made to the accompanying drawings, which form a part hereof. The description is provided to enable those of ordinary skill in the art to make and use the invention and to set forth several, non-limiting embodiments of the invention. Structural, logical, and procedural changes can be made without departing from the spirit and scope of the invention.
The imager device 100 also includes read out circuitry. In particular, the row lines are selectively activated by a row driver 110 in response to a row address decoder 120. The column select lines are selectively activated by a column selector 180 in response to a column address decoder 170. The pixel array 105 is operated by a timing and control circuit 160, which controls the address decoders 120, 170 to select the appropriate row and column lines for image capture and pixel signal read out. The pixel signals, which for a CMOS imager include a pixel reset signal (Vrst) and a pixel image signal (Vsig) for the read out of each pixel cell, are read by one or more sample and hold circuit 161 associated with the column selector 160.
A differential signal (Vrst-Vsig) is produced by differential amplifier 162 for each pixel. The differential signal is amplified and digitized by analog-to-digital converter (ADC) 175 to produce pixel data representing a pixel value for each pixel cell. For example, if the imager device 100 employs 10-bit pixel data, then the differential signal of each pixel cell is digitized to a 10-bit number representing a pixel value of 0 to 1023. Pixel data for one or more image lines is referred to as a line or lines of image data.
The analog-to-digital converter 175 supplies the pixel data to an image processing circuit 200.
The processing stage 380 of
In order to reduce the size of the line buffers 320A-E, related co-pending application Ser. No. ______, filed Jul. 25. 2007, and entitled “Method, Apparatus, and System for Reduction of Line Processing Memory Size Used in Image Processing” (attorney docket no. M4065.1379) discloses a method and apparatus for compressing incoming lines of image data before they are stored to multiple line buffers for image processing.
The line memory 300B attains a significant reduction in the size of the line buffers 340A-E (as compared to line buffers 320A-E). However, it is possible to achieve a further reduction in line buffer size by making a plurality of different compression techniques available and selecting the best available compression technique for each incoming line of image data. In accordance with embodiments of the invention, the selection is based on an assessment of the lengths of compressed lines of image that would be respectively generated by applying the available compression techniques to the incoming line of image data. Thus, the compression technique applied to one incoming line of image data may be different than the compression technique applied to the next incoming line of image data.
The selection is performed by an assessor 310A based on calculations of the respective lengths of the compressed lines of image data that would be generated by applying those available compression techniques to the incoming line of image data. For added clarity, these respective lengths are hereinafter referred to as “predicted lengths L” for an incoming line of image data because, as explained below, the predicted lengths L are calculated without actually performing compression. As the name implies, the predicted lengths L need not be precisely determined, but rather may be estimated or approximated by methods known to those skilled in the art.
The assessor 310A may use different criteria to determine which available compression technique should be used by compressor 330B to compress the incoming line of image data. As indicated, one criterion is ensuring that the compressed line of image data will fit within the line buffers 360A-E. Another criterion may be selecting a compression technique that will generate a “best fit” between the size of the compressed line of image data and the size of the line buffers 360A-E. By selecting from one of multiple compression techniques, the assessor 310 increases the maximum amount of compression that can be achieved for each incoming line of image data. By using the “best fit” criterion, the assessor 310A also maximizes the image quality that can be achieved for any particular size of the line buffers 360A-E and nature of incoming data. Accordingly, the assessor 310A provides imager device designers the capability to further reduce the size of the line buffers 360A-E (as compared to the line buffers 340A-E) with less or no sacrifice in image quality.
The assessor 310A provides compression information to the compressor 330B, which uses the compression information to determine and apply the selected compression technique. The lines of image data are compressed by the compressor 330B, stored to the line buffers 360A-E in FIFO order, decompressed by the decompressors 350A-E, and then read out for image processing at processing stage 380. Because each incoming line of image data may be compressed by a different compression technique, the assessor 310A provides respective compression information to each of the decompressors 350A-E. In this example, the assessor 310A distributes the compression information to both the compressor 330B and decompressors 350A-E by appending the compression information to the compressed line of image data, e.g., as header data. Alternatively, however, the compressor 330B may distribute the compression information directly to the compressor 330A and decompressors 350A-E by transmitting a compression information signal. As another alternative, instead of distributing the compression information to the compressor 330B and decompressors 350A-E, the assessor 310A may provide the compression information to a common storage accessible to those components. The distribution of compression information is further explained in the following discussion of assessor 310A, which is described with reference to embodiments shown in
Prior to providing examples of the assessor 31A, a brief explanation of lossless and lossy compression is provided below. In lossless compression, all bits of the pixel data are restored to their initial state by decompression. Thus, as implied by the name, no image information is lost. Entropy encoding is an example of lossless compression. In entropy encoding, pixel data are assigned bit representations of varying bit-lengths. More particularly, entropy encoding assigns bit representations having smaller bit lengths to pixel values of greater frequency, and assigns bit representations having longer bit lengths to pixel values of lesser frequency. For example, instead of representing each pixel value with 10-bit pixel data, the most frequently repeated pixel values may be represented with 3-bit pixel data, the next most frequently repeated pixel values represented with 4-bit pixel data, and so on until all pixel values are assigned a new bit representation. Because of the similarity in brightness amongst pixels within a small portion of an image, the same pixel values will be frequently repeated within a line of image data. As the amount of repetition increases, the corresponding compression from entropy encoding increases as well.
The assignments of bit representations are tracked by an entropy encoding table, which is used during decompression to perfectly restore the pixel data of the uncompressed line of image data. A new entropy encoding table may be generated for each compressed line of image data. However, as this may consume a significant amount of processing resources, a preset entropy encoding table may be used to assign the new bit representations. Preset entropy encoding tables do not perfectly assign new bit representations to the pixel values, i.e., they do not perfectly match the bit lengths of the bit representations to the frequencies of the pixel values. Therefore, the selection process of the assessor 310A may involve selecting a compression technique which employs a preset entropy encoding table predicted to generate a “best fit” between the compressed line of image data and the size of line buffers 360A-E.
Unlike lossless compression, lossy compression does not perfectly restore pixel data to its pre-compressed state. Thus, some image information is lost. Quantization is an example of lossy compression. In quantization, pixel data having one value is forced to a different though close pixel value. Related application Ser. No. ______ (attorney docket no. M4065.1379) provides an example of quantization in which 10-bit pixel data representing values 0 to 1023 is scaled by a scaling factor to 8-bit pixel data representing values 0 to 255. In that example, during compression, multiple pixel values of the 10-bit pixel data scale to the same pixel value of the 8-bit pixel data. Consequently, the 10-bit pixel data cannot be perfectly restored during decompression.
Many compression techniques, e.g., JPEG, employ a combination of quantization and subsequent entropy encoding. The quantization is used to increase the frequency of pixel values within a line of image data, which in turn increases the amount of compression resulting from entropy encoding. For example, rounding each pixel value within a line of image data down to the nearest multiple of 2 will make all of the pixel values even and thereby increase the frequency of even pixel values within the line of image data. During entropy encoding, the shortest bit representations can be assigned to the most frequent even pixel values (no odd pixel values remain).
The first predictor unit 410A determines the predicted length L for a compression technique in which no quantization is performed prior to entropy encoding. The second 410B, third 410C, and fourth 410D predictor units respectively determine the predicted length L for a compression technique in which the quantization rounds the pixel values down to a multiple of 2, 4, or 8 prior to entropy encoding. Thus, the second predictor unit 410B corresponds to a compression technique in which the pixel data is rounded down to the nearest even pixel value, and then new bit representations are assigned to the even pixel values. Similarly, the predictor units 410C-D respectively correspond to compression techniques in which the pixel data is rounded down to the nearest multiple of 4 and 8, and then new bit representations are assigned to pixel values that are multiples of 4 and 8.
In this example, as the incoming line of image data is streamed through the predictor units 410A-D, they each determine a predicted length L by summing the bit lengths of all bit representations that would be placed in the compressed line of image data by the respective compression technique. The running total may be calculated via a “counting table” indicating only the bit lengths of the bit representations corresponding to each possible pixel value within the incoming line of image data. If the incoming line of image data has a 10-bit format, then the possible pixel values would be 0 to 1023.
The counting table may be extrapolated from the quantization and entropy encoding operations of the compression technique. For instance, the compression technique of predictor unit 410B entails rounding the pixel data to the nearest multiple of 2 and then entropy encoding the quantized pixel data. If the preset entropy encoding table respectively assigns 2-bit representations of 00, 01, and 11 to the three highest frequency pixel values of 500, 502, and 504, and respectively assigns 3-bit representations of 000, 001, 010, 011, 100, 101, 110, and 111 to the next highest frequency pixel values of 402, 404, 406, 408, 506, 508, 510, and 512, then the counting table would indicate that 2 bits are added to the running total of the predicted length L for each pixel value of 500-505 within the incoming line of image data (because these pixel value are rounded to 500, 502, or 504 by the quantization), and that 3 bits are added to the running total of the predicted length L for each pixel value of 402-409 and 506-513 within the incoming line of image data (because these pixel values are rounded to 402, 404, 406, 408, 506, 508, 510, or 512 by the quantization). A counting table may be similarly formulated for each of the predictor units 410B-D.
By using a counting table or the like, each of the predictor units 410A-D can perform a running calculation of the respective predicted length L as the pixel data streams past, e.g., adding 2 bits to the predicted length each time a pixel value of 500-505 streams through predictor unit 410B. The running calculation therefore allows the predictor units 410A-D to determine the predicted length L without storing the entire incoming line of image data. However, the assessor 310A is not limited to storing counting tables or performing running calculations. The above-described use of counting tables is only one example of the manner in which the predicted length L may be determined for the available compression techniques. Other methods should be apparent to those skilled in the art in view of this disclosure.
The predicted lengths L are read out to a selector 420, which compares the predicted lengths L against the size of the line buffers 360A-E. If a predicted length L is less than or equal to the size of the line buffers 360A-E, then the corresponding technique is determined to be a “viable compression technique”, i.e., determined to be a compression technique that would compress the incoming line of image data to a compressed size fitting within the line buffers 360A-E. The selector 420 then selects a viable compression technique, from among the other viable compression techniques, having the greatest predicted length L. The assessor 310A thus ensures that the selected compression technique, as compared to the other available compression techniques, will generate a “best fit” between the size of the compressed line of image data and the size of the line buffers 360A-E.
The selector 420 also reads out compression information, which indicates the selected compression technique, to an appending unit 440 that adds the compression information to the beginning of the incoming line of image data. As some delay is associated with selecting a compression technique, the assessor 310A includes a buffer 430A that correspondingly delays receipt of the incoming line of image data by the appending unit 440. This delay provides the predictor units 410A-D and selector 420 sufficient time to perform their functions and distribute the compression information. In this example, as the compression information indicates only one of four compression techniques, the compression information may be represented by 2 bits of data, namely, 00, 01, 10, or 11, which is appended to the beginning of the incoming line of image data. The compressor 330B may accordingly use the first two bits of each incoming line of image data to determine which one of the four available compression techniques has been selected. The compressor 330B may also leave the 2 bits of compression information appended to the beginning of the compressed line of image data, such that the decompressors 350A-E may also utilize the appended compression information to determine the appropriate decompression technique.
The embodiments described herein can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the invention. Accordingly, the claimed invention is not limited by the foregoing description or drawings, but is only limited by the scope of the appended claims.