Patent Grant
10063891
The present application claims priority to European Patent Application No. 15202608.4, filed Dec. 23, 2015, the disclosure of which is hereby incorporated by reference herein in its entirety.
The present disclosure relates to data processing, and more specifically to the encoding and compression of image data.
The amount of images produced worldwide is in constant growth. Photographic and video data, by its nature, consumes a large part of available digital resources, such as storage space and network bandwidth. Image compression technology plays a critical role by reducing storage and bandwidth requirements. There is therefore a need for high compression ratios whilst ensuring a quantifiably negligible loss of information.
Imaging is the technique of measuring and recording the amount of light L (radiance) emitted by each point of an object, and captured by the camera. These points are usually laid-out in a two-dimensional grid and called pixels. The imaging device records a measured digital value dadc(x, y) that represents L(x, y), where (x, y) are the coordinates of the pixel. The ensemble of values dadc(x, y) forms a raw digital imageM. Typically M consists of several tens of millions pixels and each dadc(x, y) is coded over 8 to 16 bits. This results in each raw image requiring hundreds of megabits to be represented, transferred and stored. The large size of these raw images imposes several practical drawbacks: within the imaging device (photo or video camera) the size limits the image transfer speed between sensor and processor, and between processor and memory. This limits the maximum frame rate at which images can be taken, or imposes the use of faster, more expensive and more power consuming communication channels. The same argument applies when images are copied from the camera to external memory, or transmitted over a communication channel. For example:
Lossy image compression methods, such as those in the JPEG standard or used in wavelet compression, create distortion and artifacts. Distortion is the degree by which the decoded image Q, i.e. the result of compressing and decompressing the original image, differs from the original image M, and is usually measured as the root-mean-square of the differences in the values of each corresponding pixel between Q and M. Lossy compression typically also introduces artefacts, which are a particularly bad type of distortion that introduces image features in Q, i.e. correlations between different pixels, that are not present in M. Examples of artefacts are the “block” artifacts generated by block image compression algorithms like JPEG, but also ringing, contouring, posterizing. These artifacts are particularly nefarious, as they can be mistaken for image features. Lossy image compression algorithms, as described for example in prior art documents [1] and [2] cited below, typically consist of several steps. First, the image is transformed into a representation where the correlation between adjacent data point values is reduced. This transformation is reversible, so that no information is lost. For example, this step would be similar to calculating the Fourier coefficients, i.e. changing an image that is naturally encoded into a position space, into a special frequency space. The second step is referred to as quantization. This step truncates the value of the calculated coefficients to reduce the amount of data required to encode the image (or block, consisting of e.g. 16×16 pixels). This second step is irreversible, and will introduce quantization errors when reconstructing the image from the aforementioned coefficients. The quantization error will cause errors in the reconstructed image or block. Predicting or simulating what the error would be for any image or block is technically impossible, due to the extremely large number of values that such image or block may take. For example, a 16×16 block, with 16-bits per pixel, may take 216×16×16≅104932 different values, impossible to test in practice. If the transformation is not restricted to blocks, but rather is a function of the entire image, the number of possible values becomes considerably larger.
More particularly, in standard lossy image compression techniques (e.g. document [1]), a lossless “image transformer” is first applied to the image, resulting in transformed image, where each data-point is a function of many input pixels and represents the value of a point in a space that is not the natural (x,y) position space of the image, but rather is a transformed space, for example a data-point might represent a Fourier component. A lossy “image quantizer” is applied on the transformed image as a second step. The amount of information loss cannot be accurately quantified in standard lossy image compression techniques for the above mentioned reason of the extremely large number of values that such image or block may take and is additionally complicated by the fact that the lossy operation is applied on a space that is not the natural image space.
Attempts have been made at characterizing the quality of the output of image compression algorithms. These efforts have typically focused on characterizing the quality of the reconstructed compressed image with respect to the human visual system, as reviewed in documents [2, 3, 4, 5] described below. Several metrics have been devised to characterize the performance of the compression algorithm, however, these metrics have significant drawbacks. One such drawback is that as they relate quality to the human visual system and human perception, which are highly dependent on the subject viewing the image, on the viewing conditions (e.g. eye-to-image distance, lighting of the image, environment lighting conditions, attention, angle, image size) as well as on the image rendering algorithms (e.g. debayering, gamma correction) and characteristics of the output device, such as display, projector screen, printer, paper etc. A second drawback of characterizing the quality of image compression algorithms with respect to the human visual systems or models thereof is that for such methods to be relevant, no further image processing must take place after image compression, as such processing would make some unwanted compression artifacts visible. For example, a compression algorithm might simplify dark areas of an image, removing detail, judging that such detail would not be visible by the human eye. If it is later decided to lighten the image, this detail will have been lost, resulting in visible artifacts. Yet another problem of the above-mentioned methods is that they are unsuitable for applications not aimed solely at image reproduction for a human observer, as for example are images for scientific, engineering, industrial, astronomical, medical, geographical, satellite, legal and computer vision applications, amongst others. In these applications data is processed in a different way as by the human visual system, and a feature invisible to the untrained human eye and therefore removed by the above-mentioned image compression methods, could be of high importance to the specific image processing system. For example, inappropriate image compression of satellite imagery could result in the addition or removal of geographical features, such as roads or buildings.
More recently, attempts at a more quantitative approach on the information loss in digital image compression and reconstruction have been made [6], in particular trying to examine information loss given by the most common image compression algorithm: Discrete Wavelet Transform (DWT), Discrete Fourier Transform (DFT) and 2D Principal Component Analysis (PCA). Several “quantitative” metrics are measured on several images compressed with several algorithms, for example quantitative measures of “contrast”, “correlation”, “dissimilarity”, “homogeneity”, discrete entropy, mutual information or peak signal-to-noise ratio (PSNR). However, the effect of different compression algorithms and parameters on these metrics highly depends on the input image, so that it is not possible to specify the performance of an algorithm, or even to choose the appropriate algorithm or parameters, as applying them on different images gives conflicting results. A further drawback of the methods described in [6], is that although these methods are quantitative in the sense that they output a value that might be correlated with image quality, it is unclear how this number can be used: these numbers cannot be used as uncertainties on the value, the compression methods do not guarantee the absence of artifacts, and it is even unclear how these numbers can be compared across different compression methods and input images. Also, these quantitative methods do not distinguish signal from noise, so that for example, a high value of entropy could be given by a large amount of retained signal (useful) or retained noise (useless).
The impossibility for the above-mentioned methods to achieve compression with quantifiable information loss arises from the fact that the space in which the algorithm looses data is very large (e.g. number of pixels in the image or block times the number of bits per pixel), and cannot therefore be characterized completely, as we have mentioned earlier.
Image-processing systems that act independently on individual (or small number of) pixels have been known and used for a long time. These systems typically provide a “look-up table” of values so that each possible input value is associated with an “encoded” output value. The main purpose of such processing has been to adapt the response curves of input and output devices, such as cameras and displays respectively. The functions used to determine the look-up tables have typically been logarithmic or gamma law, to better reflect the response curve of the human visual system as well as the large number of devices designed to satisfy the human eye, starting from legacy systems like television phosphors and photographic film, all the way to modern cameras, displays and printers that make use of such encodings. Although not their primary purpose, these encodings do provide an insignificant level of data compression, such as for example encoding a 12-bit raw sensor value over 10-bits, thus providing a compression ratio of 1.2:1, as described, for example in [7].
In view of the above, it is therefore an object of the invention to provide a data compression method that permits to compress data with a high compression ratio with minimal information loss and to provide a technique for determining the information loss or the uncertainty with respect to the compression ratio.
A further object of the invention is to provide a decompression method adapted to decompress the data compressed by the compression method of the invention.
A further object of the invention is a data processing method comprising the compression method and the decompression method.
Here are described the embodiments of an image compression system and method that have a number of advantages:
To achieve the above advantages, the embodiments presented here distinguish the amount of image information from the amount of image data. Information is the useful knowledge acquired about the photographed object by means of the camera. Data is the digital representation of the measurement result, and contains information as well as noise and redundancy. In general, the amount of data captured will be much larger than the amount of information, so that it is possible to eliminate a large amount of data whilst eliminating only very little information.
Before going into its details in the next section, an overview of an embodiment is provided. First, the imaging system and compression method (e.g. sensor and processing) are characterized, for example by calibration, by simulation or theoretically, to determine the optimal pre-compression parameters. In contrast to the standard lossy image compression techniques, an appropriately-tuned lossy pre-compression, or quantizer, is applied directly on the acquired raw image data, as a pre-compression step. The amount of information lost by this quantizer can be calculated or measured, and appropriate parameters chosen to guarantee that such information loss remains negligible. Pre-compression removes most of the noise, which cannot, by its nature, be losslessly compressed, however, thanks to the properly chosen parameters, this step only removes a negligible amount of information. The appropriately-quantized image data, that now contains mostly information and little noise, is fed to a lossless compression algorithm as a second step. This algorithm will work very effectively, as more redundancy can be found in this clean data than could be found in the raw image sensor data that contained a large amount of noise. As now the second step is lossless, it will not reduce the amount of information present in the image and therefore does not need to be characterized for information loss.
The amount of information lost can be characterized as the increase in the uncertainty (i.e. the standard deviation, or “error bar”) in the measured value of the radiance after the application of the compression and decompression algorithm, and define that this loss is negligible when the uncertainty on pixel values after decompression remains smaller than the standard deviation between the pixel values of consecutive photographs of the same subject.
The invention relates to a data compression method wherein the data comprises noise and information, comprising a data acquisition step, a pre-compression parameter selection step wherein said pre-compression parameters are linked to an information loss, a compression step, and a storage step, said compression step being characterized in that it comprises a lossy pre-compression for removing some noise of the data, carried out using the pre-compression parameters determined in the calibration step followed by lossless compression for compressing the remaining data.
Preferably, the compression method comprises a parameter determination step for determining pre-compression parameters by establishing a plot of the information loss vs. said pre-compression parameters.
Preferably, the parameter determination step determines pre-compression parameters adapted to remove most of the noise.
Preferably, the parameter determination step comprises a system characterization step outputting the amount of image information that is lost by the compression algorithm as a function of pre-compression parameters and a selection step, wherein a user chooses appropriate pre-compression parameters.
Preferably, the system characterization step comprises
The invention also relates to a data decompression method comprising
where:
Δ(L, g), Δ is the relative increase in uncertainty, here as a function of light hitting the pixel and pre-compression parameter, and on its own,
do is the de-compressed digital value,
σo (x, y) is the uncertainty (standard deviation) associated with the measurement estimate of the radiance L(x, y) at point (x, y), after compression,
σi(x, y) is the uncertainty (standard deviation) associated with the measurement estimate of the radiance L(x, y) at point (x, y), before compression,
Lσo(x, y) is the estimate of radiance L(x,y) as derived from measurement, after compression,
Lσi(x, y) is the estimate of radiance L(x,y) as derived from measurement, before compression,
g is the pre-compression parameter, and
nj is the number of samples.
The invention also relates to a data processing method comprising the compression method followed by the decompression method described above.
The invention also relates to a computer program run on a computer or a dedicated hardware device, like a FPGA, adapted to carry out the data processing method described above.
n
.
How an image is typically acquired is described more in detail as follows. This is illustrated in dADC(x, y)
can be related to the mean value of the number of photoelectrons
n
by proportionality constant ζ and an offset constant c:
dADC(x,y)
=ζ
(n(x,y)
+c (EQ1)
Similarly, the mean number of photoelectrons n
is proportional to the number of photons γ(x, y) that impinge on the pixel during the integration time T with a proportionality constant called the quantum efficiency QE:
n(x,y)
=QE×
γ(x,y)
(EQ2)
The number of photons γ(x, y) is proportional to the radiance L(x, y), i.e. the amount of luminous power emitted by the object by unit area and solid angle. γ(x, y) is also proportional to other constants, such as the observed area A (function of the imaging system (304) and pixel size), observed solid angle Ω (function of the imaging system (304)), exposure time T, and inversely proportional to photon energy E=hc/λ where h is Planck's constant and λ is the wavelength of the incident light. This can be summarized as:
Substituting Equation (EQ3) into Equation (EQ2) and the resulting equation into Equation (EQ1), one obtains:
Setting Z=ζQEAΩT/E,dADC(x,y)
=ZL(x,y)+c (EQ5)
So that L(x, y) can be directly evaluated from dADC(x, y)
as:
And on for a single shot (i.e. without averaging) one may evaluate Lσi(x, y) as:
Equation (EQ7) serves as an example for a system that is very linear. If this is the case, and the sensor, amplifier and converter result in a shot-noise limited measurement, the number of photoelectrons will be Poisson-distributed, so that the uncertainty in photoelectron number will be Δn=√{square root over (n)}, and the relative uncertainty
If the sensor, amplifier or ADC presents non-linearity, it is necessary to calibrate them, for example by creating a look-up-table LUTd→L and LUTd→σi that associates a value of L and σi to each possible value of dADC. How this is done in practice is described as follows.
In conclusion, the raw data will be the recorded digital value dadc(x, y), the information acquired about the object radiance L(x, y) will have value Lσi(x, y) with uncertainty σi(x, y). When image data undergoes compression and decompression, resulting in the possibility to infer an output Lσo(x, y) with an uncertainty σo(x, y), it is then possible to quantify the amount of information lost by looking at the relative increase in uncertainty: σo(x, y)/σi(x, y), i.e. the relative increase in error bars.
An overview of a preferred embodiment of the method of the present invention will now be described with reference to
This preferred embodiment is a data processing method which is preferably split into 3 phases as depicted in
1) The first phase also called “parameter determination” (250) and represented in
2) The second phase is the compression method of the present invention represented in
3) The third phase is the decompression method of the present invention represented in
It is important to note that even if the data processing method of the invention comprises these three phases, each single phase can be carried out independently from each other. Such that the present invention comprises a parameter determination method, a compression method, a decompression method as well as a data processing method comprising one or more of these three methods.
We will now describe the first phase of parameter determination.
Parameter Determination Phase for System Calibration
As shown in
A sample output of this step is shown in n
).
In the description below, the pre-compression parameters is a single parameter relating to a pre-defined pre-compression function; however, in general, it may be represented by multiple parameters, or the look-up-tables LUTd→L and LUTd→σi. The selected parameters and associated uncertainty increase bounds are saved to be reused in the next phase: Compression.
System Characterization
An embodiment of this procedure is shown in
For clarity reason, a list of the symbols used here is available at the end of the present specification.
First, the amount of light to be measured is set, starting from L=0 (800), then, a number of images is acquired (801). This can be done using the real image acquisition system, by simulating such a system, or by theoretical modeling. In this example, the data arising from a shot-noise limited linear sensor is simulated. Assuming linearity means that the number of photoelectrons n will be proportional to the radiance L to be measured (at constant T, Ω, QE, A, E): n
=LZ/ζ. For simplicity of the simulation, ζ is set equal to 1 and c=0 so that dADC=n. For a shot-noise limited system, n will be distributed according to a Poisson distribution N(LZ/ζ)=N(
n
) that has mean
n
, variance
n
and standard deviation √{square root over (n)}. With the above simplifications, the relative error σi/
Li
will be equivalent to the relative uncertainty in the measurement of the number of photoelectrons √{square root over (n)}/n=1/√{square root over (n)}. To simulate sensor data in (801) for a specific value of
n
, we generate a number of samples n1 issued from N(
n
). Instead of simulating data, (801) may acquire data from the imaging system.
The relative input uncertainty σi(L)/L is calculated or numerically computed as root-mean-square (RMS) difference between each Li, the measured value of L, and the actual (set-point) value of L, divided by L itself (802).
A pre-compression parameter, or set of pre-compression parameters is chosen (804). In our simple example, a single parameter g is used. However, more in general, this could be a look-up-table (LUT) associating an output value of Lo to each possible value of Li.
Lossy pre-compression is then applied to each input measured value of Li (808). In this embodiment, the pre-compressed digital value dc is obtained from the function
where n is the number of photoelectrons on that sample, i.e.
with respect to the ADC output, n=(dADC−c)/ζ. The double-line “brackets” symbolize taking the nearest integer. In this case, g represents the reduction factor in the number of bits required to encode each possible number of photoelectrons. For example, encoding a number of photoelectrons between 0 and 216−1=65535 requires 16 bits. For g=1, no pre-compression would arise, and 16 bits would be required. For g=2, the number of encoding bits is reduced by a factor of 2, so that the result of the pre-compression can be encoded over 8 bits. More in general, any LUT can be applied.
A lossless compression algorithm may then be applied (810), either on a pixel-by-pixel basis, or on larger blocks, or on the entire pre-compressed image. The data to which this lossless compression is applied is called F. As after the lossy pre-compression is applied, the amount of data is reduced, but also, the entropy of such data is reduced (noise is removed) so that lossless compression will be very effective. In our example, we have used several algorithms, ranging from the universal and simple Lempel-Ziv-Welch (LZW) [8] to image-content-based algorithms such as minimum-rate predictors (MRP) [9]. The algorithm here can be chosen at will, as being lossless, it will not lose any information, and will not affect the uncertainty of the image. In some instances, if compression speed is of the essence, this step may be a simple “pass-through” not affecting the pre-compressed data at all.
In the next step (812), the output f of this lossless compression is decompressed to obtain F, then, the inverse function of the lossy compression is applied (or the inverse LUT). For example, the inverse of
is do→dcg.
From the de-compressed output do it is possible to calculate a value Lo for the radiance, e.g. Lo=doζ/Z. In (814) the error that was introduced as the difference Lo−Li is calculated. This error can be estimated for several samples of the acquired image data, and the root-mean-square (RMS) of this error calculated as σo(L, g), and compared to the input error (uncertainty) σi(L) to obtain a well-defined, properly quantified, relative increase in uncertainty Δ(L, g)=σo(L, g)/σi(L). The compression ratio r can also be calculated as the size of the original image data divided by the size of output f of the lossless compression.
By repeating the above process for a number of image acquisition samples, for a number of compression factors g (816, 824) and for a number of Radiances, integration times or generated photoelectrons (818, 822), one obtains a characterization of the compression system (820), i.e. the amount of information lost Δ(L, g) as a function of pre-compression parameters g and pixel radiance L (or equivalently, photoelectron number n or pixel value dc or do, as all these are relate by known equations).
Sample results of this characterization are shown in n
, and the pre-compression is held fixed at g=2. The results show that Δ(L, g)=σo(L, g)/σi(L)=1.15 across the range for all these settings. This is smaller than what was defined as the bound for negligible information loss, i.e. the RMS of the difference between consecutive measurements (“photographs”) of the same pixel, of exactly the same subject taken with an ideal shot-noise-limited sensor, that is √{square root over (2)}≅1.41.
Testing this pre-compression factor g=2, and therefore Δ=1.15 on an image of a 10 EUR note, acquired with an ATIK383L scientific camera, the compression factors were as follows:
Original image file: 5,164,276 bytes
Lossless compressed original file (ZIP): 4,242,951 bytes.
Lossy Pre-compressed image: 2,580,219 bytes
Lossless compressed pre-compressed image (ZIP): 1,015,808 bytes
Lossless compressed pre-compressed image (MRP [9]): 544,336 bytes
To give a compression ratio of 9.5:1 and increasing the uncertainty by a factor of only 1.15.
It has to be noted that, when keeping parameters A, Ω, T, E constant, shot noise will manifest itself with the number of photoelectrons n following a Poisson distribution with mean and variance n
and with standard deviation σi=√{square root over (n)}.
Once, the system characterization is achieved and the pre-compression parameters are available, the user can select the required parameters and continue with the compression phase.
Compression Method
An embodiment of this compression method is illustrated in
Once, the compression is achieved and the compressed data are stored or transmitted (that actually can be considered as a mobile storage in a channel), the user can decide to decompress the data to recover the data and therefore pass to the last phase that is the decompression method.
De-Compression Method and Usage
An embodiment of this method is illustrated in
As a matter of fact, using the example of the compression and decompression functions described in section “System characterization”, the decompressed image data uncertainty (230) is calculated as σo/Lo
=Δ(g, Lo)×σi/
Li
=1.15×1/√{square root over (n)}=1.15×1/√{square root over (do)} where do is the decompressed data output for that pixel and Δ is preferably 1.15.
This data (228) and, optionally, the associated uncertainty (230), can then be used to effectively process the image (232), resulting in data to be output to display, printing, or further analysis or processing (234).
χ
| Number | Date | Country | Kind |
|---|---|---|---|
| 15202608 | Dec 2015 | EP | regional |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5014134 | Lawton et al. | May 1991 | A |
| 7139437 | Jones | Nov 2006 | B2 |
| 7194143 | Sowa | Mar 2007 | B2 |
| 7251343 | Dorrell | Jul 2007 | B2 |
| 7474800 | Moskvitin | Jan 2009 | B2 |
| 7583861 | Hanna | Sep 2009 | B2 |
| 7684483 | Coleman | Mar 2010 | B2 |
| 7706624 | Beazley | Apr 2010 | B2 |
| 7869667 | Wu | Jan 2011 | B1 |
| 8086062 | Lin | Dec 2011 | B2 |
| 8279308 | Borg | Oct 2012 | B1 |
| 8462991 | Clements | Jun 2013 | B1 |
| 8612436 | Grundman | Dec 2013 | B1 |
| 20100189180 | Narroschke et al. | Jul 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 1 215 909 | Jun 2002 | EP |
| Entry |
|---|
| Narroschke, Matthias, “Extending H.264/AVC by an adaptive coding of the prediction error,” The 25th Picture Coding Symposium, 6 pgs. (Apr. 24, 2006.). |
| Ma, Siwei, “Rate-Distortion Analysis for H.264/AVC Video Coding and its Application to Rate Control,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 15, No. 12, pp. 1533-1544 (Dec. 2005). |
| Ohm, Jens-Rainer, et al., “Comparison of the Coding Efficiency of Video Coding Standards—Including High Efficiency Video Coding (HEVC),” IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, No. 22, pp. 1669-1684 (Dec. 2012). |
| Li, Jin-Chao, et al., “Noise Estimation in Video Surveillance Systems,” 2009 World Congress on Computer Science and Information Engineering, pp. 578-582 (Mar. 31, 2009). |
| Algazi, V. Ralph, et al., “Quality measurement and use of pre-processing in image compression,” Signal Processing, 70(3), pp. 215-229 (1998). |
| Eskicioglu, Ahmet M., et al., “Image Quality Measures and Their Performance,” IEEE Transactions on Communications, vol. 43, No. 12, pp. 2959-2965 (Dec. 1995). |
| Hanhart, Philippe, et al., “Subjective quality evaluation of the upcoming HEVC video compression standard,” SPIE Optical Engineering+Applications, International Society for Optics and Photonics, 13 pgs. (2012). |
| Lin, Weisi, et al., “Perceptual visual quality metrics: A survey,” J. Vis. Commun. Image R., 22, pp. 297-312 (2011). |
| Ye, Zhengmao, et al., “Quantitative Analysis of Information Loss in Digital Image Compression and Reconstruction,” Communication Software and Networks (ICCSN), 2011 3rd International Conference, pp. 398-402 (May 2011). |
| Welch, Terry A., “A technique for High-Performance Data Compression,” Computer, 17(6), pp. 8-19 (Jun. 1984). |
| Matsuda Ichiro, et al., “Lossless Coding of Still Images Using Minimum-Rate Predictors,” Image Processing, 2000, Proceedings 2000 International Conference, vol. 1, pp. 132-135 vol. 1 (2000). |
| Number | Date | Country | |
|---|---|---|---|
| 20170208347 A1 | Jul 2017 | US |
