Patent Grant
7454074
This invention relates to systems and methods for transmission of still images over relatively low-speed communication channels. More specifically the invention relates to progressive image streaming over low speed communication lines, and may be applied to a variety of fields and disciplines, including commercial printing and medical imaging, among others.
In a narrow bandwidth environment, a simple transfer to the client computer of any original image stored in the server's storage is obviously time consuming. In many cases the user only wishes to view a low resolution version of the image and perhaps several high-resolution details, in these instances it would be inefficient to transfer the full image. This problem can be overcome by storing images in a compressed format. Examples of such formats include standards such as Progressive JPEG (W. Pennebaker and J. Mitchel, “JPEG, still image data compression standard”, VNR, 1993) or the upcoming JPEG2000 (D. Taubman, “High performance scalable image compression with EBCOT”, preprint, 1999). These formats allow progressive transmission of an image such that the quality of the image displayed at the client computer improves during the transmission.
In some applications such as medical imaging, it is also necessary that whenever the user at the client computer is viewing a portion of the highest resolution of the image, the progressive streaming will terminate at lossless quality. This means that at the end of progressive transmission the pixels rendered on the screen are exactly the pixels of the original image. The current known “state-of-the-art” wavelet algorithms for progressive lossless streaming all have a major drawback: their rate-distortion behavior is inferior to the “lossy” algorithms. The implications of this include:
1. Whenever the user is viewing any low resolution version of the image (at low resolutions the term “lossless” is not well defined) more data needs to be sent for the same visual quality.
2. During the progressive transmission of the highest resolution, before lossless quality is achieved, more data needs to be sent for the same visual quality.
Researchers working in this field are troubled by these phenomena. F. Sheng, A. Bilgin, J. Sementilli and M. W. Marcellin state in “Lossy to Lossless Image Compression Using Reversible Integer Wavelet Transform”, Proc. IEEE International Conf. On Image Processing, 1998: “ . . . Improved lossy performance when using integer transforms is a pursuit of our on-going work.” An example is provided in Table 1.
As can be seen from Table 1, state of the art progressive lossless coding is inferior to lossy coding by more than 1 dB at high bit rates.
Indeed, intuitively, the requirement for lossless progressive image transmission should not affect the rendering of lower resolutions or the progressive “lossy” rendering of the highest resolution before lossless quality is obtained. The final lossless quality should be a layer that in some sense is added to a lossy algorithm with minor (if any) effect on its performance.
The main problem with known lossless wavelet algorithms, such as Set Partitioning in Hierarchical Trees (SPIHT) A. Said and W. Pearlman, “A new, fast and efficient image codec based on set partitioning”, IEEE Trans. Circuits and Systems for Video Tech. 6 (1996), 243-250 and compression with reversible embedded wavelets (CREW) A. Zandi, J. D. Allen, E. L. Schwartz and M. Boliek, “CREW: Compression with reversible embedded wavelets”, Proc. of Data Compression Conference (Snowbird, Utah), 212-221, 1995 is that they use special “Integer To Integer” transforms (see “Wavelet transforms that map integers to integers”, A. Calderbank, I. Daubechies, W. Sweldens, B. L. Yeo, J. Fourier Anal. Appl., 1998). These transforms mimic “mathematically proven” transforms that work well in lossy compression using floating-point arithmetic implementations. Because they are constrained to be lossless, they do not approximate their related floating-point algorithms sufficiently well. Although in all previous work there have been attempts to correct this approximation in the progressive coding stage of the algorithm the bad starting point and an inefficient transform prevented previous authors from obtaining acceptable rate-distortion behavior.
The system and method of the present invention solves the rate-distortion behavior problem. Using the fact that images are two-dimensional signals, novel 2D lossless Wavelet transforms are disclosed that better approximate their lossy counterparts. As an immediate consequence the lossless progressive coding algorithm of the present invention has the same rate-distortion of a lossy algorithm during the lossy part of the progressive transmission.
The imaging system that is described below is directed to a lossless image streaming system that is different from traditional compression systems and overcomes the above problems. By utilizing a lossless means of progressive transmission, the pixels rendered on the screen at the end of transmission are exactly the pixels of the original image that were transmitted. The imaging system disclosed herein eliminates the need to store a compressed version of the original image, by streaming ROI data using the original stored image. The imaging system of the present invention also avoids the computationally intensive task of compression of the full image. Instead, once a user wishes to interact with a remote image, the imaging server performs a fast preprocessing step in near real time after which it can respond to any ROI requests also in near real time. When a ROI request arrives at the server, a sophisticated progressive image-encoding algorithm is performed, but not for the full image. Instead, the encoding algorithm is performed only for the ROI. Since the size of the ROI is bounded by the size and resolution of the viewing device at the client and not by the size of the image, only a small portion of the full progressive coding computation is performed for a local area of the original image. This local property is also true for the client. The client computer performs decoding and rendering only for the ROI and not for the full image. This real time streaming architecture (known commercially as Pixels-On-Demand™) requires different approaches even to old ideas. For example, similarly to some prior art, the present imaging system is based on wavelets. But while in other systems wavelet bases are selected according to their coding abilities, the choice of wavelet bases in the present imaging system depends more on their ability to perform well in the real time framework. The system of the present invention supports several modes of progressive transmission: by resolution, by accuracy and by spatial order.
1. Notation and Terminology
The following notation is used throughout this document
The following terminology and definitions apply throughout this document.
2. Overview of the Invention
Referring to
In one embodiment, the client computer 110 and server computer 120 may comprise a PC-type computer operating with a Pentium-class microprocessor, or equivalent. Each of the computers 110 and 120 may include a cache 111, 121 respectively as part of its memory. The server may include a suitable storage device 122, such as a high-capacity disk, CD-ROM, DVD, or the like.
The client computer 110 and server computer 120 may be connected to each other, and to other computers, through a communication network 130, which may be the Internet, an Intranet (e.g., a local area network), a wide-area network, or the like. Those having ordinary skill in the art will recognize that any of a variety of communication networks may be used to implement the present invention.
With reference to
3. New Reversible Wavelet Transform
Several benefits of the rate-distortion behavior of the progressive lossless algorithm of the present invention are discussed below. Lossless wavelet transforms, must be integer-to-integer transforms, such that round-off errors are avoided. In order to demonstrate the difference between lossy and lossless transforms, let us look at the simplest wavelet, the Haar wavelet. Let x(k) be the kth component of the one-dimensional discrete signal x. The first forward Haar transform step, in its accurate “mathematical” form, is defined by:
where s is a low-resolution version of x, and d is the “difference” between s and x. In the case of lossless transform, applying the above transform results in round-off error. One possibility is to apply the transform step suggested by A. Calderbank, I. Daubechies, W. Sweldens and B. L. Yeo, “Wavelet transforms that map integers to integers”, Applied and Computational Harmonic Analysis 5 (1998), 332-369:
The notation └∘┘ denotes the floor function meaning “greatest integer less than or equal to ∘”, e.g. └0.5┘=0, └−1.5┘=−1, └2┘=2, └−1┘=−1.
The one-dimensional transform step is generalized to a 2D separable transform step by applying the 1D transform step twice, first in the X-direction and than (on the first stage output) in the Y-direction as described in
In (3.2) two properties are kept:
On the other hand, the important scaling property, is not kept in (3.2). Observe that the value of s(n) computed by (3.2), is smaller than its “real mathematical value” as computed in (3.1), by factor of √{square root over (2)}. Since s(n) should be rounded to an integer number, the fact that s(n) is smaller than what it should be, increases the round-off error. In low resolutions, the error is accumulated through the wavelet steps.
If we take the error as a model of “white noise” added to the i-th resolution in a multi-resolution representation of the image, i.e. Xi in
Our referenced computation, i.e. the accurate computation, is the Haar transform step defined in (3.1). We concentrate on the LL-subband coefficients, because the low-resolution subbands are computed from them. LL-subband coefficients are the result of a 2D-transform step (
where m and n are the indices of the row and column of the coefficient respectively.
As described in
for each input row x(k,·).
Under assumption 1 mentioned above, we can write s(k,n) as
where e is a random variable with a probability density function (PDF) p (·) defined by
We then apply the Y-direction transform by
As in (3.5) we can represent s (2m +1, n) and s (2m, n) by:
Now we can write:
where e1, e2, e′ are independent (assumption 2 above) random variables with expectation
Variance
and
Therefore,
e represents the approximation error of the LL-subband coefficients, results from one 2D transform step. The error relates to the accurate floating-point computation.
This was a description of a single 2D-transform step assuming that the input coefficients are without any error. Now we wish to evaluate the error accumulated after several steps.
At an arbitrary step i≧0, we can assume that an input coefficient can be written as:
xi(k,l)=xi(accurate)(k,l)+ei,
where xi(accurate)(k,l) is the accurate value achieved by floating-point computation for all the previous steps, i.e., a step defined by
instead of the integer-to-integer computation in (3.2). Observe that if xi(accurate)(k,l) is the i-th resolution image coefficient, using (3.14) as the 1D Wavelet step, then
where lli−1(accurate)(k,l) is the normalized ( L2−norm) LL-subband coefficient resulting from the i-th 2D transform step using (3.1) as the 1D Wavelet step (see Figure ). ei is the difference between xi(k,l) and xi(accurate)(k,l) (I.e., the approximation error of the integer computation made until now). E.g. e0=0(x0(k,l) is an original image pixel), while e, is a random number with expectation
and variance
(see (3.12)).
Using (3.11), we get:
where ei+1 is defined by
and corresponds to the LLi subband.
Consequently
Observe that
As a result, we can write recursive formulas for the error expectation and variance after i steps.
The explicit solutions to these formulas are
By replacing xi(accurate)(m,n) with
we get
Thus, the approximation to lli(accurate)(m, n) is
2i+1lli(CDSI)(m,n)=lli(accurate)(m,n)+2i+1ei+1.
The approximation error expectation is
The approximation error variance and standard deviation are
Let us now evaluate the approximation error of the 3 other subbands:
where
Thus,
The approximation to lhi(accurate)(m,n) is
2ilhi(CDSI)(m,n)=lhi(accurate)(m,n)+2iei1.11.
The approximation error variance and standard deviation are:
A similar approximation error estimation can be calculated with the HL and HH subbands.
The approximation error evaluation results are summarized in the following table where the error is the difference between the normalized (in L2−norm) coefficients according to Calderbank et al. (referenced supra) reversible transform and the “mathematical” transform (defined in (3.1)).
The above table assumes a low-bit rate transmission where only the coefficients whose absolute value belongs to the range └2h,2h+1┘ are encoded, for every resolution i, where i is greater than b (less or more). It is noted that the large error implies a significant loss of coding efficiency.
Instead, we propose a new family of reversible transforms. The proposed family of integer wavelet transforms has all three properties:
1. Reversibility
2. De-correlation
3. Scaling—i.e. improved approximation of the “mathematical” transform.
Our 2D transform step is separable also, but the one-dimensional transform step, which the 2D transform is based on, is different for the X-direction (step 1901), the Y-direction step applied on the low output of the X-direction step (step 2001) and the Y-direction step applied on the high output of the X-direction step (step 2002) as described in
The full 2D Wavelet transform is applied by using the 2D Wavelet transform step iteratively in the classic Mallat decomposition of the image (
In order to achieve the third property (improved approximation of the “mathematical” transform), we define an extra matrix we call the “Half bit-matrix” which enables the reversibility of the High Y-transform step (step 2002). The elements that belong to this matrix are bits, such that each bit corresponds to an HH-subband coefficient in the following interpretation. Let us describe this by the following example.
Supposing
s(n)=7,d(1)(n)=9
are a coefficient pair resulting from a reversible de-correlated 1D-wavelet step
Now, d(1)(n) has to be multiplied by
in order to be scaled.
The binary form of d(1)(n)=9 is
d(1)(n)=10012.
If we now divide d(1)(n) by 2 in a floating-point computation we get
Let us call the bit, located on the right side of the floating point the “Half Bit”. Observe that the Half Bit of dFP(n) is the LSB of d(1)(n). Therefore, an equivalent way to do this in an integer computation without loosing the Half-Bit is to first calculate the LSB of d(1)(n) by
HalfBit(n)=d(1)(n)mod2=9mod2=1,
then to shift-write d(1)(n) by
d(n)=d(1)(n)>>1=1001>>1=100.
By saving d(n) and HalfBit(n) we can restore d(1)(n).
In the proposed transform, this Half-bit is needed in the HH-subband coefficient computation. Therefore in our wavelet decomposition for every HH-subband coefficient (in all scales) there is a corresponding bit, which is the coefficient's Half-bit. The Half bit matrix is hidden in the HH-subband in the description of
We now present our integer-to-integer versions of the Haar transform and the CDF (1,3) transform for the 2-dimensional case.
3.1 Reversible Haar and (CDF) (1,3) Transforms
3.1.1 Haar Transform
With respect to
3.1.1.1 Step 1901: X-Direction
With respect to
3.1.1.2 Step 2001: Y-Direction—Low Forward Step
Remarks:
1. s(n) is a scaled LL-subband coefficient.
2. s(n) and d(1)(n) are de-correlated and a reversible couple (can be transformed back to x(2n) and x(2n+1)), but d(1)(n) is not scaled (it is half its “real value”). Thus, d(1)(n) is multiplied by 2. Nevertheless, the LSB of the LH-subband coefficient d (n) is known to be 0 and not encoded.
With respect to
3.1.1.3 Step 2002: Y-Direction—High Forward Step
Remark: d(1)(n) and s(n) are de-correlated and reversible couples, but d(1)(n) is not scaled (It is twice its “real value”). Therefore, d(1)(n) is divided by 2. By doing that, we lose its least significant bit, which cannot be restored. To solve this problem, as explained before, we save this bit as the “Half-Bit”. Giving this name to that coefficient means that its weight is
in the “real mathematical scale”, and it is the least significant (from the approximation point of view).
Inverse Step
3.1.2 CDF (1,3) Transform
3.1.2.1 Step 1901: X-Direction
With respect to
With respect to
3.1.2.2 Step 2001: Y-direction—Low Forward Step
Remark: See remarks for (3.22).
With respect to
3.1.2.3 Step 2002: Y-Direction—High Forward Step
We now compute the approximation error probabilities of our method, and show that it is significantly smaller. We start with the LL-subband error. Assuming ei is the approximation error of the LL-subband in the i-th resolution (
where
Consequently
By knowing that e−1=0 we get
Now we can easily evaluate the approximation error of the 3 other subbands:
where
Hence
Similar estimation can be done for the HL and the HH subbands.
The error estimation (for all subbands) are summarized in the following table where the error is the difference between the normalized (in L2−norm) coefficients according to our new reversible transform and the “mathematical” transform (defined in (3.1)).
The results indicate that at low bit rates, where only large coefficients are encoded, the error is negligible.
4. Imaging Protocol and Distributed Database
Dividing the Data into Tiles and Bit-Planes:
For the purpose of efficient rendering the coefficients may be sub-divided into tiles. The tiles of this invention differ from previous art as shown in
Remark:
Since the LH-subband contains only even coefficients, their LSB must be zero and is not coded.
associated with HH-subband coefficients as their last “half bit” (See (3.24) or (3.30)).
5. The Progressive Subband Coding Algorithm
5.1 The Encoding Algorithm
The encoding algorithm of the present invention is performed at the server 120. In the present imaging system this rather time consuming task is performed locally in near real-time for a ROI, and not on the full image. The encoding algorithm is described for images with a single color component, such as grayscale images, but of course may also be applied to images with multiple color components. The straightforward generalization for an arbitrary number of components will be explained later.
The lossless algorithm receive as input the following parameters:
The coding strategy is similar in some sense to that described in A. Said and W. Pearlman, “A new, fast and efficient image codec based on set partitioning”, IEEE Trans. Circuits and Systems for video Tech., Vol. 6, No. 3, pp. 243-250, 1996, but the preferred embodiment uses no “Zero Tree” data. For all the data blocks with t_bitplane≧2, we use the lossy encoding algorithm described in previous art with the parameters:
Remark: The lossy algorithm encodes all the bit-plane information for t_bitPlane≧2. For t_bitPlane≦1, i.e. the least significant bit plane (of the lossless algorithm) and the half bit plane, we use a different algorithm described in 5.1.3.
5.1.1 Encoding Algorithm Initialization
The lossless encoding algorithm initialization is the same as the lossy algorithm of § 4.1.1 in the above-cited Ser. No. 09/386,264, now U.S. Pat. No. 6,314,452, which disclosure is incorporated herein by reference. In order to initialize the encoding algorithm, the following procedure is performed:
1. Assign to each coefficient coef (x, y) its bit plane b (x, y) such that:
|coef(x, y)|ε[εc2h,εc2h+1)
2. Compute the maximum bit plane over all such coefficients:
3. Write the value of maxBitPlane(tile) using one byte as the header of the data block:
4. Initialize all the coefficients as members of their corresponding Type 16 group.
5. Initialize a list of significant coefficients to be empty.
6. Initialize a coefficient approximation matrix
as zero.
5.1.2 The Outer Loop
The outer loop of the encoding algorithm scans the bit planes from b=maxBitPlane(tile) to b=0. The output of each such bit plane scan is the subband data block. Since the last stage of the encoding algorithm is arithmetic encoding of given symbols, at the beginning of each scan the arithmetic encoding output module is redirected to the storage area allocated for the particular data block. Once the bit plane scan is finished and the data block has been encoded, the output stream is closed and the bit plane b is decremented. After the outer loop is finished the following stages are performed:
1. Least significant bit plane is encoded (t_bitPlane=1).
2. Half bit plane is encoded (t_bitPlane=0).
The output of the least significant bit plane scan is the data block (
The half bit plane data block is:
For t_bitPlane≧2, the framework of the bit plane scan is described in
Remark: The encoder method isLastBitPlane( ) is associated to the t_bitPlane=2.
For the least significant bit plane, a pseudo code is described in
Regarding the least significant bit encoding algorithm, the following is noted:
For the half bit plane, a pseudo code is described in
5.2 The Decoding Algorithm
Obviously, this algorithm is a reversed step of the encoding algorithm of section 5.1, performed in the server 120. The client computer 110 during the progressive rendering operation performs the decoding algorithm. Similar to the encoding algorithm, the decoding algorithm is described for an image with one component (such as a grayscale image), but of course could also be used with an image with more than one component. The input parameters to the lossless algorithm are given below:
For all the data blocks with t_bitPlane≧2, a “lossy” decoding algorithm is utilized. The input parameters for the lossy algorithm are:
5.2.1 Decoding Algorithm Initialization
Upon the completion of the outer loop in 5.1.2 the following stages are preformed:
The preferred embodiment follows the lossy prior art of the above-cited Ser. No. 09/386,264, now U.S. Pat. No. 6,314,452, which disclosure is incorporated herein by reference, for t_bitPlane≧2. The scan, for a given level b, decodes all of the coefficients' data corresponding to the absolute value interval [ε2h,ε2h+1).
Pseudo codes of the least significant bit plane scan and half bit plane scan are described in
With reference to
6.1 Step 401: Receiving the ROI Parameters
The imaging module on the client computer 120 receives from the GUI interface module view parameters detailed in Table 6. These parameters are used to generate a request list for the ROI. The same parameters are used to render the ROI.
The basic parameters of a ROI are worldPolygon and scale which determine uniquely the ROI view. If the ROI is to be rendered onto a viewing device with limited resolution, then a worldPolygon containing a large portion of the image will be coupled by a small scale. In the case where the rendering is done by a printer, the ROI could be a strip of a proof resolution of the original image that has arrived from the server computer 120. This strip is rendered in parallel to the transmission, such that the printing process will terminate with the end of transmission. The other view parameters determine the way in which the view will be rendered. The parameters deviceDepth and viewQuality determine the quality of the rendering operation. In cases where the viewing device is of low resolution or the user sets the quality parameter to a lower quality, the transfer size can be reduced significantly.
The parameter luminanceMap is typically used in medical imaging for grayscale images that are of higher resolution than the viewing device. Typically, screens display grayscale images using 8 bits, while medical images sometimes represent each pixel using 16 bits. Thus, it is necessary to map the bigger pixel range to the smaller range of [0,255].
Lastly, the parameter progressiveMode determines the order in which data blocks should be transmitted from the server 120. The “Progressive By Accuracy” mode is the best mode for viewing in low bandwidth environments. “gressive By Resolution” mode is easier to implement since it does not require the more sophisticated accuracy (bit plane) management and therefore is commonly found in other systems. The superiority of the “progressive by accuracy” mode can be mathematically proven by showing the superiority of “non-linear approximation” over “linear approximation” for the class of real-life images. See, e.g., R. A. DeVore, “Nonlinear approximation”, Acta Numerica, pp. 51-150, 1998.
The “Progressive by Spatial Order” mode is designed, for example, for a “print on demand” feature where the ROI is actually a low resolution “proof print” of a high resolution graphic art work. In this mode the image data is ordered and received in a top to bottom order, such that printing can be done in parallel to the transmission.
Since lossless compression is frequently required in medical images transmission, where typically more than 8 bits images are used, the curve (luminanceMap hereinabove) which defines the mapping from the original image gray scale range (typically 10,12,16 bits) to an 8-bit screen is discussed in more detail. Furthermore, in viewing medical images, regardless of the original image depth, mapping is required in order to control the brightness and contrast of the image.
6.1.1 Luminance Mapping
Mapping from original image depth (e.g. 10,12,16 bits ) to screen depth (typically 8-bits), is defined by a monotonic function (FIG. 17):
f[0,2original
The curve influences not only the mapping, i.e. the drawing to the screen, but also the request from the server. To understand this, let us focus on in the maximal gradient of the curve (
where
We consider the worst case of the RMS increasing factor i.e.:
If the RMS increasing factor is greater than 1, it means that the “new RMS” may be greater than we consider as visually negligible error. Thus, the request list should be increased (i.e. more bit-planes should be requested from the server) in order to improve the approximation accuracy. Conversely, if the RMS increasing factor is smaller than 1, the request listing should be reduced. The exact specification of this is given in the following section.
6.2 Step 402: Creating the Request List
In step 402 using the ROI view parameters, the client imaging module at the client computer 110 calculates the data block request list ordered according to the particular progressiveMode selected. Given the parameters worldPolygon and Scale, it may be determined which subband tiles in the “frequency domain” participate in the reconstruction of the ROI in the “time domain”. These tiles contain all the coefficients that are required for an “Inverse Subband/Wavelet Transform” (IWT) step that produces the ROI. First, the parameter dyadicResolution (ROI) is computed, which is the lowest possible dyadic resolution higher than the resolution of the ROI. Any subband tiles of a higher resolution than dyadicResolution (ROI) do not participate in the rendering operation. Their associated data blocks are therefore not requested, since they are visually insignificant for the rendering of the ROI. If scale≧1, then the highest resolution subband tiles are required. If scale≦21-number( )Resolutions then only the lowest resolution tile is required. For any other value of scale we perform the mapping described below in Table 7.
Once it has been determined which subband tiles participate in the rendering of the ROI, it is necessary to find which of their data blocks are visually significant and in what order they should be requested. Using well known rate/distortion rules from the field of image coding (such as is described in S. Mallat and F. Falzon, “Understanding image transform codes”, Proc. SPIE Aerospace Conf., 1997), an optimal order can be determined in which the data blocks should be ordered by the client imaging module (and thus delivered by the server 120). This optimal order is described in steps 301-310 of
First, the subband coefficients with largest absolute values are requested since they represent the most visually significant data such as strong edges in the image. Note that high resolution coefficients with large absolute values are requested before low resolution coefficients with smaller absolute values. Within each given layer of precision (bit plane) the order of request is according to resolution; low resolution coefficients are requested first and the coefficients of highestSubbandResolution are requested last.
The main difficulty of this step is this: Assume a subband tile is required for the rendering of the ROI. This means that t_resolution≦dyadicResolution (ROI) and the tile is required in the IWT procedure that reconstructs the ROI. It must be understood which of the data blocks associated with the subband tile represent visually insignificant data and thus should not be requested. Sending all of the associated data blocks will not affect the quality of the progressive rendering. However, in many cases transmitting the “tail” of data blocks associated with high precision is unnecessary since it will be visually insignificant. In such a case, the user will see that the transmission of the ROI from the server 120 is still in progress, yet the progressive rendering of the ROI no longer changes the displayed image.
Additionally, the influence of the luminance mapping on the accuracy level of the requested data block is described below. Supposing for some t_x, t_y and t_resolution, the set
The number of bit planes reduced (added) from the request list is
I.e., for those t_x, t_y and t_resolution mentioned before, the following set is requested:
1. Given
The number of bit planes reduced from the request list is:
2. Given a luminance mapping with Maximal gradient=2
The number of bit planes reduced from the request list is:
Thus, one bit plane is added to the original set.
6.3 Step 403: Encoding the Request List
The client imaging module in the client computer 110 encodes the request list into a request stream that is sent to the server computer 120 via the communication network 130 (
{(t_x,t_y,t_resolution,t_bitplane),nx,ny}, nx,ny≧1 (1.3)
Each such structure represents the nx×ny data blocks
{(t_x+i,t_y+j,t_resolution,t_bitPlane)}, 0≦i<nx,0≦j<ny
The encoding algorithm attempts to create the shortest possible list of structures, collecting the data blocks to the largest possible rectangles can achieve this. It is important to note that the algorithm insures that the order of data blocks in the request list is not changed, since the server 120 will respond to the request stream by transmitting data blocks in the order in which they were requested. A good example of when this works well is when a user zooms in into a ROI at a high resolution that was never viewed before. In such a case the request list might be composed of hundreds of requested data blocks, but they will be collected to one (x,y) rectangle for each pair (t_resolution, t_bitPlane).
6.4 Step 404: Receiving the Data Blocks
The client computer 110 upon receiving an encoded stream containing data blocks from the server computer 120, decodes the stream and inserts the data blocks into their appropriate location in the distributed database using their ID as a key. The simple decoding algorithm performed here is a reversed step of the encoding scheme described infra. Since the client 110 is aware of the order of the data blocks in the encoded stream, only the size of each data block need be reported along with the actual data. In case the server 120 indicates an empty data block, the receiving module marks the appropriate slot in the database as existing but empty.
Recall that the subband tile associated with each data block is denoted by the first three coordinates of the four coordinates of a data block (t_x,t_y,t_resolution). From the subband tile's coordinates the dimensions are calculated of the area of visual significance; that is, the portion of the ROI that is affected by the subband tile. Assume that each subband tile is of length tileLengthand that the wavelet basis used has a maximal filter size maxFilterSize, then defining hFilterSize:=┌maxFilterSize/2┐ and factor:=numberOfResolutions−t_resolution+1, we have that the dimensions of the affected region of the ROI (in the original image's coordinate system) are
[t_x×tilelengthfactor−hFilterSizefactor,(t_x+1)×tileLengthfactor+hFilterSizefactor]×[t_y×tilelengthfactor−hFilterSizefactor(t_y+1)×tilelengthfactor+hFilterSizefactor]
These dimensions are merged into the next rendering operation's region. The rendering region is used to efficiently render only the updated portion of the ROI.
6.5 Progressive Rendering
During the transmission of ROI data from the server to the client, the client performs rendering operations of the ROI. To ensure that these rendering tasks do not interrupt the transfer, the client runs two program threads: communications and rendering. The rendering thread runs in the background and uses a pre-allocated “off-screen” buffer. Only then does the client use device and system dependant tools to output the visual information from the “off-screen” to the rendering device such as the screen or printer.
The rendering algorithm performs reconstruction of the ROI at the highest possible quality based on the available data at the client. That is, data that was previously cached or data that “just” arrived from the server. For efficiency, the progressive rendering is performed only for the portion of the ROI that is affected by newly arrived data. Specifically, data that arrived after the previous rendering task began. This “updated region” is obtained using the method of step 404 described in §6.4.
The parameters of the rendering algorithm are composed of two sets:
The rendering algorithm computes pixels at the dyadic resolution dyadicResolution(ROI). Recall that this is the lowest possible dyadic resolution that is higher than the resolution of the ROI. The obtained image is then resized to the correct resolution. Using a tiling of the multiresolution representation of the ROI, the steps of the algorithm are performed on a tile by tile basis as described in
6.5.1 The Rendering Rate
As ROI data is transmitted to the client 110, the rendering algorithm is performed at certain time intervals of a few seconds. At each point in time, only one rendering task is performed for any given displayed image. To ensure that progressive rendering does not become a bottleneck, two rates are measured: the data block transfer rate and the ROI rendering speed. If it is predicted that the transfer will be finished before a rendering task, a small delay is inserted, such that rendering will be performed after all the data arrives. Therefore, in a slow network scenario (as the Internet often is), for almost the entire progressive rendering tasks, no delay is inserted. With the arrival of every few kilobytes of data, containing the information of a few data blocks, a rendering task visualizes the ROI at the best possible quality. In such a case the user is aware that the bottleneck of the ROI rendering is the slow network and has the option to accept the current rendering as a good enough approximation of the image and not wait for all the data to arrive.
6.5.2 Memory Constraint Subband Data Structure
This data-structure is required to efficiently store subband coefficients, in memory, during the rendering algorithm. This is required since the coefficients are represented in either long integer precision (i.e. lossless coding mode) or floating-point precision (i.e. lossy coding mode) which typically requires more memory than pixel representation (1 byte). In lossy mode, the coefficients at the client side 110 are represented using floating-point representation, even if they were computed at the server side 120 using an integer implementation. This minimizes round-off errors.
At the beginning of the rendering algorithm, coefficient and pixel memory strips are initialized. dyadicWidth(ROI) may be denoted as the width of the projection of the ROI onto the resolution dyadicResolution (ROI) . For each component and resolution 1<j≦dyadic Resolution (ROI), four subband strips are allocated for the four types of subband coefficients: hl, lh, hh and Halfbit. The coefficient strips are allocated with dimensions
For each component and resolution 1≦j<dyadicResolution a pixel strip is allocated with dimensions
Beginning with the lowest resolution 1, the algorithm proceeds with recursive multiresolution processing from the top of the ROI to bottom (i.e. y direction). Referring to
6.5.3 Step 1101: Decoding and Memory Caching
The subband coefficients data structure described previously in section 6.5.2 is filled on a tile basis. Each such subband tile is obtained by decoding the corresponding data blocks stored in the database or by reading them from the memory cache. The memory cache is used to store coefficients in a simple encoded format. The motivation is this: the decoding algorithm described previously in section 5.2 is computationally intensive and thus should be avoided whenever possible. To this end the rendering module uses a memory cache 111 where subband coefficients are stored in a very simple encoded format which can be decoded very quickly. For each required subband tile, the following extraction procedure is performed, described in
6.5.4 Step 1102: Inverse Lossless Wavelet Transform
This is an inverse step to step 603 performed in the server (see section 7.1.5). Following
6.5.5 Step 1103: Inverse Color Transform
This is an inverse step to step 603 performed at the server 120. It is performed only for tiles of pixels at the resolution highestSubbandResolution. At this stage, all of the pixels of each such tile are in the outputColorSpace and so need to be transformed into a displaying or printing color space. For example, if the original image at the server 120 is a color image in the color space RGB, the pixels obtained by the previous step of inverse subband transform are in the compression color space YUV. To convert back from YUV to RGB, we use the inverse step described in
6.5.6 Step 1104: Image Resize
In case the resolution of the ROI is not an exact dyadic resolution, the image obtained by the previous step must be re-sized to this resolution. This can be accomplished using operating system imaging functionality. In most cases the operating system's implementation is sub-sampling which produces in many cases an aliasing effect which is not visually pleasing. To provide higher visual quality, the imaging system of the present invention may use a method of linear interpolation, such as described in J. Proakis and D. Manolakis, “Digital signal processing”, Prentice Hall, 1996. The output of the linear interpolation step is written to the off-screen buffer. From there it is displayed on the screen using system device dependant methods.
6.5.7 Step 1105: Mapping to 8-Bit Screen
When luminanceMap is active mapping to 8-bit screen is performed using the mapping function described in section 6.1.1.
7. Server Worflow
With reference to
Once the client computer 110 requests to view or print a certain image, the server performs the preprocessing step 501. This step is a computation done on data read from the original digital image. The results of the computation are stored in the server cache device 121. After this fast computation a “ready to serve” message is sent from the server to the client containing basic information on the image.
In step 502, the server receives an encoded stream of requests for data blocks associated with a ROI that needs to be rendered at the client. The server then decodes the request stream and extracts the request list.
In step 503, the server reads from cache or encodes data blocks associated with low resolution portions of the ROI, using the cached result of the preprocessing stage 501.
If the ROI is a high-resolution portion of the image, the server, in step 504, reads from cache or performs a “local” and efficient version of the preprocessing step 501. Specifically, a local portion of the uncompressed image, associated with the ROI, is read from the storage 122, processed and encoded. In step 505, the data that was encoded in steps 503-504 is progressively sent to the client in the order it was requested.
7.1 Step 501: Preprocessing
The preprocessing step is now described with respect to
7.1.1 Preprocessing Parameters
Given an input image, the parameters described in Table 8 are computed or chosen. These parameters are also written into a header sent to the client 110 and are used during the progressive rendering step 405 (see section 6.5, described infra). The important parameters to select are:
Yeo, J. Fourier Anal. Appl., 1998).
Referring to Table 8, losslessMode is set to true. Threshold (c, j) is not in use, since in lossless mode, there is no thresholding. The rest of the variables have the same meaning as in the lossy algorithm.
7.1.2 Memory Constraint Multiresolution Scan Data Structure
Most prior art wavelet coding algorithms do not address the problem of memory complexity. Usually these prior art algorithms assume there is sufficient memory such that the image can be transformed in memory from a time domain representation to a wavelet frequency domain representation. The upcoming JPEG2000 standard will likely address this issue, as did its predecessor, the JPEG standard. The preprocessing algorithm also requires performing subband transforms on large images, although not always on the full image, and thus requires careful memory management. This means that the memory usage of the algorithm is not of the order of magnitude of the original image, as described in J. M. Shapiro, “An embedded hierarchical image coder using zero-trees of wavelet coefficients”, IEEE Trans. Sig. Proc., Vol. 41, No. 12, pp. 3445-3462, 1993.
Given an uncompressed image we allocate the following number of memory strips
numberOfComponents×(numberOfResolutions−jumpSize)
of sizes
[2−(numberOfResolutions−j)imageWidth,3×tileLength/2+maxFilterSize]
for 1≦j≦numberOfResolutions−jumpSize−1 and
[imageWidth,tileLength+2×maxFilterSize]
for j=numberOfResolutions−jumpSize
That is, the memory usage is proportional to 2−jumpSize×imageWidth. Each such strip stores low-pass coefficients in the color space outputColorSpace at various resolutions.
Referring to
7.1.3 Step 601: Lossless Color-transform
This step uses the conversion formula described in
7.1.4 Step 602: Lossless Wavelet Low Pass
The motivation for the low pass step is explained in § 6.1.4 in the above-cited U.S. Pat. No. 6,314,452, which disclosure is incorporated herein by reference. Several important aspects of lossless mode are emphasized below.
In step 602, the low pass filter of the transform subbandTransformType(j), numberOfResolutions−jumpSize<j≦numberOfResolutions, are used to obtain a low resolution strip at the resolution numberOfResolutions−jumpSize (as can be seen in
In the lossless mode of operation, the jumpSize parameter defines the number of lossless wavelet low pass filtering steps that should be done. A single low pass filtering step is the same for Haar and CDF (1,3) and defined by the following two stages (taken from equations (3.20) and (3.22):
Namely, in a 2D representation, the low pass step is defined by
For jumpSize=1 and jumpSize=2 (other sizes typically are not needed), the server performs these steps efficiently (almost like the lossy algorithm) by a single operation that simulates exactly jumpSize low pass steps defined in (7.1). As noticed from (7.1), the simplicity of the formula makes filters such as Haar and CDF (1,3) “optimal” with respect to low pass efficiency.
7.1.5 Step 603: Forward Lossless Wavelet Transform
In Step 603 we perform one step of an efficient local lossless wavelet transform (§3), on a tile of pixels at the resolution 1≦j≦numberOfResolutions−jumpSize . The type of transform is determined by the parameter losslessWaveletTransformType( j). As described in
The subband transform of step 603 outputs three types of data: scaling function (low pass), wavelet (high pass) and Halfbits. The wavelet coefficients are treated in step 604 while the scaling function coefficients are treated in step 605. Note that tiles of pixels which are located on the boundaries sometimes need to be padded by extra rows and/or columns of pixels, such that they will formulate a “full” tile of length tileLength.
7.1.6 Step 604: Variable Length Encoding and Storage
In step 604, the subband coefficients that are calculated in step 603 are variable length encoded and stored in the cache 121. If maxBitPlane (tile)=0 we do not write any data. Else we loop on the coefficient groups {coef (2×i+x,2×j+y)}x,y=0,1. For each such group we first write the group's variable length length (i, j) using log2 (maxBitPlane(tile)) bits. Then for each coefficient in the group we write length (i, j)+1 bits representing the coefficient's value. The least significant bit represents the coefficient's sign: if it is I then the variable length encoded coefficient is assumed to be negative. The HalfBit subband coefficients are written one bit per coefficient.
7.1.7 Step 605: Copying Low Pass Coefficients into the Multiresolution Strip Structure
In step 503, unless losslessMode is true, the subband coefficients calculated in step 604 are quantized. This procedure is performed at this time because it is required that the coefficients computed in the previous step be stored in the cache 121. To avoid writing huge amounts of data to the cache, some compression is required. Thus, the quantization step serves as a preparation step for the following variable length encoding step. It is important to point out that the quantization step has no effect on compression results. Namely, the quantization step is synchronized with the encoding algorithm such that the results of the encoding algorithm of quantized and non-quantized coefficients are identical.
A tile of an image component c at the resolution j is quantized using the given threshold threshold (c, j): for each coefficient x, the quantized value is └x/threshold (c, i)j┘. It is advantageous to choose the parameters threshold (c, j) to be dyadic such that the quantization can be implemented using integer shifts. The quantization procedure performed on a subband tile is as follows:
Note that for subband tiles located at the boundaries we can set to zero subband coefficients that are not associated with the actual image, but only with a padded portion. To do this we take into account the amount of padding and the parameter maxFilterSize. The motivation for the “removal” of these coefficients is coding efficiency.
7.2 Step 502: Decoding the Request Stream
This is the inverse step of section 6.3. Once the request stream arrives at the server 120, it is decoded back to a data block request list. Each data structure the type representing a group of requested data blocks is converted to the sub-list of these data blocks.
7.3 Step 503: Encoding Low Resolution Part of ROI
Step 503, described in
Step 701 is the inverse step of step 604 described in §7.1.6. In the preprocessing algorithm subband tiles of lower resolution, that is resolutions lower than numberOfResolutions−jumpSize, were stored in the cache using a variable length type algorithm. For such a tile we first need to decode the variable length representation. The algorithm uses the stored value maxBitPlane(tile).
For each group of four coefficients {coef (2×i+x,2×j+y)}x,y=0,1, we read log2 (maxBitPlane(tile)) bits representing the variable length of the group.
Assume the variable length is length (i, j). For each of the four coefficients we then read length (i, j)+1 bits. The least significant bit represents the sign. The reconstructed coefficient takes the value:
In step 702 we use the encoding algorithm described in §5.1 to encode the requested data blocks associated with the extracted subband tile.
7.4 Step 504: Processing High Resolution Part of ROI
Step 504 is described in
7.5 Step 505: Progressive Transmission of ROI
In the final step, the encoded data tiles are sent from the server 120 to the client 110, in the order they were requested. In many cases, data blocks will be empty. For example, for a region of the original image with a constant pixel value, all of the corresponding data blocks will be empty, except for the first one that will contain only one byte with the value zero representing maxBitPlane(tile)=0. For a low activity region of the image, only the last data blocks representing higher accuracy will contain any data. Therefore, to avoid the extra side information, rectangles of empty data blocks are collected and reported to the client 110 under the restriction that they are reported in the order in which they were requested. For blocks containing actual data, only the data block's size in bytes need be reported, since the client 110 already knows which data blocks to expect and in which order.
The present invention has been described in only a few embodiments, and with respect to only a few applications (e.g., commercial printing and medical imaging). Those of ordinary skill in the art will recognize that the teachings of the present invention may be used in a variety of other applications where images are to be transmitted over a communication media.
This application is a continuation of U.S. application Ser. No. 09/837,862 filed Apr. 17, 2001, now U.S. Pat. No, 7,024,046 entitled “System and Method for the Lossless Progressive Streaming of Images Over a Communication Network,” now U.S. Pat. No. 7,204,046, which claims priority to U.S. application Ser. No. 09/386,264, filed Aug. 31, 1999, now U.S. Pat. No. 6,314,452, entitled “System and Method for Transmitting A Digital Image Over A Communication Network” and U.S. Provisional Application No. 60/198,017, filed Apr. 18, 2000, entitled “Lossless Progressive Streaming of Images Over the Internet”, all of which are incorporated herein by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3580655 | Leith et al. | May 1971 | A |
| 3950103 | Schmidt-Weinmar | Apr 1976 | A |
| 4136954 | Jamieson | Jan 1979 | A |
| 4155097 | Lux | May 1979 | A |
| 4190861 | Lux | Feb 1980 | A |
| 4223354 | Noble et al. | Sep 1980 | A |
| 4393456 | Marshall, Jr. | Jul 1983 | A |
| 4569075 | Nussbaumer | Feb 1986 | A |
| 4599567 | Goupillaud et al. | Jul 1986 | A |
| 4652881 | Lewis | Mar 1987 | A |
| 4663660 | Fedele et al. | May 1987 | A |
| 4674125 | Carlson et al. | Jun 1987 | A |
| 4701006 | Perlmutter | Oct 1987 | A |
| 4760563 | Beylkin | Jul 1988 | A |
| 4785348 | Fonsalas et al. | Nov 1988 | A |
| 4785349 | Keith et al. | Nov 1988 | A |
| 4799179 | Masson et al. | Jan 1989 | A |
| 4805129 | David | Feb 1989 | A |
| 4815023 | Arbeiter | Mar 1989 | A |
| 4817182 | Adelson et al. | Mar 1989 | A |
| 4821223 | David | Apr 1989 | A |
| 4827336 | Acampora et al. | May 1989 | A |
| 4829378 | LeGall | May 1989 | A |
| 4837517 | Barber | Jun 1989 | A |
| 4839889 | Gockler | Jun 1989 | A |
| 4864398 | Avis et al. | Sep 1989 | A |
| 4868868 | Yazu et al. | Sep 1989 | A |
| 4894713 | Delogne et al. | Jan 1990 | A |
| 4897717 | Hamilton et al. | Jan 1990 | A |
| 4904073 | Lawton et al. | Feb 1990 | A |
| 4922544 | Stansfield et al. | May 1990 | A |
| 4929223 | Walsh | May 1990 | A |
| 4936665 | Whitney | Jun 1990 | A |
| 4974187 | Lawton | Nov 1990 | A |
| 4982283 | Acampora | Jan 1991 | A |
| 4985927 | Norwood et al. | Jan 1991 | A |
| 4987480 | Lippman et al. | Jan 1991 | A |
| 4999705 | Puri | Mar 1991 | A |
| 5000183 | Bonnefous | Mar 1991 | A |
| 5001764 | Wood et al. | Mar 1991 | A |
| 5014134 | Lawton et al. | May 1991 | A |
| 5018210 | Merryman et al. | May 1991 | A |
| 5049992 | Citta et al. | Sep 1991 | A |
| 5049993 | LeGall et al. | Sep 1991 | A |
| 5068911 | Resnikoff et al. | Nov 1991 | A |
| 5072308 | Lin et al. | Dec 1991 | A |
| 5073964 | Resnikoff | Dec 1991 | A |
| 5081645 | Resnikoff et al. | Jan 1992 | A |
| 5095447 | Manns et al. | Mar 1992 | A |
| 5097331 | Chen et al. | Mar 1992 | A |
| 5101280 | Moronaga et al. | Mar 1992 | A |
| 5101446 | Resnikoff et al. | Mar 1992 | A |
| 5103306 | Weiman et al. | Apr 1992 | A |
| 5109451 | Aono et al. | Apr 1992 | A |
| 5121191 | Cassereau et al. | Jun 1992 | A |
| 5124930 | Nicolas et al. | Jun 1992 | A |
| 5128757 | Citta et al. | Jul 1992 | A |
| 5128791 | LeGall et al. | Jul 1992 | A |
| 5148498 | Resnikoff et al. | Sep 1992 | A |
| 5152953 | Ackermann | Oct 1992 | A |
| 5156943 | Whitney | Oct 1992 | A |
| 5173880 | Duren et al. | Dec 1992 | A |
| 5182645 | Breeuwer et al. | Jan 1993 | A |
| 5235434 | Wober | Aug 1993 | A |
| 5241395 | Chen | Aug 1993 | A |
| 5262958 | Chui et al. | Nov 1993 | A |
| 5335016 | Nakagawa | Aug 1994 | A |
| 5347479 | Miyazaki | Sep 1994 | A |
| 5381145 | Allen et al. | Jan 1995 | A |
| 5412741 | Shapiro | May 1995 | A |
| 5420891 | Akansu | May 1995 | A |
| 5453945 | Tucker et al. | Sep 1995 | A |
| 5495292 | Zhang et al. | Feb 1996 | A |
| 5497435 | Berger | Mar 1996 | A |
| 5534925 | Zhong | Jul 1996 | A |
| 5537493 | Wilkinson | Jul 1996 | A |
| 5546477 | Knowles et al. | Aug 1996 | A |
| 5563690 | Hasegawa et al. | Oct 1996 | A |
| 5602589 | Vishwanath et al. | Feb 1997 | A |
| 5606359 | Youden et al. | Feb 1997 | A |
| 5666161 | Kohiyama et al. | Sep 1997 | A |
| 5699458 | Sprague | Dec 1997 | A |
| 5710835 | Bradley | Jan 1998 | A |
| 5832300 | Lowthert | Nov 1998 | A |
| 5838377 | Greene | Nov 1998 | A |
| 5861920 | Mead et al. | Jan 1999 | A |
| 5867602 | Zandi et al. | Feb 1999 | A |
| 5872965 | Petrick | Feb 1999 | A |
| 5886733 | Zdepski et al. | Mar 1999 | A |
| 5968465 | Koveal et al. | Oct 1999 | A |
| 5982362 | Crater et al. | Nov 1999 | A |
| 5999664 | Mahoney et al. | Dec 1999 | A |
| 6038257 | Brusewitz et al. | Mar 2000 | A |
| 6049342 | Nielson et al. | Apr 2000 | A |
| 6049821 | Therialut et al. | Apr 2000 | A |
| 6314452 | Dekel et al. | Nov 2001 | B1 |
| 6711297 | Change et al. | Mar 2004 | B1 |
| Number | Date | Country |
|---|---|---|
| 0510933 | Oct 1992 | EP |
| 0593013 | Apr 1994 | EP |
| 0611051 | Aug 1994 | EP |
| 0622741 | Nov 1994 | EP |
| 0701375 | Mar 1996 | EP |
| 2211691 | Jul 1991 | GB |
| 2284121 | May 1995 | GB |
| 2285374 | Jun 1995 | GB |
| WO 8810049 | Dec 1988 | WO |
| WO 9103902 | Mar 1991 | WO |
| WO 9118361 | Nov 1991 | WO |
| WO 9423385 | Oct 1994 | WO |
| WO 9519683 | Jul 1995 | WO |
| WO 9609718 | Mar 1996 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20050271283 A1 | Dec 2005 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60198017 | Apr 2000 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 09837862 | Apr 2001 | US |
| Child | 11184365 | US |
