Patent Grant
7474791
This application is related to the co-pending application entitled Header-Based Processing Of Images Compressed Using Multi-Scale Transforms, concurrently filed on Jan. 10, 2002, U.S. patent application Ser. No. 10/044,420, assigned to the corporate assignee of the present invention.
The invention relates generally to the field of image processing. More specifically, the invention relates to creating a smaller representation of an image.
Thumbnails are an extremely desirable graphical user interface component for most multimedia and document applications. A thumbnail is a resized smaller version of an image representative of the full image which can be displayed in some applications by clicking on the thumbnail. The resizing is typically done by traditional smoothing followed by downsampling. In most traditional applications, such as listings on a web page, the size of a thumbnail is fixed. A common problem with those thumbnail displays is that the image information is often not recognizable for the viewer and does not provide the desired usefulness.
Newer multimedia communication tools allow a free-format composition of images of various sources on a representative canvas. In this case, the size of a thumbnail is allowed to be variable. In such an application, besides the question of what to display in a fixed-size thumbnail, the additional question arises of what constitutes a suitable thumbnail size or shape for a given image. For example, in representing a photo of a person as captured from a visitor's kiosk, a downsampled image of the face-portion of the photo would be sufficient for a thumbnail, whereas for a web document the title at full resolution might be a useful display.
The authors in Mohan, R., Smith, J. R., and Li, C.-S., “Adapting Multimedia Internet Content for Universal Access,” IEEE Trans. Multimedia, Vol. 1, no. 1, pp. 104-114, 1999, describe a method for the transcoding of images to specific devices such as monitors or cell phones. In Lee, K., Chang, H. S., Choi, H., and Sull, S., “Perception-based image transcoding for universal multimedia access,” which appeared in ICIP 2001, Proceedings of International Conference on Image Processing, Thessalonihi, Greece, 2001. This approach is extended to sending only specifically selected parts of an image at a specific resolution. The specification is performed by the sender and does not happen automatically.
There exist several software packages providing for thumbnail creation. These software packages focus on speed, but all of these resize the entire image to user or application defined sizes using traditional downsampling. Thus, image information is often not recognizable to the viewer.
In Woodruff, A., Faulring, A., Rosenholtz, R., Morrison, J., and Pirolli, P., “Using Thumbnails to Search the Web,” Proceedings of SIGCHI'01, Seattle, April 2001, enhanced thumbnails are introduced to provide a better representation of documents. The enhancement consists of lowering the contrast in traditionally created thumbnails and superimposing keywords in larger fonts that were detected via an Optical Character Recognition (OCR) system. The result is a limited improvement at best and is only applicable to images that contain text.
The authors in Burton, C. A., Johnston, L. J., and Sonenberg, E. A., “Case study: an empirical investigation of thumbnail image recognition”, Proceedings of Visualization Conference 1995, found that the filtering of images (contrast enhancement, edge enhancement) before downsampling increases a viewer's ability to recognize thumbnails. Even so, image information is often not recognizable to the viewer.
For creation of video summaries there exist methods that display groupings of video frames where the individual frames have specific sizes/resolution, as in Uchihashi, S., Foote, J., Girgensohn, A., and Boreczky, J., “Video Manga: Generating Semantically Meaningful Video Summaries,” Proceedings of Seventh ACM International Multimedia Conference, Orlando 1999. The decision for resizing the frames is made by measuring the frames importance in the video sequence, not by the actual image content.
The area of page layout has been discussed in many papers. Typically, the authors assume, disadvantageously, that the image content of documents is mostly text with perhaps some small images or graphics, and perform text specific operations involving clustering techniques to determine connected components. A common method for page layout is the one described in O'Gorman, L., “The Document Spectrum for Page Layout Analysis,” IEEE Trans. Image Proc., Vol. 15, no. 11, pp. 1162-1173, 1993.
One existing image file format stores multiple resolutions of an image (e.g., as created by a Laplacian pyramid). As a result, this image file format is usually disadvantageously larger than a file of a wavelet coded image. This image file format has the option of incorporating specific parameters into the file. Those parameters could be, for example, result aspect ratio, rectangle of interest, filtering or contrast adjustment. The parameters are set by the creator of the file and may or may not be used by the receiver.
A method and apparatus to receive an image and to create a smaller representation of the image from a wavelet representation of the image is described. The form of display (e.g., size, shape, multiscale collage) of the smaller representation of the image is selected to compensate for the content of the image and physical properties of the display device to display the smaller representation of the image.
The present invention will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the invention, which, however, should not be taken to limit the invention to the specific embodiments, but are for explanation and understanding only.
A method and apparatus for automatic creation of smaller-sized representations of images is described that takes into account the image content as well as the physical properties of a display device, such as, for example, but not limited to, dots per inch (dpi) resolution, absolute pixel resolution, pixels per viewing angle, gain function (gamma response), contrast, and brightness, gain function to compute an image representation that contains information that is visually recognizable when being displayed on the specific display device. The techniques described herein use a wavelet representation of an image and parameters that characterize the image content.
The techniques described herein also use parameters linked to physical properties of a display device. In a typical Internet application, an image may be displayed on several different display devices such as a monitor, a media board, a digital camera display, a personal digital assistant, or a cell phone. These devices may differ in physical properties such as, for example, dots per inch, numbers of pixels, contrast, brightness, etc.
In the following description, numerous details are set forth to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.
Some portions of the detailed descriptions which follow are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
The present invention also relates to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.
A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable medium includes read only memory (“ROM”); random access memory (“RAM”); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.); etc.
Overview
Referring to
Determination of a Display Device Dependent Range of Downsampling Factors
A wavelet representation separates the image information into its information at various levels of resolution 1 . . . L.
Depending on display device parameters, a range of suitable display levels between level 1 and L can be determined as follows. In one embodiment, if the size of the image is larger than the pixel size of the display device (e.g., the image is of size 2048×2048 and the monitor resolution is of size 1024×768), then the image should be downsampled at least twice to be completely visible on the monitor display. Therefore, the absolute display resolution provides a lower bound on the number of downsampling factors (i.e., on the display level L, so called Lmin).
Another factor that influences the visibility of images on displays is the relative display resolution given in dots per inch (dpi). For example, a monitor has a relative resolution of 75 dpi, whereas a printer has a resolution of 600 dpi and a large display has an estimated resolution of 25 dpi. Other tests may have other results (e.g., text recognition). As determined from subjective tests, the minimal size of a visible recognizable object or structure is a diameter of approximately 2 mm. This size can be translated into a dot size for a given dpi resolution. For a 75 dpi monitor, the minimal object diameter in dot numbers would be 6, whereas for a 600 dpi printer, it would be 47 dots. In one embodiment, the formula for determination of minimal visible object diameter in dot numbers is given by
Dmin=(minimal object diameter in mm)*(dpi resolution)/25.4.
Depending on physical properties of the display device, such as contrast and brightness, the choice of Dmin may vary a bit. In one embodiment, this variation is expressed as a constant Cdevice in the formula and tuned for a specific display. In one embodiment, if contrast or brightness is small, then the constant Cdevice is large.
Dmin=Cdevice*(minimal visible object diameter in mm)*(dpi resolution)/25.4. (1)
Dmin also depends on the viewing distance and angle. In one embodiment, the ratio between dpi resolution and view distance is constant.
Small scale structures show up as wavelet coefficients in decomposition levels over a number of decomposition levels associated with high resolution scales. In one embodiment, to be visibly recognizable after n-times downsampling, small scale structures have a dot size D that is larger than Dmin after n-times downsampling. Therefore, the inequality D/2n>Dmin is valid. As a result, only structures that correspond to wavelet coefficients being nonzero at levels n . . . L qualify to be visibly recognizable at the chosen display device. Thus, the levels 1 . . . n−1 do not need to be considered at all during the inverse transform, since they will not provide visually recognizable information. The level
Lmax(D)=log2(D/Dmin)=log2(D)−log2(Dmin) (1)
provides an upper bound on the downsampling factor of an object of dot size diameter D. As an immediate consequence,
└Lmax┘=└log2(min(N,M)/Dmin)┘=└log2(min(N,M))−log2(Dmin)┘ (2)
is an upper bound on the downsampling factor for an entire image of size N×M. Lmax and Lmin form a range of possible display scales
λ=[Lmin, Lmax]. (3)
The bound Lmax is most likely too large for most images. Often, images contain almost equal proportion of coarse and fine structures, such as natural images, or a lot of fine structures, such as text images. Depending on the ratio of coarse to fine structures, a natural image may be downsampled more times than a text image and still display visually recognizable information. Therefore, in one embodiment, the information contained in wavelet coefficients is used to determine, for a given image, the maximal downsampling factor Ldisplay under the constraints that
Lmin≦Ldisplay≦Lmax (4)
and that the LL component at level Ldisplay contains important image information.
Determination of the Image and Device Dependent Display Scale LDisplay
The display scale Ldisplay is computed from the distribution of wavelet coefficients over the device dependent display range [Lmin,Lmax]. In one embodiment, large coefficients correspond to important edges in the image and small coefficients correspond to unimportant small-scale structures and noise. In one embodiment, a level in the decomposition is determined such that the LL component at that level contains the most important information to allow for visual recognition of the image. Two approaches are described below.
In one embodiment, the best display scale is determined for the entire image from certain resolution levels (i.e., global scale selection). In an alternative embodiment, the image (or its wavelet decomposition) is partitioned into cells and the best display scale for each cell is determined independently (i.e., local scale selection). These approaches are further described below.
Global Scale Selection
In one embodiment, a mathematical formulation for the global scale selection problem is as follows. Consider a wavelet decomposition of an image of size M×N. WmLL(X) denotes the set of all LL coefficients of the image X at level m. Similarly, WmHL(X), WmLH(X), WmHH(X) denote the set of all wavelet coefficients in the HL, LH and HH bands at level m.
M denotes a measure that reflects the importance of information contained in the low resolution version Xm=WmLL(X). This importance measure may be defined based on one or more factors, such as, for example, but not limited to, energy, entropy, etc.
The best display level Ldisplay is computed by
Ldisplay=arg maxmελ{Cost(M(WmLL(X))}. (5)
The actual resulting image is then defined as WLdisplayLL(X)=XLdisplay.
There are a number of different approaches. Two different approaches include scale selection by maximal energy or entropy and scale selection by two-class labeling.
Scale Selection by Maximal Energy/Entropy
In one embodiment, the display scale is determined by characterizing trends of wavelet coefficients over the levels of resolution and detect levels of trend change. One criterion for a trend is the energy in subbands. If ELH[m] denotes the energy of wavelet coefficients in the subband LH at level m for a text image, then the trend is typically ELH[L]<ELH[L−1]< . . . <ELH[1]. The same trend is valid for the other two subband directions, HL and HH. The trend for a photographic natural image has the opposite behavior, ELH[L]>ELH[L−1]> . . . >ELH[1]. An image that contains mainly text in a small font will follow the text trend for all scales. An image with medium size text will follow the text trend for some large scales m1 . . . L, and then will switch to the photo trend for the small scales 1 . . . m1−1. Therefore, the smaller scales contain less information than the larger scales and downsampling to scale ml still keeps most of the important information in the image. The display scale is then chosen to be m1. In one embodiment, considering the energy as a function of level E(m), the scale at which the global maximum is located [Lmin,Lmax] is determined by
Ldisplay=arg maxmε[Lmin,Lmax](E(m)). (6)
In one embodiment, E(m) is the energy of a selected subband (e.g., LH or HL or HH). In an alternative embodiment, E(m) is the energy of a sum of weighted subbands [e.g. αELH(m)+βEHL(m)+γEHH(m). In yet another embodiment, E(m) is the maximum of energy from all subbands (e.g., max(ELH(m),EHL(m),EHH(m)).
For energy, e.g., comparing formula (6) with (5), the importance measure M in Eq. 1 measures the energy of wavelet coefficients at level m and is expressed as
Cost(M(WmLL(X))=αE(WmHL(X))+βE(WmLH(X))+γE(WmHH(X)) or (7)
Cost(M(WmLL(X))=max[E(WmLH(X)), E(WmLH(X)), E(WmHH(X))] (8)
In case of entropy M measures entropy H of subbands,
Cost(M(WmLL(X))=αH(WmHL(X))+βH(WmLH(X))+γH(WmHH(X)) or (9)
Cost(M(WmLL(X))=max[H(WmHL(X)), H(WmLH(X)), H(WmHH(X))]. (10)
Otherwise computations are the same.
In images that contain a lot of noise, the determination of Ldisplay could result in a scale containing mostly noise. Therefore, an additional criterion for distinguishing between noise and the image is useful to eliminate this case. In one embodiment, noise is removed from the wavelet coefficients using wavelet denoising. One denoising technique that may be used for this is described in Donoho, D. L., “Denoising by soft-thresholding,” IEEE Transactions on Information Theory, vol. 41, no. 3, pp. 613-627, 1995). Alternatively, another technique that may be used is described in U.S. patent application Ser. No. 10/044,420, entitled “Header-Based Processing Of Images Compressed Using Multi-Scale Transforms”, concurrently filed Jan. 10, 2002, and assigned to the corporate assignee of the present invention. In one embodiment, after denoising, the determination of the display scale is performed as set forth in Eq. (6) above.
In an alternative embodiment, in order to lower computational complexity, wavelet coefficients are not considered at all possible scales (levels) simultaneously, but are considered starting with the coarsest scale first, incorporating the next finer scales one by one and checking after incorporating each level whether a display scale Ldisplay can be determined. One embodiment of such an approach is described below.
In one approach, processing logic determines the energy as a function of level E(m). For increasing m, as long as E(m−1)≦E(m), processing logic computes E(m+1). If m=Lmin, then processing logic sets Ldisplay=Lmin.
If E(m−1)>E(m), then processing level neglects level m as the possible display scale Ldisplay if the ratio of error between denoised and original coefficients in one or more subbands at level m−1 and the original coefficients is larger than a predetermined threshold. For example, in one embodiment, the predetermined threshold is 0.7. Otherwise, processing logic chooses Ldisplay=m−1.
In an alternative embodiment, instead of determining the trend based on the importance measure energy, the importance measure entropy could be used. The entropy determines how many bits are spent to encode coefficients. The display scale is determined in a similar fashion as described in Eq. (8) by substituting entropy for energy of coefficients at level m.
In one embodiment, some alternatives for energy include sum of entropy in all subbands, using the maximum energy, and using linear combination, such as described above.
Scale Selection by Maximal Significance Ratio
In an alternative embodiment, a different criterion than that of Eq. (6) is used to determine a display scale. This criteria is based on a two-class labeling in which all wavelet coefficients are segregated into classes of significant and insignificant coefficients and then the scale that has the highest ratio of significant to insignificant coefficients is determined such that
Ldisplay=arg maxm=Lmin, . . . , Lmax(μ(significant coefficients)/μ(insignificant coefficients)) (12)
where μ measures a class of data. In one embodiment, this measure is a count of data points in the class. In an alternative embodiment, this measure is a weighted counting (e.g., weighting the center of the image more).
In two-class labeling, M measures percentage of significant wavelet coefficients at level m. The significance can be determined by checking whether a coefficient has magnitude larger than a predefined threshold or not. Another method would be to incorporate information from scales larger than m in the decision. One example is to check whether a coefficient is larger than a given percentage of the energy of coefficients at the next larger scale. The percentage threshold could depend on the display device or the application. Another example is to estimate a local entropy for each coefficient and check whether that value is larger than a given percentage of the entropy of coefficients at the next larger scale.
In one embodiment, the cost function in Eq. 1 is as follows:
Cost(M(WmLL(X))=0 if percentage of significant coefficients at level m is smaller than a given threshold T, or given that percentage of significant coefficients at level n≧m is smaller than T.
Cost(M(WmLL(X))=percentage of significant coefficients at level m if that is larger or equal than a given threshold T, given that percentage of significant coefficients at level n≧m is larger than T.
Approaches to solving the two-class labeling problem are well known in the field of pattern recognition. For example, see Duda, R. O., Hart, P. E., and Stork, D. G., Pattern Classification (2nd ed.), Wiley, New York, 2000. The two-class labeling problem for each scale solved may be solved using Bayesian decision theory. Image data that can enter into the probability models are magnitude of coefficients or local entropies.
Local Scale Selection
Local scale selection needs a partition in the image or wavelet domain and needs to consider groupings of coefficients or pixels. The grouping can be either given by the application or the user or is determined from the image data.
Partition of Image Domain into Segments
In one embodiment, the image is divided into two dimensional segments (e.g., tiles in J2K) and then global scale selection is performed as described below.
In one embodiment, instead of selecting a display scale and then choosing the LL component at that scale as the image representation, a part of the image at a specific scale (e.g., text at fine scale, background at coarse scale) is selected. In order to perform such a local scale selection, the image is partitioned into segments. In one embodiment, the segments are individual coefficients. In an alternative embodiment, the segments are cover groups of coefficients, shaped like, for example, squares, rectangles, etc.
After a partition (given, e.g., by the segment size and shape size) is chosen, the same approach on global scale selection as described above is applied to each of the segments S of the partition. As a result, each S(i) has an assigned display scale Ldisplay(i). In terms of pattern recognition, the result is an (Lmax−Lmin)-class labeling of the segment.
In one embodiment, only parts (e.g., segments) of the image that are associated with a particular display scale (e.g., Ldisplay=2) are displayed. The selected segments may be grouped or embedded into connected components. For example, for most image displays a rectangular display of image information is preferred because most image displays are rectangular. There are many approaches to selecting rectangles depending on an m-class labeling result. In one embodiment, the largest bounding box that contains all segments with Ldisplay=2 is chosen. The disadvantage of this embodiment is that isolated outliers for the box might be very large. According to an alternative embodiment, an approach that avoids this effect is used by choosing the rectangle such that the ratio
R(2)=μ(segments in rectangle with Ldisplay=2)/μ(segments in rectangle with Ldisplay≠2) (13)
is maximized, where μ denotes a measure as in Eq. (12). An example of one such rectangle is given in
R(m)=μ(segments in rectangle with Ldisplay=m)/m(segments in rectangle with Ldisplay≠m) (14)
over all mελ.
Differently stated, (Lmax−Lmin)-class labeling problems combining global and local scale selections lead to design choices for reduced images in specific applications that are discussed in further detail below.
Partition of Wavelet Domain into Cells
When partitioning the wavelet domain into cells, the wavelet domain is divided into wavelet cells. The metric M(x) measures information content for a group of data points a low-resolution Xm.
Let xεXm. The information content of x is computed by M(x) from measuring the contribution of energy or entropy of all the wavelet cells level larger or equal M that include x in their support at resolution level m. The energy or entropy may be scaled depending on the resolution of the wavelet cell and its size. The cost function sums over the measured information content of all data points that fit into a predescribed shape (e.g., a rectangle) or a given size. By choosing those points that produce the largest output of the cost function a part of the original image at a specific resolution is chosen for a reduced size image with recognizable information.
As an extension the maximization could be computed over a pre-described shape, but variable size or flexible shape of fixed size or flexible shapes of variable size.
Below is a mathematical formulation of the above.
A partition P of a set X is a set of subsets of X such that
and p1∩p2=0 for p1≠p2 and every pair of sets p1, p2εP.
ImHL={i,j)|i=0, . . . M2−m−1, j=N2−m, . . . N2−2m−1} (15)
ImLH={i,j)|i=M2−m, . . . M2−2m−1, j=0, . . . N2−2m−1} (16)
ImHH={i,j)|i=M2−m, . . . M2−2m−1, j=N2−m, . . . N2−2m−1} (17)
Let PmHL, PmLH and PmHH be partitions of ImHL, ImLH and ImHH, respectively.
A reduced size image with recognizable content is given by an output border and output image content. The border is given by a shape (e.g., rectangle) and a size (e.g., 68×80 pixels). In order to fill the border with image content, a resolution of the image and an anchor point position to locate the border is determined (e.g., coordinate (10,20) in resolution image Xm). An anchor point for a rectangle or square could be the upper left corner, while the anchor point for a circle may be the center.
In a first case, given a predetermined shape and size of the output border, the maximization problem is stated as
(Ldisplay, Adisplay)=arg maxmελ,aεA{Cost(M(Xm|output
The reduced size image with recognizable content is then defined as
{xεXLdisplay|output
In a second case, in addition, if the size of the template of the reduced size image with recognizable content is variable the maximization problem is
(Ldisplay, Adisplay, Ndisplay)=arg maxmελ,aεA,nεN{Cost(M(Xm|output
The reduced size image with recognizable content is then defined as
{xεXLdisplay|output
In a third case, if anchor point of the output border, the size and shape of the template of the reduced size image with recognizable content are variable the maximization problem is stated above.
(Ldisplay, Adisplay, Ndisplay, Sdisplay)=arg maxmελ,nεN,sεS{Cost(M(Xm|output
The reduced size image with recognizable content is then defined as
{xεXLdisplay|output
In this case, the vector (Ldisplay, Adisplay, Ndisplay, Sdisplay,) determines the reduced size image with recognizable content.
In the first case, the location and resolution selection for the fixed reduced size image with recognizable content are image content and display device dependent. The size and shape of the reduced size image with recognizable content border are application or user dependent. In the second case, location, resolution and reduced size image with recognizable content size, and in the third case location, resolution size and shape of the reduced size image with recognizable content are image content and display device dependent.
The four-dimensional vector (Ldisplay, Adisplay, Ndisplay, Sdisplay) determines the reduced size image with recognizable content. An extension is possible by allowing the entries in the vector to be vectors. In this case a reduced size image with recognizable content would be a collection of selected resolution segments with specific output borders. The segments may be chosen such that their full-resolution versions cover the entire image, either overlapping or non-overlapping. This is referred to herein as a Multiscale Collage.
In general, given an image the method creates an optimal thumbnails for a given set of constrains on the vector (Ldisplay, Adisplay, Ndisplay, Sdisplay). In this case, a reduced size image with recognizable content is a combination of specifically selected resolution segments. In the scalar and vector case, postprocessing can be applied to the resolution segments, such as interpolation, extrapolation, warping, blurring, etc.
Scale Selection Using Meta Data Created by a JPEG2000 Image Coder
The codestream of an image created by a JPEG2000 (“J2K”) image coder orders the wavelet domain image data into organizational units, such as packets, codeblocks, or tiles. Those units contains header information including features such as a number of zero bitplanes or unit code length. In one embodiment, those features are used to determine the importance measure M (e.g., entropy) and the cost function Cost. The partition of wavelet domain image data is then given by the choice of the code units (e.g., code blocks). Those data are easily accessible without decoding the actual image data and lead to very fast implementations of reduced size image creations.
In order to perform scale selection using J2K meta data, the partitioning of the wavelet domain into cells is given by codeblocks. The image domain may be given by tiles.
For global scale section, the entropy of a subband is given as the sum of length of codeblocks contained in the subband. The metrics described above in conjunction with global scale selection may be applied and the reduced size image with recognizable content computed.
For local scale selection, if the image is given in tiles, then the local scale selection approach described above with respect to partitioning the image domain into segments may be used. Alternatively, for local scale selection using J2K meta data, the local scale selection approach described above with respect to partitioning the wavelet domain into cells may be used where the wavelet cells are the code blocks and M measures the entropy of a codeblock.
A combination of global and local scale section may be used. For example, if the output border size and shape are not constrained, global selection may be attempted initially. After doing so, if the calculated resolution is out of the range x, then local scale selection may be started. In one embodiment, local scale section is started under a constraint (e.g., a rectangular shape). For example, if image is square, then global scale selection produces a square. If global scale selection does not work, then local scale selection should be applied given the constraint that the shape should be a square not bigger than a specific size.
The following is an example for combined global and local scale selection for two different displays. Two different display devices are considered, one is a CRT desktop monitor and one an LCD display of a PDA device. The physical properties of those displays are listed in Table 1.
Table 2 lists the calculated parameters Lmin, Dmin, and Lmax. The parameter Cdevice is set, e.g., by tuning or by using the ratio of the display contrast ratio to the contrast ratio of a reference display.
Global scale selection determines Ldisplay=5 for the CRT device. For the LCD device global scale selection is Ldisplay=5. But that is larger than Lmax. Therefore local scale selection starts, a square shape of 300×300 pixels is fixed and it is searches for the location and resolution L in the interval [0,Lmax] that results in the best fit, i.e. maximizes the cost function.
Applications
There are several ways to incorporate the concept of global and local scale selection into possible design choices for creating reduced size image. In general, they can be separated into five groups.
In one embodiment, an image is created that contains information of only one image segment such that the LL component of this segment at its calculated display scale fits into a given shape of fixed size (e.g., a 68×88 pixel rectangle). In this manner, the image then contains the LL component of the segment at a calculated display scale.
In one embodiment, a reduced size image is created that contains information of only one segment of the image at a particular LL resolution level Ldisplay. The shape of the reduced size image is fixed, but its size is flexible. For example, a reduced size image shape could be a rectangle of a certain width/height ratio R. The reduced size image then contains the image segment at its associated resolution, such that it satisfies the “best-fit” criterion over all considered rectangles of ratio R.
In one embodiment, a reduced size image is created that contains information of only one segment of the image at a particular resolution. The reduced size image shape and size are then flexible.
In one embodiment, a reduced size image is created that contains several segments of the image, each displayed at the LL component at its display level (i.e., a multiscale collage).
In one embodiment, a reduced size image is created that contains several segments of the image. Each segment is displayed at the LL component at its display level, with post-processing further performed on the LL component that affects the values of LL coefficients as well as the shapes of the displayed segments. Examples of such post-processing operations include extrapolation, blurring, stylizing, enhancing, fusing, warping, morphing, etc. The design space for reduced size images is symbolized in Table 3.
In one embodiment, selected image segments 2715-2721 are upsampled from an LL component by a factor. In the resulting multiscale collage, for example, segments 2722, 2726, and 2727 are upsampled by a factor of 2× from their scale selected LL components. Segments 2723 and 2724 are upsampled by a factor of 1× from their scale selected LL components. Segment 2728 is upsampled by a factor of 3× from its scale selected LL components. Finally, segment 2725 remains at the original size from its original counterpart, segment 2718.
In one embodiment, selected image segments 2729-2735 are upsampled in the same manner as segments 2715-2721. Furthermore, each corresponding segment 2736-2742 in a resulting multiscale collage is post-processing enhanced, warped, blurred, etc.
The embodiments above may be applied to digital camera displays, digital photo album software, image browsing, video summaries, page layout, and web page design. In applications such as these, the display range parameters Lmin and Lmax are in one embodiment set by the user and are potentially adjusted during the use of the reduced size image creation system. In one embodiment operating system, parameters such as absolute pixel resolution are extracted from the display settings without requiring any user effort. In the above description, only downsampling by a factor of two was considered due to its natural appearance in the wavelet decomposition. It is clear to one skilled in the art, however, that the approaches to reduced size image creation can also be extended to sampling ratios other than powers of two by interpolation or extrapolation of selected segments at specific resolutions.
The techniques described herein may be extended to color. For color images, all these techniques can be used on one chromatic band (e.g., L* inCIEL*uv, Yin Yuv, or G in sRGB) or on some or all. If more than one chromatic band is used, a technique can be used to combine results. Also, the techniques described herein may be extended to time (e.g., motion, video). In such a case, objects may be identified to track in thumbnail and/or key frame may be located, and then subsequently processed.
An Exemplary Computer System
System 2900 further comprises a random access memory (RAM), or other dynamic storage device 2904 (referred to as main memory) coupled to bus 2911 for storing information and instructions to be executed by processor 2912. Main memory 2904 also may be used for storing temporary variables or other intermediate information during execution of instructions by processor 2912.
Computer system 2900 also comprises a read only memory (ROM) and/or other static storage device 2906 coupled to bus 2911 for storing static information and instructions for processor 2912, and a data storage device 2907, such as a magnetic disk or optical disk and its corresponding disk drive. Data storage device 2907 is coupled to bus 2911 for storing information and instructions.
Computer system 2900 may further be coupled to a display device 2921, such as a cathode ray tube (CRT) or liquid crystal display (LCD), coupled to bus 2911 for displaying information to a computer user. An alphanumeric input device 2922, including alphanumeric and other keys, may also be coupled to bus 2911 for communicating information and command selections to processor 2912. An additional user input device is cursor control 2923, such as a mouse, trackball, trackpad, stylus, or cursor direction keys, coupled to bus 2911 for communicating direction information and command selections to processor 2912, and for controlling cursor movement on display 2921.
Another device that may be coupled to bus 2911 is hard copy device 2924, which may be used for printing instructions, data, or other information on a medium such as paper, film, or similar types of media. Furthermore, a sound recording and playback device, such as a speaker and/or microphone may optionally be coupled to bus 2911 for audio interfacing with computer system 2900. Another device that may be coupled to bus 2911 is a wired/wireless communication capability 2925 to communication to a phone or handheld palm device.
Note that any or all of the components of system 2900 and associated hardware may be used in the present invention. However, it can be appreciated that other configurations of the computer system may include some or all of the devices.
Whereas many alterations and modifications of the present invention will no doubt become apparent to a person of ordinary skill in the art after having read the foregoing description, it is to be understood that any particular embodiment shown and described by way of illustration is in no way intended to be considered limiting. Therefore, references to details of various embodiments are not intended to limit the scope of the claims which in themselves recite only those features regarded as essential to the invention.
This is a continuation of application Ser. No. 10/044,603, filed on Jan. 10, 2002 now U.S. Pat. No. 7,095,907, entitled “Content and Display Device Dependent Creation of Smaller Representations of Images,” incorporated by reference herein and assigned to the corporate assignee of the present invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3580655 | Leith et al. | May 1971 | A |
| 3950103 | Schmidt-Weinmar | Apr 1976 | A |
| 4136954 | Jamieson | Jan 1979 | A |
| 4155097 | Lux | May 1979 | A |
| 4190861 | Lux | Feb 1980 | A |
| 4223354 | Noble et al. | Sep 1980 | A |
| 4393456 | Marshall, Jr. | Jul 1983 | A |
| 4437087 | Petr | Mar 1984 | A |
| 4569075 | Nussbaumer | Feb 1986 | A |
| 4599567 | Goupillaud et al. | Jul 1986 | A |
| 4652881 | Lewis | Mar 1987 | A |
| 4663660 | Fedele et al. | May 1987 | A |
| 4674125 | Carlson et al. | Jun 1987 | A |
| 4701006 | Perlmutter | Oct 1987 | A |
| 4751742 | Meeker | Jun 1988 | A |
| 4760563 | Beylkin | Jul 1988 | A |
| 4785348 | Fonsalas et al. | Nov 1988 | A |
| 4785349 | Keith et al. | Nov 1988 | A |
| 4799179 | Masson et al. | Jan 1989 | A |
| 4805129 | David | Feb 1989 | A |
| 4815023 | Arbeiter | Mar 1989 | A |
| 4817182 | Adelson et al. | Mar 1989 | A |
| 4821223 | David | Apr 1989 | A |
| 4827336 | Acampora et al. | May 1989 | A |
| 4829378 | Le Gall | May 1989 | A |
| 4837517 | Barber | Jun 1989 | A |
| 4839889 | Gockler | Jun 1989 | A |
| 4858017 | Torbey | Aug 1989 | A |
| 4864398 | Avis et al. | Sep 1989 | A |
| 4868868 | Yazu et al. | Sep 1989 | A |
| 4881075 | Weng | Nov 1989 | A |
| 4894713 | Delogne et al. | Jan 1990 | A |
| 4897717 | Hamilton et al. | Jan 1990 | A |
| 4899147 | Schiavo et al. | Feb 1990 | A |
| 4904073 | Lawton et al. | Feb 1990 | A |
| 4918524 | Ansari et al. | Apr 1990 | A |
| 4922544 | Stansfield et al. | May 1990 | A |
| 4929223 | Walsh | May 1990 | A |
| 4929946 | O'Brien et al. | May 1990 | A |
| 4936665 | Whitney | Jun 1990 | A |
| 4973961 | Chamzas et al. | Nov 1990 | A |
| 4974187 | Lawton | Nov 1990 | A |
| 4982283 | Acampora | Jan 1991 | A |
| 4985927 | Norwood et al. | Jan 1991 | A |
| 4987480 | Lippman et al. | Jan 1991 | A |
| 4999705 | Puri | Mar 1991 | A |
| 5000183 | Bonnefous | Mar 1991 | A |
| 5001764 | Wood et al. | Mar 1991 | A |
| 5014134 | Lawton et al. | May 1991 | A |
| 5018210 | Merryman et al. | May 1991 | A |
| 5049992 | Citta et al. | Sep 1991 | A |
| 5049993 | Le Gall et al. | Sep 1991 | A |
| 5068911 | Resnikoff et al. | Nov 1991 | A |
| 5072308 | Lin et al. | Dec 1991 | A |
| 5073964 | Resnikoff | Dec 1991 | A |
| 5081645 | Resnikoff et al. | Jan 1992 | A |
| 5095447 | Manns et al. | Mar 1992 | A |
| 5097261 | Langdon, Jr. et al. | Mar 1992 | A |
| 5097331 | Chen et al. | Mar 1992 | A |
| 5101280 | Moronaga et al. | Mar 1992 | A |
| 5101446 | Resnikoff et al. | Mar 1992 | A |
| 5103306 | Weiman et al. | Apr 1992 | A |
| 5109451 | Aono et al. | Apr 1992 | A |
| 5121191 | Cassereau et al. | Jun 1992 | A |
| 5124930 | Nicholas et al. | Jun 1992 | A |
| 5128757 | Citta et al. | Jul 1992 | A |
| 5128791 | Le Gall et al. | Jul 1992 | A |
| 5148498 | Resnikoff et al. | Sep 1992 | A |
| 5152953 | Ackermann | Oct 1992 | A |
| 5156943 | Whitney | Oct 1992 | A |
| 5173880 | Duren et al. | Dec 1992 | A |
| 5182645 | Breeuwer et al. | Jan 1993 | A |
| 5223926 | Stone et al. | Jun 1993 | A |
| 5235434 | Wober | Aug 1993 | A |
| 5241395 | Chen | Aug 1993 | A |
| 5262958 | Chui et al. | Nov 1993 | A |
| 5276525 | Gharavi | Jan 1994 | A |
| 5315670 | Shapiro | May 1994 | A |
| 5321776 | Shapiro | Jun 1994 | A |
| 5335016 | Nakagawa | Aug 1994 | A |
| 5347479 | Miyazaki | Sep 1994 | A |
| 5349348 | Anderson et al. | Sep 1994 | A |
| 5379355 | Allen | Jan 1995 | A |
| 5381145 | Allen et al. | Jan 1995 | A |
| 5384869 | Wilkinson et al. | Jan 1995 | A |
| 5412741 | Shapiro | May 1995 | A |
| 5414780 | Carnahan | May 1995 | A |
| 5416604 | Park | May 1995 | A |
| 5420891 | Akansu | May 1995 | A |
| 5442458 | Rabbani et al. | Aug 1995 | A |
| 5453945 | Tucker et al. | Sep 1995 | A |
| 5455874 | Ormsby et al. | Oct 1995 | A |
| 5481308 | Hartung et al. | Jan 1996 | A |
| 5495292 | Zhang et al. | Feb 1996 | A |
| 5497435 | Berger | Mar 1996 | A |
| 5511151 | Russell et al. | Apr 1996 | A |
| 5534925 | Zhong | Jul 1996 | A |
| 5537493 | Wilkinson | Jul 1996 | A |
| 5541594 | Huang et al. | Jul 1996 | A |
| 5546477 | Knowles et al. | Aug 1996 | A |
| 5563960 | Shapiro | Oct 1996 | A |
| 5566089 | Hoogenboom | Oct 1996 | A |
| 5602589 | Vishwanath et al. | Feb 1997 | A |
| 5610729 | Nakajima | Mar 1997 | A |
| 5631977 | Koshi | May 1997 | A |
| 5638498 | Tyler et al. | Jun 1997 | A |
| 5657085 | Katto | Aug 1997 | A |
| 5701367 | Koshi et al. | Dec 1997 | A |
| 5717789 | Anderson et al. | Feb 1998 | A |
| 5754793 | Eom et al. | May 1998 | A |
| 5808683 | Tong et al. | Sep 1998 | A |
| 5809176 | Yajima | Sep 1998 | A |
| 5850482 | Meany et al. | Dec 1998 | A |
| 5867602 | Zandi et al. | Feb 1999 | A |
| 5880856 | Ferriere | Mar 1999 | A |
| 5966465 | Keith et al. | Oct 1999 | A |
| 5995644 | Lai et al. | Nov 1999 | A |
| 6020975 | Chen et al. | Feb 2000 | A |
| 6026198 | Okada | Feb 2000 | A |
| 6041143 | Chui et al. | Mar 2000 | A |
| 6055340 | Nagao | Apr 2000 | A |
| 6088062 | Kanou et al. | Jul 2000 | A |
| 6101279 | Nguyen et al. | Aug 2000 | A |
| 6118902 | Knowles | Sep 2000 | A |
| 6121970 | Guedalia | Sep 2000 | A |
| 6128413 | Benamara | Oct 2000 | A |
| 6141452 | Murao | Oct 2000 | A |
| 6160846 | Chiang | Dec 2000 | A |
| 6201897 | Nixon | Mar 2001 | B1 |
| 6226412 | Schwab | May 2001 | B1 |
| 6229929 | Lynch et al. | May 2001 | B1 |
| 6236765 | Archarya | May 2001 | B1 |
| 6237010 | Hui et al. | May 2001 | B1 |
| 6263109 | Ordentlich et al. | Jul 2001 | B1 |
| 6263120 | Matsuoka | Jul 2001 | B1 |
| 6327392 | Li | Dec 2001 | B1 |
| 6330666 | Wise et al. | Dec 2001 | B1 |
| 6339658 | Moccagatta et al. | Jan 2002 | B1 |
| 6350989 | Lee et al. | Feb 2002 | B1 |
| 6356668 | Honsinger et al. | Mar 2002 | B1 |
| 6363381 | Lee et al. | Mar 2002 | B1 |
| 6456340 | Margulis | Sep 2002 | B1 |
| 6463181 | Duarte | Oct 2002 | B2 |
| 6466698 | Creusere | Oct 2002 | B1 |
| 6483946 | Martucci et al. | Nov 2002 | B1 |
| 6546143 | Taubman et al. | Apr 2003 | B1 |
| 6625321 | Li et al. | Sep 2003 | B1 |
| 6650782 | Joshi et al. | Nov 2003 | B1 |
| 6668090 | Joshi et al. | Dec 2003 | B1 |
| 6707948 | Cosman et al. | Mar 2004 | B1 |
| 6785411 | Kitajima et al. | Aug 2004 | B1 |
| 6975753 | Matsuura et al. | Dec 2005 | B2 |
| 6978211 | Soliman et al. | Dec 2005 | B2 |
| 6985630 | Kajiwara | Jan 2006 | B2 |
| 7024045 | McIntyre | Apr 2006 | B2 |
| 7212678 | Brown et al. | May 2007 | B2 |
| 7236637 | Sirohey et al. | Jun 2007 | B2 |
| 20010021223 | Andrew | Sep 2001 | A1 |
| 20010047517 | Christopoulos et al. | Nov 2001 | A1 |
| 20020057385 | Sun et al. | May 2002 | A1 |
| 20030110299 | Larsson et al. | Jun 2003 | A1 |
| Number | Date | Country |
|---|---|---|
| 0510933 | Oct 1992 | EP |
| 0593013 | Apr 1994 | EP |
| 0611051 | Aug 1994 | EP |
| 0622741 | Nov 1994 | EP |
| 701375 | Mar 1996 | EP |
| 0967556 | Dec 1999 | EP |
| 1035511 | Sep 2000 | EP |
| 1164781 | Dec 2001 | EP |
| 2 211 691 | Jul 1989 | GB |
| 2 284 121 | May 1995 | GB |
| 2 285 374 | Jul 1995 | GB |
| 2 293 733 | Apr 1996 | GB |
| 2 293 734 | Apr 1996 | GB |
| 2 303 030 | Feb 1997 | GB |
| 2 303 031 | Feb 1997 | GB |
| 2 341 035 | Mar 2000 | GB |
| 406038193 | Jul 1992 | JP |
| 06-245077 | Sep 1994 | JP |
| 6-350989 | Dec 1994 | JP |
| 7-79350 | Mar 1995 | JP |
| WO 8810049 | Dec 1988 | WO |
| WO 9103902 | Mar 1991 | WO |
| WO 9118361 | Nov 1991 | WO |
| WO 9310634 | May 1993 | WO |
| WO 9417492 | Aug 1994 | WO |
| WO 9423385 | Oct 1994 | WO |
| WO 9519683 | Jul 1995 | WO |
| WO 9609718 | Mar 1996 | WO |
| WO 0049571 | Aug 2000 | WO |
| WO 0116764 | Mar 2001 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20070047841 A1 | Mar 2007 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10044603 | Jan 2002 | US |
| Child | 11334175 | US |
