Patent Application
20090295789
This invention relates to methods for displaying of images, and in particular, to methods for displaying images having certain characteristics (e.g. a panoramic image) on digital display devices such as digital picture frames.
Digital display devices (“DDDs”) such as digital picture frames (“DPFs”) allow the digital display of a collection of photos or images. The advancement in the mass production of LCDs resulted in the lowering of the cost of the LCDs and therefore of the DDDs as well. As DDDs become more and more popular, particular problems associated with DDDs are becoming apparent and require customized solution. There are several factors to consider with respect to DDDs, for example image quality, ease of setup, ease of use, and image presentation.
Ideally, DDDs should be able to accept source images from a variety of capture devices or external images. These source images may also have a variety of properties such as having a variety of heights, widths, aspect ratios, and resolutions. When the properties of the source image do not conform with the properties of the DDD such as the size of a display window, the DDD needs to have the capability of processing the source image and displaying the resulting digital image. For example, if the provided image has an image size of 1024×768 pixels and the particular DDD has a display window size of 720×480 pixels, the provided image need to be resized before it can be properly displayed on the display window of the DDD. At present, most DDDs only allow for the limited processing of the original image. They may be able to reduce the resolution of the original image to conform to the resolution of the DDD. They may also be able to perform limited resizing by reducing the size of the image such that the entire source image fits into the display window of the DDD, or in the alternative, crop the original digital image such the only a portion of the source image is displayed.
However, these types of resizing methods do not allow the DDD to display the digital image adequately. For example,
b illustrates a second type of resizing method of the prior art. In this type of resizing methods, only a small portion of the image is displayed. Again, this type of resizing methods is unsatisfactory because the display fails to properly display the effect of panoramic view of the source image.
Therefore, it is desirable to provide methods for displaying digital images on the display window of a DDD that would take into account the special properties of a digital image.
An object of this invention is to provide methods for dynamically displaying digital images having special characteristics with special effects on a digital display device.
Another object of this invention is to provide methods for automatically adjusting the mode of display of a digital image as a function of the properties of that digital image.
Another object of this invention is to provide methods for the smooth panning of a digital image at velocities that are accurate up to fractional pixels per unit time.
Another object of this invention is to enable the viewing of images of unlimited sizes and resolutions on digital display devices with display windows that are limited in size and resolution.
The present invention relates to methods for dynamically displaying a source image on a display window of a digital display device, wherein the source image having source image characteristics, comprising the steps of: processing the source image into a canvas image having canvas pixels as a function of the source image characteristics; upon the canvas image meeting one or more conditions for the panning of an image, selecting a panning velocity for the canvas image; defining a viewing window with respect to the canvas image wherein said viewing window having a viewing image, said viewing image is to have a plurality of viewing pixels; calculating the viewing pixels for said viewing image; displaying said viewing image for said viewing window on said display window; redefining said viewing window with respect to the canvas image as a function of said panning velocity; and repeating the calculating and displaying steps for said viewing window.
An advantage of the methods of this invention is that they can dynamically display digital images having special characteristics with special effects on a digital display device.
Another advantage of the methods of this invention is that they can automatically adjust the mode of display of a digital image as a function of the properties of that digital image.
Another advantage of the methods of this invention is to provide for the smooth panning of digital images at velocities that is fractional pixels per unit time.
Another advantage of the methods of this invention is that they enable the viewing of images of unlimited sizes and resolutions on a digital display device with a display window that is limited in size and resolution.
The foregoing and other objects, aspects and advantages of the invention will be better understood from the following detailed description of preferred embodiments of this invention when taken in conjunction with the accompanying drawings in which:
a is an illustration of a prior art method for displaying the panoramic image of
b is an illustration another prior art method for displaying the panoramic image of
a through 3e illustrate the moving of the viewing window for displaying the image of
a through 4e illustrate the images as selected by the viewing windows (as illustrated by
a through 5e illustrate the image displayed by the display window of the digital display device during the different time periods using the methods of this invention.
a through 6f illustrate the panning of an image using a method of this invention.
a-8b illustrate the moving of the viewing pixels by a pixel unit.
a-9b illustrate the moving of fractional viewing pixels to a position that does not coincide with a canvas pixels.
a-10b illustrate the positions and canvas pixels on a canvas that may be used to calculate the viewing image characteristics at position (a,b).
The presently preferred methods of the present invention provide methods for displaying an image as a function of the properties of that image and for a particular digital display device (such as a digital picture frame).
In the discussions herein, the following terminology may be used, and it shall be understood that the methods and embodiments of the present invention are not to be limited by the definition of the terminology provided below:
Source image. The image obtained from a source. Sources may include a capturing device such as a camera or a storage device.
Canvas and canvas image. The canvas image is the image obtained after the source image has been prepared for use in a DDD. The canvas image is defined by the canvas pixels. The canvas image characteristics between the canvas pixels, herein referred to as canvas positions, are virtual positions that can be defined.
Canvas properties. Any parameter relating to the canvas or canvas image is a canvas property. Examples of canvas properties may be the width, height, or aspect ratio of the canvas, or the characteristics of the canvas image.
Viewing window. A viewing window is a portion of the canvas that is to be displayed by the display window at any one time during panning. The characteristics of the viewing image within the viewing window are defined by a set of viewing pixels. Depending on the location of the viewing window, the viewing pixels may or may not to be calculated. A viewing pixel is illustrated in the drawings described herein as a small rectangle. (□)
Viewing properties. Any parameter relating to the viewing window is a viewing property. Examples of viewing properties may be the width, height, or aspect ratio of the viewing window, or the characteristics of the viewing image within the viewing window.
Pij. The characteristics of a pixel of an image, either a canvas image or a viewing image, is represented as Pij where i and j refers to the position of the pixel in the image. For example, P00 may refers to the image characteristic at the pixel (0,0) while Pab refers to the image characteristics at position (a,b).
Display window. A display window may be the window of the DDD that displays a display image as defined by a viewing widow.
Display properties. Display properties may be any parameter relating to the display window including but nor limited to the height and width of the display window.
“x” dimension. The dimension that defines the width of the image, also referred to herein as the first dimension.
“y” dimension. The dimension that defined the height of the image, also referred to herein as the second dimension.
Panning. Panning may mean the moving of the viewing window having a viewing image at panning velocity.
In a presently preferred method of the present invention, a first step is to determine the conditions that trigger panning of an image. With pre-determined conditions, the DDD may be programmed such that panning automatically occurs once these conditions are triggered. The conditions for triggering panning may differ. In one preferred method, all images are to be panned. In another preferred method, the conditions for panning may depend on the size, aspect ratio, height or width of the canvas and the display window. (In the discussions herein, the term aspect ratio is used in the same manner as is commonly used in the field of image processing or display.) For example, if the horizontal dimension (the first dimension) is the dimension for the width of the canvas or the width of the display window, one criteria for triggering panning in the horizontal direction may be that the aspect ratio of the canvas divided by the aspect ratio of the display window is greater than 2. Similarly, if the vertical dimension (the second dimension) is the dimension for the height of the canvas, the criteria for the triggering of the panning in the vertical direction may be that the aspect ratio of the canvas divided by the aspect ratio of the display window is less than one-half (½).
The preparing of the source image to form the canvas image may include one or more of the following: obtaining or downloading the source image from a capture device or external storage; obtaining the properties of the source image, such as its width, height, aspect ratio, width/height, or height/width; decoding, if necessary, the source image into an internal format that the DDD may use; decoding, if necessary, a high resolution image to a lower resolution image in order to reduce the storage size; resizing the image; storing information relating to the decoded image and the source image in the buffer memory of the DDD. The decoded image is referred to as the canvas image and may be stored in the buffer. In preferred embodiments, this canvas may have the same aspect ratio as the source image. For example, if the source image is a 100×1000 (JPEG/BMP), after decoding and preparation, the canvas can be a 50×500 image having the internal format of the DDD.
After preparation and selection of a particular canvas image to display, the selected canvas image is then examined in order to decide if it meets one or more of the conditions for panning. If the image is not to be panned, then the display of the image may follow the traditional methods for statically display of a digital image. The canvas image may be processed for static display. This processing may involve resizing the canvas such that the entire or selected portions of the canvas are displayed in the display window.
If the image is determined to be dynamically displayed, then the velocity of panning, i.e., the direction of the panning as well as the speed of the panning, need to be determined. The panning velocity may be a function of many parameters, including canvas properties, display properties, viewing properties, and time. For example, the panning velocity may be a sinusoidal function of time. It may also depend on the position of the viewing image on the display window. For example, the panning velocity may be slower if only one part of the viewing image is displayed and the panning velocity may be increased as the area of the viewing image displayed is increased.
In another example, the direction of panning may be determined by the height, width or aspect ratio of the canvas and the display window. For example, if the canvas image is that of a panoramic view such as that illustrated in
The canvas image illustrated in
As described above, the panning velocity may be dependent of the size of the canvas image, viewing image or display window. It may even be dependent on the properties of the viewing image that is currently being displayed in the display window. For example, the panning velocity may be inversely proportional to the brightness of an image or a portion of an image that is currently on display such that the panning velocity is reduced and the image moves slower across the display window when an image is brighter.
However, in other preferred embodiments, the panning velocity and speed may be nonlinear and/or object based. The panning velocity may be object based when it depends on the properties of an object or a number of objects within the viewing image (“objects”) that are being displayed in the display window. The panning velocity is nonlinear when it varies during panning of an image. When this variation is dependent on objects, then, the panning velocity is non-linear and is object based. For example, the panning velocity may be reduced when faces in a panoramic school graduation picture are being displayed while the panning velocity may be increased when the buildings within that picture are being displayed.
The different modes of presentation created by panning or moving an image at a variable velocity may be infinite, limited only by the creativity of the DDD maker. As another example, a digital image that is not a panoramic image and that can fit into a part of the display can be panned at a two-dimensional velocity for presentational effect such that it may look as if it is slowly entering the display window from the edge of the display window, floats around the window for a while, and then leave the display window at some other edge. In this example, the panning velocity may be set to be time dependent or dependent on the position of the viewing image on the display screen. An example of such a display is illustrated by
Another method of display may have an image slowly entering the display window as the previous image is slowly leaving the display window from some other location of the display window. The panning velocity of this method of display may not only be time dependent but may also depend on the panning velocity of the previous image and the position of the previous image on the canvas.
The viewing window is a portion of the canvas and formed by selecting a portion of the canvas as the viewing window. The size of the viewing window may depend on multiple factors such as the height or width of the canvas and/or the panning speed. In one preferred method, the viewing window is cropped such that it has the same aspect ratio as the display window. Another preferred method is to display an image such that the image is maximized for display on the display window at any one time. To do so, given a panning velocity, the viewing window may be selected such that the dimension of the image that is not being panned is maximized for display. For example, if the image is of a panoramic view such that its width is much larger than its height, the image may be panned in the x direction as indicated by
The example illustrated by
With the size of the viewing window determined, the initial position to begin panning may need to be selected by selecting a particular viewing window. In a preferred embodiment, the panning starts on an edge or corner of the canvas. For example, if it is decided that the panning is to be from left to right for a panoramic image, then, in the preferred embodiment, the first viewing window may be positioned at the left end of a canvas.
Once this viewing window is selected, the characteristics of the viewing image will then have to be calculated. If the viewing pixels in this viewing window coincide with the canvas pixels, i.e., in situations where the viewing pixels of the viewing window coincide with the canvas pixels (or the panning velocity are in integer multiples of the canvas pixels per unit time), then the viewing pixels of a viewing image can be easily determined, where they may simply equal to the corresponding canvas pixels of the canvas image.
However, if the viewing pixels for a viewing image fall on virtual canvas positions (e.g. in between canvas pixels), then the viewing pixels may have to be estimated or calculated. The methods for calculating the viewing pixels are described below.
Processing of the viewing image for display may also occur, sometimes during the determination of the viewing image characteristics. Processing may involve resizing or zooming by cropping the canvas and the viewing window appropriately. After processing, this processed viewing image is then displayed by the display window.
In order to provide smooth panning of the viewing image, it may be desirable to pan at less than a whole pixel per unit time. If the panning velocity is (a, b) pixels per unit time, i.e., if the panning speed is “a” pixels (or first pixels) per unit time in the x (first) dimension and “b” pixels (or second pixels) per unit time in the y (second) dimensions, then after the defined unit of time, the viewing window may be moved “a” (first) pixels in the x dimension and “b” (second) pixels in the y dimension. The characteristics of the new viewing image corresponding to the new viewing window is calculated, the new viewing image is processed, and displayed as previously described. These steps are repeated until the panning for this image is completed.
One method of panning is based upon the moving of the viewing window at multiples of pixels per unit time. Thus, for example, as illustrated in
One of the advantages of the preferred methods of the present invention is that the viewing window can be moved more than just multiples of pixels; they can be moved a fractional of pixels per unit time. That is, the viewing window may be moved in fractional pixels per unit time. This added refinement in the movement of viewing window and the panning velocity allows for much smoother panning and presentation of the images. In a preferred method, the viewing window can be moved to the accuracy of 1/16th or 1/64th of a pixel per unit time. For example, in one preferred method, if the unit of time used for panning is 30 milliseconds the viewing image may be panned at 1/64th of a pixel every 30 millisecond, that may provide a much smoother panning of the image.
a and 9b illustrates the moving of a viewing windows of 1¼ pixels per unit time. In the example in
Interpolation, either non-adaptive or adaptive interpolation algorithms, may be used for this estimation or determination. Non-adaptive algorithms that may be used include: nearest neighbor, bilinear, bicubic, spline, sinc, and lanczos methods. For example, a bicubic algorithm may assume that, in two dimensions, the surrounding 4×4 pixels may contribute to the characteristics at the viewing pixel in question. These different methods use different numbers of surrounding canvas pixels for calculating the respective viewing pixel. Depending on the quality of the image desired, these interpolation methods may use from 0 to 256 or more adjacent pixels during interpolation. The more adjacent pixels that are used, the more accurate these algorithms may be in approximating or estimating the true image characteristics. However, the more adjacent pixels that are used, the longer the processing time is required for calculating or estimating the viewing image. Therefore, as a compromise between processing time and quality of the viewing image, a preferred method here uses the bicubic method of interpolation.
Adaptive algorithms may also be used. They apply special algorithms, sometimes on a pixel-by-pixel basis when the presence or an edge is detected in order to minimize the interpolation artifacts in the regions that are most apparent.
Interpolation uses the characteristics of the canvas image at surrounding pixels to estimate the viewing image for viewing pixels that do not coincide with the canvas pixels. Different interpolation algorithms give different weight to the surrounding canvas pixels in influencing the characteristics of the viewing image. They may use different filters that put different relative weight on the surrounding pixels for use to calculated to the viewing image characteristics. Filters that can be used may include the box, tent, Gaussian, catmull-rom and sinc. Preferred method uses a tent or triangular filter and a catmull-rom filter.
For example, in a one dimensional bicubic estimation, i.e., an estimation that only takes into account the canvas image in one dimension such as that illustrated in
The weight or influence of the characteristics of each of the pixel can vary depending on the filter chosen that may define the desired sharpness of the characteristics of the viewing image. Once the sharpness, and thus the weight of each of the canvas pixels, is determined, the coefficients, a reflection of the weight for that coefficient can then be determined and pre-calculated in a table. The viewing image characteristics P(a,0) is then a sum of the coefficients over the canvas pixels used for the respective method of interpolation where a filter that defines the sharpness of the viewing image may be selected. Different filters may be used to produce the desired effect for the transitions of the viewing images. Once the filter is selected, the coefficients, i.e., the weighing of the different surrounding pixels, can be determined, and the resulting characteristics is the sum of the surrounding pixels weighed by their coefficients.
The following illustrates how coefficients can be obtained for the example illustrated in
The following 64 sets of coefficients are generated where Phase (i) provides for the coefficient of a viewing pixel on a canvas position that is (i/64th of a pixel) from its neighboring canvas pixel.
Phase0: 0, 128, 0, 0,
Phase1: 0, 126, 2, 0,
Phase2: 0, 124, 4, 0,
Phase3: 0, 122, 6, 0,
Phase4: 0, 120, 8, 0,
Phase5: 0, 118, 10, 0,
Phase6: 0, 116, 12, 0,
Phase7: 0, 114, 14, 0,
Phase8: 0, 112, 16, 0,
Phase9: 0, 110, 18, 0,
Phase10: 0, 108, 20, 0,
Phase11: 0, 106, 22, 0,
Phase12: 0, 104, 24, 0,
Phase13: 0, 102, 26, 0,
Phase14: 0, 100, 28, 0,
Phase15: 0, 98, 30, 0,
Phase16: 0, 96, 32, 0,
Phase17: 0, 94, 34, 0,
Phase18: 0, 92, 36, 0,
Phase19: 0, 90, 38, 0,
Phase20: 0, 88, 40, 0,
Phase21: 0, 86, 42, 0,
Phase22: 0, 84, 44, 0,
Phase23: 0, 82, 46, 0,
Phase24: 0, 80, 48, 0,
Phase25: 0, 78, 50, 0,
Phase26: 0, 76, 52, 0,
Phase27: 0, 74, 54, 0,
Phase28: 0, 72, 56, 0,
Phase29: 0, 70, 58, 0,
Phase30: 0, 68, 60, 0,
Phase31: 0, 66, 62, 0,
Phase32: 0, 64, 64, 0,
Phase33: 0, 62, 65, 0,
. . .
Phase63: 0, 2, 126, 0.
The coefficients herein listed are normalized to 128. That is the characteristics using these coefficients may have to be divided by 128 to obtain the normalized characteristics.
Other filters can also be used. For example, the catmulll-spline filter is defined as:
As a compromise between processing time and accuracy, another preferred method uses the bicubic method of interpolation.
Pa0=Ch0*P00+Ch1*P10+Ch2*P20+Ch3*P30.
For example, if the triangular filter described above is used and the fractional part of “a” is ¼ or 4/16 away from (0,1), the coefficients from phase 16 can then be chosen such that
P0a=(0*P00+96*P10+32*P20+0*P30)/128=96*P10+32*P20.
The characteristics of the image at (0,a) using other filter coefficients or using other interpolative methods utilizing different number of canvas pixels can be similarly obtained.
Likewise, a two dimension interpolation can be obtained using a two-step one dimensional interpolation process to obtain the viewing image characteristics P(ab) at canvas position (a, b) as a function of N*M canvas pixels if those pixels are selected for filtering. For example, for a bicubic interpolation, i.e., for N=4 and M=4 where, as illustrated in
P0b=Cv0*P00+Cv1*P01+Cv2*P02+Cv3*P03
P1b=Cv0*P10+Cv1*P11+Cv2*P12+Cv3*P13
P2b=Cv0*P20+Cv1*P21+Cv2*P22+Cv3*P23
P3b=Cv0*P30+Cv1*P31+Cv2*P32+Cv3*P33
Where Cvi is the vertical weighing or filtering coefficients at row “i”. These coefficients may be pre-calculated and stored in a table.
The above for image characteristics may again be combined with the corresponding Chi, the horizontal weighing or filtering coefficients at column “i” to obtain the image characteristics at (a,b) such that:
Pab=Ch0*P0b+Ch1*P1b+Ch2*P2b+Ch3*P3b
The filtering or weighing coefficients Chi and Cvi are dependent on the fractional pixel value of a and b. The size of the source image, the ratio of the sharpness/blurriness desired, and other factors may dictate the desired fractional resolution between two pixels or the fractional accuracy of the positions needed. In the preferred methods where the positions within the canvas are accurate up to 1/64 of a pixel, i.e., where the fractional resolution between two pixels is 64 steps, there may be 64 table items for each Chi or Cvi. If a tent filter is used, then these coefficients may be derived from the coefficients listed herein in this specification.
While the present invention has been described with reference to certain preferred embodiments or methods, it is to be understood that the present invention is not limited to such specific embodiments or methods. Rather, it is the inventor's contention that the invention be understood and construed in its broadest meaning as reflected by the following claims. Thus, these claims are to be understood as incorporating not only the preferred methods described herein but all those other and further alterations and modifications as would be apparent to those of ordinary skilled in the art.
