Patent Grant
9230297
Embodiments described herein generally relate to image processing.
In order to produce color images, such as photograph and video images, demosaicing and image warping are often performed in succession on images resulting in rectified color images.
A demosaicing algorithm is a digital image process that is used to reconstruct a full color image from incomplete color output from an image sensor overlaid with a color filter array in order to render the image into a viewable format. As the raw color image includes an intensity of only one color of the three filter colors (red, blue, and green) per pixel, demosaicing is used to determine intensities of the other two colors for each pixel. Many algorithms exist for demosaicing a color image. These algorithms vary in complexity and may result in a tripling of the data involved.
Image warping is the process of smoothly displacing the pixels in an image. In a warped image, the value at each pixel is usually computed from values sampled in the original image. Since the location in the raw image does not necessarily have integer coordinates, some interpolation is typically needed. Warping is used, for example, to correct for distortions present in an image. An image produced by an optical system is subject to various forms of geometric distortion. The distortion is often significant enough that it is desirable to warp the image into a more usable form. For example, objects in the center of an image obtained using a wide-angle or fish-eye lens will appear enlarged compared to objects at the edges. This is called barrel distortion. Warping the image may correct for the barrel distortion, resulting in a proportionally correct image that would appear to have been produced by a perspective camera. Warping can also be used to correct other types of distortions, or even to purposely cause distortions.
Both demosaicing and warping introduce some image defects, or artifacts, because they typically each involve interpolation. Performing demosaicing and warping individually in succession compounds this issue. In addition to the processing time needed to perform the two operations, demosaicing and warping in succession requires computer memory to store an intermediate image and also requires interpolations being performed twice.
In the drawings, the leftmost digit(s) of a reference number may identify the drawing in which the reference number first appears.
A color or multi-spectral image may consist of many measurements, at each pixel, of the intensity of the light of a certain color reaching that pixel. In most color images, each pixel has a measurement of the intensity of the red light reaching that pixel, a measurement of the green light reaching that pixel, and a measurement of the blue light reaching that pixel. It may be stated that such a color image has a red, green and blue channel. In other types of images, there may be more and/or different channels. In this document, a reference to “c” may refer to a color channel.
In this document, when distinction is not needed, the term “image” may refer to an RGB (red green blue) image or multispectral image or more generally an image with multiple channels.
In this document, a reference to “(i,j)” may refer to a location, or pixel, in an image. A reference to “I” or “J” may each refer to an image, and a reference to “I(i,j,c)” may refer to the value of channel c at column i and row j in image I.
Demosaicing
Many contemporary imaging sensors only produce a single intensity measurement at each pixel, corresponding to one of the channels. For example, in a sensor equipped with a “Bayer pattern” color filter array, each pixel will only measure either the red, green or blue channel. In order to obtain a color or multi-spectral image from the output of such a sensor, it is necessary to somehow estimate the value of the missing channels at each pixel. This process is called demosaicing.
In this document, reference to a “color filter array” may refer to the data, consisting of one measurement per pixel, that are used as input to a demosaicing algorithm. A color filter array may be stored in memory or read directly from the output of an imaging sensor.
In this document, a reference to “A” may refer to a color filter array, and references to “A(i,j)” may refer to the value, or measurement, of pixel (i,j) at column i and row j in color filter array A.
Many algorithms exist for demosaicing a color image. A common method of demosaicing the output of a Bayer pattern imaging sensor consists of copying the available channel to the output image, and interpolating the missing channels from neighboring pixels where those channels are available. The choice and design of an interpolation method has been the subject of much research.
Since demosaicing involves determining values that are not observed, the “true” value will usually differ from the estimated value. In some cases, this discrepancy may introduce undesirable visible artifacts.
Demosaicing a color filter array A may result in an image I. In this document, demosaicing may be considered an operation that performs the following:
Demosaicing will now be described. For the sake of example, it may be assumed without loss of generality that channel 0 is red, channel 1 is green, and channel 2 is blue. It may also be assumed that location (i,j) is not on the border of the array. If i and j are even numbers, the following may be defined:
I(i,j,0)=A(i,j) Eq. 1
I(i,j,1)=(A(i−1,j)+A(i+1,j)+A(i,j−1)+A(i,j+1))/4 Eq. 2
I(i,j,2)=(A(i−1,j−1)+A(i−1j+1)+A(i+1,j−1)+A(i+1,j+1))/4 Eq. 3
In these definitions, it is clear that I(i,j,0) is a measurement of red; that I(i,j,1) is computed from measurements of green exclusively; and that I(i,j,2) is computed from measurements of blue exclusively.
If i is even and j is odd, the following may be defined:
I(i,j,0)=(A(i,j−1)+A(i,j+1))/2 Eq. 4
I(i,j,1)=A(i,j) Eq. 5
I(i,j,2)=(A(i−1,j)+A(i+1,j))/2 Eq. 6
If i is odd and j is even, the following may be defined:
I(i,j,0)=(A(i−1j)+A(i+1,j))/2 Eq. 7
I(i,j,1)=A(i,j) Eq. 8
I(i,j,2)=(A(i,j−1)+A(i,j+1))/2 Eq. 9
If i is odd and j is odd, the following may be defined:
I(i,j,0)=(A(i−1,j−1)+A(i−1,j+1)+A(i+1,j−1)+A(i+1,j+1))/4 Eq. 10
I(i,j,1)=(A(i−1,j)+A(i+1,j)+A(i,j−1)+A(i,j+1))/4 Eq. 11
I(i,j,2)=A(i,j) Eq. 12
This method of demosaicing is one of the many methods that are well known to one skilled in the art of image processing or of digital photography.
It is well known to one skilled in the art of image processing or digital photography that this method of demosaicing, like all demosaicing methods, may occasionally introduce undesirable visible artifacts in the resulting image.
Interpolation
In this document, the term “interpolation” may refer to an operation that determines the value of a color channel c, at a location (i,j) in an image I, when the numbers i and j are not necessarily integer numbers. When i or j are not integer numbers, the sought value is not stored in the image. Thus, it becomes necessary to estimate the value of color channel c at location (i,j) in image I. This process may be referred to as “determining by interpolation the value I(i,j,c)”, or “interpolating I(i,j,c).”
One possible method of interpolation may consist of determining “round(i)” and “round(j)”, the nearest integer values to i and j, and then determining the value I(round(i),round(j)). This method may be referred to as “nearest neighbor interpolation.”
Bilinear interpolation of channel c of an image I at a location (i′,j′), where i′ and j′ are not necessarily integer numbers. The notation [i′] represents the largest integer number not less than i′, and likewise [j′] represents the largest integer number not less than j′. The interpolated value I(i′,j′,c) is determined from the four values I([i′],[j′],c), I([i′]+1,[j′],c), I([i′],[j′]+1,c) and I([i′]+1,[j′]+1,c), using the expression:
where “*” denotes multiplication, wi is the fractional part of i′ (i.e., wi is equal to i′−[i′]), and wj is the fractional part of j′ (i.e., wj is equal to j′−[j′]). This expression is well known to those skilled in the arts of image processing and computer graphics.
Since interpolation involves determining values that are not observed, there exist no “true” values, and different interpolation methods will often obtain different results. In some cases, interpolation may introduce undesirable visible artifacts. In addition, interpolation may exacerbate artifacts that were present in the original image.
One way of viewing an image may be to consider that the image is a function that associates, to an integer location (i,j) and a channel c, a value I(i,j,c). Since the locations (i,j) are pairs of integer numbers, it may be stated that the image (or function) is only defined on a rectangular grid of pixels. The operation of interpolation, then, may allow one to extend the image (or function) to the whole rectangular domain within the original grid of pixels. This insight will be used later in the description of the invention.
Image Warping
The operation of image warping is essentially a transformation from one image into another image as defined by some warping function. Most often a warping function computes, for each location in the new image, the location of that pixel in the original image. Since the locations in the original image do not necessarily have integer coordinates, some interpolation is needed, which may introduce some interpolation artifacts.
A common use of warping is to correct for geometric distortion. An image produced by an optical system may be subject to various forms of geometric distortion. For example, images obtained from fish-eye lenses are often so distorted that they do not appear “natural” to the viewer. In image-based metrology, it is usually necessary to warp the images so they appear to have been produced by a perspective camera. When distortion is significant, and unwanted, it is necessary to somehow undo this distortion. For this purpose, an operation called “image warping” may be used.
Image warping may be used in a wider context than the correction of optical distortion. For example, a class of warping called “homographies” may be applied to make an image appear as if it were produced by a camera with a different orientation (than the camera that produced the original image). Warping may also be applied to make it appear that the original sensor had a different layout (for example a log-polar layout) or a different resolution.
Warping an image I may result in a new image J. In this document, image warping may be considered an operation that performs the following:
At 1.1, the location (i′,j′) may take the form:
i′=P(i,j) Eq. 14
j′=Q(i,j) Eq. 15
where P and Q are functions. Common classes of functions used may include polynomials and rational functions, but other types of functions have been used, for example to correct for distortions introduced by optical systems. Although one may use the same functions P and Q for all channels, in some cases, for example to correct for color aberration, it may be desirable to use a different function for each channel.
A common example of a pair of simple polynomial functions may be:
i′=P(i,j)=i′0+(1+k0r2)(i−i0) Eq. 16
j′=Q(i,j)=j′0+(1+k0r2)(j−j0) Eq. 17
where
r2=(i−i0)2+(j−j0)2, and
At 1.2, since the locations (i′,j′) are not necessarily integer numbers, one cannot in general copy directly the values from the original image I. It is thus necessary to use interpolation. Any of the many available interpolation methods may be used, for example nearest neighbor interpolation, bilinear interpolation or bicubic interpolation.
Demosaicing and Warping Used Together
Demosaicing and image warping are often applied in succession. Most images produced by digital cameras (e.g., but not limited to, single-lens reflex (SLR), point-and-shoot, camera-phone, webcam, surveillance camera, etc.) are obtained by demosaicing a Bayer color filter array. Whenever image warping is applied to such an image, the operations of demosaicing and warping are thus successively applied in order to produce a warped image.
In order to obtain a warped image J from a color filter array A, the following is usually performed, in what may be referred to as a “classical method” for successively demosaicing a color filter array and warping an image:
In this classical method, one should note that it is necessary to store image I. This storage is needed even when the two operations are performed in quick succession, for example when one only wants to keep the warped image J, and discard the image I. For example, when a camera has a wide-angle or fish-eye lens with known optical distortion, it is possible, and often desirable, to warp the image within the camera. For example, in a SLR or point-and-shoot camera, one would keep the image J in which the lens distortion is corrected, and discard image I. In another example, in a wide-angle surveillance camera, one could transmit only image J and again discard image I. In both cases, however, it is necessary that the camera have temporary memory to store image I. Since memory is costly, the requirement of a temporary memory is a disadvantage.
Another disadvantage of the classical method is that interpolation is performed at two stages of the method: interpolation occurs first during demosaicing, where it may introduce artifacts. Then, during warping, interpolation may introduce new artifacts, in addition to exacerbating the artifacts that were introduced by demosaicing.
Compound Image Demosaicing and Warping
The methods, systems, and computer program products described herein may allow for image demosaicing and warping in such a manner as to reduce the amount of memory and processing time needed, as well as to minimize artifacts and reduce overall cost. Embodiments of these methods, systems, and computer program products are now described with reference to the figures, where like reference numbers may indicate identical or functionally similar elements. While specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. A person skilled in the relevant art will recognize that other configurations and arrangements can be used without departing from the scope of the description. It will be apparent to a person skilled in the relevant art that this can also be employed in a variety of other systems and applications than what is described herein.
In an embodiment, in order to obtain a warped image J from a color filter array A, the following may be performed, as depicted in
At 1.1 (302 in
At 1.2 (306 in
A key insight is that each color channel nevertheless does provide a grid (usually, but not always, rectangular) of measurements. As a consequence, for each color channel, one may interpolate a value for any location within the grid at which measurements of that channel are available.
A description of the determining of the color value of color channel c at (i′,j′) in color filter array A (1.2/306) follows, according to embodiments. Without loss of generality, this example will use a Bayer color filter array as previously described, and will use bilinear interpolation by way of example. One skilled in the art may readily generalize the following description to other color filter arrays and to other types of interpolation.
The following describes interpolation of the value of color channel 0 (red) at a location (i′,j′), where i′ and j′ are numbers that are not necessarily integers, according to an embodiment. In this example, [[i′]] represents the largest even integer number that is not less than i′. Similarly, [[j′]] represents the largest even integer number that is not less than j′. The following may then be defined:
where wi is equal to (i′−[[i′]])/2 and wj is equal to (j′−[[j′]])/2. In this example, it is clear, from the layout of the Bayer color filter array, that A([[i′]],[[j′]]), A([[i′]],[[j′]]+2), A([[i′]]+2,[[j′]]) and A([[i′]]+2,[j′]+2) are all measurements of red light reaching the imaging sensor. Consequently, J(i′,j′,0) is indeed a measurement of red light. One skilled in the art would readily recognize that J(i′j′,0) defined above is the bilinear interpolation at (i′,j′) of the four measurements A([[i′]],[[j′]]), A([[i′]],[[j′]]+2), A([[i′]]+2,[[j′]]) and A([[i′]]+2,[[j′]]+2).
The following describes interpolation of the value of color channel 1 (green) at a location (i′,j′), where i′ and j′ are numbers that are not necessarily integers, according to an embodiment. Since green measurements are available on a grid that is rotated by 45 degrees with respect to the grid of the color filter array, the interpolation process may be slightly more complicated than in the case of the red and blue channels.
In order to interpolate the green channel, a change in coordinates may be defined by:
u′=i′+j′ Eq. 19
v′=i′−j′ Eq. 20
The above transformation is invertible, the inverse transformation may be shown as follows:
i′=(u′+v′)/2 Eq. 21
j′=(u′−v′)/2 Eq. 22
In this example, it can be seen that the locations at which u′ and v′ are both odd integer numbers correspond to locations at which either i′ or j′ is odd, but not both. Conversely, if either i′ or j′ is odd, but not both, then u′ and v′ are both odd. Consequently, recalling the layout of the Bayer color filter array, the locations at which u′ and v′ are both odd integer numbers correspond to locations in the color filter array at which green light is measured.
In this example, [[[u′]]] represents the largest odd integer number that is not less than u′. Similarly, [[[v′]]] represents the largest odd integer number that is not less than v′. The following may then be defined:
i″=([[[u′]]]+[[[v′]]])/2 Eq. 23
j″=([[[u′]]]−[[[v′]]])/2 Eq. 24
By construction, i″ and j″ are both integer numbers, and location (i′,j′) is contained within the square defined by the four locations (i″,j″), (i″+2,j″), (i″+1,j″−1), (i″+1,j″+1).
In this example, it can also be seen that all the vertices of this square fall on a location where green light is measured in the considered Bayer color filter array.
Continuing this example, the following may be defined:
where wu is equal to (u′−[[[u′]]])/2 and wv is equal to (v′−[[[v′]]])/2. One skilled in the art would readily recognize that J(i′,j′,1) defined above is the bilinear interpolation at (i′,j′) of the four measurements A(i″,j″), A(i″+1,j″+1), A(i″+1,j″−1) and A(i″+2,j″).
The following describes interpolation of the value of color channel 2 (blue) at a location (i′,j′), where i′ and j′ are numbers that are not necessarily integers, according to an embodiment. In this example, [[[i′]]] represents the largest odd integer number that is not less than i′. Similarly, [[[j′]]] represents the largest odd integer number that is not less than j′. The following may then be defined:
where wi is equal to (i′−[[[i′]]])/2 and wj is equal to (j′−[[[j′]]])/2. In this example, it is clear, from the layout of the Bayer color filter array, that A([[[i′]]],[[[j′]]]), A([[[i′]]],[[[j′]]]+2), A([[[i′]]]+2,[[[j′]]]) and A([[[i′]]]+2,[[[j′]]]+2) are all measurements of blue light reaching the imaging sensor. Consequently, J(i′,j′,2) is indeed a measurement of blue light. One skilled in the art would readily recognize that J(i′,j′,2) defined above is the bilinear interpolation at (i′,j′) of the four measurements A([[[i′]]],[[[j′]]]). A([[[i′]]],[[[j′]]]+2), A([[[i′]]]+2,[[[j′]]]) and A([[[i′]]]+2,[[[j′]]]+2).
Embodiments of the disclosed method that simultaneously perform demosaicing and warping of a Bayer color filter array A that results in a warped image J have now been described.
Contrary to the classical method of demosaicing and warping described earlier, the disclosed method of compound demosaicing and warping may not need an additional intermediary image I. Consequently, additional memory to store image I may not be needed, and thus a system that implements the disclosed method may require less memory than a system that implements the classical method.
In addition, contrary to the classical method of demosaicing and warping, only one interpolation process takes place in the disclosed method. Therefore, exacerbation of interpolation artifacts of one interpolation process during a second interpolation process may be avoided. Since interpolation artifacts are detrimental to image quality, the disclosed method may be preferred over the classical method.
Further Sampling Examples
Another way of describing the above-discussed sampling of locations in a color filter array using bilinear interpolation will now be described. In the following examples, the color filter array used will have the classic Bayer pattern, as shown in
A main idea behind the following approach is to take into account where in the color filter array (here, a Bayer pattern) one is, and given an understanding of adjacent color pixels, apply an appropriate interpolation to sample red, green, and blue values at the sample point. In an embodiment, an interpolation may be done on a sample point located within a two-by-two quadruplet (or “quad”) of pixels. For example, for a Bayer color filter array, a sampled location may fall within one of four possible types of quads 420, 422, 424, and 426, as shown in
In an embodiment, in order to interpolate a value of the color red for sample point 532 in quad 534, horizontal interpolations of the red values at F and H (indicated at location 644 in
In this example, the upper and lower horizontal interpolations may be shown as in the following equations:
Upper=F+(H−F)*u/2 Eq. 27
Lower=N+(P−N)*u/2 Eq. 28
The vertical interpolation may then be shown as in the following equation:
r=Upper+(Lower−Upper)*v/2 Eq. 29
The resulting interpolated value of r for sample point 532 may then be shown as in the following equations:
r=F+(H−F)*u/2+(N+(P−N)*u/2−F−(H−F)*u/2)*v/2 Eq. 30
r=F+(H−F)*u/2+(N−F)*v/2+(P+F−N−H)*u*v/4 Eq. 31
In this way, a single set of interpolation calculations may be needed, as opposed to requiring a set of interpolation calculations for demosaicing and another set of interpolation calculations for warping. For any sample points found within the other three types of quads (422, 424, and 426 of
Determining the green value at a similarly situated sample point differs due to the frequency and placement of the green values in the color filter array. An example of determining the green value of a similarly situated sample point 732 in color filter array 736 may be shown in
An alternative example of determining the green value of a sample point 862 may be shown in
In the examples shown in FIGS. 8A-B/9A-B, coordinates of each sample point may be denoted as (u,v), where u may be the horizontal distance between the upper left pixel of the quad and the sample point, and v may be the vertical distance between the upper left pixel of the quad and the sample point. This is shown in the example shown in
In order to determine the green values at the sample points, a pair (u′,v′) may be defined for each of the four green examples shown in
u′=u+v+1.0 and v′=1.0+u−v Eqs. 32
(For example, in the example of this quad configuration shown in
u′=u+v−1.0 and v′=1.0+u−v Eqs. 33
In the third of the four examples, where the sample point is below a falling diagonal of green pixels (Example C):
u′=u+v and v′=2.0+u−v Eqs. 34
In the last of the four examples, where the sample point is above a falling diagonal of green pixels (Example D):
u′=u+v and v′=u−v Eqs. 35
The green value interpolated from the four green source pixels (designated as N, S, E, and W) of any of the four examples may then be calculated as:
g=W+(S−W)*u′/2.0+(N−W)*v′/2.0+(W−N−S+E)*u′*v′/4.0 Eq. 36
As discussed above, and as would be recognized by those of skill in the relevant art, by using the above disclosed methods for the demosaicing and warping of images, multiplicity of interpolation operation sets may be avoided, saving memory, reducing processing time, minimizing artifacts, and reducing cost.
Exemplary Systems
One or more features disclosed herein may be implemented in hardware, software, firmware, and combinations thereof, including discrete and integrated circuit logic, application specific integrated circuit (ASIC) logic, and microcontrollers, and may be implemented as part of a domain-specific integrated circuit package, or a combination of integrated circuit packages. The terms software and firmware, as used herein, refer to a computer program product including at least one computer readable medium having computer program logic, such as computer-executable instructions, stored therein to cause a computer system to perform one or more features and/or combinations of features disclosed herein. The computer readable medium may be transitory or non-transitory. An example of a transitory computer readable medium may be a digital signal transmitted over a radio frequency or over an electrical conductor, through a local or wide area network, or through a network such as the Internet. An example of a non-transitory computer readable medium may be a compact disk, a flash memory, or other data storage device.
In an embodiment, some or all of the processing described herein may be implemented as software or firmware. Such a software or firmware embodiment may be illustrated in the context of a computing system 1302 in
The systems, methods, and computer program products described above may be a part of a larger information system.
In embodiments, system 1500 comprises a platform 1502 coupled to a display 1520. Platform 1502 may receive content from a content device such as content services device(s) 1530 or content delivery device(s) 1540 or other similar content sources. A navigation controller 1550 comprising one or more navigation features may be used to interact with, for example, platform 1502 and/or display 1520. Each of these components is described in more detail below.
In embodiments, platform 1502 may comprise any combination of a chipset 1505, processor 1510, memory 1512, storage 1514, graphics subsystem 1515, applications 1516 and/or radio 1518. Chipset 1505 may provide intercommunication among processor 1510, memory 1512, storage 1514, graphics subsystem 1515, applications 1516 and/or radio 1518. For example, chipset 1505 may include a storage adapter (not depicted) capable of providing intercommunication with storage 1514.
Processor 1510 may be implemented as Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors, x86 instruction set compatible processors, multi-core, or any other microprocessor or central processing unit (CPU). In embodiments, processor 1510 may comprise dual-core processor(s), dual-core mobile processor(s), and so forth.
Memory 1512 may be implemented as a volatile memory device such as, but not limited to, a Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), or Static RAM (SRAM).
Storage 1514 may be implemented as a non-volatile storage device such as, but not limited to, a magnetic disk drive, optical disk drive, tape drive, an internal storage device, an attached storage device, flash memory, battery backed-up SDRAM (synchronous DRAM), and/or a network accessible storage device. In embodiments, storage 1514 may comprise technology to increase the storage performance enhanced protection for valuable digital media when multiple hard drives are included, for example.
Graphics subsystem 1515 may perform processing of images such as still or video for display. Graphics subsystem 1515 may be a graphics processing unit (GPU) or a visual processing unit (VPU), for example. An analog or digital interface may be used to communicatively couple graphics subsystem 1515 and display 1520. For example, the interface may be any of a High-Definition Multimedia Interface, DisplayPort, wireless HDMI, and/or wireless HD compliant techniques. Graphics subsystem 1515 could be integrated into processor 1510 or chipset 1505. Graphics subsystem 1515 could be a stand-alone card communicatively coupled to chipset 1505.
The graphics and/or video processing techniques described herein may be implemented in various hardware architectures. For example, graphics and/or video functionality may be integrated within a chipset. Alternatively, a discrete graphics and/or video processor may be used. As still another embodiment, the graphics and/or video functions may be implemented by a general purpose processor, including a multi-core processor. In a further embodiment, the functions may be implemented in a consumer electronics device.
Radio 1518 may include one or more radios capable of transmitting and receiving signals using various suitable wireless communications techniques. Such techniques may involve communications across one or more wireless networks. Exemplary wireless networks include (but are not limited to) wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. In communicating across such networks, radio 1518 may operate in accordance with one or more applicable standards in any version.
In embodiments, display 1520 may comprise any television type monitor or display. Display 1520 may comprise, for example, a computer display screen, touch screen display, video monitor, television-like device, and/or a television. Display 1520 may be digital and/or analog. In embodiments, display 1520 may be a holographic display. Also, display 1520 may be a transparent surface that may receive a visual projection. Such projections may convey various forms of information, images, and/or objects. For example, such projections may be a visual overlay for a mobile augmented reality (MAR) application. Under the control of one or more software applications 1516, platform 1502 may display user interface 1522 on display 1520.
In embodiments, content services device(s) 1530 may be hosted by any national, international and/or independent service and thus accessible to platform 1502 via the Internet, for example. Content services device(s) 1530 may be coupled to platform 1502 and/or to display 1520. Platform 1502 and/or content services device(s) 1530 may be coupled to a network 1560 to communicate (e.g., send and/or receive) media information to and from network 1560. Content delivery device(s) 1540 also may be coupled to platform 1502 and/or to display 1520.
In embodiments, content services device(s) 1530 may comprise a cable television box, personal computer, network, telephone, Internet enabled devices or appliance capable of delivering digital information and/or content, and any other similar device capable of unidirectionally or bidirectionally communicating content between content providers and platform 1502 and/display 1520, via network 1560 or directly. It will be appreciated that the content may be communicated unidirectionally and/or bidirectionally to and from any one of the components in system 1500 and a content provider via network 1560. Examples of content may include any media information including, for example, video, music, medical and gaming information, and so forth.
Content services device(s) 1530 receives content such as cable television programming including media information, digital information, and/or other content. Examples of content providers may include any cable or satellite television or radio or Internet content providers. The provided examples are not meant to limit embodiments of the invention.
In embodiments, platform 1502 may receive control signals from navigation controller 1550 having one or more navigation features. The navigation features of controller 1550 may be used to interact with user interface 1522, for example. In embodiments, navigation controller 1550 may be a pointing device that may be a computer hardware component (specifically human interface device) that allows a user to input spatial (e.g., continuous and multi-dimensional) data into a computer. Many systems such as graphical user interfaces (GUI), and televisions and monitors allow the user to control and provide data to the computer or television using physical gestures.
Movements of the navigation features of controller 1550 may be echoed on a display (e.g., display 1520) by movements of a pointer, cursor, focus ring, or other visual indicators displayed on the display. For example, under the control of software applications 1516, the navigation features located on navigation controller 1550 may be mapped to virtual navigation features displayed on user interface 1522, for example. In embodiments, controller 1550 may not be a separate component but integrated into platform 1502 and/or display 1520. Embodiments, however, are not limited to the elements or in the context shown or described herein.
In embodiments, drivers (not shown) may comprise technology to enable users to instantly turn on and off platform 1502 like a television with the touch of a button after initial boot-up, when enabled, for example. Program logic may allow platform 1502 to stream content to media adaptors or other content services device(s) 1530 or content delivery device(s) 1540 when the platform is turned “off.” In addition, chipset 1505 may comprise hardware and/or software support for 5.1 surround sound audio and/or high definition 7.1 surround sound audio, for example. Drivers may include a graphics driver for integrated graphics platforms. In embodiments, the graphics driver may comprise a peripheral component interconnect (PCI) Express graphics card.
In various embodiments, any one or more of the components shown in system 1500 may be integrated. For example, platform 1502 and content services device(s) 1530 may be integrated, or platform 1502 and content delivery device(s) 1540 may be integrated, or platform 1502, content services device(s) 1530, and content delivery device(s) 1540 may be integrated, for example. In various embodiments, platform 1502 and display 1520 may be an integrated unit. Display 1520 and content service device(s) 1530 may be integrated, or display 1520 and content delivery device(s) 1540 may be integrated, for example. These examples are not meant to limit the invention.
In various embodiments, system 1500 may be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 1500 may include components and interfaces suitable for communicating over a wireless shared media, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, and so forth. An example of wireless shared media may include portions of a wireless spectrum, such as the RF spectrum and so forth. When implemented as a wired system, system 1500 may include components and interfaces suitable for communicating over wired communications media, such as input/output (I/O) adapters, physical connectors to connect the I/O adapter with a corresponding wired communications medium, a network interface card (NIC), disc controller, video controller, audio controller, and so forth. Examples of wired communications media may include a wire, cable, metal leads, printed circuit board (PCB), backplane, switch fabric, semiconductor material, twisted-pair wire, co-axial cable, fiber optics, and so forth.
Platform 1502 may establish one or more logical or physical channels to communicate information. The information may include media information and control information. Media information may refer to any data representing content meant for a user. Examples of content may include, for example, data from a voice conversation, videoconference, streaming video, electronic mail (“email”) message, voice mail message, alphanumeric symbols, graphics, image, video, text and so forth. Data from a voice conversation may be, for example, speech information, silence periods, background noise, comfort noise, tones and so forth. Control information may refer to any data representing commands, instructions or control words meant for an automated system. For example, control information may be used to route media information through a system, or instruct a node to process the media information in a predetermined manner. The embodiments, however, are not limited to the elements or in the context shown or described in
As described above, system 1500 may be embodied in varying physical styles or form factors.
As described above, examples of a mobile computing device may include a personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet or smart television), mobile internet device (MID), messaging device, data communication device, and so forth.
Examples of a mobile computing device also may include computers that are arranged to be worn by a person, such as a wrist computer, finger computer, ring computer, eyeglass computer, belt-clip computer, arm-band computer, shoe computers, clothing computers, and other wearable computers. In embodiments, for example, a mobile computing device may be implemented as a smart phone capable of executing computer applications, as well as voice communications and/or data communications. Although some embodiments may be described with a mobile computing device implemented as a smart phone by way of example, it may be appreciated that other embodiments may be implemented using other wireless mobile computing devices as well. The embodiments are not limited in this context.
As shown in
Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.
One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium which represents various logic within the processor, which when read by a machine causes the machine to fabricate logic to perform the techniques described herein. Such representations, known as “IP cores” may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that actually make the logic or processor.
The following examples pertain to further embodiments.
Example 1 may include an image processing system, comprising a first circuit logic unit configured to, for each pixel location (i,j) and for each color channel in a color image, determine a warped location (i′,j′) in a color filter array; and a second circuit logic unit configured to, for each pixel location (i,j) and for each color channel in the color image, determine a color value of the color channel at location (i′,j′) in the color filter array.
In Example 2, the subject matter of Example 1 may optionally include a third circuit logic unit configured to, for each pixel location (i,j) and for each color channel in the color image, store the determined color value.
Example 3 may include the subject matter of any one of Examples 1-2, wherein the determining the color value by the second circuit logic unit comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array.
Example 4 may include the subject matter of Example 3, wherein the interpolating the color value comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array by bilinear interpolation.
Example 5 may include the subject matter of Example 3, wherein the interpolating the color value comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array by bicubic interpolation.
Example 6 may include the subject matter of Example 3, wherein the interpolating the color value comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array by nearest neighbor interpolation.
Example 7 may include the subject matter of any one of Examples 1-6, wherein the color filter array comprises a Bayer pattern.
Example 8 may include a computer program product for image processing that includes at least one non-transitory computer readable medium having computer program logic stored therein, the computer program logic including logic to cause a processor to, for each pixel location (i,j) and for each color channel in a color image, determine a warped location (i′,j′) in a color filter array; and logic to cause the processor to, for each pixel location (i,j) and for each color channel in the color image, determine a color value of the color channel at location (i′,j′) in the color filter array.
Example 9 may include the subject matter of Example 8, wherein the computer program logic further includes logic to cause the processor to, for each pixel location (i,j) and for each color channel in the color image, store the determined color value.
Example 10 may include the subject matter of any one of Examples 8-9, wherein the logic to cause the processor to determine the color value comprises logic to cause the processor to interpolate the color value of the color channel at location (i′,j′) in the color filter array.
Example 11 may include the subject matter of Example 10, wherein the logic to cause the processor to interpolate the color value comprises logic to cause the processor to interpolate the color value of the color channel at location (i′,j′) in the color filter array by bilinear interpolation.
Example 12 may include the subject matter of Example 10, wherein the logic to cause the processor to interpolate the color value comprises logic to cause the processor to interpolate the color value of the color channel at location (i′,j′) in the color filter array by bicubic interpolation.
Example 13 may include the subject matter of Example 10, wherein the logic to cause the processor to interpolate the color value comprises logic to cause the processor to interpolate the color value of the color channel at location (i′,j′) in the color filter array by nearest neighbor interpolation.
Example 14 may include the subject matter of any one of Examples 8-13, wherein the color filter array comprises a Bayer pattern.
Example 15 may include an apparatus for image processing, comprising means for, for each pixel location (i,j) and for each color channel in a color image, determining a warped location (i′,j′) in a color filter array; and means for, for each pixel location (i,j) and for each color channel in the color image, determining, by the image processing device, a color value in the color channel at location (i′,j′) of the color filter array.
In Example 16, the subject matter of Example 15 may optionally include means for, for each pixel location (i,j) and for each color channel in the color image, storing the determined color value.
Example 17 includes the subject matter of any one of Examples 15-16, wherein the means for determining the color value comprises means for interpolating the color value of the color channel at location (i′,j′) in the color filter array.
Example 18 includes the subject matter of Example 17, wherein the means for interpolating comprises means for interpolating the color value of the color channel at location (i′,j′) in the color filter array by bilinear interpolation.
Example 19 includes the subject matter of Example 17, wherein the means for interpolating comprises means for interpolating the color value of the color channel at location (i′,j′) in the color filter array by bicubic interpolation.
Example 20 includes the subject matter of Example 17, wherein the means for interpolating comprises means for interpolating the color value of the color channel at location (i′,j′) in the color filter array by nearest neighbor interpolation.
Example 21 includes the subject matter of any one of Examples 15-20, wherein the color filter array comprises a Bayer pattern.
Example 22 may include a method of image processing, comprising, for each pixel location (i,j) and for each color channel in a color image, determining, by an image processing device, a warped location (i′,j′) in a color filter array, and determining, by the image processing device, a color value in the color channel at location (i′,j′) of the color filter array.
In Example 23, the subject matter of Example 22 may optionally include, for each pixel location (i,j) and for each color channel in a color image, storing the determined color value.
Example 24 may include the subject matter of any one of Examples 22-23, wherein the determining the color value comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array.
Example 25 may include the subject matter of Example 24, wherein the interpolating the color value comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array by bilinear interpolation.
Example 26 may include the subject matter of Example 24, wherein the interpolating the color value comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array by bicubic interpolation.
Example 27 may include the subject matter of Example 24, wherein the interpolating the color value comprises interpolating the color value of the color channel at location (i′,j′) in the color filter array by nearest neighbor interpolation.
Example 28 may include the subject matter of any one of Examples 22-27, wherein the color filter array comprises a Bayer pattern.
Example 29 may include at least one machine readable medium comprising a plurality of instructions that in response to being executed on a computing device, cause the computing device to carry out a method according to any one of Examples 22-28.
Example 30 may include a computer system to perform the method of any one of Examples 22-28.
Example 31 may include an apparatus configured to perform the method of any one of Examples 22-28.
Example 32 may include a machine to perform the method of any one of Examples 22-28.
Example 33 may include an apparatus comprising means for performing the method of any one of Examples 22-28.
Example 34 may include a computing device comprising a chipset according to any one of Examples 22-28 and memory to process images.
Methods and systems are disclosed herein with the aid of functional building blocks illustrating the functions, features, and relationships thereof. At least some of the boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries may be defined so long as the specified functions and relationships thereof are appropriately performed.
While various embodiments are disclosed herein, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various changes in form and detail may be made therein without departing from the scope of the methods and systems disclosed herein. Thus, the breadth and scope of the claims should not be limited by any of the exemplary embodiments disclosed herein.
As may be used in this application and in the claims, a list of items joined by the term “one or more of” can mean any combination of the listed terms. For example, the phrases “one or more of A, B or C” can mean A; B; C; A and B; A and C; B and C; or A, B and C.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2013/032050 | 3/15/2013 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2013/169384 | 11/14/2013 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7492396 | Kim et al. | Feb 2009 | B2 |
| 20030118251 | Hamza | Jun 2003 | A1 |
| 20070058179 | Kuroki et al. | Mar 2007 | A1 |
| 20080122946 | Sung et al. | May 2008 | A1 |
| 20100182466 | Chang et al. | Jul 2010 | A1 |
| 20110013056 | Myhrvold | Jan 2011 | A1 |
| 20110090381 | Côté et al. | Apr 2011 | A1 |
| Number | Date | Country |
|---|---|---|
| 2175656 | Apr 2010 | EP |
| 2013169384 | Nov 2013 | WO |
| Entry |
|---|
| International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2013/032050, mailed on Jun. 28, 2013, 12 pages. |
| International Preliminary Report on Patentability and Written Opinion received for PCT Patent Application No. PCT/US2013/032050, mailed on Nov. 20, 2014, 9 pages. |
| EP Search Report mailed Oct. 16, 2015, for EP Patent Application No. 13788458.1-1902/2847998. |
| Number | Date | Country | |
|---|---|---|---|
| 20140056513 A1 | Feb 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61645995 | May 2012 | US |
