The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. These and/or other aspects and advantages of the invention will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the present invention by referring to the figures.
Advantages and features of the present invention and methods of accomplishing the same may be understood more readily by reference to the following detailed description of preferred embodiments and the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of the invention to those skilled in the art, and the present invention will only be defined by the appended claims. Like reference numerals refer to like elements throughout the specification.
The present invention will now be described more fully with reference to the accompanying drawings, in which preferred embodiments of the invention are shown.
The originals images produced by the camera module 200 may have same luminance or different luminances. The plurality of original images produced by the camera module 200 are supplied to the image processing module 800 which will be described later. The camera module 200 will be described in detail with reference to
The image processing module 800 processes the plurality of original images supplied from the camera module 200 to produce final images. The image processing module 800 will be described in detail with reference to
The display module 900 displays the final images produced by the image processing module 800. Even though such a display module 900 is embodied as a flat panel display or a touch screen, the display module 900 is not limited thereto.
The lens module 300 may include a plurality of lens 310, 320, 330, and 340 condensing incident light. However, the number of lenses is not limited, and the lenses are arranged on the same plane in various forms. For example, a plurality of lens 310, 320, 330, and 340 are arranged in one line in a vertical direction or a horizontal direction, or arranged in a matrix. Hereinafter, for convenience sake, it is described an example in which four lenses are arranged in 2×2 matrix.
The image sensor module 500 senses light condensed by the lenses to produce a plurality of original images. In order to perform the above operation, the image sensor module 500 includes a photo sensitive module 51, a decoder 53, a converting module 54, and an exposure control module 52.
The photo sensitive module 51 senses light condensed by the lens module 300 to convert the light into an electrical signal and then convert the electrical signal into a voltage signal. The photo sensitive module 51 will be described in detail with reference to
Referring to
A metal wiring layer 590 is formed above the light receiving elements 560 to form a circuit. An IMD (inter-metal dielectric) 580a is formed between the light receiving elements 560 and the metal wiring layer 590. The metal wiring layer 590 may be formed so as not to block a path of light incident onto the light receiving elements 560. In
A planarizing layer 585a and a color filter layer 575 are formed on the IMD 580b in this order. The color filter layer 575 includes a red color filter, a green color filter, and a blue color filter, and the individual color filters perform to filter on the light condensed by the plurality of lenses 310, 320, 330, and 340 to represent the original colors. Each of the color filters may be formed in various patterns, and an example that the red color filter, the green color filter, and the blue color filter are formed in a Bayer pattern will be described.
Above the color filter layer 575, a planarizing layer 585b planarizing the color filter layer and an ML (micro lens) 595 to increase the photo sensitivity of the light receiving element 560 are formed in this order. Generally, the light receiving element 560 does not occupy the entire region of the unit pixel, but occupies only a part of the unit pixel. Therefore, a fill factor that indicates an area of the pixel occupied by the light receiving element 560 is 1 or less, which indicates that a part of incident is lost. In contrast, when the micro lens 595 is formed on the uppermost portion of the IMD 589b, since the incident light is condensed by the micro lens 595, it is possible to increase the amount of light converged to the light receiving element 560.
A plurality of the pixels configured as mentioned above form sensing area 510, 520, 530, and 540. In this case, the sensing area 510, 520, 530, and 540 is divided into plurality of sub sensing areas as shown in
The decoder 53 reads the voltage signal indicated by a pixel in a predetermined sub sensing area. For its sake, the decoder 53 includes a row decoder 53_1 to read information concerning pixels disposed in a horizontal direction, and a column decoder 53_2 to read information concerning pixels disposed in a vertical direction. Such a row decoder 53_1 and a column decoder 53_2 are provided for every sub sensing area, or assembled as a hardware. The voltage signals of the pixels are amplified by an amplifier 53_3 and then supplied to the converting module 54.
The converting module 54 converts the amplified voltage signal into a digital signal. The converting module 54 may be provided for every sub sensing area the same as the decoder, or may be assembled as a hardware.
The exposure control module 52 adjusts an exposure condition of each sub sensing area. Examples of exposure conditions include an exposure starting time, an exposure time, and a gain. The exposure time refers to a time when the sub sensing area is exposed to external light to accumulate electric charges. If the exposure times of the sub sensing areas are equal to each other, the same amount of electric charges is accumulated.
According to an exemplary embodiment of this invention, the exposure control module 52 sets such that the gains and the exposure times of sub sensing areas are equal to each other and the exposure starting times of the respective sub sensing area are different from each other. For example, as shown in
According to an embodiment of the present invention, the exposure control module 52 may set such that the exposure starting times are the equal to each other, but the exposure time of the sub sensing area are different from each other. For example, as shown in
According to another example of the invention, the exposure control module 52 may set such that the exposure starting times and the exposure time of the sub sensing areas are equal to each other, and the gains are different from each other. As same as the example that the exposure times of the sub sensing areas are different from each other, in this example, it is possible to simultaneously obtain a plurality of images whose luminances are different from each other through one capturing process. Specifically, when a gain of a predetermined sub sensing area is controlled, the sensitivity of the corresponding sub sensing area increases in proposition to the gain. When the sensitivity of the sub sensing area is higher, more photons are emitted with the same intensity. Therefore, even though the other exposure conditions of each of the sub sensing areas are equal, when the gains are different from each other, the sensitivities of the sub sensing areas becomes different from each other. As a result, it is possible to simultaneously obtain a plurality of original images having different luminances due to the difference in the sensitivities of the sub sensing areas.
In addition to the above components, the image sensor module 500 may selectively include an infrared ray blocking filter (not shown) to block an infrared ray. The photo sensitive module 51 is sensitive to the infrared ray as well as a visible ray. Therefore, when using the infrared ray blocking filter, the infrared ray that reaches the photo sensitive module 51 is blocked, thereby preventing damage on information of images in the visible ray area.
Next, referring to
The plurality of original images from the camera module 200 is input to the input module 810. In detail, to the input module 810, the first original image obtained by the first sub sensing area 510, the second original image obtained by the second sub sensing area 520, the third original image obtained by the third sub sensing area 530, and the fourth original image obtained by the fourth sub sensing area 540 are input. The plurality of input original images functions to provide information concerning colors and luminances that are used to produce final images using the final images producing module 830 which will be described later.
The intermediate image producing module 820 performs a de-mosaic process on the plurality of input original images to produce a plurality of intermediate images. In here, the de-mosaic process refers a process that restores color information that is not included in a predetermined pixel using color information of the pixel and adjacent pixels.
The final images producing module 830 produces final images on the basis of the pixel information included in each of the pixels of the plurality of intermediate images. In this case, the pixel information may include color information, luminance information of a predetermined pixel.
The final images producing module 830 multiplies the pixel information of the pixel of each of the intermediate images by a predetermined weight. In this case, the predetermined weights that are multiplied to the pixel information may be the same, or varied depending on the luminances of the pixels. Thereafter, the final images producing module 830 produces final images on the basis of the pixel information of a pixel selected from pixels of the intermediate images that are disposed in the same position. For example, among the pixels of the intermediate images which are in the same position, the final images producing module 830 selects one pixel that has pixel information within a predetermined threshold value, and produces the final images on the basis of the pixel information of the selected pixel. Further, the final images producing module 830 may produce the final images on the basis of an average value of the pixel information of pixels of the intermediate images in the same position.
The image processing module 800 may further include a filter module 840. When the exposure control module 52 sets the gains of the sub sensing areas to be different from each other, a plurality of original images having different luminances can be obtained. However, the noise increases in proportion to the gain. Therefore, the noises are necessarily removed from the plurality of original images having different luminances. The filter module 840 removes the noises included in the plurality of original images by filtering the plurality of original images having different luminances. A higher weight may be applied to an original image that is obtained by a sub sensing area to which a higher gain is applied according to an aspect of the present invention.
Next, referring to
At first, the exposure control module 52 sets such that the gains and the exposure times of sub sensing areas are equal to each other and the exposure starting times of the respective sub sensing area are different from each other (S81). For example, the exposure control module 52 sets the exposure starting times of the sub sensing areas so that the first sub sensing area 510, the second sub sensing area 520, the third sub sensing area 530, and the fourth sub sensing area 540 are sequentially exposed in this order. Specifically, the exposure control module 52 sets the exposure starting time of the first sub sensing area 510 as A, the exposure starting time of the second sub sensing area 520 as B, the exposure starting time of the third sub sensing area 530 as C, and the exposure starting time of the fourth sub sensing area 540 as D, as shown in
When taking a moving object 10 in this state, light reflected from the object 10 is condensed by four lenses 310, 320, 330, and 340 (S82).
The light condensed by the lenses 310, 320, 330, and 340 is converged into the corresponding sub sensing areas 510, 520, 530, and 540.
In this case, the sub sensing areas are sequentially exposed according to the previously set exposure starting times. That is, as shown in
Thereafter, the electrical signals generated by light converged into the sub sensing areas are converted into the voltage signals, and amplified, and converted into digital signals to output sequentially (S83). In this case, the resolution of an original images captured by a predetermined sub sensing area is 4×4, which is a quarter of a resolution of the sensing area 510, 520, 530, and 540. The plurality of original images 511, 512, 513, and 514 captured by the sub sensing areas are supplied to the image processing module 800.
The input module 810 of the image processing module 800 is supplied with the plurality of original images 511, 512, 513, and 514, and then supplies the original images to the intermediate image producing module 820.
The intermediate image producing module 820 interpolates the plurality of input original images 511, 521, 531, and 541 to produce a plurality of intermediate images 512, 522, 532, and 542 (S84).
When the plurality of intermediate images 512, 522, 532, and 542 are produced, the final images producing module 830 rearranges the plurality of intermediate images 512, 522, 532, and 542 according to the captured order of the original images 511, 521, 531, and 541 to produce final images 700 (S85).
The final images 700 produced by the final image producing module 830 are displayed on the display module 900 (S86). In this case, since the displayed final images 700 have a higher frame rate than that of the related art, it is possible to naturally represent the motion of the object 100. Specifically, when each of the sub sensing areas captures 6 images per second, the final images 700 can represents 24 images per second. Therefore, the motion of the object 100 can be more naturally represented as compared with the related art.
Next, referring to
The exposure control module 52 sets the sub sensing areas such that the gains and the exposure starting times are equal to each other, and the exposure times are different from each other (S91). For example, as shown in
When taking a picture in this state, light reflected from the object is condensed by four lenses (S92), and then converged into the corresponding sub sensing areas.
In this case, all of the sub sensing areas start to be simultaneously exposed at time A. Thereafter, the exposure of the first sub sensing area 510, the second sub sensing area 520, the third sub sensing area 530, and the fourth sub sensing area 540 is completed in this order. When the exposure of a predetermined sub sensing area is completed, the electrical signals generated by the light converged into the corresponding sub sensing area are converted into voltage signals, then amplified, and converted into digital signals to output sequentially (S93).
After the exposure of the fourth sub sensing area 540 is completed, when comparing the original images captured through the sub sensing areas with each other, it is known that the luminance becomes higher as the exposure time of the sub sensing area becomes longer. That is, the luminances become higher in the order of the first original image 513, the second original image 523, the third original image 533, and the fourth original image 543.
Therefore, when a plurality of original images 513, 523, 533, and 543 having different luminances are captured, the intermediate image producing module 820 de-mosaics the plurality of original images 513, 523, 533, and 543 having different luminances to produces a plurality of intermediate images 514, 524, 534, and 544 having different luminances (S94).
Thereafter, the final images producing module 830 produces final images on the basis of the pixel information included in each of the pixels of the plurality of intermediate images 514, 524, 534, and 544 having different luminances (S95).
The final images producing module 830 multiplies the pixel information of the pixel of each of the intermediate images 514, 524, 534, and 544 by a predetermined weight. In this case, the weights that are multiplied to the pixel information may be the same, or varied depending on the luminances of the pixels.
Thereafter, the final images producing module 830 produces final images 710 on the basis of the pixel information of a pixel selected from pixels of the intermediate images 514, 524, 534, and 544 that are disposed in the same position. For example, among the pixels of the intermediate images 514, 524, 534, and 544 that are in the same position, the final images producing module 830 selects one pixel that has pixel information within a predetermined threshold value, and produces the final images 710 on the basis of the pixel information of the selected pixel. Further, the final images producing module 830 may produce the final images 710 on the basis of an average value of the pixel information of pixels of the intermediate images 514, 524, 534, and 544 in the same position.
According to the above method, since it is possible to acquire a plurality of original images 513, 523, 533, and 543 by only one imaging process, it is possible to realize a clear image under a condition in which illuminance difference is large. That is, it is possible to realize WDR function.
Next, referring to
The exposure control module 52 sets the exposure starting time and the exposure time of all of the sub sensing areas to be equal to each other. In this case, the exposure time is preferably set to a time that can prevent the motion blur due to the hand shaking of a user, for example 1/30 second or less.
The exposure control module 52 sets the sub sensing areas to have different gains (S11). For example, gains of the first sub sensing area 510, the second sub sensing area 520, the third sub sensing area 530, and the fourth sub sensing area 540 are 1, 2, 3, and 4, respectively.
When taking a picture in this state, light reflected from the object 10 is condensed by four lenses (S12), and then converged into the corresponding sub sensing areas. In detail, the light condensed by the first lens 310 is converged into the first sub sensing area 510, and the light condensed by the second lens 320 is converged into the second sub sensing area 520.
Thereafter, the sub sensing areas are simultaneously exposed for a predetermined time, for example, 1/30 second.
After the exposure of a predetermined sub sensing area is completed, the electrical signals generated by the light converged into the corresponding sub sensing area are converted into voltage signals, then amplified, and converted into digital signals to output. As a result, it is possible to obtain a plurality of original images having different luminances (S13).
Therefore, when comparing the original images captured through the sub sensing areas with each other, it is known that the luminance becomes higher as the exposure time of the sub sensing area becomes longer, as shown in
The plurality of original images 515, 525, 535, and 545 having different luminances are captured (S13), the intermediate image producing module 820 interpolates the plurality of input original images 515, 525, 535, and 545 to produce a plurality of intermediate images 516, 526, 536, and 546 having different luminances.
Thereafter, the filter module 840 filters the plurality of intermediate images 516, 526, 536, and 546. In this case, filter module 840 preferably filters an intermediate image captured by a sub sensing area having high gain by applying a high weight. It is because the noises increases in proportion to the gain set in the corresponding sub sensing area.
Thereafter, the final image producing module 830 produces final images 720 on the basis of pixel information of each of the pixels of the plurality of filtered intermediate images 517, 527, 537 and 547. For this, the final images producing module 830 multiplies the pixel information of the pixel of each of the intermediate images 517, 527, 537 and 547 by a predetermined weight. In this case, the weights that are multiplied to the pixel information may be the same, or varied depending on the luminances of the pixels. Thereafter, the final images producing module 830 produces final images 720 on the basis of the pixel information of a pixel selected from pixels of the intermediate images 517, 527, 537 and 547 that are disposed in the same position. For example, among the pixels of the intermediate images 517, 527, 537 and 547 that are in the same position, the final images producing module 830 selects one pixel that has pixel information within a predetermined threshold value, and produces the final images 720 on the basis of the pixel information of the selected pixel. Further, the final images producing module 830 may produce the final images 720 on the basis of an average value of the pixel information of pixels of the intermediate images 517, 527, 537 and 547 in the same position.
According to the above method, it is possible to acquire a plurality of original images 515, 525, 535, and 545 by only one imaging process by controlling the gains of the sub sensing areas to be different from each other. As a result, it is possible to realize a clear image under a condition in which illuminance difference is large.
Although the present invention has been described in connection with the exemplary embodiments of the present invention, it will be apparent to those skilled in the art that various modifications and changes may be made thereto without departing from the scope and spirit of the invention. Therefore, it should be understood that the above embodiments are not limitative, but illustrative in all aspects.
According to the image display method and apparatus to obtain a high quality image, the following effects can be obtained.
First, since the high speed imaging can be performed with using the high sensitivity sensor, it is possible to obtain a high quality moving image.
Second, by controlling exposures conditions for a plurality of image sensing area, it is possible to simultaneously acquire a plurality of images having different luminance.
Third, since it is possible to simultaneously acquire a plurality of images having different luminance, it is possible to prevent the blurring or the false color during the image processing operations, and to realize the clear image under a condition in which illuminance difference is large.
Although a few embodiments of the present invention have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.
Number | Date | Country | Kind |
---|---|---|---|
10-2006-0078872 | Aug 2006 | KR | national |