The present invention is related with a cellular phone camera with three-dimensional imaging function using a micromirror array lens.
There are many three-dimensional imaging devices proposed and developed. Most common three-dimensional imaging method illustrated in
Other method with one camera was developed using “depth from focus” criterion and fast response variable focal length lens which can be found in T. Kaneko et al., 2000, “Quick Response Dynamic Focusing Lens using Multi-Layered Piezoelectric Bimorph Actuator”, Proceeding of SPIE Vol. 4075: 24-31. In this system, a variable focal length lens comprised of two thin glass diaphragms and transparent working fluid of high refractive index inside. Multi-layered piezoelectric bimorph actuator adjusts the shape of the diaphragms for varying the focal length of the lens. “Depth from focus” method achieves an all-in-focus image and depth information from multiple two-dimensional images with different focal planes. Since the variable focal length lens has a slow time response (150 Hz), it cannot accomplish real-time three-dimensional image. Also in a dimension of the size, multi-layered piezoelectric actuator type has too large a size (14 mm×14 mm×16 mm) to fit into the pocket-sized applications like cellular phones. The lens has a small focal plane variation in the range of a few millimeters, which limits the possible range of depth and the depth resolution of the three-dimensional image. To get a good quality three-dimensional image, a high speed, compact variable focal length lens with large focal length variation should be introduced.
Three-dimensional imaging device with a high quality and real-time image in a cellular phone needs an imaging system performed with compact device and a high speed, compact variable focal length lens with large focal length variation.
The objective of the present invention is to provide a real-time three-dimensional imaging system for a cellular phone camera and to solve the difficulties for embedding three-dimensional imaging function in a pocket-sized cellular phone by use of variable focal micromirror array lens.
The real-time three-dimensional imaging system for a cellular phone camera generates an all-in-focus image with depth information. The system comprises a variable focal length lens, an imaging unit for capturing images with different focal planes, which are changed by the variable focal length lens, an image processing unit for processing the images taken by the imaging unit and converting the processed images to three-dimensional displayable type, and three-dimensional display unit for displaying the processed three-dimensional images.
The variable focal length lens comprises a micromirror array lens. The focal plane of the imaging device is changed by varying the focal length of the micromirror array lens as described in U.S. patent application Ser. No. 10/855,287 for “Variable focusing lens comprising micromirrors with two degrees of freedom rotation and one degree of freedom translation” filed on May 27, 2004, Ser. No. 10/857,280 for “Variable focusing lens comprising micromirrors with one degree of freedom translation” filed on May 28, 2004, and Ser. No. 10/983,353 for “Variable focal length lens comprising micromirrors” filed on Nov. 8, 2004.
The variable focal length lens also performs auto-focusing and zooming function of the camera as described in U.S. patent application Ser. No. 10/896,141 for “High Speed Automatic Focusing System” filed on Jul. 21, 2004 and U.S. patent application Ser. No. 10/806,299 for “Small and Fast Zoom System” filed on Mar. 22, 2004. Zooming function devised by the micromirror array lens is compact and does not have macroscopic mechanical motions of lenses. Since the zoom system with the micromirror array lens has no macroscopic mechanical motions of lenses, the system consumes minimal power to perform the function. Another advantage of the invention is to provide a zoom system that can compensate various optical distortions or aberrations.
The imaging unit includes one or more two-dimensional image sensors taking two-dimensional images at different focal planes.
The image processing unit generates the all-in-focus image with depth information from the two dimensional images taken by the imaging unit and converts the all-in-focus image into three-dimensional displayable data format. It is desirable that all the processes are achieved within a unit time which is less than or equal to the afterimage time of the human eyes.
The three-dimensional display unit displays the three-dimensional images and all-in-focus images processed by the image processing unit. The display unit also displays the two-dimensional images taken from the imaging unit. For three-dimensional display, lenticular three-display method or other three-display method can also be used.
The micromirror array lens includes a plurality of micromirrors. The translation and/or rotation of each micromirror of the micromirror array lens are controlled to vary the focal length of the variable focal length lens.
The micromirrors of the micromirror array lens are arranged to form one or more concentric circles.
Each micromirror of the micromirror array lens may have a fan shape to enhance the optical efficiency.
The reflective surface of each micromirror of the micromirror array lens is substantially flat. Alternatively, the reflective surface of each micromirror of the micromirror array lens can have a curvature. The curvature of the micromirrors can be controlled.
Preferably, the reflective surface of the micromirror is made of metal.
Each micromirror of the micromirror array lens is actuated by the electrostatic force and/or electromagnetic force.
The micromirror array lens further includes a plurality of mechanical structures upholding the micromirrors and actuating components for rotating and translating the micromirrors. The mechanical structures and the actuating components are located under the micromirrors for maximize the reflecting surface area to enhance the optical efficiency.
The micromirror array lens is a reflective Fresnel lens, and the micromirrors are arranged in a flat plane. Each micromirror has the same function as a mirror. The array of micromirrors works as a reflective focusing lens by making all light scattered from an object converge into a focal plane and meet periodic phase condition among the lights from different micromirrors. In order to perform this procedure, the micromirrors are electrostatically and/or electromagnetically controlled by actuating components to have desired positions. The focal length of the lens is changed by controlling translation of micromirrors, by controlling rotation of micromirrors, or by controlling both translation and rotation of micromirrors.
The micromirror array lens is an adaptive optical component. The micromirror array lens compensates for phase errors of light introduced by the medium between an object and its image.
The three-dimensional imaging system includes a micromirror array lens and a image sensor. Micromirror array lens focuses the image onto a image sensor. Alternatively, a beam splitter can be used between the micromirror array lens and an image.
The micromirror array lens in three-dimensional imaging system is an adaptive optical component. The micromirror array lens compensates for phase errors of light introduced by the medium between an object and its image and/or corrects the defects of the three-dimensional imaging. Also, an object which does not lie on the optical axis can be imaged by the micromirror array lens without any macroscopic mechanical movement of the three-dimensional imaging system.
In order to obtain a color image, the micromirror array lens is controlled to satisfy the same phase condition for each wavelength of Red, Green, and Blue (RGB), respectively. The three-dimensional imaging system may further include a plurality of bandpass filters, photoelectric sensor for color imaging. Photoelectric sensor includes Red, Green, and Blue (RGB) sensors.
A color image is obtained by processing electrical signals from the Red, Green, and Blue (RGB) sensors according to an image processing method. The processing of electric signals from the Red, Green, and Blue (RGB) sensors is synchronized and/or matched with the control of the micromirror array lens to satisfy the same phase condition for each wavelength of Red, Green, and Green (RGB), respectively.
The micromirror array lens includes micromirrors and actuating components, and uses a very simple mechanism to control the focal length. The focal length of the micromirror array lens can be changed by translation and/or rotation of each micromirror.
Since micromirror has a tiny mass, the lens comprising micromirrors has a very fast response time down to hundreds of microseconds. The lens also has a large focal length variation and a high optical focusing efficiency. In addition, the lens design makes a large size lens possible, makes the focusing system very simple, and requires low power consumption. The lens has a low production cost because of the advantage of mass productivity.
The micromirror array lens can compensate for optical effects introduced by the medium between the object and its image and/or corrects the defects of a lens system.
The micromirror array lens can have a polar array of micromirrors. For the polar array, each micromirror has a fan shape to increase the optical efficiency by expanding active reflecting region. The aberration of the micromirror array lens can be reduced by micromirrors with curvatures. The optical efficiency of the micromirror array lens can be increased by locating a mechanical structure upholding the micromirror and the actuating components under the micromirror to enhance the optical efficiency by increasing the active area of reflecting surface. Electric circuits to operate the micromirrors can be replaces with know semiconductor technologies such as MOS and CMOS.
The micromirror array lens used in present invention has the following advantages: (1) the micromirror array lens has a very fast response time thanks to the tiny mass of the micromirror; (2) the lens has compactness in size suitable for a cellular phone applications; (3) the lens has a large focal length variation because large numerical aperture variations can be achieved by increasing the maximum rotational angle of the micromirror; (4) the lens has a high optical efficiency; (5) the lens can have a large size aperture without losing optical performance. Since the micromirror array lens includes discrete geometry comprising micromirrors, the increase of the lens size does not enlarge the aberration caused by the shape error of a lens; (6) the cost is inexpensive because of the advantage of mass productivity of microelectronics manufacturing technology; (7) the lens can compensate for phase errors introduced by the medium between the object and the image and/or corrects the defects of the lens system; (8) the lens makes the focusing system much simpler; (9) the lens requires small power consumption when electrostatic actuation is used to control it.
The invention of cellular phone camera with three-dimensional imaging function has the following advantages: (1) the device can make a real-time three-dimensional image (2) the device has an auto-focusing function; (3) the device has an optical zooming function without any macroscopic mechanical movement; (4) the device can capture real-time all-in-focus image; (5) the device can process and covert image into three-dimensional displayable format; (6) the device can convert two-dimensional images into lenticular type three-dimensional display or other three-dimensional displayable formats; (7) the device has a large range of depth; (8) the device can have high depth resolution; (9) the cost is inexpensive because the micromirror array lens is inexpensive; (10) the device has a high optical efficiency; (11) the device can compensate for phase errors introduced by the medium between the object and its image and/or correct the defects of a lens system; (12) the device is very simple because there is no macroscopic mechanical displacement or deformation of the lens; (13) the device is compact; (14) the device requires small power consumption when the micromirror array lens is actuated by electrostatic force.
Although the present invention is brief summarized herein, the full understanding of the invention can be obtained by the following drawings, detailed description, and appended claims.
These and other features, aspects, and advantages of the present invention will become better understood with reference to the accompanying drawings, wherein:
The focal plane of the imaging device is changed by change of focal length of the micromirror array lens 42.
The imaging unit 43 includes one or more two-dimensional image sensor taking original two-dimensional images 45 with different focal planes.
The image processing unit 44 generates the all-in-focus image 46 and the depth information for in-focus image from two-dimensional images. All the processes are achieved within a unit time which is less than or equal to the afterimage time of the human eye.
While the micromirror array lens 42 varies the focal length of the micromirror array lens, the image sensor 43 takes two-dimensional images 45 with corresponding optical depth information. The desired number of depths is determined by the depth resolution and the range of depth of the object to be imaged. To get real-time three-dimensional motion images, two-dimensional images should be taken and processed within the afterimage time of human eyes. There are several methods for the image processing to obtain all-in-focus image 46 and depth information for each pixel of the image. After having the all-in-focus image, the image data can be converted into various three-dimensional displayable image formats.
The mechanical structures upholding each micromirror and the actuating components to rotate and translate the micromirrors 72 are located under the micromirrors 72 so that the micromirrors 72 have larger active area.
The focal length f of the micromirror array lens 81 is adjustable by controlling the rotation and/or translation of the micromirror 84. The operation of the micromirror array lens 81 is possible by controlling only rotation regardless of the phase condition. In this case, the quality of the image generated by the micromirror array lens is degraded by the aberration. Also translation only without rotation can form a Fresnel diffraction lens with the aberration. The smaller the sizes of the micromirrors 84 can reduce the aberration. Even though the focusing ability of the one motion by either rotation or translation is not powerful, the lens with one motion has the advantage of simple control and fabrication.
Since the ideal shape of the conventional lens 62 has a curvature even in the small size of the micromirror, it is strongly desired that each of the micromirrors 84 has a curvature itself. However, since the aberration of the lens with flat micromirrors 84 is not much different from the lens with curvature if the size of each micromirror is small enough, there is not much need to control the curvature.
The micromirror array lens is an adaptive optical component with the phase of light compensated by the translations and/or rotations of micromirrors. The micromirror array lens can correct the phase errors of light introduced by the medium between the object and its image and/or corrects the defects of a lens system.
The phase matching condition implicits that the light used should be monochromatic. To get a color image, the micromirror array lens in cellular phone camera with three-dimensional imaging function, the same phase condition for each wavelength of Red, Green, and Blue (RGB), should be satisfied respectively, and the system can use bandpass filters to make monochromatic lights at the wavelength of Red, Green, and Blue (RGB).
If a color photoelectric sensor is used as an image sensor in cellular phone camera with three-dimensional imaging function using a micromirror array lens, a color image can be obtained by processing electrical signals from Red, Green, and Blue (RGB) sensors with or without bandpass filters, which should be synchronized with the control of micromirror array lens.
The cellular phone camera with three-dimensional imaging function can have the optical zooming function by the focal length variation of the micromirror array lens without any macroscopic mechanical movement. The variable focal length lens also performs auto-focusing and zooming function of the camera.
Zooming function devised by the variable focus lens of the micromirror array lens is compact and does not have macroscopic mechanical motions of lenses. Since the zoom system with the micromirror array lens has no macroscopic mechanical motions of lenses, the system consumes minimal power to perform the function. Another advantage of the invention is to provide a zoom system that can compensate various optical distortions or aberrations.
The fast response of the micromirror enables the auto-focusing function if applied with the focus finding algorithm in image processing unit. Image processing unit processes the taken two-dimensional image to determine whether the object image lies on the focus or not. If the focus and object image lies in different planes, the active feedback can track for finding appropriate focus by varying the focal lens of the micromirror array lens. Since the response of the micromirror array lens is very fast, the auto focusing function can perform the process within very short time scale.
While the invention has been shown and described with reference to different embodiments thereof, it will be appreciated by those skills in the art that variations in form, detail, compositions and operation may be made without departing from the spirit and scope of the invention as defined by the accompanying claims.