Patent Application
20230336858
The application relates generally to the field of high-magnification photography and imaging techniques. More specifically, the application relates to systems and methods for correction of image aberrations in high magnification cameras or imaging devices that include the use of partial reflectors. The application also relates to imaging devices or cameras built into a cell phone, smartphone, tablet, laptop or any other mobile devices.
Digital cameras are widely used in mobile devices. Their performance is a major differentiator for consumers and is a driver for market share. Hence mobile device makers and suppliers strive to improve camera performance.
One key aspect of mobile device camera performance is high-magnification (or high-zoom) photography. Typically, cameras or imaging devices that achieve high-magnification photography have long and large lens assemblies (e.g. zoom lenses in DSLR cameras). But mobile devices are thin and compact and cannot contain long lens assemblies, which has traditionally limited their magnification and zoom capabilities.
US Patent Application 2021/0211563 (by Edo Waks, Benjamin Shapiro) discloses systems and methods for producing high-magnification images of objects for a camera or imaging device that is thin and can fit inside the thickness of a cell phone, smart phone, tablet, laptop or other mobile devices. In particular it discloses the use of partial reflections between partially-reflective surfaces inside the camera or imaging system, to increase the path light length, and to thereby enable a long focal length and a high magnification inside a thin camera or imaging system.
The application provides solutions to reduce image aberrations for cameras or imaging devices that include partial reflectors, and that can be incorporated into mobile devices such as cell phones, smartphones, tablets, laptops, etc. The application includes both hardware solutions (selection of optical elements and their placement and properties) and software solutions (selection of imaging methods and algorithms).
This application discloses systems and methods for reducing image aberrations in cameras with partial reflectors. All camera and imaging systems suffer to some degree from image aberrations. Image aberrations refer to defects or imperfections in a formed image due to the inability of cameras or imaging devices or optical systems to take perfect images. Image aberrations can include out-of-focus and focus shift effects, coma, astigmatism, field curvature, distortion, spherical and chromatic aberrations, wavefront errors, field distortion (pincushion or barrel distortion), vignetting, ghosting or flare, diffraction, and other types of aberrations. The methods and systems can produce a high-magnification image with reduced image aberrations. The disclosed imaging or camera systems can fit into thin and compact mobile devices, and make use of partial reflectors. Particularly, the application provides methods and systems for reducing image aberrations for cameras or imaging devices that include partial reflectors, to enable high-magnification high-quality photography for mobile devices.
Other aspects provide for systems with optical elements (e.g., at least one lens, two partial reflectors) and an imaging sensor (e.g., time-integrating sensor), wherein the light that forms an in-focus image on the sensor completes at least one round-trip partial reflection between the partial reflectors. The properties of the optical elements (location, size, shape, materials, coatings) are selected to enable high-magnification and reduced image aberrations in a small volume device. Image processing algorithms are provided to extract the in-focus component of the image formed on the sensor, and to process it to reduce image aberrations.
This application will now be described more fully with reference to the accompanying drawings in which preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiment set forth herein.
It is known that there are always some distortions and aberrations in the images formed by any camera due to the physical and optical limitations of lenses, of other optical elements, and due to limitations of image processing firmware and software. Image aberrations can include out-of-focus and focus shift effects, coma, astigmatism, field curvature, distortion, spherical and chromatic aberrations, wavefront errors, field distortion (pincushion or barrel distortion), vignetting, ghosting or flare, diffraction, and other types of aberrations.
Embodiments includes systems and methods for improving and correcting images from cameras or imaging devices or optical systems with partial reflectors. These include disclosing configurations of the camera or imaging system, including exemplary location, size, shape, materials and material properties, surface coatings, focal lengths, indexes of refraction, absorption, and scattering, and other aspects of system optical elements such as lenses, partial-reflectors, prisms, and mirrors. Additionally disclosed are exemplary methods to select the configuration of cameras or imaging systems with partial reflectors, and also disclosed are exemplary designs. Further disclosed are imaging processing methods and algorithms to extract in-focus reduced-aberration images from the images that are formed on the sensor (e.g. time-integrating sensor). Particularly, embodiments of the present invention provide methods and systems for reducing image aberrations for cameras or imaging devices that include partial reflectors to enable high magnification in a small or thin mobile device.
Although a single spherical lens with a large focal length can achieve high magnification, in US Patent Application 2021/0211563 (by Edo Waks, Benjamin Shapiro) it was disclosed that adding partial reflectors and image processing allows a single spherical lens to achieve high magnification in a small or thin device. But a single spherical lens typically also creates strong spherical aberrations. Hence it is advantageous to use multiple lenses, to correct for aberrations. For example, a sequence of even longer focal length lenses can be used to achieve the net desired focal length, but with reduced aberrations, e.g., with reduced spherical aberrations.
As disclosed herein, the partial reflectors create new and novel opportunities to correct image aberrations. In particular, optical systems with partial reflectors allow light to visit optical elements multiple times, meaning that a single optical element can interact with a light beam multiple times. This fundamentally changes how optical elements interact and provides for the inventive configurations, component selections and shapes, and image processing algorithms disclosed herein.
Ordinarily, if a certain focal length is desired, that focal length can be achieved by having two or more lenses with longer focal lenses and using them together to achieve the desired focal length. The benefit of doing so is that lenses with longer focal lenses create less aberration, hence multiple lenses with longer focal lenses achieve the same desired focal length, but with less spherical aberration. Ordinarily, light interacts with each lens one time, as it passes from scene, through each lens, to the imaging sensor.
We disclose a method for reducing and correcting image aberrations, such as spherical aberrations, by exploiting multiple partial reflections that cause light to interact with one, some, or all of the optical elements multiple times. For example, since a lens acting on the light multiple times will function like many copies of the same lens acting on that light, hence a lens with an even longer focal length can be employed. Thus aberrations may be reduced further than in conventional camera systems, by designing optical elements to exploit partial reflections, as disclosed herein.
An exemplary system is illustrated in
Thus as disclosed in the exemplary
This application further discloses using curved partially-reflective mirrors to achieve a desired camera focal length with reduced aberrations effect. With partial reflections, one or more curved partial reflectors can be used with a longer focal length, to achieve a desired camera focal length but incur less aberrations. This is because the light path will visit a curved partially-reflective element multiple times, e.g. K+1 times for K round-trip partial reflections, and each visit will cause additional focusing, but with reduced aberration (because of the longer focal length of the curved partial reflector). In addition, curved reflectors also do not introduce chromatic aberration, so this enables mitigating multiple aberrations simultaneously.
Also disclosed is using a combination of lenses and flat or curved partial reflectors. Then the partial-reflections visit longer focal length lenses, and/or longer focal length curved reflectors, or both, multiple times, and thus create the desired camera focal length but with reduced aberrations.
In an embodiment, the present invention provides for shaping (e.g. curving) of optical elements for systems with partial reflectors, to reduce aberrations. In one exemplary embodiment a system with two shaped lenses, two partial reflectors whose reflective surfaces are flat but their other surfaces are shaped, and a sensor is disclosed, such that a focused image is formed on the sensor after one round-trip reflection between the two partial reflectors. And the image is formed with reduced image aberrations.
The exemplary placement and shape of the optical elements (lenses, partial reflectors) is shown in
This embodiment was selected by a combination of experience, innovation and mathematical optimization. Specifically, optical element placement, size, and shapes were selected to enable high performance, as well as manufacturability and mass production. For example, the degree of curvature of lenses was selected to enable their manufacture by standardly available lens mass manufacturing methods.
In this embodiment, both surfaces (211 and 212) of the first lens (201), and both surfaces (241 and 242) of the second lens (204), and the non-reflecting surfaces (221 and 232) of both partial reflectors (202 and 203), have been shaped (curved) to minimize image aberrations. We disclose that these curved shapes can be aspherical (curve not approximately a portion of a circle), and have been chosen by the optimization methods disclosed herein. But in this embodiment the partially-reflective surfaces (222 and 231) of both partial reflectors (202 and 203) have been left flat. It will be obvious to someone familiar with the art of optics and lens design that other combinations of lenses and partial reflectors are possible, and that other combinations of surfaces can be chosen to be shaped (curved) or left flat, and that the intended number of partial round-trip reflections for a focused image could differ (e.g. 2, 3, 4, etc. instead of 1). The embodiment disclosed is an example, and it is understood other combinations and choices are possible and are covered by this disclosure.
Now we disclose example methods and procedures to select the shape of optical elements for systems with partial reflectors.
In an exemplary case, the number and type of optical elements (e.g. the 4 optical elements in
The method for optimizing the optical elements involves optimizing the selected free parameters in order to minimize a selected camera merit function. Selecting which parameters to set as free, and which to hold fixed, as well as selecting an appropriate camera merit function, is an art. An inventive component of the current application is this art of optimization for cameras with partial reflections. The free parameters of surfaces can include, but are not limited to: surface curvature radius (for a spherical optical element), or conic coefficients or higher order polynomial coefficients (for aspherical elements). In addition to free parameters, constraints can also be imposed. For example, the focal length can be constrained to match the distance the light travels from the first optical element to the sensor, when there is one round-trip reflection of light between the two partial reflectors (as in
We disclose that a merit function is selected to reflect the desired system optical performance. For example the merit function can include parameters that represent effective focal length, MTFs (modulus of the optical transfer functions) at one or various object distances and settings, spot size at a point or across a portion of or the entire optical field, out-of-focus and focus shift effects, coma, astigmatism, field curvature, distortion, spherical and chromatic aberrations, wavefront errors, field distortion (pincushion or barrel distortion), vignetting, ghosting or flare, diffraction, and other types of aberrations, physical length constraints on a portion of or the entire system, etc. One, some, or all of these quantities can be multiplied by different weighting amounts. For example, if it is desired that MTF performance be enforced more strongly than vignetting, then the weight of the former can be made stronger than the weight of the latter.
Once the variable parameters of all of the optical elements, the merit function parameters and their weightings have been selected, then the optical system can be optimized as disclosed herein. This can be done either manually by adjusting values, or by using automated routines. The camera embodiments disclosed herein have been selected by an inventive art, by using inventiveness to select the number and type of optical elements, and further by manual adjustments and by the disclosed automation optimization steps, combined together into an iterative and inventive process. Selections and combinations of the above are possible, and are disclosed. It is further understood that the disclosed designs are exemplary, and many modifications, additions, and variations are possible.
This application discloses solving an optimization problem to help select the placement, shape, and materials of optical elements for exemplary systems. The mathematical part of the optimization problem can be solved by means known in the art of optical design or mathematical optimization, including by methods such as gradient methods, nonlinear algorithms, probabilistic search methods, branch and bound methods, neural nets, genetic algorithms, machine learning, AI (artificial intelligence) methods, etc. Once a solution is found, the number of elements can be changed, and the process repeated to arrive at a design selection. For example, this has been done for the selection of elements shown in the embodiment of
Inventive features of the current example include aspherical surfaces of the optical elements. For example, adding higher order polynomial modifications to the lens surface can result in significantly improved aberration reduction. We further disclose that the partially reflective surfaces do not need to be flat or planar. The partially-reflective surfaces can be made to have spherical or aspherical curvature. By allowing the partially reflective surfaces to deviate from planar, further improvements in optical performance can be realized. This is especially helpful for optical systems where the physical length is restricted by packaging requirements of mobile devices.
We note that the exemplary disclosure of our design selection methods is meant to be illustrative and not limiting. It is understood that other design selections are possible, including changing the number of elements, and using mathematical optimization to find other optical elements position, shape, and material choices. In particular, the number of light round-trips reflections that form a focused image on the sensor can also change. Designs can be selected for one, two, three, four, etc. partial reflection round trips.
In an embodiment, the present invention provides for the reflective surfaces to also be shaped (curved) to reduce image aberrations. In an exemplary embodiment, a system with shaped lenses, two partial reflectors with a shaped (curved) reflective surface, and a sensor is disclosed such that a focused image is formed on the sensor after one round-trip reflection between the two partial reflectors, and is formed with reduced image aberration.
Optical element placement, size, and shapes were selected to enable high performance, as well as manufacturability and mass production. For example, the degree of curvature of lenses was selected to enable their manufacture by standardly available lens mass manufacturing methods.
The exemplary placement and shape of the optical elements (lenses, partial reflectors) is shown in
This application discloses that the reflective surfaces (222 and 231) of the partial reflectors have been shaped, by the methods disclosed herein, in addition to shaping of the lenses and the other sides of the partial reflectors (as previously disclosed in the context of
This application discloses that systems with partial reflectors can include prisms or angled mirrors. In an embodiment, the present invention provides for a system with a prism (can also be an angled mirror), two lenses, two partial reflectors whose partially-reflective surfaces are shaped, and a sensor, such that a focused image is formed on the sensor after one round-trip reflection between the two partial reflectors, with reduced image aberration. Because the light interacts with the partial reflector surfaces multiple times, a slight change in these surface shapes can have a significant impact on the overall optical performance and on aberration reduction. By allowing the partially reflecting surfaces to be non-planar it is possible to further improve the optical performance while at the same time keeping the optical length of the system short.
This embodiment was selected by a combination of experience, innovation and mathematical optimization. Specifically, optical element placement, size, and shapes were selected to enable high performance, as well as manufacturability and mass production. For example, the degree of curvature of lenses was selected to enable their manufacture by standardly available lens mass manufacturing methods.
The exemplary placement and shape of the optical elements (prism, lenses, partial reflectors) is shown in
This application discloses that front and back surfaces of the two lenses (201, 204) and front and back surfaces of the two partial reflectors (202, 203) have been shaped, by methods disclosed herein, so as to bring the light in-focus to the image plane (401) after one round-trip reflection and further to reduce image aberrations and improve image quality.
In an embodiment, the present invention provides for 90 degree turning of the light path that occurs between the partial reflectors. Thus in another exemplary embodiment, a system with lenses, two partial reflectors, an angled mirror at substantially 45 degrees between them, and a sensor, is designed such that a focused image is formed on the sensor with reduced image aberration after one round-trip reflection between the two partial reflectors. Here the optical element (angled mirror) that turns the light substantially 90 degrees is located between the partial reflectors. Placing the turn mirror between sets of optics can reduce the number of optical elements and decrease the total volume occupied by the optical system. Additionally, because the light paths now reflect off the mirror surface multiple times for one or multiple round-trips, hence small changes of the angle of this mirror will have significant influence on the position of the image on the sensor, making active electro-mechanical control of this mirror a more effective method of achieving image stabilization and object tracking.
Optical element placement, size, and shapes were selected to enable high performance, as well as manufacturability and mass production. For example, the degree of curvature of lenses was selected to enable their manufacture by standardly available lens mass manufacturing methods.
The exemplary placement and shape of the optical elements (lenses, partial reflectors, angled mirror) is shown in
In this example, this application discloses that the front and back surfaces of the two lenses (201, 205) and the non-reflecting surfaces of the two partial reflectors (202, 204) have been shaped, whereas the partially-reflecting surface of each partial reflector has been left flat. Shaping was carried out by the methods and procedures disclosed herein, to bring the light in-focus (106) to the image plane (401) after one round-trip reflection, and to reduce image aberrations.
In addition to selecting the shape and placement of optical elements for systems with partial reflections, we further disclose selecting other parameters to reduce image aberrations. For example, this application discloses selecting optical properties such as indexes of refraction, reflectivity, transmissivity, scattering and absorption. These parameters can be selected from ranges, or from a list of available values. Ranges may correspond to ranges that are available or manufacturable. For example, there may be a range of curvatures that can be manufactured for glass or plastic lenses, or for aspherical elements there may be a range of manufacturable shapes that can be described by a range of conic or polynomial function parameters. Selection from a list may correspond to a list of available lens or partial-reflector shapes, and/or it may correspond to a list of materials parameters (indexes of refraction, refraction, reflectivity, transmissivity, scattering and absorption, and/or parameters for available surface coatings) for available glass or plastic materials. This application discloses that selecting such values is a part of the art, methods and procedures to optimize systems with partial reflections, and includes optimizing them for reducing aberrations and for aiding image correction and processing.
In embodiments, the present invention provides for using traditional or non-traditional lens materials and geometries, in order to further improve the image quality. For example, GRIN (GRadient INdex) lenses in which the refractive index of the material is made to vary in a systematic way as a function of position within the lens, such lenses could be used. Parameters to describe GRIN lenses can be made free parameters in our disclosed optimization approach, and can be optimized for systems with partial reflectors.
Disclosed next is an improved method of changing focal length (focusing, zooming). In a conventional camera, at least one lens or optical element can be moved back and forth to change the focal length of the system. In a cell phone or mobile device, the distance that a lens or optical element can travel back and forth is limited by the thickness of the device and/or by the thickness of the camera it contains, and also by the space taken up by the other elements of the camera (other lenses, sensor, aperture, shutter, filters, PCB board, etc). Thus the degree of focusing or zooming possible in a conventional cell phone or mobile device camera can be limited. In particular, it can be difficult or not possible to focus on objects that are near the phone or mobile device.
Methods and systems disclosed herein can overcome the above limitation. A small back and forth motion of an optical element or elements for a system with partial reflectors can result in a larger change in focusing or zooming (e.g. in focal length) than would be the case for a system without partial reflections. Referring to
In contrast, referring now to
This application also discloses that other optical elements can be moved, singly or in various combinations, with lenses. For example, moving just the second partial reflector 213 by Δx will change the light-path distance between the two lenses 211 and 214 by a distance of 2 Δx for one round trip partial reflection, by 4 Δx for two round trips, etc. Hence there is a beneficial multiplicative effect for moving partial-reflectors, just as there is for moving lenses. This application discloses that this can be used to improve image stabilization and auto-focus in cameras and imaging systems for mobile devices, because now small movements of optical elements will improve the available amount of focal length change for zooming, focusing, and image stabilization.
Thus for a system with partial reflections, if a partial reflector is moved by a distance d, then the corresponding change in the light path length is 2K d where K is the number of round-trip partial reflections. This means that to effect a change in the light path length of ΔL, we need only mechanically move the partial reflector by d=ΔL/2K. With partial reflections, the required mechanical motion is 1/(2K) shorter than without partial reflections. For example, with K=2 round trip partial reflections, one quarter the motion will provide the same zoom capability as a full motion of an optical element for a system without partial reflections.
Further, this application discloses that one or more lenses that are actuated can be arranged between the partial reflectors. In such an embodiment, light can pass through said actuated lens or lenses multiple times and that can provide a multiplicative effect for zooming, focusing or image stabilization. The optimization methods disclosed herein can be used to design systems with such one or more actuated lenses between the partial reflectors.
Thus this application discloses using partial reflections to allow the available travel distance for one or more optical elements to enable covering a larger focusing or zooming range than would be possible for a conventional camera. We further disclose that the optimization methods described above can be used to help select the shape (e.g. curvature or aspheric shape) of partial reflectors and lens or lenses, and also to select which elements will be moved how much, to ensure zooming and focusing can be maximized and thus varied effectively over a much larger range than would be possible in a conventional camera. Overall, this application discloses that a system with partial reflectors wherein light interacts with at least one optical element more than once, such a system enables improved zooming or focusing range, and in particular also enables focusing on objects that are near the camera (“macro photography”).
With regard to “macro photography”, modern cell phone cameras have limitations for taking an in-focus image of an object too close to the mobile device. For example, most conventional phone cameras cannot be brought within approximately less than 4 centimeters of an object and still achieve an in-focus photograph. This is because the size of the phone camera and the number of lenses in that camera limits how much the lenses inside that camera can be made to travel mechanically to accommodate focusing on near-in objects.
This application discloses using partial reflections to overcome this limitation and both get closer to the object and achieve higher magnification, thus accruing a double benefit (e.g. a small object is both closer when still in focus and is magnified more, thus doubly improving photography of nearby objects). Hence our disclosed system can, for example, be brought closer to a small insect or a flower than the 4 centimeters that is possible for current phones, and also magnify the insect or flower more than is possible with current cell phones.
Thus for a small volume camera where the mechanical motion of optical elements is restricted by its size, like in a cell phone, this application discloses that partial reflections provide an advantage. Now the same mechanical range of motion will allow a greater range of focusing and zooming. With partial reflections, a cell phone camera could therefore focus onto an object that is substantially closer than, for example, 4 centimeters away (thus enabling improved “macro photography”). Alternatively, if the same zoom range is all that is desired, then with partial reflections, less mechanical motion will achieve that same zoom range, and the camera design can be simplified providing savings in cost and complexity. Thus partial reflections can either decrease the amount that optical elements need to be moved, or conversely, for the same range of motion, can increase the range of zoom and focusing for an available amount of optical element mechanical motion range.
In embodiments, the present invention provides for image processing methods and algorithms for high-magnification camera systems with partial reflectors, including algorithms that reduce image aberrations. This application discloses that algorithms that include deconvolution, that include characterization and use of point spread functions, of unsharp masks, of high-pass filtering, and other methods, that these algorithms can extract the in-focus component of the image and can be used to process images and improve image quality. In particular, they can be used to reduce image aberrations. Such algorithms may be incorporated into an imaging system such as for a mobile device (cell phone, tablet, laptop), digital video camera, a digital still-image camera, or any other imaging system with partial reflections.
Referring now to
One embodiment includes a system or method in which the image extraction uses deconvolution via an optical transfer function for a system with partial reflections. A transfer function is essentially the response of the imaging optics to a plane wave input. For a single lens, the ideal transfer function is a tightly focused spot (106), as shown in
Conversely, for a system with one or more partial reflections, the transfer function appears as in
where R(a, b) is the in-focus image, and h is the transfer function. In the frequency domain this convolution becomes a product
C(u,v)=H(u,v)R(u,v).
To obtain the in-focus image for our system with one or more partial reflections, this application discloses applying a deconvolution algorithm
R(u,v)=C(u,v)/H(u,v).
Also disclosed is using other algorithms which may also be applied to perform this deconvolution, for example in a numerically more efficient way, or to eliminate or reduce numerical errors.
We further disclose using unsharp masks to extract the in-focus image component for systems with partial reflections. In this approach we subtract a blurred version of the image on the sensor from the original image. The blurred version provides an estimate for the out-of-focus images. The blur radius and weighting are adjustable parameters that are optimized to achieve the best image quality.
Also disclosed is using high pass filtering image correction for systems with partial reflections. In a system with partial reflections, the out-of-focus components of the image will have largely low-frequency spatial components. But the in-focus image will have high-frequency components. By eliminating the low-frequencies we may effectively remove the out-of-focus components while preserving most of the in-focus image which resides in the higher frequency ranges. Thus this application discloses using high-pass filtering to extract the in-focus image.
In addition to increasing the zoom range by moving optical elements (lenses, partial reflectors, or prisms/angled-mirrors), we additionally disclose increasing the zoom range even further by utilizing different numbers of partial-reflection round trips. By moving optical elements within our camera, so that we achieve an in-focus image for a different number of round trips (e.g. K+1 round trips rather than K), then we may achieve a larger focal length without increasing the form-factor of the imaging system. For example, the system may first extract an in-focus image after one round-trip, then extract it after two round-trips, then after three, etc., or in reverse. The resulting in-focus image would be extracted using the same methods as previously disclosed. This would combine the benefits of both moving optical elements and using the most desirable number of round trip partial reflections to maximize zoom and focusing range.
This application discloses an inventive method to reject out-of-focus components of the image formed on the sensor by engineering the spectral transmission of one or more partial reflectors. In an exemplary case, if the partial reflectors fully transmit red light (do not reflect any red light), but partially reflect light at other colors (e.g. partially reflect blue and green light), then the in-focus image that relies on K round-trip partial reflections will not contain any red light. Thus red light can be used to determine which parts of the image should be subtracted as out-of-focus “single-pass” components, and light at other frequencies can be used to determine which part of the image is the desired in-focus part. Thus exploiting color selection in the partial reflectors can aid in extracting the in-focus part of the image.
A person knowledgeable in the art of photography, optics, and image processing will recognize that the above inventive method can be implemented in many ways, including in ways that provide for high image quality. For example, the partial-reflectors may transmit light at a narrow frequency band (e.g. only a specific shade of red), so that the spectrum of light that is not available for the in-focus part is minimized. Other modifications are also possible, including selecting colors that are or are not partially-reflected, and extraction of the in-focus image component by exploiting different colors and frequencies of light, for systems with partial reflections.
Hence we further disclose, for instance, a partial reflector that is partially reflective at some wavelengths (e.g. blue and green), but is transmissive at other wavelengths (e.g. red). The red channel then provides an estimate of the out-of-focus image which can be used to correct the blue and green channels.
In an embodiment, the present invention also provides for a system with a lens, partial reflectors, and time-integrating sensor array, wherein some elements of the sensor array measure light intensity in order to set the dark level for the other sensor elements.
This application discloses features and elements to enable dark level correction for systems with partial reflectors, including dark level correction prior to analog to digital conversion. When integrating multiple defocused images on the sensor due to one or more partial reflections, the formed image can acquire a large background level. After analog to digital conversion, this background will reduce the pixel depth by compressing the relevant pixel levels to the top end of the digital values.
To eliminate this reduction of pixel depth, this application discloses performing dark-level correction prior to digital-to-analog conversion using additional pixels whose purpose is to sample the intensity at each point on the sensor. Thus this application discloses adding additional pixels to, in part, measure background intensity. The signal from these pixels can be used to adaptively set the dark level of the sensor elements (e.g. pixels in a CCD or CMOS camera), allowing the reconstruction of the original image without loss of pixel depth. For instance, the values of these pixels can be used to determine a voltage offset for each signal prior to analog to digital conversion, which will eliminate the unwanted background induced by multiple out-of-focus reflections. Each pixel could act as an individual input for the offset level, or alternately each offset can be set by averaging multiple nearby pixels.
Further disclosed is a method to reconstruct a focused image without losing pixel depth by taking multiple exposures and using low exposures to correct for the out of focus component of the higher exposures. This application discloses taking multiple exposures to get estimates of image and background, for systems with partial reflectors. For an exemplary system with partial reflectors, different light paths will visit these partial reflectors a different number of times. This application discloses matching the dynamic range of the sensor to selection of transmission, for example match sensor dynamic range to a desired number K of round-trip partial reflections.
This application discloses using HDR (high dynamic range) methods for systems with partial reflections. In an exemplary setting, this application discloses taking one image of the scene, then taking another image of the same scene but with double the exposure, and then taking a third image of the scene with quadruple the exposure. The first image will capture well the background level generated by the out of focus images, while the higher exposures will progressively capture more detail from the in-focus image. This information can then be used to reconstruct the in-focus image from all of the exposures. Then can splice the images together to create one composite image that has a high dynamic range, that captures both dimly and brightly illuminated parts of the scene (that maximizes contrast). A person skilled in the art of optics, image processing, and photography, will recognize that there are many variations of such a method, e.g. including more than 3 images, varying exposure times, etc. and those variations are included in this disclosure. Specific embodiments may be operated and executed with different computer processors, memory configurations, and data structures.
The embodiments of the invention described herein can be implemented as logical steps in one or more computational processing systems. The implementation is a matter of choice, dependent on the performance requirements of the processing system implementing the invention. Accordingly, the logical operations making up the embodiments of the invention described herein are referred to variously as operations, steps, objects, or modules. Furthermore, it should be understood that logical operations may be performed in any order, unless explicitly claimed otherwise or unless a specific order is inherently necessitated by the claim language.
The foregoing description of embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. The embodiments were chosen and described in order to explain the principles of the invention and its practical application to enable one skilled in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated.
| Number | Date | Country | |
|---|---|---|---|
| 63126942 | Dec 2020 | US | |
| 63127047 | Dec 2020 | US | |
| 63127052 | Dec 2020 | US | |
| 63127081 | Dec 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/US2021/073018 | Dec 2021 | US |
| Child | 18337917 | US |
