Patent Application
20190076010
This disclosure relates generally to lens defogging of digital imaging systems, including lightfield otoscopes.
Digital camera systems are sometimes required to work in extreme environmental conditions. These environmental conditions include high temperatures and relative humidity, such as in a body cavity, where temperatures can reach 45° C. (113° F.). The relative humidity is a function of environment, and in vivo the relative humidity can reach 100%.
A lightfield otoscope is a clinical imaging system that collects 3D information about the patient's ear canal. The lightfield otoscope's primary imaging target is the tympanic membrane. The otoscope is required to work in a high-temperature and high-humidity environment. If a patient has a high fever in a wet season, common environmental factors associated with disease states, fog and condensation may form on the front surface of the lightfield otoscope during the exam. The fog will prevent capture of good quality images of the ear drum.
In one aspect, a lightfield otoscope includes a housing with a tip configured to receive a disposable speculum. The otoscope also includes a microlens array, a sensor array and an optical train contained within the housing. The optical train includes an objective lens and a relay lens. The objective lens is positioned at least partially within the tip. The relay lens is used to relay an image plane of the objective lens to the microlens array and to relay a pupil plane of the objective lens to the sensor array. An active heating element is also contained within the housing and positioned to heat the front surface, thereby reducing fogging and/or condensation on the front surface.
Other aspects include components, devices, systems, improvements, methods, processes, applications, computer readable mediums, and other technologies related to any of the above.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
Embodiments of the disclosure have other advantages and features which will be more readily apparent from the following detailed description and the appended claims, when taken in conjunction with the accompanying drawings, in which:
The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.
The figures and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.
The main housing 210 contains the optical train, which is shown in more detail in
The optical train forms two overlapping imaging subsystems. One imaging subsystem images the object onto image plane I1, which is then relayed to the microlens array 330 located at conjugate plane I2. The other imaging subsystem images the pupil plane P1 onto the sensor array 340 located at conjugate plane P3. The system in its entirety forms spatially multiplexed and interleaved optical images at the sensor array 340. For convenience, the image captured by the sensor array 340 will be referred to as a lightfield image. For convenience, the locations of images, apertures and their optical conjugates will be referred to as planes (e.g., image plane, pupil plane), but it should be understood that the surface does not have to be perfectly planar.
The lightfield image has the following structure. The object is imaged onto I2. In conventional imaging, a sensor array at I2 would capture this image. In the lightfield otoscope, a microlens array is located at I2. Each microlens is illuminated by light from a certain region of the object. For example, the center microlens is illuminated by light from the center region of the object. The lightfield image has a structure of superpixels corresponding to the microlenses. Each superpixel captures light from a certain region of the object. The superpixels contain many individual subpixels, which typically correspond to individual sensors in the sensor array. Each subpixel within a superpixel captures light from the same region of the object, but at different propagation angles.
In other words, the object generates a four-dimensional light field L(x,y,u,v), where L is the amplitude, intensity or other measure of a ray originating from spatial location (x,y) propagating in direction (u,v). Each subpixel in the lightfield image captures light from a certain volume of the four-dimensional light field. The subpixels are sampling the four-dimensional light field. The shape or boundary of such volume is determined by the characteristics of the lightfield imaging system.
In certain lightfield imaging system designs, the sample volumes are hyperrectangles. That is, every subpixel within a superpixel captures light from the same rectangular (x,y) region associated with the superpixel, and each subpixel within the superpixel captures light from a different rectangular (u,v) region. However, this is not always the case. For convenience, the superpixels will be described as capturing light from a certain region of the object (even though subpixels within that superpixel may capture light from slightly different regions), and the subpixels will be described as capturing light from a certain range of propagation directions (even though the range may be different for different subpixels, or even for different (x, y) points captured by the same subpixel). Regardless of the details, the sensor array 340 captures a lightfield image, which maps (x,y) spatial locations and (u,v) propagation directions to subpixels. This is in contrast to a conventional image, which maps (x,y) spatial locations to pixels but loses information about the (u,v) propagation directions.
Optionally, a filter module can be positioned at the pupil plane (or one of its conjugates P2 in this example). The filter module can contain a number of spatially multiplexed filter cells, which allows filtering of the lightfield image.
Because the lightfield image contains information about the four-dimensional light field produced by the object, the processing module 280 can be used to perform different types of analysis, such as depth estimation, three-dimensional reconstruction, synthetic refocusing, extending the depth of focus, spectral analysis and other types of multi-view analysis.
The characteristics of a lightfield imaging system can be used advantageously in otoscopes to image the interior of the ear. By using a lightfield imaging system, three-dimensional (3D) shapes, translucency and/or color information can be captured and extracted.
For example, the lightfield otoscope may be operable in a depth imaging mode. In the depth imaging mode, the lightfield image captured by the sensor array is processed to provide a three-dimensional depth image of an inside of an ear. Alternately or additionally, a lightfield otoscope is operable in a spectral imaging mode. In the spectral imaging mode, lightfield data captured by the sensor array is processed to provide two or more different spectral images of an inside of an ear. Disparity or depth maps can also be determined. The lightfield otoscope may be switchable between the depth imaging mode and the spectral imaging mode or operate in both.
The lightfield data can be processed to produce enhanced imagery of the ear interior. Data based on the enhanced imagery can then be used to assist a person in making a medical diagnosis. This diagnostic data could be the enhanced imagery itself or it could involve further processing of the enhanced imagery.
Enhanced imagery of the tympanic membrane is a good example. A lightfield otoscope can simultaneously capture depth and spectral information about the tympanic membrane. A depth map of the tympanic membrane can produce information regarding its shape—whether it is bulging or retracting, and the estimated curvature. Spectral information can include an amber or yellow image, which is especially useful to diagnose conditions of the tympanic membrane.
For example, Table 1 lists some features distinguishing the conditions of acute otitis media (AOM), otitis media with effusion (OME), and otitis media with no effusion. As can be seen from Table 1, the three conditions of the ear are different and they can be distinguished from one another based on one or more of the following features: color, position (e.g., 3D shape), and translucency. In order to make correct diagnosis of the ear condition, otoscopic images capturing accurate information about color, 3D shape and translucency of an inside of an ear (e.g., a tympanic membrane in an ear canal) are desirable. These can all be captured simultaneously by a lightfield otoscope.
Lightfield data also includes multiple views of the same image. This allows the user to refocus to different depths in the image and to view the same image from different viewpoints. For example, the effect of occluding objects may be reduced by taking advantage of multiviews. This could be accomplished by refocusing. Alternately, it could be accomplished by segmenting the lightfield (multiple views) into depth layers.
However, as mentioned previously, the ear canal can be hot and humid. Fogging and condensation on the front surface of the lightfield otoscope can degrade the quality of the captured images.
In more detail, the two ends 442A,B of the wire 440 are located in the base of the tip (i.e., the part that is away from the ear). The wire 440 runs along the lens barrel from the base to the front surface and back, so that a mid-section of the wire is located near the front surface. The two ends 442 are connected to the controller, for example electronic circuitry that regulates the current running through the wire 440. Because this is an otoscope application, the controller controls the heating to avoid any injury to the patient. In addition, the controller has an operating range that includes at least 36-40 degrees Celsius, since those are typical conditions to be encountered.
Heating can be manually controlled. For example, the otoscope might have a button that is pressed by the operator to activate heating. Alternatively, the active heating element may run continuously during operation of the otoscope to maintain the front surface 410 at a high temperature to avoid fogging, even without manual instructions from the operator. In some cases, the controller may control the heating element based on environmental conditions in the vicinity of the front surface 410, for example the temperature or humidity in this vicinity.
In yet another approach, heating is controlled based on the quality of the captured images.
In
Measuring the power spectra of the R, G, and B channels over time monitors the intensity of spatial frequency contributions in the image. This can be used to control the heating element. This is just one example based on the power spectra of the images. In other approaches, control can be based on a wavelet analysis of the images, or on a gradient analysis of the images, or on a contrast of the images, or on other measures of the spatial frequency content of the images. In these cases, the controller may be a microcontroller or embedded processor, rather than a simple feedback loop.
Experiments using the ear canal phantom were repeated over a range of 36-45° C. and 70-105% humidity, with the heating element on and off. For each trial, we observe whether the image remains foggy or clear. A clear image is a success and is marked by a circle in
Although the detailed description contains many specifics, these should not be construed as limiting the scope of the invention but merely as illustrating different examples and aspects of the invention. It should be appreciated that the scope of the invention includes other embodiments not discussed in detail above. For example, the approach described above is not limited to nichrome wires. Other active heating elements include metal heating elements, ceramic heating elements, resistive heating elements, thin film heating elements, wire heating elements. Examples of materials include kanthal and cupronickel in addition to nichrome.
In addition, this approach can also be used with other imaging systems, including other lightfield systems and also conventional imaging systems. For example, imaging systems used in body cavities include endoscopes and otoscopes. Other imaging systems may be designed for use in steam environments, or high temperature or high humidity environments. The ability to automatically defog a camera removes the need for a human to access a camera in a location that is difficult to access. Temperature and humidity sensors can be used to automatically regulate the temperature of the optics based on the surrounding environmental conditions. One example is an outdoor security camera, where weather is constantly changing and condensation can occur naturally.
Various other modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus of the present invention disclosed herein without departing from the spirit and scope of the invention as defined in the appended claims. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents.
