Patent Grant
11950981
Embodiments of the present invention relate to the field of imaging and, in particular, to a system and method for performing confocal imaging of a three dimensional surface.
A great variety of methods and systems have been developed for direct optical measurement of teeth and the subsequent automatic manufacture of dentures. The term “direct optical measurement” signifies surveying of teeth in the oral cavity of a patient. This facilitates the obtainment of digital constructional data necessary for the computer-assisted design (CAD) or computer-assisted manufacture (CAM) of tooth replacements without having to make any cast impressions of the teeth. Such systems typically include an optical probe coupled to an optical pick-up or receiver such as charge coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) sensor and a processor implementing a suitable image processing technique to design and fabricate virtually the desired product.
One type of system that performs intra-oral scans is a system that uses confocal imaging to image a three dimensional surface. Such systems that use confocal imaging typically include field lenses to flatten an imaging field and enable flat focal planes for emitted light beams. Such flat focal planes ensure that the surface topology of scanned three dimensional surfaces is accurate. However, the field lenses are diverging lenses that open the rays of the light beams. This causes the optics of the confocal imaging apparatus to be enlarged. Additionally, the field lenses should be aligned to ensure accuracy. Such alignment can be a time consuming and challenging process.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.
Described herein is a confocal imaging apparatus having a non-flat focal surface. The non-flat focal surface may be caused by the optics of the confocal imaging apparatus lacking a field lens. As is discussed in greater detail below, the lack of a field lens in the confocal imaging apparatus introduces challenges but also provides numerous advantages. For example, a confocal imaging apparatus without a field lens is smaller, lighter and easier to manufacture than a confocal imaging apparatus having a field lens. Embodiments discussed herein show how to overcome the challenges in designing and using a confocal imaging apparatus lacking a field lens.
Also described herein is a large field confocal imaging apparatus having focusing optics that change a magnification of a focal surface with changes in a focusing setting. As is discussed in greater detail below, the change in magnification introduces challenges that are overcome in embodiments.
In one embodiment, a confocal imaging apparatus includes an illumination module to generate an array of light beams. Focusing optics of the confocal imaging apparatus perform confocal focusing of an array of light beams onto a non-flat focal surface and direct the array of light beams toward a three dimensional object to be imaged. A translation mechanism of the confocal imaging apparatus adjusts a location of at least one lens to displace the non-flat focal surface along an imaging axis. A detector of the confocal imaging apparatus measures intensities of an array of returning light beams that are reflected off of the three dimensional object and directed back through the focusing optics. Intensities of the array of returning light beams are measured for locations of the at least one lens for determination of positions on the imaging axis of points of the three dimensional object. Detected positions of one or more points are adjusted to compensate for the non-flat focal surface. Thus, an object may be accurately imaged despite the non-flat focal surface of the confocal imaging apparatus.
Referring now to
The confocal imaging apparatus 20 further includes a unidirectional mirror or beam splitter (e.g., a polarizing beam splitter) 40 that passes the array of light beams 36. A unidirectional mirror 40 allows transfer of light from the semiconductor laser 28 through to downstream optics, but reflects light travelling in the opposite direction. A polarizing beam splitter allows transfer of light beams having a particular polarization and reflects light beams having a different (e.g., opposite) polarization. In one embodiment, the unidirectional mirror or beam splitter 40 has a small central aperture. The small central aperture may improve a measurement accuracy of the confocal imaging apparatus 20. In one embodiment, as a result of a structure of the unidirectional mirror or beam splitter 40, the array of light beams will yield a light annulus on an illuminated area of an imaged object as long as the area is not in focus. Moreover, the annulus will become a completely illuminated spot once in focus. This ensures that a difference between measured intensities of out-of focus points and in-focus points will be larger.
Along an optical path of the array of light beams after the unidirectional mirror or beam splitter 40 are confocal focusing optics 42, and an endoscopic probing member 46. Additionally, a quarter wave plate may be disposed along the optical path after the unidirectional mirror or beam splitter 40 to introduce a certain polarization to the array of light beams. In some embodiments this may ensure that reflected light beams will not be passed through the unidirectional mirror or beam splitter 40. Confocal focusing optics 42 may additionally include relay optics (not shown). Confocal focusing optics 42 may or may not maintain the same magnification of an image over a wide range of distances in the Z direction, wherein the Z direction is a direction of beam propagation (e.g., the Z direction corresponds to an imaging axis that is aligned with an optical path of the array of light beams 36). The relay optics enable the confocal imaging apparatus 20 to maintain a certain numerical aperture for propagation of the array of light beams 36. The confocal focusing optics 42 and endoscopic probing member 46 are discussed in greater detail with reference to
The endoscopic probing member 46 may include a rigid, light-transmitting medium, which may be a hollow object defining within it alight transmission path or an object made of alight transmitting material, e.g. a glass body or tube. In one embodiment, the endoscopic probing member 46 include a prism such as a folding prism. At its end, the endoscopic probing member 46 may include a mirror of the kind ensuring a total internal reflection. Thus, the mirror may direct the array of light beams towards a teeth segment 26 or other object. The endoscope probing member 46 thus emits array of light beams 48, which impinge on to surfaces of the teeth section 26.
The array of light beams 48 are arranged in an X-Y plane, in the Cartesian frame 50, propagating along the Z axis. As the surface on which the incident light beams hits is an uneven surface, illuminated spots 52 are displaced from one another along the Z axis, at different (Xi, Yi) locations. Thus, while a spot at one location may be in focus of the confocal focusing optics 42, spots at other locations may be out-of-focus. Therefore, the light intensity of returned light beams of the focused spots will be at its peak, while the light intensity at other spots will be off peak. Thus, for each illuminated spot, multiple measurements of light intensity are made at different positions along the Z-axis. For each of such (Xi, Yi) location, the derivative of the intensity over distance (Z) may be made, with the Zi yielding maximum derivative, Z0, being the in-focus distance. As pointed out above, the incident light from the array of light beams 48 forms a light disk on the surface when out of focus and a complete light spot when in focus. Thus, the distance derivative will be larger when approaching in-focus position, increasing accuracy of the measurement.
The light scattered from each of the light spots includes a beam travelling initially in the Z axis along the opposite direction of the optical path traveled by the array of light beams 48. Each returned light beam in an array of returning light beams 54 corresponds to one of the incident light beams in array of light beams 36. Given the asymmetrical properties of unidirectional mirror or beam splitter 40, the returned light beams are reflected in the direction of detection optics 60.
The detection optics 60 may include a polarizer 62 that has a plane of preferred polarization oriented normal to the plane polarization of polarizer 32. Alternatively, polarizer 32 and polarizer 62 may be omitted in some embodiments. The array of returning light beams 54 may pass through imaging optics 64 in one embodiment. The imaging optics 64 may be one or more lenses. Alternatively, the detection optics 60 may not include imaging optics 64. In one embodiment, the array of returning light beams 54 further passes through a matrix 66, which may be an array of pinholes. Alternatively, no matrix 66 is used in some embodiments. The array of returning light beams 54 are then directed onto a detector 68.
The detector 68 is an image sensor having a matrix of sensing elements each representing a pixel of the image. If matrix 66 is used, then each pixel further corresponds to one pinhole of matrix 66. In one embodiment, the detector is a charge coupled device (CCD) sensor. In one embodiment, the detector is a complementary metal-oxide semiconductor (CMOS) type image sensor. Other types of image sensors may also be used for detector 68. The detector 68 detects light intensity at each pixel.
In one embodiment, detector 68 provides data to computing device 24. Thus, each light intensity measured in each of the sensing elements of the detector 68, is then captured and analyzed, in a manner to be described below, by processor 24.
Confocal imaging apparatus 20 further includes a control module 70 connected both to semiconductor laser 28 and a motor 72, voice coil or other translation mechanism. In one embodiment, control module 70 is or includes a field programmable gate array (FPGA) configured to perform control operations. Motor 72 is linked to confocal focusing optics 42 for changing a focusing setting of confocal focusing optics 42. This may adjust the relative location of an imaginary non-flat focal surface of confocal focusing optics 42 along the Z-axis (e.g., in the imaging axis). Control module 70 may induce motor 72 to axially displace (change a location of) one or more lenses of the confocal focusing optics 42 to change the focal depth of the imaginary non-flat focal surface. In one embodiment, motor 72 or confocal imaging apparatus 20 includes an encoder (not shown) that accurately measures a position of one or more lenses of the confocal focusing optics 42. The encoder may include a sensor paired to a scale that encodes a linear position. The encoder may output a linear position of the one or more lenses of the confocal focusing optics 42. The encoder may be an optical encoder, a magnetic encoder, an inductive encoder, a capacitive encoder, an eddy current encoder, and so on. After receipt of feedback that the location of the one or more lenses has changed, control module 70 may induce laser 28 to generate a light pulse. Control unit 70 may additionally synchronize image-capturing module 80 from
Referring now to
Depth determiner 90 determines the relative intensity in each pixel over the entire range of focal settings of confocal focusing optics 42 from received image data. Once a certain light spot associated with a particular pixel is in focus, the measured intensity will be maximal for that pixel. Thus, by determining the Zi corresponding to the maximal light intensity or by determining the maximum displacement derivative of the light intensity, for each pixel, the relative position of each light spot along the Z axis can be determined for each pixel. Thus, data representative of the three-dimensional pattern of a surface in the teeth segment 26 or other three dimensional object can be obtained.
In embodiments, the confocal focusing optics 42 of confocal imaging apparatus 20 lack field lenses. The purpose of the field lens is to flatten a focal field and thus produce a flat focal pane for the array of light beams. For confocal imaging apparatuses with field lenses, each light beam from the array of light beams focuses on the same flat focal pane. However, without such field lenses the array of light beams focus on an imaginary non-flat focal surface (e.g., on a curved focal surface). This causes the Z axis information that depth determiner 90 computes to be distorted for many pixels.
Field compensator 92 compensates for the curved field caused by the lack of a field lens. Field compensator 92 may also compensate for changes in a position of the curved focal surface caused by temperature and/or for magnification changes caused by changes in a focusing setting. Field compensator 92 applies a field curvature model 94 and/or other optics compensation model (not shown) to each Z axis measurement of each pixel to correct for field curvature, temperature and/or magnification changes. In one embodiment, a different field curvature model 94 (or other optics compensation model) is applied for each focusing setting of the confocal imaging apparatus 20. This is because the amount of field curvature and/or magnification may change with changes in the focusing setting. Alternatively, a single field curvature model 94 (or other optics compensation model) may account for the changes in the field curvature caused by changes in the focusing setting and/or for changes in magnification caused by changes in the focusing setting. For each combination of an X, Y pixel location and a focusing setting (e.g., a z-axis position of one or more lenses of the focusing optics), a particular depth adjustment may be applied based on the field curvature model or models. Additionally, an X location adjustment and/or a Y location adjustment bay be applied based on the field curvature model and/or other optics compensation model. In one embodiment, for each combination of an X, Y pixel location, a focusing setting, and a temperature reading or a z-axis position of a measured element whose position changes with changes in temperature, a particular depth adjustment may be applied based on the field curvature model or models. The adjusted depth (z-axis) values represent the actual z-axis values of the imaged surface.
A three-dimensional representation may be constructed based on the corrected measurement data and displayed via a user interface 84. The user interface 84 may be a graphical user interface that includes controls for manipulating a display of the three-dimensional representation (e.g., viewing from different angles, zooming-in or out, etc.). In addition, data representative of the surface topology of the scanned object may be transmitted to remote devices by a communication module 88 for further processing or use (e.g., to generate a three dimensional virtual model of the scanned object).
By capturing, in this manner, an image from two or more angular locations around the structure, e.g. in the case of a teeth segment from the buccal direction, from the lingual direction and optionally from above the teeth, an accurate three-dimensional representation of the teeth segment may be reconstructed. This may allow a virtual reconstruction of the three-dimensional structure in a computerized environment or a physical reconstruction in a CAD/CAM apparatus. For example, a particular application is imaging of a segment of teeth having at least one missing tooth or a portion of a tooth. In such an instance, the image can then be used for the design and subsequent manufacture of a crown or any other prosthesis to be fitted into this teeth segment.
The optics 200 include an illumination module 38, a unidirectional mirror or beam splitter 40, a series of lenses that may correspond to confocal focusing optics 42, and folding prism 220 arranged along an optical path traversed by an array of light beams 225. The optical path is shown to be a linear path. However, in embodiments one or more of the components of optics 200 may change a direction of the optical path. For example, the folding prism 220 may include a mirror (not shown) that may reflect light beams at an angle. An example of such a folding prism is shown in
Illumination module 38 is a source of multiple light beams. In one embodiment, illumination module is a micro lens array that divides an incoming light beam into array of light beams 225. In one embodiment, the array of light beams output by the illumination module 38 is an array of telecentric light beams. Accordingly, chief rays of the array of light beams may be parallel to each other. Unidirectional mirror or beam splitter 40 is disposed along the optical path of the array of light beams, and passes the array of light beams received from the unidirectional mirror or beam splitter 40.
In one embodiment, the confocal focusing optics are divided into a series of lens groups including a first lens group 205, a second lens group 215 and a third lens group 210. First and/or second lens groups 205, 215 may act as relay optics. The first and second lens groups 205, 215 are configured to focus the array of light beams and compensate for optical aberrations. Optical aberrations that may be corrected include shape aberrations, coma, stigmatism, and so forth. In one embodiment, the first and second lens groups 205, 215 are configured to produce an approximately rectangular field having minimal optical distortion. The first lens group 205 and second lens group 215 may have a fixed position relative to each other and to other components of the optics 200. The third lens group 210 has a variable location that may be adjusted to change a location of a curved focal surface produced by the optics 200.
The third lens group 210 is movable along the imaging axis (z axis), but has a fixed position normal to the imaging axis. A focusing setting of the focusing optics can be adjusted by moving the third lens group 210 along the imaging axis. Third lens group 210 may be adjusted to perform scanning of an object. To scan an object, the third lens group 210 may be displaced to numerous different locations (encoder positions) along the imaging axis 240, and images may be taken at each location. In one embodiment, an axial gain of the focusing optics is approximately 7×. Accordingly, a displacement of the third lens group 210 adjusts a location of a curved focal surface 230 by seven times the amount of displacement. For example, a 1 mm displacement of the third lens group 210 causes a position of the curved focal surface (also referred to as a curved focal plane) by 7 mm. This enables the optics 200 to be compact and minimizes movement during operation.
In one embodiment, second lens group 215 focuses the array of light beams 225 into prism 220, which may be a folding prism. Prism 220 may be configured to provide an appropriate refractive index (e.g., that corresponds to a refractive index of glass).
The optics 200 lack any field lens. A field lens is used to flatten a focal surface (flatten an imaging field) to achieve a flat focal plane. As shown, there is no field lens between the illumination module 38 and the unidirectional mirror or beam splitter 40. Nor is there a field lens near prism 220 or a field lens between the unidirectional mirror or beam splitter 40 and a detector (not shown). The lack of a field lens introduces numerous advantages over confocal imaging apparatuses that use field lenses. The field lens is a diverging lens that causes a radius of the lenses used for the focusing optics and/or for relay optics to be larger. This in turn increases the amount of material (e.g., glass) used in the lenses and thus increases a weight of the confocal imaging apparatus. Additionally, the larger lenses cause a thickness of the confocal imaging apparatus to be larger. For example, an example confocal imaging apparatus with a field lens includes a largest lens having a distance from an optical axis to an outer perimeter of the lens of about 15 mm. In contrast, the same confocal imaging apparatus without a field lens may include a largest lens having a distance from the optical axis to an outer perimeter of the lens of less than 15 mm (e.g., less than 13 mm or about 9 mm in embodiments).
In a confocal imaging apparatus having a field lens, the field lens may be positioned between the illumination module 38 and the unidirectional mirror or beam splitter 40. This causes a spacing between the illumination module 38 and the unidirectional mirror or beam splitter 40 to be about 7 mm. Additionally, a corresponding field lens would be placed between the unidirectional mirror or beam splitter 40 and a detector (not shown) at a distance of about 7 mm. In contrast, by eliminating the field lens, the distance 235 between the illumination module 38 and the unidirectional mirror or beam splitter 40 may be less than 7 mm (e.g., less than 5 mm or about 2 mm in embodiments). This further reduces the size of the confocal imaging apparatus.
As mentioned, if a field lens is used in a confocal imaging apparatus, then in actuality two field lenses are used. These two field lenses should be matching field lenses and should be carefully aligned to one another. This alignment can be a time consuming process. Additionally, failure to exactly align these field lenses introduces inaccuracy into the confocal imaging apparatus. Accordingly, an accuracy of the confocal imaging apparatus can be improved and an ease of manufacture for the confocal imaging apparatus can be improved by eliminating the field lens.
The lack of a field lens causes the focal surface 230 to be a curved focal surface (or other non-flat focal surface). The shape of the curved focal surface 230 may depend on the focusing setting of the focusing optics (e.g., the location of the third lens group 210). The curved focal surface may introduce significant error into the confocal imaging apparatus, which accounts for the inclusion of field lenses in prior confocal imaging apparatuses. However, embodiments of the present invention provide a field compensator (see, e.g., field compensator 92 of
As shown, the confocal focusing optics is a non-telecentric optical system. Accordingly, magnification of an imaged object may change with changes in depth and/or in changes of focal settings. However, such magnification changes (and any accompanying distortion) may be accommodated and corrected by the field compensator based on application of a field curvature model. Alternatively, the confocal focusing optics may operate in a telecentric mode, and distance-introduced magnification changes may be avoided.
The array of light beams 270 are focused onto a curved focal surface 275. Though the optics 250 are not telecentric, magnification is preserved (fixed) with changes in focusing settings because the array of light beams are collimated between first lens group 255 and second lens group 265. For optics 250, axial gain is 1×. Accordingly, a displacement of 1 mm of the second lens group 265 causes a displacement of the curved focal surface of 1 mm.
An object may be placed along the beam path to be imaged. The array of light beams 285 reflect off of the object and an array of returning light beams return back through the series of lens groups. The array of returning light beams 285 is then reflected by the unidirectional mirror (or beam splitter) 40 onto detector 68. As shown, the optics 250 lack a field lens between the unidirectional mirror or beam splitter 40 and the illumination module 38 and further lack a field lens between the unidirectional mirror or beam splitter 40 and the detector 68. Accordingly, the focal surface for the optics 250 is a curved focal surface 275.
Embodiments have been discussed herein with reference to a confocal imaging apparatus that lacks a field lens and that has a curved focal surface. However, in some embodiments the confocal imaging apparatus includes one or more field lenses and thus has a flat focal surface. For such embodiments, the confocal imaging apparatus operates in a non-telecentric mode, and magnification at a focal plane changes with changes in focusing settings of the confocal imaging apparatus.
The array of light beams 298 are focused onto flat focal plane 299. Magnification at the flat foal plane 299 changes with changes in focusing settings. The changes in magnification may introduce significant error into the confocal imaging apparatus. Accordingly, the focusing optics for some large field confocal imaging apparatuses maintain the same magnification with changes in focusing settings (e.g., with changes in a position of one or more lenses along an imaging axis). However, embodiments of the present invention provide a field compensator (see, e.g., field compensator 92 of
Various components of the confocal imaging apparatus may dissipate considerable amounts of heat relative to a size of the confocal imaging apparatus. For example, the confocal imaging apparatus may include a CMOS sensor and an FPGA, both of which may produce heat. Accordingly, internal temperatures of the confocal imaging apparatus may rise over time during use. At any given time, different portions of the confocal imaging apparatus may have different temperatures. A temperature distribution within the confocal imaging apparatus is referred to as a thermal state of the confocal imaging apparatus. The thermal state of the confocal imaging apparatus may affect various optical parameters. For example, the thermal state may cause the positions of one or more optical components to move within the confocal imaging apparatus due to expansion of the various components in accordance with thermal expansion coefficients of these components. Additionally, the refractive coefficient of one or more lens of the confocal imaging apparatus may change with changes in the thermal state. Such changes cause measurements produced by the confocal imaging apparatus to change with changes in the internal thermal state. Some regions of the confocal imaging apparatus are more sensitive to thermal change than others (e.g., due to a high optical gain). For example, some optical elements may have an axial gain of up to about 7.5 in an embodiment. For such optical elements, a 10 μm movement due to changes in the thermal state could cause up to a 75 μm shift in a measurement. Accordingly, in some embodiments, as shown in
Probing member 370 additionally includes internal target 380 secured to the anterior segment 371 of the probing member 370 within a field of view (FOV) of the probing member 370. The internal target 380 may be a rigid reflective material that will reflect light beams. The internal target 380 may be secured at a fixed position within the probing member 300. Since the internal target 380 is a part of the probing member 370, the location of the internal target 380 should remain constant. In one embodiment, the internal target 380 takes up approximately 500 μm to 1 mm of the FOV.
During measurement, an array of light rays or beams 390-392 is projected out of the anterior segment 371. As can be seen, the internal target 380 is in the path of light beams 390. Accordingly, the light beams 390 are reflected off of the internal target 380, which provides a depth (z-axis) measurement of the internal target 380. Since the internal target 380 is at a fixed position, the measured depth of the internal target 380 should not change. Accordingly, any measured change in the position of the internal target 380 reflects changes in internal optics associated with the thermal state of the confocal imaging apparatus.
The light beams 392 project through the window 376 and focus on a non-flat focal surface 310, the position of which can be changed by the focusing optics (not shown in this figure). Alternatively, the internal target 380 may be included in an imaging apparatus with a flat focal surface (e.g., an imaging apparatus with a field lens). Such an imaging apparatus may or may not be a confocal imaging apparatus. These light beams 392 may be used to measure the position of an object in the FOV of the confocal imaging apparatus. The measured change in the position of the internal target 380 can be used to correct for measurement errors caused by the thermal state. Any apparent change in the z-axis position of the internal target 380 may be used to apply an adjustment factor to other z-axis measurements of the imaged object to compensate for changes in the focusing optics caused by temperature. Additionally, a change in the z-axis position of the internal target may be used to apply an adjustment to the X and Y pixel measurements in embodiments. In one embodiment, the z-axis position of the internal target and measured points of an object are input into a thermal state compensation model to compensate for the thermal state. In one embodiment, the thermal state compensation model is a three dimensional polynomial function.
Parent beam 452 may include multiple different wavelengths, with a different wavelength being transmitted from each laser 454A-C. Thus, parent light beam 452 and one or more incident light beams in the array of light beams 458 may be composed of multiple different light components. Alternatively, each light beam in the array of light beams may include a single wavelength from the multiple wavelengths of parent beam 452. Lasers 454A-C may be arranged such that each light beam focuses on a different curved focal surface, PA, PB and PC, respectively. In the position shown in
In an alternative embodiment, only a single wavelength of light is emitted (e.g., by a single laser). Thus, parent beam 452 and the array of light beams 458 may include a single wavelength. In such an embodiment, each of the light beams in the array of light beams 458 focuses on the same curved focal surface PC. Thus in the position shown in
The confocal imaging apparatus described in embodiments herein has a non-flat (e.g., curved) focal surface. This curved focal surface introduces inaccuracies in depth measurements of points of a scanned object. For example, a first point of the object at a center of the confocal imaging apparatus' imaging field may be in focus and thus cause a highest intensity measurement at a depth Zi. However, a second point of the object at an edge of the imaging field that has a same depth as the first point may be in focus and cause a highest intensity measurement at a depth Zi+X due to the non-flat focal surface, where X represents the difference between the focal point at the center of the imaging field and the focal point at the edge of the imaging field. Thus, the non-flat imaging field will cause measurements of the first and second points to yield different depth values even though they are at the same depth. In one embodiment, calibration method 500 is performed to calibrate the confocal imaging apparatus so that the error introduced by the non-flat focal surface can be eliminated.
At block 505 of method 500, a calibration object is measured by the confocal imaging apparatus. The calibration object is a high accuracy object with known X, Y and Z coordinates for every point of the calibration object. The accuracy level of the calibration object may define the final accuracy of the confocal imaging apparatus. In one embodiment, the X, Y and Z coordinates for the calibration object are accurate and known to a level of accuracy that is a degree of magnitude higher than a final desired accuracy of the confocal imaging apparatus. For example, if the confocal imaging apparatus is to have a final accuracy to 5 microns, then the calibration object may be accurate to 0.5 microns.
Various calibration objects may be used, a few examples of which are set forth herein. One example calibration object is a sphere with a very accurate radius on an accurate X-Y-Z stage. Another example calibration object is a flat plate with a grid of horizontal and vertical lines printed on a surface of the plate. A flatness of the plate and the line spacing may be very accurate. Another example calibration object is a flat plate with circles or dots printed on a surface of the plate. The flatness of the plate and the size and spacing of the circles may be very accurate. Many other calibration objects may also be used.
Referring back to
In one embodiment, the calibration object is mounted to a calibration jig, which may precisely move the calibration object in one or more dimensions. For example, the calibration object 590 may be mounted to the calibration jig, and the calibration jig may be moved along the z-axis. In one embodiment, the calibration jig moves the calibration object in 1 mm increments, with an accuracy of 1 μm. The calibration jig may move the calibration object in such a way as to cover more than the full field of view of the confocal imaging apparatus (e.g., the calibration object may be larger than the FOV of the confocal imaging apparatus) and to cover more than the range for the depth of scanning of the confocal imaging apparatus.
In the example of the calibration object 590, the calibration object 590 may be scanned in two ways. A first scan may be performed at each depth position of the calibration object 590 using regular confocal scanning. This will provide a z-position for each dot in the coordinate system of the confocal imaging apparatus (e.g., based on the coordinates of the encoder that positions the lens). A second scan may be performed to generate an image of the dots at focus for each focal setting. The image may be used to determine an X, Y position for the center of each dot in pixel coordinates and with sub-pixel accuracy.
At block 510, the measurements of the calibration object (measurements of the calibration object's surface topology) are compared to a known surface topology of the calibration object. Each point in the calibration object (e.g., each dot in calibration object 590 having a measured x-pixel, y-pixel and encoder value) may be paired to a corresponding real world point (point in a world coordinate system) from the calibration object, where the world coordinate system corresponds to known X, Y, Z coordinates of the calibration object. For example, the X and Y coordinates for calibration object 590 would correspond to known fixed positions of the dots, and the Z coordinate for calibration object 590 would depend on a setting of a calibration jig. For each point of the calibration object, a difference between a measured depth value and a known depth value may be determined. Additionally, for each point of the calibration object, a difference between a measured X and Y position and a known X and Y position may be determined. This may be performed for each focusing setting of the confocal imaging apparatus.
At block 515, the determined differences of the multiple points may be applied to a smooth function (e.g., to a polynomial function such as a three dimensional polynomial function) that may be used to model the field curvature of the confocal imaging apparatus' non-flat focal surface. The function is referred to herein as a un-distortion function. In one embodiment, the determined differences are applied to solve for the constants in a bivariate quadratic polynomial of the form:
ZField Curvature(object)(x,y,Zoptics)=a1x2+a2y2+a3x+a4y+a5xy+a6 (1)
Where x and y are the X, Y coordinates for points on a plane normal to the imaging axis. Alternatively, a higher order polynomial may be used. The smooth function with the solved constants may then be used as an accurate field curvature model. Every parameter may be a polynomial that depends on the focusing setting (z-axis value) of the confocal imaging apparatus. This may result in an 18 parameter field curvature model if the above described bivariate quadratic polynomial is used.
Alternatively, the determined differences may be applied to solve for the constants in another smooth function (e.g., a function describing a conic shape). In such an embodiment, a generated model may have a different number of parameters (e.g., 12 parameters if a function describing a conic shape is used). Linear minimization methods (e.g., linear least square method) and/or non-linear minimization methods (e.g., Broyden-Fletcher-Goldfarb-Shanno (BFGS) method) may be applied to find the best values for the constants. As mentioned, this process may be performed for each focusing setting. This is because the amount of field curvature may change with different focusing settings of the confocal imaging apparatus. Accordingly, a separate field curvature model may be generated for each focusing setting. Alternatively, a single field curvature model may be generated that accounts for the changes to the field curvature model due to changes in the focusing setting.
In embodiments, X and Y positions are solved for at the same time that the depth is solved for. For example, differences in X and Y position at different focus settings may also be applied to solve for the constants in the smooth function. Additionally, other types of geometric correction may be solved for as well using this technique. All such geometric corrections may be solved for together. Other types of phenomena that may be corrected for using this technique include magnification change, optical distortion (e.g., non-constant magnification in x and y), optical aberrations, and so on. All such distortions may be solved for together.
At block 525, a temperature dependence of the confocal imaging apparatus (e.g., the focusing optics and of a lens housing for the focusing optics) is determined. In one embodiment, the operations of one or more of blocks 505-515 are performed at multiple temperatures over a temperature operating range of the confocal imaging apparatus to determine the temperature dependence. Changes in temperature may cause differences in the measured depth values. Accordingly, a temperature dependency may be determined and applied to the field curvature model to create a thermal state correction model. For example, the field curvature model may be modified from x, y, z=F(i, j, encoder) to x, y, z=F(i, j, encoder, Tstate), where x, y and z represent real world coordinates, i represents an x-pixel, j represents a y-pixel, encoder represents a focal setting (encoder position), and Tstate represents a thermal state. For such a model that takes into account the thermal state, an estimate of the thermal state should be obtained for each measurement. A thermal state correction model may also be generated for an imaging apparatus with a flat focal surface using the same process as described herein for an imaging apparatus with a curved focal surface.
In one embodiment, opto-mechanical simulation is performed to determine a relationship between temperature and adjustments in calibration of the focusing optics. This relationship may be used to determine a correction that may be applied to all parameters of the generated field curvature model or models, where the amount of correction is based on a current temperature.
In one embodiment, the main change in the focusing optics due to temperature is a focus shift. Curvature of the non-flat focal surface may be practically unchanged by changes in temperature. In one embodiment, a shift in focus for focusing settings may be determined by scanning one or more elements (e.g., an internal target such as internal target 380 of
In one embodiment, separate field curvature models are generated for each temperature value or range of the confocal imaging apparatus at a particular focusing setting. Alternatively, a single model may be generated for each focusing setting that accounts for changes in temperature. Alternatively, a temperature dependent adjustment factor may be determined and applied to the field curvature model or models based on a measured temperature.
In one embodiment, a simple model may be used that assumes that optical change caused by the thermal state is primarily due to a linear shift in the focal setting (e.g., a backward motion in the encoder position). For such a model, changes caused by the thermal state may be corrected by adding the difference between a current measured internal target position and a reference value to every focal setting (encoder value) before applying the un-distortion function. The simple model may have the form of:
x,y,z=F(i,j,encoder−(internal target position−reference target position)) (2)
where F is the un-distortion function, such as function (1) above.
In another embodiment, a more complex model is used that assumes internal target effects are caused by the focal shift of encoder, but in a complex way. Such a model may have the form of:
x,y,z=F(i,j,f(encoder,internal target position)) (3)
In another embodiment, a model that corrects for distortions caused by the thermal state assumes that the thermal state changes all optics by a small amount that can be linearly estimated. Such a model may have the form of:
where Fhot is the un-distortion function under a hot condition, Fcold is the un-distortion function under a cold condition, a is the internal target position in the hot condition, b is the internal target position in the cold position, and p is the measured internal target position.
At block 535, the one or more generated field curvature models for the confocal imaging apparatus are stored. The field curvature models may be stored in a memory of the confocal imaging apparatus and/or in a memory of a computing device that processes data from the confocal imaging apparatus. In one embodiment, the field curvature models are stored in a nonvolatile memory (e.g., a read only memory (ROM), FLASH, or other nonvolatile memory) of the confocal imaging apparatus. The Field curvature model (or models) may be applied to measurements of the confocal imaging apparatus to correct the error in the depth measurements that are introduced by the non-flat focal surface of the confocal imaging apparatus. If calibration information is stored in memory of the confocal imaging apparatus, then the field curvature models may be sent along with measurement data to a computing device when measurements are taken. The computing device may then use the received field curvature models to correct for the field curvature of the confocal imaging apparatus.
The confocal imaging apparatus described with reference to method 550 may have a non-flat (e.g., curved) focal surface or a flat focal plane. Moreover, the confocal imaging apparatus described with reference to method 550 has focusing optics that are configured so that changes in a focusing setting cause a change in magnification at the focal surface or focal plane. This change in magnification introduces inaccuracies in X and Y position measurements of points of a scanned object. For example, a point of the object might be measured to have a first X and Y position at a first focusing setting, but might be measured to have a second X and Y position at a second focusing setting. Thus, the magnification changes will cause measurements to yield different X, Y values as the focusing setting changes. In one embodiment, calibration method 550 is performed to calibrate the confocal imaging apparatus so that the inaccuracies introduced by the changes in magnification can be eliminated.
At block 555 of method 500, a calibration object is measured by the confocal imaging apparatus. The calibration object is a high accuracy object with known X, Y and Z coordinates for every point of the calibration object. The accuracy level of the calibration object may define the final accuracy of the confocal imaging apparatus. In one embodiment, the X, Y and Z coordinates for the calibration object are accurate and known to a level of accuracy that is a degree of magnitude higher than a final desired accuracy of the confocal imaging apparatus. For example, if the confocal imaging apparatus is to have a final accuracy to 5 microns, then the calibration object may be accurate to 0.5 microns. Any of the calibration objects described with reference to
The calibration object is measured at each focusing setting (encoder value) of the confocal imaging apparatus. For some types of calibration objects (e.g., the sphere), the calibration object is moved to multiple different X, Y positions for each focusing setting and/or to multiple different X, Y, Z positions for each focusing setting. For other types of calibration objects (e.g., the plates), the calibration object may be moved to multiple different Z positions for each focusing setting. Measurements may be taken for each position of the calibration object. Based on these measurements, a list of coordinates is collected in both the calibration object space (e.g., real world) and in the sensor/optics space (e.g., virtual space). In the calibration object space, each set of coordinates for a point of the object has an Xobj, Yobj and Zobj coordinate. These coordinates are known to be accurate due to the known information about the calibration object. In the sensor/optics space, each set of coordinates for a point of the object includes an Xpix, Ypix, Zoptics coordinate, where Xpix and Ypix are determined based on the pixel detecting the point and Zoptics is the lens position of the focusing optics (e.g., the focusing setting).
At block 560, the measurements of the calibration object (measurements of the calibration object's surface topology) may be compared to a known surface topology of the calibration object. For each point of the calibration object, a difference between a measured depth value, X value and/or Y value and a known depth value, X value and/or Y value may be determined. This may be performed for each focusing setting of the confocal imaging apparatus.
At block 562, it is determined whether the focusing optics have a curved focal surface. If the focusing optics do have a curved focal surface, the method proceeds to block 565. Otherwise the method proceeds to block 570.
At block 565, the determined differences of the multiple points for the X, Y and/or Z coordinates may be applied to a smooth function (e.g., to a polynomial function such as a three dimensional polynomial function) that may be used to model the field curvature of the confocal imaging apparatus' non-flat focal surface. In one embodiment, the determined differences are applied to solve for the constants in a bivariate quadratic polynomial of the form:
ZField Curvature(object)(x,y,Zoptics)=a1x2+a2y2+a3x+a4y+a5xy+a6 (5)
Where x and y are the Xpix, Ypix coordinates in the sensor space. Alternatively, the determined differences may be applied to solve for the constants in another smooth function (e.g., a function describing a conic shape), such as a polynomial of higher order. The smooth function with the solved constants may then be used for an accurate field curvature model.
At block 570, the determined differences of the multiple points for the X, Y and/or Z coordinates may be applied to a smooth function (e.g., to a polynomial function such as a three dimensional or higher dimensional polynomial function) that may be used to model the changes in magnification of the confocal imaging apparatus on an x-axis caused by changes in the focusing setting (e.g., changes in the Zoptics value). In one embodiment, the determined differences are applied to solve for the constants in a bivariate quadratic polynomial of the form:
XObject(x,y,Zoptics)=b1x2+b2y2+b3x+b4y+b5xy+b6 (6)
Where x and y are the Xpix, Ypix coordinates in the sensor space. Alternatively, the determined differences may be applied to solve for the constants in another smooth function (e.g., in another three dimensional polynomial function, such as a function describing a conic shape). The smooth function with the solved constants may then be used as an accurate magnification compensation model for the X coordinate.
At block 575, the determined differences of the multiple points for the X, Y and/or Z coordinates may be applied to a smooth function (e.g., to a polynomial function such as a three dimensional polynomial function) that may be used to model the changes in magnification of the confocal imaging apparatus on a y-axis caused by changes in the focusing setting (e.g., changes in the Zoptics value). In one embodiment, the determined differences are applied to solve for the constants in a bivariate quadratic polynomial of the form:
YObject(x,y,Zoptics)=c1x2+c2y2+c3x+c4y+c5xy+c6 (7)
Where x and y are the Xpix, Ypix coordinates in the sensor space. Alternatively, the determined differences may be applied to solve for the constants in another smooth function (e.g., in another three dimensional polynomial function, such as a function describing a conic shape). The smooth function with the solved constants may then be used as an accurate magnification compensation model for the Y coordinate.
Blocks 565, 570 and 575 have been described as three separate operations. However, in some embodiments a single operation may be performed to solve for each of the x-coordinate, the y-coordinate and the z-coordinate. For example, an un-distortion function having the following form may be solved to determine the x, y and z coordinates.
FX(x,y,z)=a0+a1x+a2y+a3z+a4x2+a5y2+a6z2+ . . . +aixy+ . . . +ajxnymzk
FY(x,y,z)=b0+b1x+b2y+b3z+b4x2+b5y2+b6z2+ . . . +bixy+ . . . +bjxnymzk
FZ(x,y,z)=c0+c1x+c2y+c3z+c4x2+c5y2+c6z2+ . . . +cixy+ . . . +cjxnymzk (8)
where FX, FY and FZ are the functions whose results in world coordinates are to be solved for, x and y are pixel coordinates measured by the confocal imaging apparatus, z is a focal setting (e.g., encoder coordinates corresponding to a focal setting), ai, bi and ci are learned parameters, and n, m and k are the maximal degree of the nominal. The function may be selected to minimize a mean square error between the world coordinates and the found positions after the function transformation. Outlier positions may be detected and removed before fitting. In one embodiment, a number of non-zero parameters is constrained.
At block 580, one or more optics correcting models are generated based on the first second and third polynomial functions (or other smooth functions), such as those represented in equations 5-8. Every parameter for equations 5-8 may be a polynomial that depends on the focusing setting (z-axis value) of the confocal imaging apparatus. In one embodiment, each parameter is modeled as a quadratic change to the Zoptics (focusing setting). For example, parameter a may be a parameter having a form:
a1(ZOptics)=A+B*ZOptics+C*ZOptics2 (9)
Parameters a2-a6, b1-b6 and c1-c6 may be similarly represented. This may result in a 54 parameter model that corrects for full curvature, magnification and distortion of the field of view (FOV).
Linear minimization methods (e.g., linear least square method) and/or non-linear minimization methods (e.g., Broyden-Fletcher-Goldfarb-Shanno (BFGS) method) may be applied to find the best values for the constants at each of blocks 565, 570 and 575. As mentioned, these processes may be performed for each focusing setting. This is because the amount of field curvature and magnification may change with different focusing settings of the confocal imaging apparatus. Accordingly, a separate model may be generated for each focusing setting. Alternatively, a single model may be generated that accounts for the changes to the model due to changes in the focusing setting. Note that temperature dependence may also be determined and included in the model as described with reference to block 525 of method 500. In one embodiment, a temperature dependence is determined, and a model that corrects for thermal state is created, as discussed above with reference to method 500.
At block 585, the one or more generated models for the confocal imaging apparatus are stored. The models may be stored in a memory of the confocal imaging apparatus and/or in a memory of a computing device that processes data from the confocal imaging apparatus. In one embodiment, the models are stored in a nonvolatile memory (e.g., a read only memory (ROM), FLASH, or other nonvolatile memory) of the confocal imaging apparatus. The model (or models) may be applied to measurements of the confocal imaging apparatus to correct the error in the depth measurements that are introduced by the non-flat focal surface as well as to correct for inaccuracies caused by changes in magnification. If calibration information is stored in memory of the confocal imaging apparatus, then the models may be sent along with measurement data to a computing device when measurements are taken. The computing device may then use the received models to correct for the field curvature and/or magnification changes of the confocal imaging apparatus.
At block 605 of method 600, processing logic receives intensity measurements generated by pixels of a detector of a confocal imaging apparatus. The detector may have a two-dimensional array of pixels, and each pixel may receive a particular light beam of an array of light beams directed at the detector. The array of light beams may be an array of returning light beams that have been reflected off of a surface of the imaged three dimensional object. Thus, each pixel of the detector is associated with a particular point of the three dimensional object and provides intensity measurements for an associated returning light beam from the array of returning light beams.
Each received intensity measurement is associated with a particular focusing setting of the confocal imaging apparatus. Intensity measurements may be received over a range of focusing settings. At block 620, processing logic determines, for each pixel, a focusing setting of the confocal imaging apparatus that provides a maximum measured intensity.
A relative distance between a probe of the confocal imaging apparatus and a focal point of the confocal imaging apparatus may be known for each focusing setting (encoder value). A point of the imaged object is known to be in focus (e.g., at the focal point) when a measured intensity for that point is maximal. Accordingly, at block 630 processing logic determines, for each pixel, a depth of a point of the three dimensional object associated with that pixel that corresponds to the focusing setting that yielded the maximal intensity. If the imaging apparatus includes an internal target in the FOV of the imaging apparatus, then some pixels will be associated with points on the internal target. Accordingly, a depth of the points of the internal target may also be determined.
As discussed previously herein, the non-flat focal surface and/or magnification changes of the confocal imaging apparatus introduce an error in the depth measurements and/or in the X, Y coordinate measurements. Accordingly, at block 640 processing logic adjusts the determined depths of points of the imaged three dimensional object based on applying the determined focusing settings for the pixels associated with those points to a field curvature model. Processing logic may additionally or alternatively determine X, Y coordinates of the points based on applying the determined focusing settings to the field curvature model or other model. One or more field curvature models and/or other models may be used. For example, a particular field curvature model and/or other model may be associated with each focusing setting. An appropriate field curvature model may be identified based on the focusing setting at which a point on the object came into focus. A particular depth adjustment for that point may then be determined by providing the X, Y coordinates of the pixel into the determined field curvature model. Alternatively, a single field curvature model may be used, and the X, Y coordinates and focusing setting may be input into the field curvature model to determine the depth displacement. In one embodiment, a temperature of the focusing optics is also measured and/or a thermal state is otherwise determined (e.g., using an internal target position), and an additional depth adjustment factor (and/or other optical adjustment) is determined based on the temperature (e.g., using a thermal state compensation model). This additional depth adjustment factor (and/or additional optical adjustment) may then be applied to the measured depths (and/or X and Y coordinates) of all points. In one embodiment, a single model is used that compensates for both the thermal state and field curvature.
At block 650, processing logic may determine a shape (e.g., surface topology) of the three dimensional object based on the adjusted depths and/or x and y coordinates. Processing logic may then create an accurate virtual three dimensional model of the imaged object.
The example computing device 700 includes a processing device 702, a main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), a static memory 706 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 728), which communicate with each other via a bus 708.
Processing device 702 represents one or more general-purpose processors such as a microprocessor, central processing unit, or the like. More particularly, the processing device 702 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 702 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processing device 702 is configured to execute the processing logic (instructions 726) for performing operations and steps discussed herein.
The computing device 700 may further include a network interface device 722 for communicating with a network 764 or other device. The computing device 700 also may include a video display unit 710 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), and a signal generation device 720 (e.g., a speaker).
The data storage device 728 may include a machine-readable storage medium (or more specifically a non-transitory computer-readable storage medium) 724 on which is stored one or more sets of instructions 726 embodying any one or more of the methodologies or functions described herein. A non-transitory storage medium refers to a storage medium other than a carrier wave. The instructions 726 may also reside, completely or at least partially, within the main memory 704 and/or within the processing device 702 during execution thereof by the computer device 700, the main memory 704 and the processing device 702 also constituting computer-readable storage media.
The computer-readable storage medium 724 may also be used to store a field compensator 750 which may correspond to field compensator 92 of
It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent upon reading and understanding the above description. Although embodiments of the present invention have been described with reference to specific example embodiments, it will be recognized that the invention is not limited to the embodiments described, but can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
This patent application is a continuation application of U.S. patent application Ser. No. 17/740,113 filed May 9, 2022, which is a continuation application of U.S. patent application Ser. No. 17/492,453 filed Oct. 1, 2021, which is a continuation application of U.S. patent application Ser. No. 17/161,503 filed Jan. 28, 2021, which is a continuation application of U.S. patent application Ser. No. 16/823,156 filed Mar. 18, 2020, which is a continuation application of U.S. patent application Ser. No. 16/708,293 filed Dec. 9, 2019, which is a continuation application of U.S. patent application Ser. No. 16/396,484 filed Apr. 26, 2019, which is a continuation of U.S. patent application Ser. No. 16/286,437, filed Feb. 26, 2019, which is a continuation application of U.S. patent application Ser. No. 15/610,515, filed May 31, 2017, which is a divisional application of U.S. patent application Ser. No. 14/825,173, filed Aug. 13, 2015, which claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 62/037,778, filed Aug. 15, 2014, all of which are herein incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7313264 | Crampton | Dec 2007 | B2 |
| 7978346 | Riza | Jul 2011 | B1 |
| 20060102833 | Eiff | May 2006 | A1 |
| 20090224174 | Netz et al. | Sep 2009 | A1 |
| 20100002313 | Duparre et al. | Jan 2010 | A1 |
| 20110134017 | Burke | Jun 2011 | A1 |
| 20150097931 | Hatzilias | Apr 2015 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20230293272 A1 | Sep 2023 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62037778 | Aug 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14825173 | Aug 2015 | US |
| Child | 15610515 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17740113 | May 2022 | US |
| Child | 18136831 | US | |
| Parent | 17492453 | Oct 2021 | US |
| Child | 17740113 | US | |
| Parent | 17161503 | Jan 2021 | US |
| Child | 17492453 | US | |
| Parent | 16823156 | Mar 2020 | US |
| Child | 17161503 | US | |
| Parent | 16708293 | Dec 2019 | US |
| Child | 16823156 | US | |
| Parent | 16396484 | Apr 2019 | US |
| Child | 16708293 | US | |
| Parent | 16286437 | Feb 2019 | US |
| Child | 16396484 | US | |
| Parent | 15610515 | May 2017 | US |
| Child | 16286437 | US |
