Patent Application
20150293342
1. Field of the Invention
The present invention relates to an image apparatus of, for example, a microscope for capturing an object image of a sample or the like.
2. Description of the Related Art
In a microscope for acquiring digital images by capturing of an image of a sample, such as a human tissue, an image capturing optical system with high resolution is being developed to determine a fine structure of the sample. However, higher resolution processing of an image capturing optical system generally decreases a depth of focus.
On the other hand, a surface shape of a sample is not necessarily flat, and usually has irregularities. If the thickness of a cover glass holding the sample in a prepared slide is uneven, large undulation appears in the surface shape of the sample. If the temperature near a sample changes, the focus state of an image capturing optical system may change.
As disclosed in Japanese Patent Laid-Open No. (“JP”) 2006-343573, conventional microscopes use a method for capturing images of an entire sample while moving, with respect to the sample, an image capturing optical system that has a field of view capable of capturing an image of only small areas of the sample and while adjusting the focus of an image capturing optical system with respect to each of the small areas. Therefore, the microscopes has a disadvantage that capturing time gets longer.
On the other hand, JP2009-3016 discloses a method for enabling a quick image capturing of an entire sample by enlarging the field of view of an image capturing optical system and capturing a plurality of small areas of the sample together using a plurality of image sensors.
However, in an image capturing optical system, if the resolution is increased while enlarging the field of view, the depth of focus decreases as mentioned above. Therefore, it is difficult to focus over entire field of view on a sample having a surface shape including the irregularities and the undulation.
The present invention provides an image capturing apparatus and an image capturing method that enables image capturing with high resolution in focusing on a sample (object) over a large field of view.
An image capturing apparatus as one aspect of the present invention performs image capturing of an object from its surface side, and includes an image capturing optical system, a deformable mirror disposed in an intermediate imaging area of the image capturing optical system, and a controller configured to divide the object into a plurality of object areas, calculate an approximate surface approximating a surface shape of the object in each of the object areas using shape data indicating the surface shape, and deform the deformable mirror to a shape corresponding to the approximate surface.
A method for performing image capturing of an object from its surface side through an image capturing optical system as another aspect of the present invention includes obtaining shape data indicating a surface shape of the object, calculating an approximate surface approximating the surface shape in each object area using the shape data by dividing the object into a plurality of object areas, and capturing the image by deforming a deformable mirror to a shape corresponding to the approximate surface, the deformable mirror disposed in an intermediate image area of the image capturing optical system.
Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings).
A description will now be given of embodiments according to the present invention.
The measuring system 100 is configured by a measuring illuminator 101, a measuring stage 102, a measuring optical system 104, and a measuring part 105.
The measuring illuminator 101 includes an illumination optical system that introduces light from an optical source (not shown) to a sample as an object in a preparation 103 disposed on the measuring stage 102. The preparation 103 is configured by a slide glass, a sample, such as a tissue, that is an observation object located on the slide glass, and a cover glass overlaid on the slide glass so as to cover (protect) the sample. The measuring stage 102 holds the preparation 103, and is driven so as to adjust a position of a preparation 103 (sample) with respect to the measuring optical system 104. The measuring part 105 measures the size of the sample, the surface shape, the thickness of the cover glass, and the like by receiving through the measuring optical system 104 light that is emitted from the measuring illuminator 101 and is reflected by or passing through the preparation 103.
The measuring optical system 104 desirably has a large field of view that covers the entire sample, and the resolving power may be low. The size of the sample can be measured by a general method, such as the binarization of a luminance distribution of a sample image formed by the measuring optical system 104 or the detection of an outline of the sample image. Moreover, the surface shape of the sample may be measured using light reflected by the sample or using an interferometer. For example, a method of utilizing a triangulation method (JPH6-011341) or a method of measuring a difference of a distance of a laser light reflected by a glass interface using a confocal optical system (JP2005-98833) may be utilized. The thickness of the cover glass can be measured using a laser interferometer or the like. The measuring part 105 transmits, to a controller 400, data indicating the measurement result.
After the measurement, the preparation 103 mounted on the measuring stage 102 is conveyed on the image capturing stage 302 by adsorption, grip, or the like due to a sample conveying equipment (not shown) (hereinafter, numerical sign of the preparation is represented by 303). A configuration where the measuring stage 102 moves and functions as the image capturing stage 302 may be adapted.
The image capturing stage 302 can move in parallel to X-axis direction and Y-axis direction orthogonal to an image capturing optical axis (Z-axis direction among X-, Y-, and Z-axis directions in
The microscope 300 performs image capturing of the sample in the preparation 103 from the surface side. The microscope 300 is configured by an image capturing illuminator 301, the above-mentioned image capturing stage 302, an image capturing optical system 304, an image capturing part 305, and a controller 400. The microscope 300 includes at least one of a temperature sensor 308 and a focus measuring part (focus measurer) 309.
The image capturing illuminator 301 includes an optical source 201, and an illumination optical system 202 that introduces, to the preparation 303 mounted on the image capturing stage 302, light emitted from the optical source 201. A halogen lamp, a xenon lamp, a LED, or the like can be used as the optical source 201. The image capturing stage 302 holds the preparation 303, and is driven so as to adjust the position of the preparation 303 with respect to the image capturing optical system 304.
The temperature sensor 308 is disposed on the image capturing stage 302 (or near the image capturing stage 302). The temperature sensor 308 measures the temperature near the preparation 303. The temperature sensor 308 may be disposed between the cover glass and the slide glass in the preparation 303 or in the image capturing optical system 304. A plurality of temperature sensors may be disposed at the above-mentioned plurality of positions.
The focus measuring part 309 measures a position of the preparation 303 in the optical axis direction of the image capturing optical system 304. In particular, the focus measuring part 309 monitors one or plural positions (specific point) arbitrarily set in the optical axis direction in the preparation 303. As a result, it is possible to detect how much the specific point in the optical axis shifts from a focus reference position of the microscope 300 (for example, best focus surface of the image capturing optical system 304), including shifts due to temperature change. Based on the shift amount and the data of the surface shape of the sample or the thickness of the cover glass, which is measured by the measuring system 100, the amount of change of focus positions over the entire preparation 303 (hereinafter referred to as “focus shift”) can be obtained.
In this embodiment, the focus measuring part 309 is disposed outside the image capturing optical system 304, but may be inside the image capturing optical system 304.
The image capturing optical system 304 forms an optical image of a sample illuminated on a plane A of the image capturing stage 302 (image captured plane) on the image capturing plane B with high resolution. The image capturing optical system 304 includes an after-mentioned deformable mirror inside.
The controller 400 calculates data of the surface shape of the sample or the thickness of the cover glass, which is measured by the measuring system 100, and the data indicating change of the temperature measured by the temperature sensor 308. The focus shift is compensated by deforming the shape of the reflection surface of the deformable mirror based on the focus shift amount.
The image capturing part 305 receives a sample image (optical image) that light reflected by or passing through the sample in the preparation 303 forms on the image capturing plane B by transmitting the image capturing optical system 304. The image capturing part 305 includes an image sensor 306 of a CCD sensor, a CMOS sensor, or the like and an electric circuit for driving the image sensor 306, and the image sensor 306 photoelectrically converts the sample image. By processing an electrical signal (imaging signal) output from the image sensor 306, image data of the sample can be obtained. The microscope 300 divides the sample into a plurality of areas (object area: hereinafter referred to as “divided sample area”), the image sensor 306 corresponding to a part of the plurality of divided sample areas is disposed along the image capturing plane B.
The focus shift that is not enough to compensate only by deformation of the reflection surface can be compensated by singly or simultaneously changing (adjusting) the position of the deformable mirror or the image sensor 306 in the optical axis direction of the image capturing optical system 304. This position adjustment amount in the optical axis direction of the deformation mirror and the image sensor 306 is referred to as focus offset amount.
Next, a description will be given of a procedure (image capturing method) of acquiring image data of a sample with reference to a flowchart shown in
In step S102, the measuring part 105 receives transmitted light (or reflected light) from a sample through the measuring optical system 104, and thereby the intensity of the transmitted light (or reflected light) and the position (coordinates) in the thickness direction of the sample are measured.
In step S103, the measuring part 105 transmits data indicating the measurement result (shape data) to the controller 400.
In step S104, the controller 400 calculates, using the shape data transmitted from the measuring part 105, an approximate curved surface (expression indicating the approximate curved surface) as an approximate surface approximated to the surface shape of the sample. While this approximate curved surface is calculated, in step S105, the preparation 103 is conveyed from the measuring stage 102 to the image capturing stage 302 via a sample conveying part. The image capturing stage 302 sets the sample at an image capturing position in X-axis and Y-axis directions.
In step S106, the temperature sensor 308 measures the temperature near the preparation 303 (or near the image capturing optical system 304). In addition to or instead of the temperature measurement, the focus measuring part 309 may measure a position of a specific point of the preparation 303 in the optical axis direction (focus position). In step S107, the temperature sensor 308 or (and) the focus measuring part 309 transmits data of the measured temperature or (and) the focus position to the controller 400.
In step S108, the controller 400 converts the temperature measured by the temperature sensor 308 into the focus shift amount of the entire preparation 303 by calculation. Moreover, if a focus position at a specific point is measured by the focus measuring part 309, the measured focus position is converted into the focus shift amount of the entire preparation 303 by calculation. The focus shift amount may be a uniform value regardless of the position (XY coordinate) in X-axis and Y-axis directions, and may be a value expressed by quadric function of the XY coordinate.
Next, in the same step, the controller 400 derives a new approximate curved surface by adding the focus shift amount to the coefficients of the expression of the approximate curved surface calculated in step S104. Furthermore, the controller 400 calculates a deformation amount of the reflection surface of the deformable mirror (hereinafter referred to as “mirror compensation amount”) and a focus offset amount, based on the new approximate curved surface.
In step S109, the controller 400 deforms the deformable mirror based on the mirror compensation amount. Further, the controller 400 adjusts a position of at least one of the image sensor 306 and the deformable mirror in the optical axis direction according to the focus offset amount.
In step S110, the image capturing illuminator 301 introduces illumination light to the preparation 303 (sample) on the image capturing stage 302. The image sensor 306 of each image capturing part 305 performs image capturing of a divided sample area corresponding to the image capturing part 305 in the entire sample. The image capturing part 305 transmits an image capturing signal output from the image sensor 306 to an image processing part (not shown). The image processing part generates image data by processing the image capturing signal. The image data is transmitted to a storage inside or outside the microscope 300 and is stored in it.
In step S111, the controller 400 determines whether the image capturing is performed in all of the plurality of divided sample areas. At this time, by preliminarily setting the number of times of the image capturing of all of the plurality of divided sample areas, it may be determined whether the actual number of times of the image capturing has reached the preliminarily set number of times of the image capturing. When the image capturing has not been performed in all of the plurality of divided sample areas, the controller 400 moves the image capturing stage 302 in at least one of X-axis and Y-axis directions to capture images of divided sample areas that have not been captured yet, and repeats the processings of step S106 to step S110. In other words, the controller 400 repeats the image capturing while changing the divided sample area, whose an image is captured, so as to obtain the image data of the entire sample, based on the size information of the entire sample transmitted from the measuring part 105.
These will be described with reference to
In an example of this figure, by performing the image capturing four times while changing four divided sample areas to be captured, that is to say the center of the field of view of the image capturing optical system 304, the image of the entire sample (sixteen divided sample areas) can be captured. 1 denotes the center of the field of view of the image capturing optical system 304 in first image capturing. Similarly, 2, 3, and 4 denote the centers of the fields of view of the image capturing optical system 304 in second, third, fourth image capturing.
After the image capturing in all of the plurality of divided sample areas is finished by the above method, in step S112, the controller 400 causes the image processing part to generate image data of the entire sample by combining the image data of all of the divided sample areas. The image processing part performs image processings, such as gamma correction, noise elimination, and compression, to the combined image data.
In step S113, the controller 400 causes the above-mentioned storage to store the image data of the entire sample, generated in the image processing part.
A description will be given of a configuration of the image capturing optical system 304 in detail with reference to
The image capturing optical system 304 divides an imaging plane (intermediate imaging plane) C of the first imaging optical system 3041 into nine areas. An image (intermediate image) of one area among the nine areas is reimaged on the image capturing plane B by the second imaging optical system 3043. In other words, the image captured plane A, the image capturing plane B, and the intermediate imaging plane C are conjugated with each other. The other intermediate imaging plane may be located on an imaging captured plane A side further than the intermediate imaging plane C.
The same number of the deformable mirrors (hereinafter also simply referred to as mirror) 3042 as the image sensor 306 are disposed in the intermediate imaging area including the intermediate imaging plane C and the vicinity thereof. As shown in
The field of view (image circle) on the intermediate imaging plane C of the first imaging optical system 3041 is represented by a large circle in
In
Light reflected by the mirror 3042 transmits the second imaging optical system 3043, and forms an image on a second imaging point B1 on the image capturing plane B and enters the image sensor 306.
Similarly, light from points A2, A3, A4 on the image captured plane A forms images on the first imaging points C2, C3, C4, and further forms images on the second imaging points B2, B3, B4 (however, A3, A4, B3, B4 are not shown).
As an example of a detection method of a focus state used for the focus position measurement in the focus measuring part 309,
Light emitted from an optical source 3091 is converted into a parallel light flux by a collimate lens 3092. Half of the parallel light flux is shield by a light shielding plate 3093, and the rest enters a half mirror 3094. Light that has entered the half mirror 3094 is collected through the first imaging optical system 3041 on the image captured plane A. The light is reflected by the image captured plane A, and then enters the half mirror 3094 through a light path opposite to an incident light path across the optical axis (upper side in
In the case of performing the focus position measurement using the above method, it is particularly preferable to use non-captured area in the field of view of the first imaging optical system 3041 as an area for the focus position measurement as mentioned above. Thereby, the optical source 3091 for the focus position measurement, an optical system (collimator lens 3092, half mirror 3094, and imaging lens 3095), and the light receiving element 3096 can be easily disposed.
In order to control the shape configured as above of the reflection surface of the mirror 3042 configured as above, as mentioned above, the controller 400 calculates an approximate curved surface approximating the surface shape (undulation shape) of the sample using the shape data from the measuring part 105. The quadric surface as the approximate curved surface is calculated by the following expression (1).
z=B
1
x
2
+B
2
xy+B
3
y
2
+B
4
x+B
5
y+B
6 (1)
Sometimes high order undulation components of third order or more is included in the surface shape of the actual sample, but it may be ignored because the amount of the undulation component of third order or more is small. The focus shift caused by the change in temperature is sufficient only by approximating low order undulation components of second order or less. Furthermore, by approximating the surface shape of the sample as quadric surface, the deformation of the reflection surface of the mirror 3042 can be easily controlled, and the number of the actuators (
The controller 400 deforms the reflection surface of the mirror 3042 according to the approximate curved surface of the surface shape of the sample, calculated by the above-mentioned method. In other words, the reflection surface of the mirror 3042 is deformed to a shape corresponding to the approximate curved surface. The controller 400 adjusts a position of the image sensor 306 or the mirror 3042 in the optical axis direction to compensate the focus shift.
A description will be given of the calculation processing of the approximate curved surface in step S104 in the flow chart shown in
In step S201, the controller 400 obtains the surface shape map, where the height position (Z position) to each XY coordinate is expressed in (X, Y, Z) form, using the measuring part 105.
In step S202, the controller 400, as shown in
The size of each divided sample area on the sample is the size of the image sensor 306 divided by the magnification of the image capturing optical system 304. For example, if the magnification m1 of the first imaging optical system is 5 and the magnification m2 of the second imaging optical system is 2, the magnification of the whole image capturing optical system 304 is 10. If the image sensor 306 is a square whose one side is 32.5 mm, the length of one side of each divided sample area on the sample is 32.5 mm/10=3.25 mm. The field of view of the first imaging optical system 3041 on the side of the image captured plane A is circular and the diameter is 14.1 mm, and the length of one side of the square inscribed inside the field of view is 10 mm. On the other hand, the field of view of the second imaging optical system 3043 on the side of the image captured plane A is circular and the diameter is 4.6 mm, and the length of one side of the square inscribed inside the field of view is 3.25 mm.
The Z position to XY coordinate (xj, yj) in each divided map area is represented by (xj, yj, zj). In step S203, the controller 400 calculates a quadric surface approximating the surface shape of the sample. At this time, a quadric surface that is approximating the surface shape in each of the divided sample areas (area number is represented by i) and is indicated by the following expression (1′) is calculated by the least-squares method using Z position data of the divided map area corresponding to the divided sample area.
z=B
1(i)x2+B2(i)xy+B3(i)y2+B4(i)x+B5(i)y+B6(i) (1′)
where i=1, 2, 3, . . . , in is satisfied.
The coefficients B1(i),B2(i), . . . ,B6(i) are calculated in each of all of the divided sample areas.
In
Next, a description will be given of a process of determining the mirror compensation amount and the focus offset amount in step S108 of the flow chart in
The controller 400, as mentioned above, converts the temperature input in step S107 into the focus shift amount of the entire sample by calculation. Alternatively, the controller 400 converts a focus position of a specific point of the sample measured by the focus measuring part 309 into the focus shift amount of the entire sample, by calculation. These focus shift amounts may be an even value in XY coordinate, or may be values represented by the quadratic function regardless of XY coordinate.
The controller 400 adds the focus shift amounts to the coefficient of the approximate curved surface obtained in step S104, and calculates a new approximate curved surface. If the focus shift amounts are an even value regardless of XY coordinate, it is required to merely add the value to only a constant term of the approximate curved surface. Moreover, if the focus shift amounts are values represented by the quadratic function in XY coordinate, it is required to merely respectively add the values to the coefficients of terms corresponding to the approximate curved surface (quadric surface).
The controller 400 calculates the mirror compensation amount and the focus offset amount from the expression of the approximate curved surface that is newly obtained. In particular, as mentioned below, it calculates the mirror compensation amount from terms other than the constant term of the expression of the approximate curved surface, and calculates the focus offset amount from the constant term.
The mirror compensation is half of B1(i) x2+B2 (i) xy+B3 (i) y2+B4 (i) x+B6 (i) y. The deformation amount (actual mirror deformation amount) of the reflection surface of the actual mirror 3042 is an amount obtained considering longitudinal magnification m12 obtained from the magnification m1 of the first imaging optical system 3041. Namely, the actual mirror compensation amount zmj at sample points j=1, . . . , nj in the divided sample area i is calculated from an approximate quadric surface as the following expression (2).
zm
j
={B
1(i)xj2+B2(i)xjyj+B3(i)yj2+B4(i)xj+B5(i)yj}·m12/2 (2)
Next, the controller 400 calculates the focus offset amount with respect to the divided sample area i.
fs(i)=B6(i)·(m1m2)2 (3)
The focus offset amount is calculated using B6(i) calculated by expression (1). The focus offset amount fs(i) of the image sensor 306 when only an image sensor 306 is moved in the optical axis direction is calculated by the following method. On the image capturing plane B, the magnification of the entire image capturing optical system 304 is calculated as (m1m2) from the magnification m1 of the first imaging optical system 3041 and the magnification m2 of the second imaging optical system 3043. On an image capturing plane B, the change amount of the focus position on the image captured plane A is an amount multiplied by the lateral magnification of the image capturing optical system 304 in a direction orthogonal to the optical axis, and is an amount multiplied by the longitudinal magnification of the image capturing optical system 304 in a direction parallel to the optical axis, and in other words the change amount is an amount multiplied by the square of the magnification. Accordingly, in the optical axis direction, as shown in the following expression (3), an amount where B6(i) is multiplied by (m1m2)2 is the focus offset amount fs(i) of the image sensor 306.
fs(i)=B6(i)·(m1m2)2 (3)
On the other hand, the focus offset amount fm(i) of the mirror 3042 when only the mirror 3042 is moved in the optical axis direction satisfies the following expression (4).
fm(i)=B6(i)·m12/2 (4)
In the case that both of the image sensor 306 and the mirror 3042 are moved in the optical axis direction by independently controlling the focus offset amounts of the image sensor 306 and the mirror 3042, these focus offset amounts fs(i), fm(i) may be determined so as to satisfy the following expression (5).
B
6(i)=fs(i)/(m1m2)2+2fm(i)/m12 (5)
A configuration where the second imaging optical system 3043 is moved in a direction orthogonal to the optical axis of the second imaging optical system 3043 along with the movement of the mirror 3042 may be incorporated.
The correction residual error Zerror in XY coordinate (xj,yj) in each divided map area is calculated using the focus offset amounts fs(i), fm(i) and the actual mirror deformation amount zmj by the following expression (6). This correction residual error is converted into the amount on the sample (image captured plane A) by calculation.
Zerror=({zj−[2(zmj+fm(i))/m12+fs(i)/(m1m2)2]}2)1/2 (6)
In addition, the magnifications of the first and second imaging optical systems 3041, 3043 used when calculating the results shown in
The average and the value of 36 of the standard score σ of the correction residual error are very small amount, and are restrained within the depth of focus.
In this embodiment, the reflection surface of the deformable mirror 3042 is deformed based on the approximate curved surface of a surface shape (undulation shape) of a sample, and thereby the image capturing can be performed in a state where all points on the surface of the sample are contained within the depth of focus of the image capturing optical system 304.
Next, as second embodiment of the present invention, a description will be given of a case using a plane surface as an approximate curved surface approximating a surface shape of a sample. If B1=B2=B3=0 is satisfied in the expression (1) of first embodiment, the surface shape of the sample is approximated as a plane surface.
When the surface shape is approximated as a plane surface, the actual mirror deformation amount zmj is an amount to which the inclination of the plane surface is adjusted by the expression (2). The inclination on the mirror 3042 is an amount where the inclination of the sample is multiplied by (the longitudinal magnification of the first imaging optical system 3041)/(lateral magnification), and therefore the magnification is (m1)2/m1=m1. The actual inclination amount of the mirror 3042 is half of its magnification. The focus offset amount is calculated using the expressions (3) to (5) explained in first embodiment.
Under the same conditions as first embodiment, an approximate plane surface is calculated for the divided sample area i in the case where there is the undulation shown in
In addition, the magnifications of the first and second imaging optical systems 3041, 3043 in the calculation of the results shown in
The average and the value of 3σ of the standard score σ of the correction residual error shown in
In each embodiment, the image capturing is performed via the deformable mirror having a shape close to a surface shape of an object, and thereby the image capturing can be performed with high resolution in focusing on an object over a large field of view.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2014-081928, filed on Apr. 11, 2014, which is hereby incorporated by reference wherein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2014-081928 | Apr 2014 | JP | national |
