Patent Application
20050163398
1. Field of the Invention
The present invention relates to an image processing apparatus and, in particular, to a microscopic image joining apparatus configured to positively join a microscopic image of a higher resolution and wider visual field with the use of a plurality of specimen images acquired in a divided way.
2. Description of the Related Art
Where a specimen is observed under a microscope, a range observable one at a time is determined mainly by the magnification of an objective lens and, as the magnification of the objective lens becomes higher, an observation range is restricted to a very small portion of a specimen. In pathological diagnosis, on the other hand, there is a demand that a specimen's whole image be grasped so as to prevent a diagnostic site from being left unnoticed. Further, with an advance of the information processing technique, an electronic information form of an image has been promoted on pathological diagnosis and there is also a demand that any microscopic observation image acquired by a camera be realized to the same high resolution level as that of a conventional camera film.
Up to this time, the following methods have been known as the method for obtaining a microscopic image of a higher resolution or such an image with a wider angle of view. A first method for example comprises, as disclosed in JPN PAT APPLN KOKAI PUBLICATION NO. 3-209415, relatively scanning a specimen-placed stage and illumination, dividing the specimen into a plurality of areas and acquiring their partial images, and combining mutually continuous images, like tiles, as a related array and creating a whole specimen image. A second method for example relates to a method for reconstituting a whole specimen image by joining together associated images and, for example, JPN PAT APPLN KOKAI PUBLICATION NO. 9-281405 discloses a microscopic system by dividing a whole specimen area into a plurality of mutually overlapped portions, acquiring the respective smaller areas with the use of a microscope, comparing the overlapped portions of the acquired respective images, detecting their positional displacement, correcting the image positions of the respective smaller areas and joining together the images to constitute a specimen's image of high accuracy and wide visual field. A third method comprises, as shown in JPN PAT APPLN KOKAI PUBLICATION NO. 2000-59606, constituting a combined whole image by acquiring a whole subject and a plurality of highly accurate, divided partial images, enlarging the whole image, detecting the positions of the divided images highly accurately acquired from the whole image and replacing the corresponding portions of the enlarged whole image by the highly accurate partial images.
In a first aspect of the present invention, there is provided an image processing apparatus comprising a lower magnification image acquisition section configured to acquire an image on a whole specimen area at a lower magnification corresponding to a first magnification; an area division information memory configured to, when the whole specimen area is divided into a plurality of smaller areas with a partially overlapped area included, store the position information of each smaller area as area division information; a higher magnification image acquisition section configured to sequentially acquire substantially the same area as the divided area at a second magnification in accordance with the area division information, the second magnification being higher than the first magnification; a positional displacement detecting section configured to, based on a lower magnification specimen image acquired by the lower magnification image acquisition section, detect a positional displacement of a higher magnification image acquired by the higher magnification image acquisition section; a positional displacement correction section configured to correct the position of each higher magnification image based on the positional displacement detected by the positional displacement detection section; and an image joining section configured to sequentially join together the respective higher magnification images corrected by the positional displacement correction section and create a higher magnification image of the whole specimen area.
In a second aspect of the present invention, there is provided an image processing apparatus according to the first aspect of the invention which preferably further comprising an image quality difference detection section configured to detect a difference between an image quality of the higher magnification image at the respective smaller area corrected by the positional displacement correction section and an image quality of a partial image of the lower magnification specimen image corresponding to the higher magnification image, and an image quality difference correction section configured to correct the image quality of the higher magnification image at the smaller area based on the image quality difference detected by the image quality difference detection section.
In a third aspect of the present invention, there is provided an image processing apparatus according to the second aspect of the present invention wherein the image quality difference is preferably generated by a difference in brightness.
In a fourth aspect of the present invention, there is provided an image processing apparatus according to the second aspect of the present invention wherein the image quality difference is preferably generated by a difference in uniformity of brightness.
In a fifth aspect of the present invention, there is provided an image processing apparatus according to the second aspect of the present invention wherein the image quality difference is preferably generated by a difference in geometric characteristics.
In a sixth aspect of the present invention, there is provided an image processing apparatus according to the first aspect of the present invention wherein preferably the lower magnification image acquisition section has a linear sensor configured to acquire a whole specimen image by scanning relative to the specimen and the higher magnification image acquisition section has an area sensor configured to acquire a portion of the specimen as an image.
In a seventh aspect of the present invention there is provided an image processing apparatus according to the first aspect of the present invention wherein, preferably, by comparing a lower magnification image acquired by the lower magnification image acquisition section and the higher magnification image acquired by the higher magnification image acquisition section, a positional relation between the lower magnification image acquisition section and the higher magnification image acquisition section is detected and, based on the detected positional relation, the area division information is corrected.
In an eighth aspect of the present invention there is provided an image processing apparatus according to the seventh aspect of the present invention wherein the positional relation is preferably determined by taking a horizontal or vertical movement into consideration.
In a ninth aspect of the present invention there is provided an image processing apparatus according to the seventh aspect of the present invention wherein the positional relation is preferably determined by taking a rotational movement into consideration.
Advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out hereinafter.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate presently preferred embodiments of the invention, and together with the general description given above and the detailed description of the preferred embodiments given below, serve to explain the principles of the invention.
The embodiments of the present invention will be described in more detail below with reference to the drawing.
First, an explanation will be made below about the first embodiment. An image processing apparatus of the present invention comprises, as a system structure, a microscope having objective lenses of a plurality of kinds of magnifications not shown, an electrically driven stage configured to move a specimen in a two-dimensional way under a visual field of the microscope, a CCD camera configured to acquire a microscopic image, and a personal computer as a controller configured to control these parts.
Then, a lower magnification image area thus acquired is divided into a plurality of partially overlapped smaller areas. The area division information thus taken at this time is converted to the coordinate of the electrically driven stage and stored in an area division information memory section 13.
Then the objective lens is switched to a higher magnification one and, by driving the stage, the specimen is moved to the coordinate as stored in the area division information memory section 13. Since, by doing so, a predetermined smaller area is moved to a position under the image acquisition range of the higher magnification objective lens, a higher magnification image of the corresponding smaller area is acquired by the higher magnification image acquisition section.
In order to accurately join together those sequentially input higher magnification images, correction has to be made by detecting any displacement from a predetermined position represented by the area division information of the acquired higher magnification image.
In a positional displacement section 14, a lower magnification partial area corresponding to the acquired higher magnification image is cut from the whole specimen image of the lower magnification in accordance with the area division information above and, by achieving a template matching to the higher magnification image using this lower magnification partial image as a reference, any positional displacement of the higher magnification image is detected. In a positional displacement correction section 15, the position of the higher magnification image is corrected in accordance with the detected positional displacement information. The corrected higher magnification image is input to an image joining section 16. The image joining section 16 sequentially joins together the corrected higher magnification image and one-previous-corrected higher magnification image in accordance with the area division information and, in this way, a higher magnification combined image is sequentially completed.
Where the smaller area L1 is acquired as a higher magnification image, the stage is driven to allow the specimen to be moved to the stage's coordinate (x1, y1) and its image to be acquired with a higher magnification lens. If the stage's coordinate reproduction is fully adequate, a target area can be acquired as a higher magnification image and hence there occurs no positional displacement of the higher magnification image. Generally, the normal coordinate reproduction accuracy of the electrically driven stage is of the order of a few μ to a few tens of μ. Even if, therefore, the area L1 is trying to be acquired as an image, there is a possibility that the acquired higher magnification image will be displaced from a predetermined position by the coordinate reproduction accuracy extent. In order to secure an accurate image joining, therefore, it is necessary to input the lower magnification image, area division information and higher magnification image to the positional displacement detection section 14 and to detect the positional displacement from a predetermined area of the higher magnification image.
The function of the positional displacement section 14 will be explained in more detail below with reference to
In the absence of any reproduction error about the stage coordinate, (x1, y1)=(xh1, yh1) but there occurs a displacement in actual practice. There, this displacement (Δx, Δy) is found by a template matching of H1 and L1. First, it is necessary that such two images of different magnifications be converted to ones of the same magnification. Let it be assumed that b represents the image acquisition magnification of the lower magnification image L1; a, the image acquisition magnification of the higher magnification image H1; and c, a template matching magnification. In this connection it is to be noted that, generally, a relation a≦c≦b exists.
L1′ shown in
In order to enhance the probability of a success of the template matching and the matching accuracy, it is necessary that the template and its search area be made as great as possible. In this case, however, it takes more time to perform such matching processing. The high-speed processing is done by performing a plural-stage template matching. First, a coarse search may be performed with an intermediate magnification d (a<d<c) and, at a marginal portion of a coarsely detected area, a fine search be made with an intermediate magnification c.
Since, in the present embodiment, the electrically driven stage is used, consideration is given, as a positional displacement, to a vertical or a horizontal movement only. If, in this case, any rotation is involved, detection is made in a way to include a rotational angle also. After the positional displacement of a higher magnification image is detected by the position detection section 14, the positional displacement of the higher magnification image is corrected with the use of the detection data. This correction is made with the use of an affine transformation.
The image joining section 16 is such that, in accordance with the area division information above, the positional displacement-corrected input high magnification images are sequentially joined together to provide a higher magnification image corresponding to a whole specimen. Since, from the area division information, it is possible to calculate the width of the overlapped area between the adjacent higher magnification images, any blank portion created at the marginal portion of the image is eliminated by the position displacement correction of the higher magnification image in the calculation of the width of the overlapped portion. At the overlapped area, blending processing is performed on the adjacent images, so that the middle of the overlapped area corresponds to a joined portion.
Now, the algorithm of the above-mentioned embodiment will be explained below.
Then the objective lens of the microscope is switched to a higher magnification one by an instruction of the controller (step S53). Then, 1 is added as an initial value to a variable N representing a processing number (step S54). Then, in order to acquire a smaller area corresponding to the processing number at a higher magnification, a specimen portion corresponding to the smaller range L is moved by the electrically driven stage to a position under an image-acquisition visual field of a higher magnification image acquisition section (step S55). And a higher magnification image is acquired (step S56).
Then, in order to detect the positional displacement of the higher magnification image corresponding to the position reproduction error of the stage, a corresponding lower magnification partial image is cut and enlarged to a predetermined intermediate magnification c at step S57 and a higher magnification image is reduced to the intermediate magnification c at step S58.
Then, with the central portion of the higher magnification image defined as a template, a corresponding block in a range of a predetermined search area of the lower magnification image is detected. A positional displacement corresponding to a predetermined position of the higher magnification image (a position at an area dividing time) is detected (step S59) by the template matching. A positional displacement of the higher magnification image is corrected based on the positional displacement detected (step S60).
The corrected higher magnification image is joined to the one-previous higher magnification image at step S61 and the higher magnification images are sequentially created to provide a higher magnification whole image. It is decided whether or not, prior to the next higher magnification processing, divide-processing are all completed at step S62. If NO, a variable N representing a processing number is incremented by one at step S64 and this processing goes back to step S55 for repetition. If YES, a finally completed higher magnification whole image is output as a specimen's whole image at step S63 and whole joining processing is completed at step S65.
Although, in the above-mentioned explanation, the coordinate system, has been explained all as being the coordinate of the electrically driven stage, there arises no problem even if it is converted to those pixel units at a time of a higher magnification. Although the stage has been explained as being electrically driven, it may be of a manual operation type. Further, in place of moving the specimen two-dimensionally with the image acquisition section fixed, an image acquisition section may be moved two-dimensionally relative to a fixed specimen to acquire divisional partial images.
According to the present embodiment, when divisionally acquired higher magnification images are joined together, the positional displacement is corrected based on the lower magnification image and, even if no specimen image exists at the overlapped area, it is possible to accurately correct any positional displacement and hence to positively join together microscopic images as a whole image. Further, it is possible to achieve less memory level necessary to the image processing and hence to provide a high resolution better image of a whole specimen with less prominent joining section even if there occur any difference in light exposure condition at the time of acquiring division images and any optical distortion involved. Further, even if the higher magnification images are sequentially input, these images are sequentially joined together and, since the joining together of the division images is done is synchronism with the image acquisition operation, it is possible to achieve high-speed processing as a whole.
An explanation will be made below about the second embodiment of the present invention.
It may be considered that, when in
Where a plurality of higher magnification images acquired under an optical image acquisition system having a prominent difference in brightness of acquired images between near an optical axis and at a marginal portion are joined together, even if an average brightness is achieved between the higher magnification images and there is no positional displacement, a joined section appears prominent on the joined image due to the shading brightness between the higher magnification images.
Where the geometric characteristics of the images differ by the distortion aberration of the image acquisition system between near the optical axis and at the marginal portion, even if any positional difference between the higher magnification images is accurately found with the central portion of the higher magnification image as a template in accordance with the method of the first embodiment, a joined section appears prominent on the joined image due to the positional displacement between the adjacent higher magnification images near the joined section.
According to the present embodiment, the higher magnification image whose positional displacement is corrected by a positional displacement correction section 15 of
Now, in connection with the image quality difference detection section 17, an explanation will be made in more detail below about the procedure of detecting a difference in brightness between the higher magnification image and the lower magnification image or a difference in brightness shading or in distortion aberration and finding correction data. In order to find the image quality difference of the higher magnification image relative to the lower magnification image, it is first necessary to convert two images of different magnifications to ones of an equal magnification as in the same method of finding the positional displacement. In this case, either an intermediate magnification between the lower magnification and the higher magnification or the lower magnification may be used as a reference magnification. Here, the calculation of the correction data is made based on the higher magnification taking a later correction to a higher magnification image into consideration.
V1=VL1−VH1
Here, VL1 and VH1 represent average values of the pixel values of the images L1 and H1. Similarly, in order to correct the average brightness of the image H2 to the same level as that of the L2, the correction value V2 may be added to the respective pixel values of the H2.
V2=VL2−VH2
Here, VL2 and VH2 represent average values of the pixel values of the images L2 and H2. In general, however, the input/output characteristics of the image acquisition element never provides a simple straight line as shown in
First, a plurality of feature points 50 are extracted from the lower magnification partial image L1 shown in
(x″, y″)=(x, y)
av[x″, y″]=hv[x′, y′]
Here, av[ ] represents a pixel value of the respective pixel in the corrected higher magnification image and hv[ ] a pixel value of the respective pixel in the higher magnification image before correction. Although, by finding the feature point 50 and corresponding point 51, the coordinate of the point 51 corresponding to the feature point 50 at the pixel position can be found, the coordinate of a corresponding point in any other position cannot be found. In order to find the positions of those pixels in the higher magnification image corresponding to all the pixels in the lower magnification image, use is made of a linear interpolation by which they are found from the coordinate of a plurality of feature points 50 near any pixel of interest and the corresponding points 51. The positions of the pixels in the higher magnification image corresponding to all the pixels in the lower magnification partial image may of course be found by estimating the coefficients of a numerical equation representing a reference distortion aberration from a plurality of feature points and corresponding points.
In order to correct the shading of the higher magnification image, a plurality of feature points are extracted from the lower magnification partial image L1. Although the pixel at a predetermined position as shown in
In the higher magnification image where the necessary corrections including the positional displacement correction, brightness correction and distortion aberration correction are made, a complete positional matching is achieved between the higher magnification image and the lower magnification partial image and the pixel at the same position of the higher magnification image becomes a corresponding point 53 relative to the feature point 52. And a ratio k[xi, yi] between the pixel value of the feature point 52 and that of the corresponding point 53 is found with (xi, yi) as the coordinate of the feature point 52 (corresponding point 53). This pixel value ratio serves as the shading data of the higher magnification image. The pixel value of the respective pixel of the corrected higher magnification image may be found by the following equation.
av[xi,yi]=hv[xi,yi]*k[xi,yi]
It is to be noted that av[ ] represents the pixel value of the respective pixel of the corrected higher magnification image and hv[ ] the pixel value of the respective pixel of the higher magnification image before correction. Although the correction data at the feature point 52 and corresponding point 53 is found, no correction data on any other position is found. The correction data on all the pixels of the higher magnification image except the corresponding point 53 is found by a linear interpolation from a plurality of correct data near the pixel of interest. The correction data on all the pixels of the higher magnification image may be course be found by estimating the coefficients of a numerical equation representing a reference shading from the pixel value ratio between a plurality of sets of feature points 52 and the corresponding point 53.
Although an explanation has been made up to this point under an assumption that the image quality difference is detected at each higher magnification image and corrected, any image quality difference found at the higher magnification image of a predetermined smaller area may of course be used for processing on the higher magnification image at any other smaller areas.
According to the present invention, as set out above, when the higher magnification image is acquired, an image quality difference of the higher magnification image relative to the lower magnification image is detected and a corresponding correction is made. Even if, therefore, the exposure condition of the higher magnification image differs and there arises any distortion aberration at the higher magnification image acquisition section and any brightness shading at the higher magnification image, a higher magnification whole image of the specimen can be generated without involving any prominent joined section.
Now an explanation will be made below about the third embodiment of the present invention. Although, in the above-mentioned respective embodiments, the explanation has been made under an assumption that the lower magnification image is acquired with the use of the lower magnification lens and the higher magnification image is acquired with the use of the higher magnification lens and the same kind of image acquisition element is used at the lower magnification image acquisition section and higher magnification image acquisition section, there is a limitation on the image-taking visual field of the lower magnification lens of the microscope. Usually, the whole specimen is wider than the visual field of the lower magnification lens. In order to acquire a whole image of the specimen, the lower magnification image acquisition section and higher magnification image acquisition section are separately provided so that a lower magnification image is acquired by a linear sensor capable of covering a specimen range and a higher magnification image is acquired by a conventional area sensor.
An explanation will be made below about the fourth embodiment of the present invention. When an image is acquired by switching a lower magnification lens and higher magnification lens without moving any electrically driven stage, there exist a shift 132 in center position due to a displacement in an optical axis of the lens as shown in
In the present invention, the whole specimen is divided into a plurality of smaller areas while referring to a lower magnification image and the stage is moved in accordance with the position information of the respective smaller area. Then a higher magnification image is acquired and a positional displacement between the higher magnification image of each smaller area and a partial image of the corresponding lower magnification image. Then the positional displacement of the higher magnification image is corrected and those higher magnification images are joined together. Herein lies the feature of the present invention. In case there occur any considerable shift in center position or any considerable rotation in center position between the lower magnification and the higher magnification, even if the whole specimen is divided into areas based on the lower magnification image and an amount of movement of the stage is instructed in accordance with the area division information at that time, it is not possible to acquire any higher magnification image of a desired position. It is, therefore, necessary to initially detect to what extent there occur a shift in center position or a rotation in center position at the time of switching the lower magnification lens and higher magnification lens.
As a method for detecting a shift in center position, without moving the position of the stage, the lower magnification image and higher magnification image are acquired at exactly the same position and, with the higher magnification image used as a template, a matching position is detected from the lower magnification image. By doing so it is possible to detect a shift between the matching position and the center of the image as a shift in center position.
As a method for detecting a rotation in center position, without moving the position of the stage, the lower magnification image and higher magnification image are acquired at exactly the same position as in the case of detecting a shift in center position and a matching position is found between the higher magnification image and the lower magnification image. Then at the matching position an angle at which a correlation coefficient becomes the greatest is found while rotating the template little by little. In this way, the rotation angle in center position is found.
As another method, with the use of the lower magnification image and two or more higher magnification images for different image acquisition positions, the respective higher magnification images allow the positions of the lower magnification image to be detected and, from these positions, a shift in rotation angle between the higher magnification and the lower magnification image acquisition section is found.
Using the detected shift in center position between the lower magnification and the higher magnification, area division information for dividing the whole specimen is corrected based the lower magnification image. With the area division information represented by the stage coordinate (xi, yi) (i represents the number of each divided area) at the center of each divided area and the detected misalignment represented by (Δx, Δy), the area division information is corrected as (xi+Δx, yi+Δy). The corrected value is stored in a memory section (for example, an area division information memory section 13 in
The lens position may be manually adjusted so that any shift in center position ceases to exist. With a representing a rotation angle in center position of the higher magnification lens relative to the lower magnification lens, the shift correcting method preferably comprises, while leaving the higher magnification image as it is, rotating the lower magnification image through an angle of α° to provide a new lower magnification image. Of course, each higher magnification image may be rotated through an angle of −α° while leaving the lower magnification image as it is or the rotation angle of the lower magnification lens are higher magnification lens may be adjusted without being rotated.
Where a plurality of images are joined together, a positive image joining can be provided as a whole without any specimen image being left at the overlapped portion and, even if the image acquisition condition differs at a division image acquisition time, it is still possible to provide an image processing apparatus in which a joined image can be constructed against a uniform background.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2003-134732 | May 2003 | JP | national |
This application is based upon and claims the benefit of priority from prior Japanese Patent Application No. 2003-134732, filed May 13, 2003, the entire contents of which are incorporated herein by reference.
