Patent Grant
11367200
The following documents are incorporated herein by reference as if fully set forth: German Patent Application No. 10 2019 116 381.1, filed Jun. 17, 2020.
A method for determining the image position of a marker point in images of an image sequence.
At least one point is marked in a first image in the known methods. Such a marked point is also referred to as a marker point below. Subsequently, image constituents corresponding to the point are determined in the first image of the image sequence and in a further image of the image sequence. This can be implemented by way of known object detection, or edge detection or similar image processing.
However, such methods are disadvantageous in that it is difficult to identify the pixel or the image constituents in the second image if the recording has been rotated, for example, or has otherwise undergone drastic changes.
By way of example, this may occur in endoscopy if fast, jerky movements are carried out or if there is a significant change in the illumination.
A transformation which converts the image constituents of the first image into the image constituents of the second image is determined in a further method. Transforming the marked pixel into the second image with the aid of this transformation is conventional. However, the low accuracy when identifying the marker points, particularly in the case of non-planar objects, is disadvantageous in this case.
It is therefore an object of the invention to create a method of the aforementioned type, which allows reliable identification of previously defined marker points.
This object is achieved by a method having one or more features of the invention.
The method according to the invention comprises the following method steps:
In particular, it is advantageous that the marked points are not, or not necessarily, used to determine the transformation.
This is because this also allows pixels that, e.g., have few characteristic properties in comparison with the remaining constituents of the scene to be easily identified in the second image. Namely, it was found that the identification of the corresponding pixel for such image marks is difficult if, for example, the image is rotated or has otherwise undergone drastic changes.
A further advantage consists of the marker point also being reliably identifiable in non-successive images. This is advantageous, in particular in the case of fast or jerky movements.
Furthermore, the identity of the marker points is maintained, and so even marker points located in similar surroundings and/or located close together are not mixed up.
If need be, relative position information from a position sensor or similar auxiliary signals can also be taken into account when determining the transformation.
In principle, the method can be carried out in a portion of an image, as a result of which the speed of the method can be increased. In the simplest case, the portion comprises the whole image.
In one embodiment, the method is further characterized by the pictorial representation of the mapped marker point in the second image. By way of example, this representation can be implemented by framing, coloring or any other type of highlighting.
In one embodiment, the method is further characterized by the output of the image position, in particular coordinates, of the mapped marker point.
In one embodiment, a marker point is set manually. In particular, a user can enter the marker points on a screen by way of touching.
It is particularly advantageous if a marker point is set in a still of the image sequence. In this way, a point of interest can be selected and marked calmly and with great accuracy.
In one embodiment, a geometric transformation with a plurality of degrees of freedom is used to determine a transformation. Such a matrix transformation allows the fast and simple calculation of a transformation between the first image and a second image. In particular, this also holds true in the case of a pronounced change in the camera position.
The use of a matrix transformation with eight degrees of freedom is particularly advantageous. This renders it possible to take account of, and recognize, scaling, rotation, translation, and perspective changes such as shearing, for example. Additional advantages include a more accurate reconstruction of the reference image and hence a more accurate localization of the marker point in the transformed image.
Moreover, this allows the compensation of effects arising from a rolling shutter of the image sensor during the image recording, for example.
The marker point is localized in the transformed image according to one of the numerous known methods. By way of example, such a method can be an algorithm for object identification and/or feature detection, for example for edge detection and/or corner detection.
The first image is transformed in one embodiment; however, the feature description for the marked pixel is calculated anew for each individual subsequent image in this case and this is complicated from a computational point of view.
Therefore, it is particularly advantageous if the transformation is performed in the second image.
An advantageous embodiment includes checking whether the image position of the marker point is located within the second image prior to the localization. By way of example, this can stop the complicated localization should there be no prospect of success.
For checking purposes, the image position of the marker point is transferred into the transformed second image and mapped into the original second image with the aid of the transformation. This is possible with relatively little computational outlay, and so it is possible to estimate whether a complicated calculation is necessary.
An advantageous embodiment comprises a check as to whether a transformation has been found prior to the localization. If not, this could mean that the image now contains a different scene, for example. For instance, this could have happened due to a significant camera movement.
In one embodiment of the invention, there is an output of an error message or any other warning to a user were the marker point to lie outside of the image.
One embodiment comprises checking, in particular during the localization, as to whether the similarity with which a corresponding marker point is found is sufficient. Should a pixel be concealed in the subsequent image, this prevents any another point in the image, which is the next best corresponding pixel, from being selected.
By way of example, the similarity can be determined by way of descriptors. To this end, it is possible, in particular, to define a threshold for the similarity.
However, the first image does not migrate along in one embodiment. This means that the first image remains constant, at least for a certain number of subsequent images, so that the second image and each subsequent image is compared to this first image. Thus, the transformation is determined between the first image and a second image, and then between the first image and a third image, etc. This facilitates a more accurate localization of the marker in the subsequent image.
In an alternative embodiment of the invention, every successful identification of a marker point in a second point is followed by using said second image as a reference, i.e., first image, for the subsequent third image of the image sequence. This means that the transformation is always determined from one image to the subsequent image.
In addition to individual points as a marker, a plurality of marker points, more particularly geometrically related marker points, can also be used in one embodiment. By way of example, these could show a certain geometric extent of the object under examination, for example a tumor.
In addition to the method, an apparatus for image processing having at least one means for carrying out the method according to the invention is also a constituent part of the invention.
The invention is explained in more detail below on the basis of an advantageous exemplary embodiment with reference to the appended drawings.
In the figures:
A first image is read in a first step S1. By way of example, a first image 1 is shown in
Now, an n-th image is read in a step S3.
Now, a transformation that maps the rotated image 4 onto the first image 1 is determined in a further step S4. By way of example, a matrix transformation with a plurality of unknowns, in particular eight unknowns, can be used to find a suitable transformation. By solving the transformation equations, it is thus possible to take account of, e.g., rotation, translation, scaling, and perspective changes.
After the transformation has been determined, the n-th image 4 is transformed with the aid of this transformation in a next step S5. The result of the transformation is shown in exemplary fashion in
Now, the marker point 3′ is localized in the transformed image 4′ in a localization step S6. This localization can be implemented with the aid of known search algorithms, for example for object identification.
Here, the search can be limited to a restricted search region 5, which is defined around the position of the marker point 3 in the first image 1.
Now, the found marker point 3′ is transformed into the n-th image 4 in a mapping step S7.
Finally, the marker point 3 can be presented in the n-th image 4 in a presentation step S8, or the coordinates, for example, could be output. As a result, the marker point 3 in the n-th image 4 is located exactly at the point originally defined in the first image 1, as shown in
Then, the procedure is repeated with the n+1-th image. As a rule, the images are taken from a video sequence. In particular, the processing of the image signals is preferably implemented so quickly that the marker can be tracked in the live video signal.
It is particularly expedient for the first image to be kept for the running image sequence such that all further images of the image sequence are respectively related to said first image.
Alternatively, the first image could also, in principle, be reset after a certain amount of time and/or after a certain number of elapsed images or be set to the last second image of the interval or to the last image with the visible marker point.
Various error situations which make it impossible to track the marker point may arise when applying the method, for example in endoscopy.
Here, too, a first image 1 is initially loaded in step S1. A check is carried out in a transformation monitoring step S9 as to whether a transformation that maps an n-th image 4 into the first image 1 has been found.
If not, an error in the scene of the camera is identified in a scene error step S10. This is the case, in particular, if there was a significant change in the camera position. By way of example, such a situation is illustrated in
If so, the marker points 3 marked in the first image 1 are transferred into the transformed n-th image 4′ and mapped into the n-th image 4 with the aid of the transformation in a transfer step S11.
In a plausibility step S12, a check is carried out as to whether the marker points 3 transferred thus are located within a valid image region, in particular within the n-th image 4.
If not, it is identified that at least one marker point 3 is located outside of the image region of the n-th image 4 in an edge error step S13. The message step S16 also follows in this case.
If so, the localization step S6 follows here. Now, a check is carried out in a similarity test step S14 as to whether the similarity to the first image 1 is sufficiently large for the ascertained point. A threshold for similarity could be defined in this case.
If not, it is identified that although the marker point is located in the valid image region it is nevertheless not visible, in particular because it is concealed, in a point error step S15.
If so, this is followed by the mapping step S7 and the presentation step S8.
| Number | Date | Country | Kind |
|---|---|---|---|
| 102019116381.1 | Jun 2019 | DE | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7421101 | Georgescu et al. | Sep 2008 | B2 |
| 7489809 | Rush | Feb 2009 | B2 |
| 20160066781 | Thompson | Mar 2016 | A1 |
| 20170221217 | Hong | Aug 2017 | A1 |
| 20180158199 | Wang | Jun 2018 | A1 |
| 20180317865 | Sakaguchi | Nov 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 10246355 | Apr 2004 | DE |
| 112004001861 | Jan 2013 | DE |
| 102009017135 | Nov 2020 | DE |
| Number | Date | Country | |
|---|---|---|---|
| 20200394805 A1 | Dec 2020 | US |
