The present invention concerns the calibration or adjustment of optical systems, and especially endoscopic devices, especially in the context of mini-invasive surgery.
More specifically, the purpose of the invention is to create a process for determining the offset or misalignment between the median axis and the optical axis of a rigid endoscope, or one consisting of at least a rigid end segment, as well as a mini-invasive investigative and/or surgical procedure.
In numerous endoscopic devices, the lenses and CCD sensor of which the camera consists are misaligned with the physical central axis (median axis) of the body of the endoscope. In other words, the optical axis is displaced in relation to the axis of rotation, either intentionally (as a result of the structure of the endoscope) or otherwise (due to a distortion of the endoscope through repeated use, a manufacturing defect or uncertainty in the manufacturing process).
This misalignment or shift between the two axes is problematic in the context of using such an endoscope in a hybrid operating theater in which intraoperative three-dimensional (3D) images (for example, those acquired during the intervention by means of CT scanner-type equipment with a rotational arm in C) and endoscopic images are used simultaneously, and even superimposed. With this aim, it is necessary to determine the spatial position of the endoscopic camera in relation to the intraoperative 3D image.
The classic approach in this situation of a person well-versed in the art, such as that disclosed in [Feuerstein, M., Mussack, T., Heining, S. M., & Navab, N. (2008). “Intraoperative laparoscope augmentation for port placement and resection planning in minimally invasive liver resection” in Medical Imaging, IEEE Transactions 27(3), 355-369], consists of introducing an optical placement system into the operating room, fixing optical markers on the camera and scanner, calibrating the position of the first marker in relation to the camera's optical center, calibrating the position of the second marker in relation to the scanner's mark, and creating conditions that will make it possible to view both markers simultaneously during the surgical intervention. These stages are difficult to perform, the accuracy of the calibration obtained making it possible to achieve precision positioning of the camera in the order of 1 mm, using the leverage effect (since the body of an endoscope is often longer than 30 cm), corresponding to an error of at least 3 mm from the nominal distance of the shot.
The inventors recently proposed and assessed an approach that would make it possible to avoid introducing an additional optical positioning system consisting of positioning the body of the endoscope facing the area of interest and acquiring the scanned image in such a way that the end of the endoscope appears in the 3D image [S. Bernhardt, S. A. Nicolau, V. Agnus, L. Soler, C. Doignon, J. Marescaux. “Automatic Detection of Endoscope in Intraoperative CT Image: Application to AR Guidance in Laparoscopic Surgery”, in IEEE International Symposium on Biomedical Imaging (ISBI 2014), pp 563-567]. The end and orientation of the endoscope can then be automatically located in the 3D images and a virtual camera is created with a view of the area of interest that is identical to that of the actual endoscope, so as to be able to “increase” the endoscopic vision using 3D intraoperative data. Nevertheless, the expected accuracy could not be validated with larger quantities after the performance of encouraging preliminary tests, due to lack of a guarantee of superimposition of the central and optical axes in the endoscopes used.
The inventors deduced that, in order to obtain a more accurate superimposition of the 3D images from those supplied by the endoscope, not only was prior determination of the intrinsic and extrinsic endoscopic camera settings (as would be known to a person well-versed in the art) required, but it would also be necessary to determine the reciprocal offsetting/misalignment of the endoscope's optical and median axes.
In particular, though not restrictively, in relation to the abovementioned context, the main purpose of the invention is to propose a simple, swift, and accurate solution for determining this last setting.
The purpose of the invention is thus a process for determining the offset or misalignment between the central or rotating axis and the optical axis of a rigid endoscope or similar camera device consisting of a rigid body with an external cylindrical casing profiled in the direction of the optical axis or consisting of at least one rigid end segment clad with such a casing,
a procedure characterized by the fact that it consists of taking a number of shots, using a camera or sensor that is part of the endoscope or similar device, with a field of vision limited by a contour that is polygonal, circular, or elliptical in shape, the positioning in relation to the central or rotating axis being defined physically and being specific to each shot, with a relative angular rotation between the contour and the endoscope or similar rotation between two successive shots. A point or pixel needs to be determined in images acquired successively, its position remaining unchanged between the various shots, with the point or pixel corresponding to the projection in the image plane of the central axis or rotation of the rigid body of the endoscope or similar instrument or of the rigid segment at the end.
The invention will be better understood through the following description, concerning the preferred methods of achieving the purpose. These are given as unrestrictive examples and explained with reference to the schematic drawings enclosed, in which:
On the representations of
In accordance with the invention, this process consists of taking, with a camera or similar sensor 3 that is part of the endoscope or similar device 1, a multiplicity of shots with a field of vision limited by a contour 4 that is polygonal, circular or elliptical in shape, whose position in relation to the median or rotating axis Δ is physically defined and specific, for each shot, as having a relative angular rotation between the contour 4 and the endoscope or similar 1 intervening between two successive shots, and determining a point or a pixel PI, CΔ in the successively acquired images whose position remains unchanged between the various shots, the point or pixel PI, CΔ corresponding to the projection in the image plane 3 of the median axis of rotation Δ of the rigid body 2 of the endoscope 1 or similar or the rigid segment at the end of the endoscope.
This point PI is invariable in the image plane, regardless of the relative angular position between the contour 4 and the body 2 of the endoscope 1 (around the rotational axis Δ of the latter) and is formed by a limited number of pixels, preferably by a single pixel, corresponding to the orthogonal projection CΔ of the said axis of rotation Δ in the said image plane.
Preferably, and in order to facilitate the processing and identification of significant elements in the shots, it is planned that, in the case of images acquired successively, the contour 4 presents with a significant contrast in relation to the scene in the viewfinder, for example in terms of the various levels of grays, the differences in color, the differences in brightness, and the difference in the level of color saturation or similar mode.
In accordance with the practical implementation of a simple process, it is also possible to allow for the said contour to be defined by an aperture or cut-out 5 of a patch 6, associated temporarily with the endoscope 1 when various different shots are taken.
More specifically, and as shown in
In practice, before taking a series of shots, the insertion would then consist of inserting a tubular part or body 6, whose internal section is larger than the external section of the cylindrical casing 2′ and that could advantageously be fitted with a non-reflecting interior surface that is dark in color on the free end 1′ of the endoscope or similar 1, in such a way that it rests lengthwise on the latter's cylindrical body 2 or its rigid end and extends beyond the free end 1′ to define a restrictive window for the shot, with a field of vision that is limited peripherally by a contour 4, the angular positioning then being relatively modified between the said tubular body 6 and the cylindrical body 2 around the said median or rotating axis Δ between two successive shots.
In connection with the construction in the
According to a characteristic mentioned above, the process can consist of applying different digital processing to each of the successive images so as to be able to remove at least the corner angles and even the major part or the entirety of the polygonal contour 4 that is visible in the various images, photographed from a variety of angles of the polygonal aperture 5, and in each image processed determining the diagonal D or the bisecting line of which one end touches the edge of contour 4 that is visible in the image in question, while superimposing the various images processed with their respective diagonal D or bisecting line selected and to determine, at least approximately, the intersection point PI shared by all the diagonals D or superimposed bisecting lines, for which the displacement between successive images has been mapped.
As an example of preliminary processing before using the images resulting from the succession of various shots, the process may consist, for each image acquired (
In the case of a square contour 4, the abovementioned operations provide two sets of diagonals D in the various images, a single set making it possible to provide point CA.
In order to choose the right set of diagonals D (see
In the case of a contour 4 that is rectangular in shape, the point PI corresponds to an intersection of the bisecting lines of the corner of the aperture 5, representing the edge of the inserted piece 6 whose adjacent edges rest on the cylindrical body 2 of the endoscope 1.
In accordance with the second method of creation shown in
In practice, the body 2 of the endoscope 1 can be rotated in the circular tube 6 and the discoidal area determined that encloses the various “windows” defined by the aperture 5 in the various rotating angular positions (see
Naturally, the software and hardware used to perform the aforementioned processing and calculations will be known to a person well-versed in the art and do not require an additional description. They can be incorporated into the imaging system used.
The invention also concerns an investigation procedure and/or mini-invasive surgical intervention that, on the one hand, uses a rigid endoscope or similar photographic device 1 consisting of a rigid body 2 inside a cylindrical outer casing 2′ profiled in the direction of the optical axis Σ, or consisting of at least one segment with a rigid end of such casing, fitted with a camera 3 and, on the other hand, a system for acquiring 3D medical images (not shown), both incorporating the area of interest ZI in their respective fields of acquisition. A segment at the end of the endoscope or similar device 1 is visible in the 3D images, thus making it possible to establish a match between the references of camera 3 of the endoscope and the references of the 3D image acquisition system, by determining the orientation of the median axis Δ of the endoscope or similar device and the position of its optical center in the reconstructed 3D images.
This process is characterized, first and foremost, by determining at least a few of the endoscope's settings or those of the similar device 1, especially the offset or misalignment between its optical axis Σ and its central or rotating axis Δ, at least at its end segment, by implementing the procedure described above.
Thanks to this preliminary measure, performed automatically, it is possible to compensate for the misalignment or offsetting between the physical axis Δ of the endoscope 1 and its optical axis Σ, and more generally in relation to the camera 3 (for example one of the CCD type).
According to an advantageous characteristic, the invention may further consist, also in advance, of acquiring successive shots with different orientations of a checkerboard pattern by means of the camera 3 on the endoscope 1, then using these various views to determine the focal distance, especially for calculating the field of vision for a virtual camera, the optical center CΣ in the image plane of camera 3 of the endoscope 1 and the distortion of the lens 7 of the said camera 3, and finally taking into account these intrinsic settings in order to perform a prior calibration of camera 3 and/or subsequent compensation while taking pictures using the endoscope or similar device 1.
The method used in practice for determining the intrinsic settings for camera 3 of the endoscope could, for example, be the one described in the document entitled “A flexible new technique for camera calibration”, Zhang Z., IEEE, Transaction on Pattern Analysis and Machine Intelligence 22, 1330-1334, 2000.
Furthermore, an accelerometer could be mounted at the end of the body 2 of the endoscope 1 to be used for measuring the angular position (pitching and rolling) of the end segment.
In practice, the procedure can consist, during an investigation and/or intervention, of using the results of prior operations to determine misalignment and the intrinsic settings for performing an adjustment and/or recalibration between the internal images supplied by the camera 3 on the endoscope 1 and any external images supplied by the 3D image acquisition system, especially in terms of position, orientation, focus, distortion, and misalignment, with a view to permitting exact superimposition of the information extracted from the external images, especially a volumetric rendering, on the internal images produced by the camera 3.
Generally, the “virtual” point of view produced by intra-operative 3D images is adopted from the viewfinder of the camera 3 on the endoscope in order to provide an increased endoscopic vision from the 3D data.
Of course, the invention is not restricted to the methods of implementation described and represented in the attached drawings. Modifications remain possible, especially from the point of view of the creation of various elements or through the substitution of technical equivalents, yet without nevertheless exceeding the scope of protection of the invention.
Number | Date | Country | Kind |
---|---|---|---|
1550792 | Feb 2015 | FR | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/FR2016/050203 | 2/1/2016 | WO | 00 |