This application claims the benefit under 35 U.S.C. 119(a) to German Patent Application No. 10 2018 124 432.0, filed Oct. 3, 2018, the disclosure of which is incorporated herein by reference in its entirety.
The present invention is directed to a system for holding an image display apparatus and to a method for displaying an image.
Within the scope of micro-invasive medical measures, the direct view through an endoscope is increasingly being replaced by a camera and displaying the image captured by the camera on one or more screens. This allows the medical staff to maintain an ergonomically more expedient and less tiring body posture. Further, a plurality of members of the medical team can simultaneously observe the medical measure on one or more screens. Further, the images of the medical measure can be recorded for documentation purposes and/or can be transmitted into a lecture hall, for example, for training purposes.
As a rule, the person guiding the endoscope directly by hand or controlling a motor-driven movement of the endoscope knows the position and orientation of the endoscope and its viewing direction at all times. All other parties involved have to resort to verbal notifications or their own observation of the movements of the displayed image and their own anatomical knowledge.
For assisting medical staff with finding their bearings in the displayed image, particularly with grasping the current orientation of the endoscope and the viewing direction in a manner that is as intuitive as possible, the suggestion of superimposing auxiliary objects into the image, said auxiliary objects indicating a predetermined and fixed reference direction, for example, has already arisen. Further, rotating or deforming the image displayed on the screen in a manner corresponding to the orientation of the endoscope has also already been proposed.
WO 95/01749 A1 (also published as EP 0 712 289 B1) has described an apparatus for rotating an image display device, facilitating an orientation of the displayed image that corresponds to the actual orientation (page 2, lines 20 to 26). The deflection electrodes of a cathode ray tube of a screen are arranged on a frame that can be rotated by means of a servomotor 22 (page 5, lines 26 to 33;
U.S. Pat. No. 8,187,167 B2 (also published as WO 2006/046809 A1; similar: US 2012-0296164 A1) has described a display apparatus for laparoscopy. The display apparatus comprises a screen for reproducing an image captured by means of a laparoscope. The screen is rotatable in motor driven fashion about the surface normal of its image display surface. The display apparatus further comprises a foot switch, by means of which medical staff can control a rotation of the screen.
An object of the present invention consists of developing an improved system for holding an image display apparatus and an improved method for displaying an image captured by means of a movable image capturing apparatus (20).
This object is achieved by the subjects of the independent claims.
Developments are specified in the dependent claims.
Embodiments of the present invention are based on the concept of moving a screen, a monitor or any other image display apparatus that reproduces an image captured by means of an endoscope, an exoscope or any other image capturing apparatus, in such a way that the orientation of the image display apparatus in space corresponds or largely corresponds to, or at least indicates, the orientation of the image capturing apparatus in space at all times.
A system for holding an image display apparatus for displaying an image captured by means of an image capturing apparatus comprises a movable holding apparatus for the alterable hold of the image display apparatus, a controllable drive device for moving the holding apparatus, comprising a control signal input for receiving a control signal, and a controller comprising a signal input for receiving a signal that represents an orientation or a change in the orientation of the viewing direction of the image capturing apparatus in space or that facilitates a determination of the orientation or the change in the orientation of the viewing direction of the image capturing apparatus, and comprising a control signal output, couplable to the control signal input of the controllable drive device, for providing a control signal for controlling the controllable drive device, wherein the controller is embodied and provided to control the controllable drive device in such a way that, within a predetermined range of possible orientations of the viewing direction of the image capturing apparatus in space, the orientation of the image display apparatus in space is a predetermined function of the orientation of the viewing direction of the image capturing apparatus in space.
In particular, neither of the image capturing apparatus and the image display apparatus are constituent parts of the system. However, the system may alternatively comprise an image capturing apparatus and may be embodied and provided for the combination with an image display apparatus that is not a constituent part of the system; or said system may comprise an image display apparatus and may be embodied and provided for the combination with an image capturing apparatus that is not a constituent part of the system; or said system may comprise both an image capturing apparatus and an image display apparatus.
In particular, the image capturing apparatus is an endoscope, an exoscope, a surgical microscope or any other apparatus for capturing a monocular image or a stereoscopic image. The image capturing apparatus may comprise one or more cameras or one or more image sensors for capturing an image or a stereo image and for producing an image signal that represents the image or stereo image. Alternatively, the image capturing apparatus may be embodied and provided to be optically coupled to one or more cameras or one or more image sensors, for example by way of an eyepiece.
In particular, the image display apparatus comprises one (or more) screen(s) or monitor(s) or one (or more) projector(s) with one (or more) assigned projection surface(s).
The holding apparatus may be embodied and provided for a permanent connection, i.e., a connection that cannot be released in non-destructive fashion or that can only be released using tools, with an image display apparatus. Alternatively, the holding apparatus may be embodied and provided for an easily releasable connection, in particular also a connection that is releasable by medical staff and/or without the use of tools, with an image display apparatus.
The holding apparatus may have one or more rotational degrees of freedom and, optionally, one or more translational degrees of freedom. In particular, the holding apparatus comprises one or more joints for each rotational degree of freedom and one or more linear guides or linear bearings, which facilitate a telescopic property, for example, for each translational degree of freedom. Alternatively, a translational degree of freedom can be realized by means of one or more joints. These joints may facilitate rotational degrees of freedom, once again at the same time.
The movable holding apparatus may respectively comprise a joint defining a horizontal pivot axis or axis of rotation and a joint defining a vertical pivot axis or axis of rotation. In particular, the movable holding apparatus comprises a joint that defines a pivot axis or axis of rotation that is vertical at all times, a joint that defines a pivot axis or axis of rotation that is horizontal at all times, said axis being pivotable about the vertical pivot axis, and a joint that defines a third pivot axis that is orthogonal to the vertical first pivot axis at all times and orthogonal to the second pivot axis that is horizontal at all times.
Further, the movable holding apparatus may comprise a plurality of linear guides or linear bearings, which each define a straight or curved path. In this case, a first linear guide may have a stationary arrangement, with one or more further linear guides being movable along the path defined by the first linear guide. By way of example, a first linear guide of the holding apparatus may define a straight horizontal translation path and a second linear guide may define a straight vertical translation path.
Alternatively, the movable holding apparatus can be embodied in such a way that not all pivot axes or axes of rotation and translation paths, or only the pivot axes or axes of rotation and translational paths in individual configurations the holding apparatus, are oriented orthogonal to one another and either vertically (i.e., in the direction of gravity) or horizontally (i.e., orthogonal to the vertical). In particular, the holding apparatus may comprise one or more hexapods and/or have a similar embodiment to the apparatuses usually referred to as industrial robots. A hexapod and a configuration in the style of an industrial robot comprising a plurality of joints, which each define a pivot axis or an axis of rotation, are examples in which even translational degrees of freedom are implemented purely by joints.
The controllable drive device may be partly or completely integrated in the holding apparatus. The controllable drive device may comprise one or more electric, pneumatic, hydraulic or other motors or actuators and one or more transmissions. In particular, a motor or actuator is assigned to each degree of freedom of the movable holding apparatus.
The controller can be embodied and provided to process, in analog or digital, electric, optical or any other fashion, originally electric, optical or other analog or digital signals and to produce electric, optical or other analog or digital signals. A power supply for providing electric, pneumatic, hydraulic or any other power for one or more motors or actuators of the drive device may be integrated into the controller. Alternatively, the controller may only be embodied and provided to actuate one or more power supplies for electric motors or actuators or to actuate valves. The controller can also be integrated into the drive device and/or also be integrated into the holding apparatus.
In particular, the orientation the viewing direction of the image capturing apparatus or of the image display apparatus in space means the orientation of the apparatus in relation to any predetermined coordinate system. By way of example, a Cartesian coordinate system with two horizontal axes and one vertical axis may be considered as a coordinate system. The direction of the gravitational force acting at the location of the system is vertical; any direction orthogonal thereto is horizontal. By way of example, the horizontal axes of the coordinate system are tangents to the circles of latitude and longitude of the global spherical coordinate system at the Earth's surface. Alternatively, one or more axes of the coordinate system may be used parallel to the edges of a cuboid space in which the system is arranged or to the basic directions (e.g., dorsal-caudal and lateral) of a patient.
By way of example, the orientation of an apparatus in space can be described by the angle between reference directions of the apparatus and axes of the coordinate system or between reference directions of the apparatus and planes spanned by the axes of a coordinate system.
In particular, the viewing direction of the image capturing apparatus is a reference direction thereof. The direction of a straight edge portion of a rectangular image or image field of an image capturing apparatus can be used as a further reference direction of the image capturing apparatus. In the case of an image capturing apparatus that produces a circular image, for example like many endoscopes, a straight edge portion of a region that is captured by a camera coupled to the image capturing apparatus can be used.
The viewing direction of the image capturing apparatus is the direction from a light entry surface of the image capturing apparatus (at the distal end, in particular) to a very distant object that lies in the center of the field of view and that is consequently imaged in the center of the image field. In the case of an image sensor whose image signals are not used in the totality thereof, the viewing direction is the direction in which a very distant object lies, said object being imaged in the center of that region of the image sensor whose image signals are in fact transmitted and displayed on the image display apparatus. The viewing direction of an image capturing apparatus can be alterable, in particular pivotable, in relation to the image capturing apparatus, for example in the case of an endoscope with a pivoting prism or a pivotable camera at its distal end.
By way of example, the surface normal or—in the case of a curved image display surface—the central surface normal of the image display surface or the surface normal of the image display surface of an image display apparatus at the center thereof and a straight portion of a rectangular edge of the image display surface can be used as reference directions for the image display apparatus.
In particular, the orientation of an apparatus can be specified by an azimuth angle (also referred to as horizontal angle), a zenith angle (also referred to as elevation or, depending on the sign, as a height angle or depth angle) and a third angle (referred to as inclination angle below). The azimuth angle is measured within a horizontal plane in a certain direction (in particular in the clockwise sense as seen from above) from a certain direction (e.g., North). The zenith angle is measured as an angle with respect to the vertical. The elevation or the height angle is measured as an angle with respect to the horizontal plane; the term depth angle is also used in the case of a negative sign. By way of example, the orientation of an apparatus can be specified as azimuth angle, zenith angle and inclination angle of the apparatus in relation to a reference orientation of the apparatus.
The predetermined range of possible orientations of the image capturing apparatus in space, within which the orientation of the image display apparatus is a predetermined function of the orientation of the image capturing apparatus, can be a manifold with a dimension of 1, 2 or 3. By way of example, the image display apparatus may only be pivotable about a vertical pivot axis or axis of rotation, wherein the controller is embodied and provided to keep the direction of the central surface normal of the image display surface of the image display apparatus parallel within a certain angle range at all times to the projection of the viewing direction of the image capturing apparatus onto a horizontal plane. As an alternative or in addition thereto, the image display apparatus can be pivotable about a horizontal axis (in particular parallel to the image display surface of the image display apparatus or orthogonal to the central surface normal of the image display surface), wherein the controller is embodied and provided to move the image display apparatus within a predetermined angle range at all times such that the zenith angle for the viewing direction of the image capturing apparatus and the zenith angle for the central surface normal of the image display surface of the image display apparatus are the same.
Limits of the predetermined range of possible orientations of the image capturing apparatus within which the orientation of the image display apparatus is a predetermined function of the orientation of the viewing direction of the image capturing apparatus emerge, for example, from limits of the mobility of the image display apparatus. By way of example, for geometric reasons, hexapod only facilitates pivoting within a restricted solid angle range (the content of which, as a rule, is less than or not much greater than 2n (2 pi) but in most cases significantly less than 4n (4 pi)). Further, limits of the predetermined range may result from the length or elasticity of bending of cables on the image display apparatus and, in particular, from the requirement that medical staff should be able to observe without impairments the image displayed by the image display apparatus without having to change their location.
The system facilitates an intuitive grasp of the current orientation of the viewing direction of the image capturing apparatus by the user or observer purely on the basis of perceiving the orientation of the image display apparatus.
In a system as described here, the controller is embodied, in particular, to control the drive device in such a way that, within a predetermined angle range, pivoting of the viewing direction of the image capturing apparatus about a first axis causes pivoting of the image display apparatus about a second axis.
In particular, the controller is embodied to control the device drive in such a way that, within a predetermined angle range, any pivoting of the viewing direction of the image capturing apparatus about a first axis causes pivoting of the image display apparatus about a second axis.
In particular, the first axis is orthogonal or substantially orthogonal to the viewing direction of the image capturing apparatus. In particular, the second axis is orthogonal or substantially orthogonal to a surface normal or—in the case of a curved image display surface of the image display apparatus—to a central surface normal or to a surface normal of the image display surface of the image display apparatus.
The first axis and the second axis may be parallel or substantially parallel. In particular, the first axis is horizontal and orthogonal to the viewing direction of the image capturing apparatus. Alternatively, the first axis is, e.g., orthogonal to the viewing direction of the image capturing apparatus and orthogonal to a horizontal that, in turn, is orthogonal to the viewing direction. In particular, the first axis may be vertical.
In a system as described here, the controller is embodied, in particular, to control the drive device in such a way that, in respect of pivoting the viewing direction of the image capturing apparatus about the first axis and pivoting the image display apparatus about the second axis within the predetermined angle range, every angle position of the viewing direction of the image capturing apparatus is associated with an angle position of the image display apparatus.
The assignment can be defined by a mathematical function or by a table (often referred to as look-up table). The assignment between the angle position of the viewing direction of the image capturing apparatus and the angle position of the image display apparatus corresponds to a monotonic or strictly monotonic mathematical function, in particular.
The predetermined angle range and its limits can take account of boundary conditions such as the length or elasticity of a cable or the admissible pivoting range of a joint. Further, limits of the predetermined angle range may result from the requirement that medical staff should be able to observe without impairments the image displayed by the image display apparatus without having to change their location.
In particular, the assignment is such that, in the center or in a central region of the predetermined angle range, any pivoting of the image capturing apparatus through a small angle causes pivoting of the image display apparatus through the same angle. In particular, the assignment is such that the ratio between the angle through which the image display apparatus is pivoted to the angle through which the image capturing apparatus is pivoted becomes ever smaller toward the edges of the predetermined angle range. Thus, on the one hand, the orientation of the image display apparatus in space can precisely reproduce the orientation of the image capturing apparatus in space in the center of the predetermined angle range and, on the other hand, an observation of an image displayed by the image display apparatus can still be facilitated at the edges of the predetermined angle range.
In a system as described here, the controller is further embodied, in particular, to control the drive device in such a way that pivoting of the viewing direction of the image capturing apparatus about the first axis further causes a movement of the image display apparatus along a path.
In particular, the path is curved, for example in arcuate fashion. In particular, the path lies in a plane or substantially in a plane orthogonal to the second axis and/or to the first axis. The controller can be embodied to control the drive device in such a way that the image display apparatus is simultaneously pivoted about the second axis and moved along the path. The position of the path may, in turn, be dependent on an angle position of the viewing direction of the image capturing apparatus in relation to a further axis which, in particular, is orthogonal to the first axis. By way of example, the image display apparatus may be simultaneously pivotable about a horizontal axis and about a vertical axis and movable on a sphere surface or on a surface of an ellipsoid or on any other curved surface. Here, the controller is embodied, in particular, to control the drive device in such a way that the central surface normal of the image display surface of the image display apparatus always points at the center of curvature of the surface along which the image display apparatus is movable at the point at which the image display apparatus is currently situated. Expressed differently, the controller is embodied, in particular, to control the drive device in such a way that the image display surface of the image display apparatus is parallel at all times to the local tangential surface of the surface along which the image display apparatus is movable.
In a system as described here, the controller is further embodied, in particular, to control the drive device in such a way that, within the predetermined angle range, a position of the image display apparatus along the predetermined path is associated with each angle position of the viewing direction of the image capturing apparatus.
In a system as described here, the holding apparatus comprises, in particular, one or more joints that allow pivoting of the image display apparatus about the second axis, wherein the drive device comprises one or more motors or actuators for pivoting an image display apparatus, held by means of a holding apparatus, about the second axis.
In a system as described here, the controller is further embodied, in particular, to control the drive device in such a way that, within a predetermined angle range, altering an angle between the viewing direction of the image capturing apparatus and the vertical causes a change in the angle between the surface normal of an image display surface of the image display apparatus and the vertical.
In particular, the controller is embodied to control the drive device in such a way that, within a predetermined angle range, any alteration of an angle between the viewing direction of the image capturing apparatus and the vertical causes a change in the angle between the surface normal of an image display surface of the image display apparatus and the vertical.
In a system as described here, the controller is embodied, in particular, to control the drive device in such a way that, within a predetermined angle range, pivoting of the viewing direction of the image capturing apparatus, more particularly any pivoting of the viewing direction of the image capturing apparatus, about a first horizontal axis causes pivoting of the image display apparatus about a second horizontal axis.
In particular, the second horizontal axis is parallel to the first horizontal axis. In particular, the first horizontal axis is orthogonal to the viewing direction of the image capturing apparatus.
In a system as described here, the controller is embodied, in particular, to control the drive device in such a way that, within a predetermined angle range, an angle between the surface normal of the image display surface of the image display apparatus and the vertical is associated with each angle between the viewing direction of the image capturing apparatus and the vertical.
In a system as described here, the controller is further embodied, in particular, to control the drive device in such a way that pivoting of the viewing direction of the image capturing apparatus about the first horizontal axis further causes a movement of the image display apparatus along a path that is vertically or substantially vertically, at least in a small portion or at a point.
In particular, the path is arcuate or curved differently. In particular, the path lies in a vertical plane. In particular, the path lies in a vertical plane that is orthogonal to the first horizontal axis. In particular, the first horizontal axis is orthogonal to the viewing direction of the image capturing apparatus.
In a system as described here, the controller is embodied, in particular, to control the drive device in such a way that, within the predetermined angle range, a position of the image display apparatus along the predetermined path is associated with each angle position of the viewing direction of the image capturing apparatus.
In a system as described here, the controller is further embodied, in particular, to control the drive device in such a way that a translational movement of the image capturing apparatus causes a translational movement of the image display apparatus.
In a system as described here, the signal input of the controller is further embodied, in particular, to receive a signal that represents a position or a change in the position of the image capturing apparatus or that facilitates a determination of the position or the change in the position of the image capturing apparatus, wherein the controller is further embodied and provided to control the controllable drive device in such a way that, within a predetermined range of possible positions of the image capturing apparatus, the position of the image display apparatus is a predetermined function of the position of the image capturing apparatus.
The position of an apparatus means the position of a reference point of the apparatus, for example of the center point of the light entry surface of the image capturing apparatus or of the center point of the image display surface of the image display apparatus. Any rotation of an apparatus about its reference point does not change its position. A purely translational movement of the apparatus changes its position but not its orientation.
The predetermined region can be a manifold with one, two or three dimensions, in particular corresponding to the same number of degrees of freedom. The position of the image display apparatus may indicate the position of the image capturing apparatus such that medical staff can deduce the position of the image capturing apparatus from the position of the image display apparatus.
In a system as described here, the signal input of the controller is embodied, in particular, to receive from a sensor a signal or a signal that has been integrated over time.
The signal input of the controller is embodied and provided, in particular, to receive a signal from a sensor for capturing the direction of the gravitational force and/or for capturing a linear acceleration and/or for capturing a rotation and/or for capturing a direction and/or a strength of a magnetic field.
A system as described here further comprises, in particular, a sensor for capturing at least either the direction of the gravitational force or of a linear acceleration or a rotation or a direction or strength of a magnetic field.
A sensor for capturing a rotation comprises, in particular, a gyroscope, a vibrating structure gyroscope or any other rate sensor. In particular, the sensor is a constituent part of the image capturing apparatus or mechanically rigidly connected or connectable to the latter.
The orientation of an image capturing apparatus relative to the vertical can be determined by capturing the direction of the gravitational force by means of a sensor in or at the image capturing apparatus. By capturing the direction of the Earth's magnetic field or, for example, an artificially produced magnetic field in an operating theater by means of a sensor arranged in or at an image capturing apparatus, it is possible to determine the orientation of the image capturing apparatus relative to the magnetic field. By capturing the direction of an artificially produced electric field or by capturing the polarization of an alternating electromagnetic field by means of a sensor arranged in or at an image capturing apparatus, it is possible to determine the orientation of the image capturing apparatus relative to said field.
Proceeding from an initial position and an initial orientation, the position and orientation of the viewing direction of an image capturing apparatus can be calculated by twofold integration of the linear or translational acceleration captured by a sensor in or at the image capturing apparatus and by a single integration of the rotation rate.
In a system as described here, the signal input of the controller is embodied, in particular, to receive an image signal from a camera for capturing an operating field, wherein the controller is embodied and provided to determine the orientation of the image capturing apparatus by evaluating an image signal received at the signal input.
A system as described here further comprises, in particular, a camera for capturing an operating field and for producing an image signal, said camera being coupled to the signal input of the controller.
The camera for capturing the operating field should not be mixed up with the image capturing apparatus. In particular, the camera for capturing the operating field is arranged above or obliquely above or next to the operating field. The camera can be a stereo camera such that an evaluation of its stereo image allows information to be obtained about the orientation and the location (in all three dimensions) of the image capturing apparatus. The camera can be embodied and provided to capture an image in the wavelength range visible to the healthy human eye or in a part thereof and/or in one or more other wavelength ranges (e.g., infrared).
The controller can be provided to receive and evaluate image signals from a plurality of cameras. In particular, the controller is provided and embodied to identify a proximal end of an image capturing apparatus or a special marking at the proximal end of an image capturing apparatus. By capturing the position and, optionally, the size of the image of the proximal end of the image capturing apparatus, too, and possibly by comparing the images from a plurality of cameras, it is possible to determine the position and the orientation of the proximal end of the image capturing apparatus. From the position and orientation of the proximal end of a rigid image capturing apparatus, it is possible to calculate the position and orientation of the distal end of the image capturing apparatus and, in particular, the orientation of the viewing direction of the image capturing apparatus.
In a system as described here, the signal input of the controller is embodied, in particular, to receive a signal from a receiver for electromagnetic waves which emanate from the image capturing apparatus, wherein the controller is embodied and provided to determine the orientation of the image capturing apparatus in space by evaluating the signal received from the receiver.
A system as described here further comprises, in particular, a receiver for receiving electromagnetic waves emanating from the image capturing apparatus.
The image capturing apparatus may comprise one or more sensors for capturing the orientation of the image capturing apparatus or the viewing direction of the image capturing apparatus and a transmitter for emitting an electromagnetic wave or a transponder for diffusely reflecting a modified electromagnetic wave. The transmitter or transponder is embodied in such a way that the emitted electromagnetic wave or the modified, diffusely reflected electromagnetic wave contains information about the orientation of the image capturing apparatus and, optionally, about its position, too.
Alternatively, the image capturing apparatus may comprise one or more reflectors for reflecting electromagnetic waves and/or one or more transmitters for emitting electromagnetic waves, wherein the controller is embodied to deduce the orientation of the image capturing apparatus from the intensity or amplitude or power density and/or from the polarization and/or from the phase angle of one or more of the electromagnetic waves reflected or emitted by the image capturing apparatus and received by the receiver.
In particular, the receiver comprises an antenna or a plurality of antennas for receiving the electromagnetic wave. In particular, the frequency of the electromagnetic wave lies between several hundred kHz and a few GHz.
In a system as described here, the controller is embodied, in particular, to control the drive device in such a way that, within a predetermined angle range, a rotation of the image capturing apparatus through a first angle about its viewing direction causes a rotation of the image display apparatus through a second angle, wherein the second angle is a predetermined fraction of the first angle or a predetermined non-constant function of the first angle or a predetermined function of the angle position of the image capturing apparatus.
Alternatively, the controller can be embodied in such a way that the second angle equals the first angle.
In particular, the controller is embodied to control the drive device in such a way that, within a predetermined angle range, each rotation of the image capturing apparatus through a first angle about its viewing direction causes a rotation of the image display apparatus through a second angle.
In particular, the controller is embodied to control the drive device in such a way that, within a predetermined angle range, any or each rotation of the image capturing apparatus about its viewing direction causes a rotation of the image display apparatus about the central surface normal of the image display surface of the image display apparatus. In particular, the controller is embodied to control the drive device in such a way that a rotation of the image capturing apparatus in a direction of rotation (in relation to the viewing direction from the image capturing apparatus (20) to the observed object) causes a rotation of the image display apparatus in the same direction of rotation (in relation to the direction from an observer to the image display apparatus).
The predetermined angle range may comprise 360 degrees, i.e., a full circle. In particular, the controller is embodied in such a way that the second angle corresponds to the first angle or to a predetermined fraction of the first angle at all times. Further, the controller can be embodied in such a way that limits or boundary conditions on the part of the image display apparatus, for example a restricted length or elasticity of a cable for transmitting an image signal or electric power, or a restricted angle range of a joint of the holding apparatus are taken into account.
In a system as described here, the controller is embodied, in particular, to control the drive device in such a way that, within a predetermined angle range, any or each rotation of a camera, which is coupled to the image capturing apparatus, relative to the image capturing apparatus through a first angle about the viewing direction of the camera causes a rotation of the image display apparatus through a second angle, wherein the second angle equals the first angle or the second angle is a predetermined fraction of the first angle or a predetermined function of the first angle or a predetermined function of the angle position of the image capturing apparatus.
By way of example, many endoscopes do not comprise a camera themselves but only comprise an eyepiece, a camera being able to be optically and mechanically coupled thereto. A rotation of the camera relative to the endoscope (or relative to any other image capturing apparatus) in a direction of rotation causes a rotation of the image represented by the image signal of the camera in the opposite direction of rotation. This rotation can be compensated by an appropriate rotation of the image display apparatus. Alternatively, the rotation of the camera relative to the image capturing apparatus can be implemented by a processing of the image signal produced by the camera, wherein the image represented by the image signal is rotated.
In principle, the function that sets the orientation of the image display apparatus in space in a system as described here may additionally also still take account of other factors. By way of example this could include the position and/or orientation of the user relative to the operating field and/or relative to the image display apparatus. To this end, means suitable for capturing the position and orientation of a person, for example a marking that is fastened to the person and is captured by a camera, or means for identifying the eyes of the user and their viewing direction may be provided. This can prevent the image display apparatus from being poorly visible or no longer visible at all to a user, and a position of the image display apparatus relative to the user can be optimized.
In a system as described here, the holding apparatus comprises, in particular, one or more hinges that allow a rotation of the image display apparatus about a central surface normal of an image display surface of the image display apparatus, with the drive device comprising one or more motors or actuators for rotating the image display apparatus about the central surface normal of the image display surface of the image display apparatus.
The holding apparatus may comprise a single joint in the form of a rotational bearing that facilitates a rotation of the image display apparatus about the central surface normal of the image display surface of an image display apparatus mechanically connected to the holding apparatus as intended.
In this case, the drive device, too, only comprises one motor or one actuator, in particular, for rotating the image display apparatus mechanically connected to the holding apparatus as intended. Alternatively, the holding apparatus can be embodied as a hexapod, for example, or can comprise one or more hexapods. In this case, a rotation of the image display apparatus is implemented, in particular, by shortening a plurality of arms and/or simultaneously lengthening other arms, each by way of a motor or actuator.
A system as described here further comprises, in particular, an image signal processing apparatus comprising an image signal input for receiving a first image signal from the image capturing apparatus and an image signal output for providing a second image signal to the image display apparatus, wherein the image signal processing apparatus is embodied to rotate the image displayed by the image display apparatus when the image capturing apparatus is rotated.
To this end, the image signal processing apparatus comprises, in particular, a signal input for receiving a signal that represents an orientation or a change in the orientation of the image capturing apparatus in space or that facilitates a determination of the orientation or the change in the orientation of the image capturing apparatus. Alternatively, the image signal processing apparatus can be embodied to identify a rotation of the image capturing apparatus by analyzing the image captured by the image capturing apparatus, i.e., the image signal produced by the image capturing apparatus or the camera thereof, and cause a corresponding rotation of the image represented by the image signal supplied to the image display apparatus.
The image signal processing apparatus can be integrated with the controller and/or with other components of the system.
A controller for a system as described here comprises, in particular, a signal input for receiving a signal that represents an orientation or a change in the orientation of the viewing direction of the image capturing apparatus in space or that facilitates a determination of the orientation or the change in the orientation of the viewing direction of the image capturing apparatus, and a control signal output, which is couplable to the control signal input of the controllable drive device, for providing a control signal for controlling the controllable drive device, wherein the controller is embodied and provided to control the controllable drive device in such a way that, within a predetermined range of possible orientations of the viewing direction of the image capturing apparatus in space, the orientation of the image capturing apparatus in space is a predetermined function of the orientation of the viewing direction of the image capturing apparatus in space.
The controller may comprise further features of the controller of a system described herein.
A method for displaying an image captured by means of a movable image capturing apparatus (20) comprises a step of capturing the orientation of the viewing direction of the image capturing apparatus (20) in space by means of a sensor and a step of setting the orientation of the image display apparatus in space depending on the orientation of the image capturing apparatus (20) in space.
The method is able to be carried out, in particular, using a system as described herein. A system as described herein is embodied, in particular, for a method as described herein. A controller as described herein is embodied, in particular, to control a method as described herein.
When capturing the orientation of the image capturing apparatus in space, in particular the direction of the gravitational force and/or a linear acceleration and/or a rotation (for example, by means of a gyroscope, a vibrating structure gyroscope or any other rate sensor) and/or a direction and/or a strength of a magnetic field are captured. In particular, the sensor is part of a image capturing apparatus or mechanically rigidly connected thereto (releasable in nondestructive fashion or not releasable in nondestructive fashion). A signal produced by the sensor can be transferred electrically, optically or by means of an electromagnetic wave to a controller for the purposes of controlling the method.
As an alternative or in addition thereto, one or more images of an operating field can be captured in order to identify the image capturing apparatus in the image or in the images and in order to determine the orientation and the position of the image capturing apparatus therefrom. As an alternative or in addition thereto, electromagnetic waves that are produced or specularly reflected or diffusely reflected by the image capturing apparatus can be captured by means of one or more antennas or other receivers. The orientation of the image capturing apparatus can be deduced from the intensity or amplitude or power density and/or from the polarization and/or from the phase difference of a plurality of waves.
Further, in a method as described here, the orientation of the image capturing apparatus in space, in particular, is presented graphically or numerically on an image display surface of the image display apparatus.
The orientation of the image capturing apparatus in space can be represented, for example, by a perspective illustration of an arrow that indicates a basic direction or by a reduced perspective illustration of a human body in the position of the patient. As an alternative or in addition thereto, angles can be output in analog fashion on scales or in digital fashion in the form of numbers.
A method as described here further comprises, in particular, a rotation of the image presented on the image display surface depending on a rotation of the image capturing apparatus about the viewing direction of the image capturing apparatus.
Thus, a rotation of the image capturing apparatus about the viewing direction of the image capturing apparatus may cause, alternatively or simultaneously, a rotation of the image presented on the image display surface of the image display apparatus relative to the image display apparatus and a rotation of the image display apparatus.
In a method as described here, the image presented on the image display surface is deformed depending on an orientation in the viewing direction of the image capturing apparatus in space.
In particular, the deformation is implemented in such a way that a perspective impression of an image that is pivoted or inclined in relation to the actual image display surface arises. By way of example, a greater pivot or inclination angle can also be presented without causing an inexpedient observation direction onto the image display surface of the image display apparatus by virtue of pivoting or inclining of the image capturing apparatus only partly causing a corresponding pivoting or inclining of the image display apparatus but additionally causing a corresponding deformation of the presented image.
Embodiments are explained in more detail below on the basis of the attached figures. In detail:
The viewing direction 28 of the endoscope 20 is the direction from a light entry surface on the distal end 22 of the endoscope 20 to an article at a great distance, which appears in the center of an image captured by the endoscope 20. The edge of the region captured by the endoscope 20 is indicated by two lines 29 in
The endoscope 20 can—inter alia—be rotated or pivoted about a pivot axis or axis of rotation 57, which is orthogonal to the plane of the drawing of
In
The endoscope 20 may have further degrees of freedom, more particularly a rotation about its viewing direction 28 and a rotation or pivoting about a further axis of rotation or pivot axis, which may be orthogonal to the pivot axis 57 and orthogonal to the viewing direction 28 of the endoscope 20. However, only the degree of freedom of the pivot movement 58 about the pivot axis or axis of rotation 57 is initially considered below.
An image captured by means of the endoscope 20 (in particular by the camera 25 or one or more image sensors of the endoscope 20) is transferred to a screen 60 as an image display apparatus by means of apparatuses not presented in
The screen 60 is held on a joint 71 which defines a pivot axis or axis of rotation 67. Thus, the screen 60 is able to be pivoted about the pivot axis 67. The pivot axis 67 is orthogonal to the plane of the drawing of
The joint 71 is part of a movable holding apparatus, otherwise not presented in
A drive device 72 for moving the screen 60, specifically for pivoting the screen 60 about the pivot axis 67, is provided at the joint 71. A control signal input 74 of the drive device 72 is connected to a signal output 47 of the controller 40.
From a device not presented in
Further, the controller 40 knows the current orientation of the screen 60, in particular, and compares the orientation of the screen 60 to the orientation represented by the received signal or the orientation of the endoscope 20 determined by the controller 40. The controller 40 produces a control signal for the drive device 72 in order at any time to move the screen 60 into an orientation that corresponds to, or is associated with, the current orientation of the endoscope 20.
In
In the example shown in
In the example presented in
The holding apparatus presented in representative fashion by the joint 71, the controllable drive device 72 and the controller 40 form a system for holding the screen 60. The screen 60 is not a constituent part of this system but connected or connectable in mechanically rigid and nondestructively releasable fashion with the holding apparatus represented by the joint 71. Alternatively, the screen 60 may be part of the system. The endoscope 20, too, is not part of the system although it may be part of said system.
The holding apparatus 70 shown in
The controller 40 is embodied to control the multi-part drive device 72 in such a way that every pivoting movement of the endoscope 20 about the pivot axis 57 causes an associated pivot movement of the screen 60 about the pivot axis 67 and, at the same time, a translational movement 78 of the screen along a curved, more particularly arcuate path 77. In particular, the controller 40 is embodied to control the multi-part drive device 72 in such a way that the surface normal 63 of the image display surface 62 of the screen 60 and the curved path 77 lie in a plane at each orientation and position of the screen 60. Two orientations and positions of the screen 60 are indicated in
In the examples presented on the basis of
In addition to the degrees of freedom presented on the basis of
In this case, the controller can be embodied to additionally pivot or rotate the screen 60 about one or more further pivot axes or axes of rotation depending on the movements of the endoscope 20. The holding apparatus 70, the multi-part drive device 72 and the controller 40 can be embodied in such a way that the screen 60, depending on the viewing direction 28 of the endoscope 20, is pivoted about two pivot axes or axes of rotation that are orthogonal to one another and not parallel to the surface normal 63 of the image display surface 62 of the screen 60 and optionally moved along a curved surface at the same time. Here, every orientation of the viewing direction 28 of the endoscope 20 is associated with an orientation and a position of the screen 60.
As an alternative or in addition thereto, the holding apparatus 70 may comprise a joint that defines an axis of rotation parallel to the surface normal 63 of the image display surface 62 of the screen 60 and the controller 40 may be embodied in such a way that a rotation of the endoscope 20 about its viewing direction 28 and/or a rotation of the camera 25 relative to the endoscope 20 causes a rotation of the screen 60 about the surface normal 63 of the image display surface 62.
An image of the distal end of the medical instrument 14 and of the organ 18 in the cavity 16 in the body of the patient is captured by means of the endoscope 20. To this end, one or more image sensors of the endoscope 20 produce an image signal that represents the captured image. Alternatively, in a manner similar to the examples presented on the basis of
The screen 60 is arranged opposite the medical member of staff 12 in such a way that the medical member of staff 12 is able to observe the image display surface of the screen 60 with a posture that is as little tiring as possible. The screen 60 receives the image signal produced by means of the endoscope (i.e., by the endoscope itself or by a camera optically coupled to the endoscope) and displays the image represented by the image signal.
The screen 60 is held by a movable holding apparatus 70 that—similar to what is presented on the basis of
The cameras 30 are directed on the operating field and capture the proximal end 24 of the endoscope 20 with the structure 26. The controller 40 receives image signals from the cameras 30 and evaluates the latter in order to identify the structure 26 therein and in order to determine the position and the orientation of the proximal end 24 of the endoscope 20. Depending on the position and orientation of the proximal end 24 of the endoscope 20 as determined by the controller 40 by means of the cameras 30, the controller 40 controls—by way of a control signal at the control signal input 74 of the drive device 72—the drive device 72 in such a way that the screen 60 always has an orientation that corresponds to, or is associated with, the simultaneously present orientation of the endoscope 20, more particularly of the viewing direction 28 of the endoscope 20.
The controller 40, the holding apparatus 70 and the drive device 72 form a system for holding the screen 60.
The endoscope 20, the controller 40, the screen 60, the movable holding apparatus 70 and the drive device 72 are similar in some features, properties and functions to the endoscope 20, the controller 40, the screen 60, the movable holding apparatus 70 and the drive device 72, which are presented on the basis of
At its proximal end 24, the endoscope 20 comprises a transponder 36 for receiving an electromagnetic wave and for transmitting an electromagnetic wave that has been modified depending on the orientation of the endoscope 20 and of the transponder 36. A signal input 42 of the controller 40 is coupled to a device 34 for transmitting an electromagnetic signal and for receiving an electromagnetic signal emanating from the transponder 36.
The transponder 36 at the endoscope 20 can be embodied so as to capture the orientation of the endoscope 20 itself and encode information about the orientation of the endoscope 20 in an electromagnetic signal sent back to the device 34. Alternatively, the transponder 36 can be embodied as a purely passive component, wherein the device 34 and/or the controller 40 derive or determine the orientation of the transponder 36 and of the endoscope 20, for example from a polarization direction, a power density or a phase angle of an electromagnetic signal transmitted from the transponder 36 to the device 34. Both the device 34 and the transponder 36 may each comprise one or more antennas for different polarizations and/or emission characteristics and/or for electromagnetic waves with different phase angles.
During a medical measure, the device 34 produces an electromagnetic wave that is specularly reflected or diffusely reflected by the transponder 36 on the endoscope 20. The device 34 receives the electromagnetic wave specularly reflected or diffusely reflected by the transponder 36. Here, the device 34 receives, in particular, the information encoded in the sent-back electromagnetic wave and/or the intensity or amplitude or power density of the received electromagnetic wave and/or the polarization of the electromagnetic wave and/or the phase angle of the electromagnetic wave. If the transponder 36 comprises a plurality of antennas or comprises a plurality of partial transponders, the device 34 can capture the amplitudes, polarizations and/or phase angles of the electromagnetic waves emanating from these antennas or partial transponders in absolute terms or at least relative to one another.
The device 34 can be embodied to determine the orientation of the transponder 36 and of the endoscope 20 from the received electromagnetic waves and to transmit this information to the controller 40. Alternatively, the device 34 can be embodied to transmit raw data to the controller 40, with the orientation of the transponder 36 and of the endoscope 20 only being determined from the raw data by the controller 40.
As an alternative to a transponder 36, the endoscope 20 can be provided with one transmitter or a plurality of transmitters that transmit one or more electromagnetic waves by way of one or more antennas. Here, the device 34 receives, in particular, the information encoded in the electromagnetic waves transmitted by the transmitter or transmitters and/or the intensity or amplitude or power density of the received electromagnetic wave and/or the polarization of the electromagnetic wave and/or the phase angle of the electromagnetic wave. If the transmitter comprises a plurality of antennas or if a plurality of transmitters are provided, the device 34 can capture the amplitudes, polarizations and/or phase angles of the electromagnetic waves emanating from these antennas or transmitters in absolute terms or relative to one another.
Similar to what was presented on the basis of
The endoscope 20, the controller 40, the screen 60, the movable holding apparatus 70 and the drive device 72 are similar in some features, properties and functions to the endoscopes 20, the controllers 40, the screens 60, the movable holding apparatuses 70 and the drive devices 72, which are presented on the basis of
In or at its proximal end 24, the endoscope 20 comprises a sensor 38. The sensor 38 is embodied and provided to capture the direction of the gravitational force and/or to capture a linear or translational acceleration and/or to capture a rotation rate and/or to capture the direction and/or the strength of the magnetic field or any other directed field. The sensor 38 is coupled to a signal input 42 of the controller 40. From the sensor 38, the controller 40 receives the captured information and determines the orientation of the sensor 38 and of the endoscope 20 in space therefrom. As an alternative thereto, the sensor 38 or a device disposed downstream of the sensor in or at the endoscope 20 determines the orientation of the sensor 38 and of the endoscope 20 in space and transmits this information to the controller 40.
In a manner similar to what was presented on the basis of
In
The screen 60 has three rotational degrees of freedom. The rotation 64 about the surface normal 63 of the image display surface 62 of the screen 60 is referred to as first degree of freedom. Pivoting movements 66, 68 about two further pivot axes or axes of rotation 65, 67 correspond to two further degrees of freedom. In the presented example, the pivot axes 65, 67 are orthogonal to the surface normal 63 of the image display surface 62 of the screen 60 and to one another. Each pivot axis 65, 67 is parallel to two straight edge portions of the rectangular image display surface 62.
The controller 40 of one of the systems presented on the basis of
A screen 60 is held by the movable holding apparatus 70. Controlled by the controller 40 and driven by the drive device 72, the movable holding apparatus 70 can produce rotational and pivot movements 64, 66, 68 about three axes and translation movements 76, 78 in two orthogonal directions along a curved surface—indicated by two paths 75, 77.
Similar to what is presented on the basis of
The screen 60 is held by a movable holding apparatus 70 that can be moved by a drive device 72. A signal input 74 of the drive device 72 is coupled to a signal output 47 of a controller 40. The controller 40, the movable holding apparatus 70 and the drive device 72 form a system with similar features, characteristics and functions as the systems described on the basis of
In
By virtue of a rotation or pivot movement 54, 56, 58 (cf.,
The rotation and/or deformation of the displayed image shown in
In a step 101, a sensor 38, which is arranged in or at an image capturing apparatus 20 and rigidly connected to the image capturing apparatus 20, captures a direction of the gravitational acceleration produced by the Earth. At the same time, or immediately thereafter, a signal that represents the direction of the gravitational force relative to the sensor is produced in a further step 111. Therefore, the orientation of the image capturing apparatus 20 relative to the vertical can be deduced from the produced signal.
In a further step 102, a sensor 38, which is arranged in or at an image capturing apparatus 20 and rigidly connected to the image capturing apparatus 20, captures the direction of the magnetic field relative to the sensor and hence relative to the image capturing apparatus 20. By way of example, the magnetic field is the Earth's magnetic field or an artificially produced magnetic field. A signal representing the direction of the magnetic field relative to the sensor 38 and hence relative to the image capturing apparatus 20 is produced in a further step 112, which occurs simultaneously or immediately thereafter. The orientation of the sensor 38 and hence the orientation of the image capturing apparatus 20 relative to magnetic field can be deduced from the signal.
In a further step 103, a sensor 38, which is arranged in or at the image capturing apparatus 20 and rigidly connected to the image capturing apparatus 20, captures a rotation rate of the sensor 38 and hence of the image capturing apparatus 20, too. This may comprise capturing the rotation rate and the direction of the current axis of rotation or capturing rotation rates in relation to a plurality of orthogonal reference axes. A signal representing the rotation rate captured in step 103 is produced in a step 113, which occurs simultaneously or immediately thereafter. In a subsequent step 123, the signal produced in step 113 is integrated over time. If the angle position of the sensor 38 and of the image capturing apparatus 20 is known at an initial time, the angle position or orientation of the sensor 38 and of the image capturing apparatus 20 present can be deduced at any later time from the integrated signal.
In a further step 104, a sensor 38, which is arranged in or at the image capturing apparatus 20 and rigidly connected to the image capturing apparatus 20, captures a linear or translational acceleration. Capturing the acceleration may comprise capturing the magnitude and the direction of the acceleration. Alternatively, capturing the acceleration may comprise capturing a plurality of components (for example, parallel to the axes of a local or global Cartesian coordinate system). A signal representing the captured acceleration is produced in a step 114, which occurs simultaneously or immediately thereafter. The produced signal is integrated in a subsequent step 124. If the position of the sensor 38 and of the image capturing apparatus 20 is known at an initial time, the position of the sensor 38 and of the image capturing apparatus 20 present at any later time can be calculated or determined at the later time from the integrated signal.
Steps 101 and 111, steps 102 and 112, steps 103, 113 and 123 and steps 104, 114, 124 represent alternatives that may be alternatively or simultaneously present in a method.
The orientation of the image capturing apparatus 20 is determined in a further step 130 on the basis of the signal or signals produced by the sensor 38 or the sensors in or at the image capturing apparatus 20.
An intended orientation of an image display apparatus 60 is determined from the orientation of the image capturing apparatus 20 in a further step 140. The intended orientation of the image display apparatus may be identical to the orientation of the image capturing apparatus or may be any desired function of the orientation of the image capturing apparatus.
An actual orientation of the image display apparatus 60 is determined in a further step 150; said actual orientation may be known, for example, from the performed number of steps of stepper motors or of position or angle sensors on joints of a holding apparatus, which holds the image display apparatus.
A necessary correction movement of the image display apparatus is determined in a further step 161 from the intended orientation and the actual orientation of the image display apparatus 60.
A control signal for a drive device 72 of the movable holding apparatus 70 for the image display apparatus 60 is produced in a further step 162 from the required correction movement.
Step 130, step 140, step 150, step 161 and step 162 are performed, in particular, by a controller 40, which may be part of the image capturing apparatus 20 or part of the drive device 72 for the movable holding apparatus or part of the image display apparatus 60 or an independent component.
The holding apparatus 70 holding the image display apparatus 60 is moved in accordance with the control signal in a further step 163.
The image to be displayed by the image display apparatus 60 is rotated in a further step 171. The image to be displayed by the image display apparatus 60 is deformed in a further step 172 in order to give an observer the impression of the displayed image being pivoted relative to the image display apparatus 60 about an axis 85, 87.
Step 163, step 171 and step 172 represent alternatives that, together or alternatively, may be part of the method shown in
The image is displayed by the image display apparatus 60 in a step 173.
The orientation of the image capturing apparatus 20 in space is presented graphically or numerically on the image display apparatus 60 in a further and optional step 180. By way of example, this can be implemented by a perspective presentation of an arrow that indicates a predetermined reference direction. The reference direction depends on the medical measure and its location, for example cranial, caudal, ventral or dorsal. Alternatively, orienting the image capturing apparatus 20 in space can be implemented, for example, by a reduced and perspective presentation of the patient's body at the edge of the image display surface 62 of the image display apparatus 60. As an alternative or in addition thereto, orienting the image capturing apparatus 20 in space can be implemented by specifying angles on analog scales or in the form of numerical values.
The steps of the method presented on the basis of
An image of an operating field, in which a proximal end of an image capturing apparatus 20 is also disposed, is produced by a camera 30 in a step 105. An image signal representing the captured image is produced in a step 115, which occurs simultaneously or immediately thereafter.
An electromagnetic wave, which emanates from a transponder or a transmitter 36 in or at an image capturing apparatus, is received in a further step 106, in particular by means of an antenna or any other receiving device 34. A signal representing the amplitude, intensity or power density and/or the polarization and/or a phase of the received electromagnetic wave is produced in a further step 116, which is carried out simultaneously or immediately thereafter. The orientation of the transponder or transmitter 36 and hence of the image capturing apparatus 20 in space can be deduced from the signal.
Steps 105, 115 and steps 106, 116 represent alternatives. A method may comprise steps 105, 115 and/or steps 106, 116.
The orientation of the image capturing apparatus 20 in space is determined in a further step 130 by evaluating the image signal produced in step 115 and/or by evaluating the signal produced in step 116.
The further steps of the method shown in
Steps 101, 111 and/or steps 102, 112 and/or steps 103, 113, 123 and/or steps 104, 114, 124 from the method presented on the basis of
Number | Date | Country | Kind |
---|---|---|---|
10 2018 124 432.0 | Oct 2018 | DE | national |