Patent Application
20240206699
The present disclosure relates to a medical visualisation system, a monitor device for a medical visualisation system and an associated method for controlling one or more functions of the medical visualisation system, such as of the monitor device. Particularly, the present disclosure relates to activation of functions of the medical visualisation system based on movement patterns determined based on series of image frames.
For medical visualisation devices, such as endoscopes, when an operator operating such device both hands may be occupied using and controlling the device. Thus, to activate functions, such as capturing of pictures or videos, the operator may need to let go of one hand or have an assisting person operate controls of an associated monitor device. In case the operator removes one hand from the device, the result may be that the visualisation device, e.g. a camera of the visualisation device, moves and is no longer in the desired position.
One suggested solution might be to add a button on a handle of the visualisation device. However, this makes single use visualisation devices more expensive, and require replacement of existing visualisation devices.
It is an object of the present disclosure to provide a solution which at least improve the solutions of the prior art. Thus, the present disclosure relates to a medical visualisation system, a monitor device for a medical visualisation system and an associated method for controlling one or more functions of the medical visualisation system, such as of the monitor device.
Accordingly, a method for controlling one or more functions in a medical visualisation system is disclosed. Also the medical visualisation system and a monitor device for such medical visualisation system are disclosed.
The medical visualisation system comprises a visualisation device having an insertion cord extending from a proximal cord end to a distal cord end. The visualisation device further comprising an image sensor configured to generate image data indicative of a view at the distal cord end of the insertion cord. A distal portion of the insertion cord may be bendable.
The monitor device comprises a processing unit, such as a GPU, CPU, FPGA or another processing unit known in the art. The monitor device is connectable to the visualisation device of the medical visualisation system, and operable to receive the image data as the image data is being generated by the image sensor. The medical visualisation system may comprise the monitor device. The monitor device may be connectable to the visualisation device by a wired connection. Alternatively or additionally, the monitor device may be connectable to the visualisation device by use of a wireless connection.
The medical visualisation system may comprise a display. For example, the monitor device may comprise a display, such as a touch sensitive display. Alternatively or additionally, the monitor device may be connectable to an external display, such as an external display of the medical visualisation system. The plurality of image frames may be displayed on the display of the medical visualisation system, such as the display of the monitor device or the external display. For example, a live view of the image frames captured by the image sensor may be displayed on the display.
The visualisation device may comprise a handle at the proximal cord end. The handle may comprise an input mechanism, e.g. adapted to receive an input, such as a touch input, in one or more input directions. The input in the one or more input directions may cause a distal portion, e.g. including the distal cord end, of the insertion cord to bend in a corresponding bending direction.
The method comprises receiving a plurality of image frames captured by the image sensor including a first series of image frames, determining based on the first series of image frames a movement pattern of the distal cord end of the insertion cord, determining whether the movement pattern satisfies one or more movement criteria, and in accordance with the movement pattern satisfying a first movement criterion of the one or more movement criteria activating a first function of the medical visualisation system. For example, the processing unit may be operable to receive the first series of image frames captured by the image sensor, determine based on the first series of image frames the movement pattern of the distal cord end of the insertion cord and whether the movement pattern satisfies one or more movement criteria, and in accordance with the movement pattern satisfying a first movement criterion of the one or more movement criteria activate the first function of the medical visualisation system.
Determining the movement pattern based on the first series of image frames may be performed concurrently with receiving the plurality of image frames. For example, an evaluation of the movement pattern may be continuously performed while receiving new image frames captured by the image sensor. Thereby, more complex movement patterns may be used to activate functionality, without significantly influencing the time before the function is activated.
Different movements may activate different functions. Thus, the user may activate a plurality of functions by movement patterns of the distal cord end. For example, in accordance with the movement pattern satisfying a second movement criterion of the one or more movement criteria a second function of the medical visualisation system may be activated. The second movement criterion may be different from the first movement criterion. The second function may be different than the first function.
The first function or the second function may be a capture image function, e.g. saving a frame received from the image sensor in memory of the medical visualisation system. The memory may be a memory of the monitor device and/or the memory may be a remote memory, e.g. located on a server. Alternatively or additionally, the first function or the second function may be starting or stopping a recording of the image data. Alternatively or additionally, the first function or the second function may be activating a filter, changing contrast of the image frames captured by the image sensor, or changing a zoom factor of the image frames captured by the image sensor.
The first function or the second function may be a video capture function comprising a start video capture function and a stop video capture function. The start video capture function and the stop video capture function may be alternately activated.
A visual indication, e.g. a symbol or an alert message, may be displayed, e.g. on the display of the medical visualisation system, such as the display of the monitor device and/or an external display coupled to the monitor device, when the first function and/or the second function is activated, such as in response to the first function and/or the second function being activated. For example, the monitor device may be configured to cause display of the visual indication. The visual indication aids the user to recognise the function having been activated and confirms to the user that the intended function was activated. For example, the user is alerted in case he mistakenly captured a still image, when the intention was to start a video recording.
The first movement criterion and/or the second movement criterion may include holding the distal cord end still for a predetermined time period. Alternatively or additionally, the first movement criterion and/or the second movement criterion may include moving the distal cord end up, e.g. followed by moving the distal cord end down, optionally followed by holding the distal cord end still for a second predetermined time period. Alternatively or additionally, the first movement criterion and/or the second movement criterion may include moving the distal cord end to a first side, e.g. followed by moving the distal cord end to a second side opposite the first side, optionally followed by holding the distal cord end still for a third predetermined time period. The predetermined time period, the second predetermined time period and/or the third predetermined time period may be between 0.5-10 seconds, such as between 1-6 seconds, such as 6 seconds, 4 seconds or 3 seconds, or such as between 0.8-1.5 seconds. The predetermined time period, the second predetermined time period and/or the third predetermined time period may be user adjustable, e.g. in a settings menu. Alternatively or additionally, the predetermined time period, the second predetermined time period and/or the third predetermined time period may be set by a learning procedure, wherein the user performs the desired movement pattern, e.g. a plurality of times, as he/she desires, and the respective time period may be set in accordance with the time period the user has held the distal cord end still during the performed movement pattern. In case of the user repeating the desired movement pattern a plurality of times, the respective time period may be set to a mean of the time period for which the user has held the distal cord end still during each of the performances of the desired movement pattern.
The determination of the movement pattern of the distal cord end may be based on software image processing, such as feature detection or motion vector.
Determining the movement pattern of the distal cord end may be achieved by comparing image features between succeeding image frames in the first series of image frames. The processing unit may be configured to determine the movement pattern of the distal cord end by comparing image features between succeeding image frames in the first series of image frames. For example, determining the movement pattern of the distal cord end may be achieved by comparing image features for each image frame in the first series of image frames with a subsequent or previous image frame in the first series of image frames. The image features of the image frames may comprise corners, SIFT (scale-invariant feature transform), SURF (speeded up robust features), blobs, edges and/or pixel colour information.
As an example, a determination that the distal cord end is held still may be based upon the comparison of the image features, wherein it is determined that the distal cord end is held still if a difference between image features of image frames in the first series of image frames are less than a pre-set threshold.
As another example, a determination that the distal cord end is moved in a particular direction (e.g. up, down, left or right) may be based upon the comparison of the image features, wherein it is determined that the distal cord end is moved in the particular direction if differences between image features of image frames in the first series of image frames have similar magnitude.
The present disclosure provides a solution which allows an operator of the visualisation device to activate functionality without having to remove a hand from the visualisation device or having to ask an assistant to activate the function. The present disclosure is further advantageous in that it may be realised by a software upgrade rather than having to replace existing or stocked visualisation devices.
Embodiments of the disclosure will be described in more detail in the following with regard to the accompanying figures. The figures show one way of implementing the present disclosure and are not to be construed as being limiting to other possible embodiments falling within the scope of the attached claim set.
Various exemplary embodiments and details are described hereinafter, with reference to the FIGS. when relevant. It should be noted that the figures may or may not be drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. It should also be noted that the figures are only intended to facilitate the description of the embodiments. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an illustrated embodiment needs not have all the aspects or advantages shown. An aspect or an advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiments even if not so illustrated, or if not so explicitly described.
A distal portion 43 of the insertion cord 4 is bendable, such that the field of view of the image sensor 5 can be changed. For example, as illustrated the distal portion 43 is bendable in a first bending direction 44 and in a second bending direction 45.
The visualisation device may comprise a handle 6 at the proximal cord end. The handle 6 may comprise an input mechanism 7 adapted to receive an input, such as a touch input, in one or more input directions, such as in a first input direction 71 and a second input direction 72, as illustrated. The touch input on the input mechanism 7 causes the distal portion 43 of the insertion cord 4 to bend in a corresponding bending direction. For example a touch input in the first input direction 71 causes the distal portion 43 to bend in the first bending direction 44. A touch input in the second input direction 72 causes the distal portion 43 to bend in the second bending direction 45. A user is therefore able to bend the insertion cord 4 by use of the input mechanism 7.
The monitor device 3 is connectable to the visualisation device 2. For example, the visualisation device 2 may be connected with a cable to an input port on the monitor device 3. Alternatively, the visualisation device 2 may be wirelessly connected to the monitor device 3. The monitor device 3 is operable to receive the image data from the visualisation device 2 as the image data is being generated by the image sensor 5. The monitor device may comprise a display 34 as illustrated, to display the received image data. However, in other exemplary medical visualisation systems, the monitor device 3 may be connected to an external display, and the monitor device 3 may cause display of the received image data on the external display. Thus, the monitor device 3 may be configured to receive real-time data from the visualisation device 2.
The processing unit 32 of the monitor device 3, may receive a first series of image frames captured by the image sensor. This first series of image frame can be a continuously changing real-time series of image frames, such that the content of the series may be continuously changing.
Based upon the first series of image frames a movement pattern that is representative of any movement (or lack thereof) of the distal end of the insertion cord may be determined, e.g. by the processing unit 32. It may further be determined, e.g. by the processing unit 32, whether the determined movement pattern satisfies one or more movement criteria. If the movement pattern satisfies one of the one or more movement criteria, a function, e.g. corresponding to the satisfied criterion, that is associated with the medical visualisation system, may be activated, e.g. by the processing unit 32. Thus, the functions of the medical visualisation system may be controlled, e.g. by the processing unit 32, based on image based detection of certain movements of the image sensor 5 of the visualisation device 2.
The functions, which may be activated, may be a capture image function causing the processing unit to save a frame received from the image sensor, e.g., in the memory 33 of the monitor device 3 or in an external memory, starting or stopping a recording of the image data to be saved, e.g., in the memory 33 or an external memory, activation or deactivation of a filter, changing of the contrast of image frames captured by the image sensor, or change of a zoom factor of the image frames captured by the image sensor 5. The functions may be functions of the visualisation device 2, monitor device 3 and/or the medical visualisation system 1 in general.
Different functions of the visualisation device 2 may have associated to them different movement criteria, such that a user can activate a number of different functions by moving the distal cord end 42 in pre-defined movement patterns. A movement pattern may satisfy a first movement criteria that activates a first function of the medical visualisation system. A movement pattern may satisfy a second movement criteria that activates a second function of the medical visualisation system.
The movement pattern of the distal cord end 42 can come from a number of different sources. It may come from bending of the bendable portion 43 of the insertion cord but may alternatively or additionally come from movement of the whole of the insertion cord 4, e.g., caused by the user rotating the entire visualisation device 2. The present disclosure allows a user of the medical visualisation system 1 to move the distal cord end 42 relative to the environment and based on the movement activate one or more functions of the medical visualisation system 1.
The movement pattern may correspond to the movement of the distal cord end 42 of the insertion cord 4 and a movement pattern could consist of a sequence of different movements, such as holding the distal cord end 42 of the insertion cord 4 still, moving the distal cord end 42 in a predefined direction or rotating it, or a combination thereof. The movement criteria to be satisfied for the activation of a function may be a specific sequence of movements. For example, an exemplary movement criteria may include holding the distal cord end 42 still for a predetermined time period, moving the distal cord end 42 up followed by moving the distal cord end 42 down, optionally followed by holding the distal cord end 42 still for a second predetermined time period, Another exemplary movement criteria may include moving the distal cord end 42 to a first side followed by moving the distal cord end 42 to a second side opposite the first side, optionally followed by holding the distal cord end 42 still for a third predetermined time period.
The movement criteria may be composed of a finite number of movements in a specified sequence order and/or for a specified amount of time, either as a whole for all the movements or for individual movements. The movement criteria could therefore be as short as 1-2 movements to as long as 6-7 movements or more. The predetermined time period, the second predetermined time period and/or the third predetermined time period could be between 0.5-10 seconds, such as between 1-6 seconds, such as 6 seconds, 4 seconds or 3 seconds, or such as between 0.8-1.5 seconds. In some examples, the time periods may be user defined. Generally, the time periods may be different. For example, an intermediate holding still criteria may be shorter than an end of sequence hold. Holding the distal cord end 42 still could also be used to distinguish between different movements, which may allow for exact separation between movements.
Some movement criteria may alternately activate and deactivate a similar functionality. For example the same movement pattern may start and stop a video capture function, based on whether the video capture function is activated or not.
Different functions could also have different movement criteria lengths. For example, some functions may have a criteria that is composed of 2-3 different movements and some may have 5-6 different movements in the criteria. The length and/or the start and/or the end of the first series of image frames may be dynamically determined based upon the sequences of movements. In some embodiments, the start of the first series of image frames may be determined based on a characteristic movement, such as holding the distal cord end 42 still, e.g. for 3 seconds. An end of the first series of image frames may be determined based on the same or another characteristic movement. In some embodiments, received image frames may be continuously evaluated to determine whether a first series of image frames is indicative of a movement pattern satisfying a movement criteria.
Determining the movement pattern of the distal cord end 42 may be achieved by comparing image features between image frames in the first series of images. For example, by comparing image features for each image frame in the first series of image frames with a subsequent or previous image frame in the first series of image frames. The determination of the movement pattern of the distal end could be based on software image processing, such as feature detection or motion vector. For example, as schematically illustrated in
In
As illustrated in
The determination that the distal end is held still may be based upon comparison of image features. For example, it may be determined that the distal end is held still if a difference between image features of image frames in the first series of image frames are less than a pre-set threshold.
A visual indicator, e.g. a symbol or an alert message, may be displayed when a function, such as the first function and/or the second function is activated. For example, the monitor device 3, as illustrated in
The monitor device 3 may further aid the user in finding and/or teaching the correct movement criteria for specific functions by showing, e.g. on the display 34, the most popular functions and their corresponding movement criteria for activating them. For example, the monitor device may show animations of the most popular functions together with a text of the associated function. Alternatively, the monitor device may show a textual guide of the movements necessary for activating an associated function.
The disclosure has been described with reference to a preferred embodiment. However, the scope of the invention is not limited to the illustrated embodiment, and alterations and modifications can be carried out without deviating from the scope of the invention. As an example the endoscope may be a rigid endoscope, e.g. a laparoscope, or a flexible endoscope adapted for bending of the distal end left-right and up-down, e.g. a gastroscope. The endoscope may be reusable, but preferably the endoscope is intended for single use.
Throughout the description, the use of the terms “first”, “second”, “third”, “fourth”, “primary”, “secondary”, “tertiary” etc. does not imply any particular order or importance but are included to identify individual elements. Furthermore, the labelling of a first element does not imply the presence of a second element and vice versa.
| Number | Date | Country | Kind |
|---|---|---|---|
| PA 2021 70195 | Apr 2021 | DK | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2022/060912 | 4/25/2022 | WO |
