Patent Application
20240115235
The embodiments in this disclosure relate to robotized imaging systems, and, in particular, though not exclusively, to methods and systems for robotized imaging systems, preferably robotized imaging systems, such as a robotized echography system, and a computer program product for executing such methods.
Cardiovascular diseases are the number one cause of mortality worldwide. An essential means for identification of such diseases is an ultrasonic imaging system known as echocardiography, an acoustic imaging technique for generating echocardiograms. More generally, sonography or ultrasonography is an important diagnostic imaging technique for many medical procedures, wherein ultrasound waves are used to create an image of internal body structures such as tendons, muscles, joints, blood vessels, and internal organs. Making and interpreting sonograms is a skilled labour. For example, ultrasound images in 14 up to 37 different view directions are captured during a full examination of the heart by a medical specialist, wherein each of these views is associated with different positions and orientations of the probe relative to the body.
During this process, a medical specialist will manually position an ultrasound probe at a predetermined position such that the targeted view direction is shown on a display. When the display shows the correct image, a short video will be recorded showing the motion of the heart. This process will be repeated for each of the required views. In order to minimize interference effects in the images due to objects and/or tissue close to the heart, such as (parts of) the rib cage and lung tissue, the pose of the patient (lying on the back or side in a certain pose) will be changed depending on the view direction that is recorded.
The result of the examination is a set of ultrasound images associated with different view directions which may be analyzed by the medical specialist. Due to a shortage of skilled medical specialists however, there is a high threshold and/or relatively long waiting time for patient to get a scan, increasing the risk of hospitalization and sometimes severe medical consequences for a patient.
To relieve pressure on medical specialists there is a need to automate medical imaging schemes, such an echocardiography scheme, for routine patients so that human interference is not needed or at least minimized. Robotized imaging systems in which a robot arm comprising an imaging probe is controlled to produce images of a patient are known. For example, WO2019/174953 describes a robotized imaging system including a robotic arm that can be controlled to position an imaging probe at a predetermined position at a body part of a patient lying on the back (in supine position) one a bed. A robotic system is configured to control the robotic arm based on a neural network system, which is trained to receive an ultrasound image generated by the probe and to produce a control signal for the robotic system to move the probe in a direction of a target position.
The neural network system includes different functional neural networks which need to be trained based on data, which is for a large part related to synthetic data produced on the basis of a phantom and for another part manually labelled clinical data. The data is subsequently used to train a neural network to position the probe at a target position that is associated with a predetermined clinical property. However, this system is not adapted to autonomously produce a medical examination of the heart of a patient in multiple view directions to produce a series of multi-view echocardiograms that is similar or substantially similar to those produced by a skilled sonographer.
A further problem relates to the fact that training a neural network for a large number of views as is required for a full examination by a sonographer requires large amounts of labelled data, preferably clinical data of a certain quality. Such data is not available and manually producing such amounts of data is difficult and expensive.
Hence, from the above, it follows that there is a need in the art for improved methods and systems for robotized ultrasound imaging system, such as an ultrasound echocardiographic system, in particular there is a need in methods and systems for robotized ultrasound imaging configured to autonomously determine a full echocardiographic set of images for different views that is of the same quality as a set of images determined by a sonographer using a conventional echocardiographic system.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.
Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable. RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java™, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU) or a graphical processing unit (GPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. Additionally, the Instructions may be executed by any type of processors, including but not limited to one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
It is an aim of the embodiments in this application to provide methods and systems for robotized medical imaging, such as ultrasound echography, allowing autonomous determination of a full set of images of different target views.
In an aspect, the embodiments relate to a robotized ultrasound imaging system comprising: a robotic system configured to position an ultrasound probe against a body part of an object, e.g. a human object, positioned on a support structure, preferably a horizontal support structure, and to move the ultrasound probe over the body part; and, a computer system comprising a data generating module configured to control the robotic system to generate clinical training data for a machine learning algorithm, the clinical training data comprising ultrasound images associated with one or more target views of internal tissue and/or organs of the object, each target view being associated with a target position of the ultrasound probe, the training data further including for each target position a plurality of training positions of the probe relative to a target position and including direction information associated with the plurality of training positions, the direction information defining for each training position a direction, preferably a vector, pointing towards the target position.
In an embodiment, the robotized ultrasound imaging system may further comprise a machine learning module comprising a trained machine learning algorithm, which is trained based on the clinical training data, the machine learning module being configured to receive one or more ultrasound images captured by the probe at a current probe position when moving over the surface of the body part of the object and to determine for the current probe position estimated direction information, preferably an estimated vector, defining a direction pointing from the current probe position towards an estimated probe position associated with the target view based on the one or more ultrasound images; and,
In an embodiment, the robotized ultrasound imaging system may further comprise an execution module configured to autonomously control the robotic system, the controlling including: moving the probe over the surface of the body part of the object to find an optimal probe position for each of the one or more target views based on estimated direction information generated by the trained machine learning algorithm for different probe positions and to capture at the optimal probe position for each of the one or more target views, one or more images of the internal tissue and/or organs.
In an embodiment, the data generating module may be further configured to determine a plurality of training positions for the probe within a predetermined data collection area defined around a target position; move the probe to each of the training positions and capture one or more images for each of the training positions; associating the one or more images associated with a training position with direction information, the direction information including coordinates of a vector pointing from the training position to the target position; and,
In an embodiment, the system may further comprise a training module configured to: provide an image associated with a training position to the input of the machine learning algorithm, the image being associated with direction information defining a target direction, preferably a target vector, pointing from the training position towards a target position; receiving estimated direction information, preferably an estimated vector, from the output of the machine learning algorithm and evaluating the estimated direction information based on the target vector and a loss function; adjust training parameters of the machine learning algorithm based on the evaluation of the estimated direction information; and, repeating the process of providing an image to the input of the machine learning algorithm, evaluating the output of the deep neural network and adjusting of the training parameters, until the evaluation indicates that the estimated vector substantially matches the vector.
In an embodiment, the execution module may be configured to: moving the probe to a first probe position against a body part of a patient and capturing one or more first images at the first probe position; providing the one or more first images to a deep neural network to determine estimated direction information associated with the first probe position, the estimated position information defining an estimated direction pointing from the first probe position towards an estimated probe position of the target view; storing first probe position and the estimate direction information in a memory; repeating the steps of moving the probe to a further probe position, determining estimated direction information for the further probe position using the trained deep neural network and storing the probe position and the associated estimated direction information until a set of probe positions and associated estimated direction information are collected; and, determining an optimal probe position of the target view based on the collected probe positions and estimated direction information and moving the probe to the optimal probe position to capture one or more images of the target view.
In an embodiment, the machine learning algorithm may be a deep neural network or a deep neural network system comprising a plurality of concatenated deep neural networks.
In an embodiment, the convolutional neural network or convolutional neural network system may be configured as a regression algorithm or a classification algorithm.
In an embodiment, the deep neural network or deep neural network system may comprise one or more convolutional neural networks for extracting features from ultrasound images captured by the probe; and/or, wherein the deep neural network or deep neural network system comprises a network of densely connected layers for transforming features associated with an ultrasound image captured at a probe position into estimated direction information, wherein the estimated direction information defines a direction pointing from the probe position towards an estimated probe position of the target position.
In an embodiment, the clinical training data further may include one or more features of one or more vital sign signals associated with the one or more captured ultrasound images, preferably the one or more features including features of an electrocardiogram ECG signal and/or features of a respiratory signal.
In an embodiment, the support structure may be configured to position the object in a prone position, the support structure comprising an opening or a recess exposing part of the object, preferably the chest of the object, to the probe, more preferably at least part of the robotic system being arranged under the support structure.
In contrast to known robotized echography systems, this embodiment, relies on the fact that a human object is positioned on a support structure that comprises an opening that provides a robotized probe to access the chest of a human object that lies on the support structure in a prone position. In certain viewing directions, scanning in prone position may be advantageous. For example, during echocardiographic imaging in the prone position the heart will less blurred by the lung tissue as is the case if a human object is lying on its back.
Thus, imaging a patient in the prone position will generate improved images, i.e. images with less distortions and/or interference due to lung tissue. Further, the opening in the support structure allows the accurately position and fixate a human object that needs to be scanned. Ultrasound images of internal tissue and/or organ of the human object in prone position that are generated by the data collector module may be used for generating training data for training the model. Training the algorithm based on the generated training data may result in a trained algorithm that can be used to make an accurately estimate of an optimal probe position for determining ultrasonic images of a target view.
A further advantage of the system relates to the safety of the human object. In contrast to known robotized imaging systems, the human object is not positioned between the support structure, the bed, and the probe. Rather, the probe accesses the patient through an opening of the support structure so that the patient cannot get stuck between the probe and the bed, thus reducing the risk that malfunctioning of the robot may lead to situations in which the patient could be injured. Further embodiments and associated advantages of the system and methods executed by the system are described hereunder in more detail.
In an embodiment, the robotic system includes may include a robot arm connected to the imaging probe for moving the probe in translational and rotational directions; or a linear robotic system, preferably arranged under the support structure, wherein the robotic system includes a linear robotic system configured to move a probe stage comprising the imaging probe in translational directions and wherein the probe stage is configured to move the imaging probe in angular directions.
In an embodiment, the estimated directional information may be used by the robotic system to move the probe in one or more first directions (preferably the x,y directions) parallel to the plane of the support structure.
In an embodiment, a sensor signal of a pressure sensor or a force sensor associated with the probe may be used to move the probe in a second direction (preferably the z-direction) substantially perpendicular to the one or more first directions, preferably the sensor signal being used to press the probe with a constant force against the body part while the probe is moving in the one or more first directions.
In an embodiment, the probe stage may include a probe holder for holding the imaging probe, the probe holder including the pressure or force sensor, preferably the pressure or force sensor including a spring structure, such that when the probe is pressed against the body part of the object, the one spring structure will be compressed, wherein the compression of the spring structure represents a value of the force with which the probe is pressed against the body part.
In an embodiment, the direction information may include a set of translational coordinates for moving the imaging probe in a translational direction and a set of angular coordinates for moving the imaging probe in an angular direction.
In an embodiment, the remote controller may be shaped as a manual echography probe, the manual echography probe comprising one or more sensors.
In an embodiment, the sensors may be one or more accelerometer sensors, one or more optical navigation sensors and/or inclination sensors for translating a position of the manual echography probe into positional information for controlling the robotic system to move the imaging probe to a position according to the positional information.
In an aspect, the invention may relate to a robotized imaging system comprising: a robotic system configured to position an imaging probe against a body part of a patient positioned on a support structure and to move the probe over the body part; a computer connected to the robotic system, the computer comprising a computer readable storage medium having computer readable program code embodied therewith; and a processor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: determine a plurality of training positions for the probe within a predetermined data collection area defined around a target position; move the probe to each of the training positions and capture one or more images for each of the training positions; associating the one or more images associated with a training position with direction information, the direction information defining a direction, for example a vector, pointing from the training position to the target position; and, storing the one or more images and associated direction information as clinical training data on a storage medium.
In an aspect, the invention may relate of a robotized imaging system comprising: a robotic system configured to position an imaging probe against a body part of a patient positioned on a support structure and to move the probe over the body part; a computer connected to the robotic system, the computer comprising a computer readable storage medium having computer readable program code embodied therewith; and a processor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising: provide an image associated with a training position to the input of the machine learning algorithm, the image being associated with direction information defining a direction, preferably a target vector, pointing from the training position towards a target position; receiving estimated direction information, preferably an estimated vector, from the output of the machine learning algorithm and evaluating the estimated direction information based on the target vector and a loss function; adjust training parameters of the machine learning algorithm based on the evaluation of the estimated direction information; and, repeating the process of providing an image to the input of the machine learning algorithm, evaluating the output of the deep neural network and adjusting of the training parameters, until the evaluation indicates that the estimated vector substantially matches the target vector.
In an aspect, the invention may relate to a robotized ultrasound imaging system comprising: a robotic system configured to position an imaging probe against a body part of a patient positioned on a support structure and to move the probe over the body part; a computer connected to the robotic system, the computer comprising a computer readable storage medium having computer readable program code embodied therewith; and a processor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform executable operations comprising:
The embodiments may also related to methods executed by a computer of the robotized ultrasound imaging system as described above.
The invention may also relate to a computer program product comprising software code portions configured for, when run in the memory of a computer, executing the method steps according to any of process steps described above.
The invention will be further illustrated with reference to the attached drawings, which schematically will show embodiments according to the invention. It will be understood that the invention is not in any way restricted to these specific embodiments.
The embodiments in this disclosure relate to robotized imaging systems, in particular medical robotized imaging system such as robotized ultrasound imaging systems. Such imaging systems are configured to determine images of internal tissue and/or organs an object, e.g. a human or an animal, for a medical procedure or physical examination. Such systems may for example include an ultrasound imaging system that is configured to determine ultrasound images, typically 2D slices, of the heart using an ultrasound imaging probe. The imaging probe produces sound waves, which enter the body and are reflected back by tissue and/or fluids, e.g. blood, which is detected by an ultrasound detector in the imaging probe. The type of the tissue and/or fluid determines the strength of the reflection, which is seen in the echo view as a brighter or darker spot. Typically, blood reflects very little, while muscle reflects more. This means that the structure of organs such as the heart. (consisting largely of muscle and blood) can be imaged using sonography taking into account that tissue of other internal tissue, e.g. rib cage or lungs, may interfere with obtaining a clear picture of (part of) the heart in different view directions.
An ultrasound probe generates images in the form of a 2D slice of whatever it is aimed at, in a fan-like pattern. This fan can be seen as protruding from the head of the probe, slicing into the body. To look at, different parts of a body part, e.g. the heart, a sonographer can manually rotate and/or laterally translate the probe around or over any axis. In echocardiography a number of predetermined “views” are defined. The axis of these view directions are fixed positions relative to the heart so that a sonographer can consistently use the different views to make a diagnosis and compare one set of images with another set of images. For instance, the so-called apical 4-chamber view (A4C) looks at the heart from below, through the apex, showing all four chambers of the heart. Another view is the parasternal long axis view (PLAX). To produce ultrasound images for this view, the ultrasound probe is placed perpendicular to the center of the chest (or sternum), and rotates it such that two chambers are visible: the left ventricle and the left atrium.
An ultrasound probe can capture images, also referred to as video frames, at a rate of more than 30 per second, thus allowing to make a video of a slice of the heart associated with a specific view direction. The result is a near-live video of the heart in motion, which allows echocardiographers diagnosing complex issues of the heart. Thus, an echography system may be configured to generate and process images of the heart according to predetermined standard view directions. Typically, images of 14 up to 37 different view directions may be captured during a full examination by a sonographer, in any case, at least five view directions are needed to reliably identify issues. A sonographer will position a probe at a predetermined position such that the targeted view direction is shown on a display. When the display shows the correct image, the sonographer will record a short video to show the motion of the heart. The sonographer will repeat this process for each of the required views, wherein for each view the sonographer will position the patient in a certain pose so in order to enable clear images. In order to minimize interference of internal tissue, such as (parts oft the rib cage and lung tissue, the sonographer will change the position of the patient (lying on the back or side in a certain pose) depending on the view direction that is recorded. The result of the examination by the sonographer is a set of ultrasound images associated with different views which can be analyzed by the cardiologist.
From the above it follows that, a full echocardiography examination by a sonographer, which requires recording of the heart in multiple view directions in which the patient has to be repositioned various times, is laborious and physically exhausting, especially when examining multiple patients in one day. Furthermore, a lack of skilled sonographers causes long waiting times for patients. The embodiments in this application aim to solve at least part of the problems related to echography examination, in particular echocardiography examination, based on conventional echography systems. In particular, the embodiments relate to a robotized echography system, a robotized ultrasound imaging system, which allows autonomous echography examination in predetermined view directions that does not require human intervention or at least minimum human intervention.
While the examples in the figures are explained with reference to a robotized echography system, it is submitted that the embodiments in the application are not limited to robotized ultrasound imaging and may also be used with other types of robotized medical imaging systems for imaging internal tissue such as photon-acoustic imaging, tomographic imaging, radiographic imaging, x-ray imaging, magnetic resonance imaging, etc.
In the embodiment of
The system further includes a computer system 109 comprising a processor 115 for executing different modules, e.g. controllers and/or modules associated with a model that are used to autonomously control the robotic system. For example, the computer system may include a robot controller 110 for controlling the robot system. The robot controller may be configured to control actuators and/or motors of the robot system to move and position the probe relative to the support system (and a patient positioned on the support system) based on information received from a machine learning model 114 or a remote controller 122. Here, the machine learning model may include a system of one or more deep neural networks that are trained to move the probe from a current position towards an estimated target position. The machine learning model 114 may be used to autonomously control the robotic system. In that case, the processor of the computer system may set the system in an autonomous mode wherein based on ultrasound images produced by the ultrasound probe, the robot controller may send instructions to the robotic system to move the probe from a first position to a second position. Alternatively, the remote controller can be used to control the system in a human-controlled mode, wherein a person may manually control the robotic system.
The robotic system may further include sensors. e.g. force-, torque-, pressure-, position sensors, and/or accelerometers, for enabling control of the probe during scanning. For example, when the probe is positioned against the body of a patient, a force sensor may be used to exert a predetermined force against the body in the z-direction. For example, the robot controller may receive the sensor signal from the robotic system and use this signal as a force (or pressure) feedback signal. The force feedback may enable the probe controller to position and move the probe with a constant force and/or torque against the body of a patient lying on the support structure. Additionally, the force feedback will make sure that the pressure exerted onto a body part of the patient does not exceed a (threshold) value so that the positioning of the probe can be performed safely both when controlled by the machine learning model or by a person using the remote controller. This way the probe may be controlled to scan a patient by moving the probe over part of the body of a patient lying on the support structure, while capturing images, which may be processed by the computer to produce coordinates, e.g. in the form of a vector. At least part of the translational coordinates (e.g. the x and y direction) and the angular coordinates may be produced by the machine learning model. Another part of the translational coordinates (e.g. the z direction) may be controlled by the force feedback system as described above. In another embodiment, an optical navigation sensor may be used to accurately determine the position, e.g. translational motion, of the probe.
In the human-controlled mode, suitable user interfaces UIs, for example remote controller 122 and display 124, connected to the system can be used by a sonographer to manually move the probe of the robotized echography system to a predetermined position in a certain view direction, while the probe produces images which are shown on the display. In an embodiment, the remote controller may be designed so that it can be used as a conventional hand-held probe which is configured detect translational and/or rotational movements. This way, the remote controller may have the look and feel of a conventional manual probe that are used in performing conventional sonography.
As will be described hereunder in greater detail, the system depicted in
The robot controller may be controlled based on information that is determined by machine learning model 114, e.g. a machine learning model such as a deep neural network or a system of deep neural networks. The model may be trained to control the robot system to move and orient the probe so that images of different target view can be determined. For each of the target views an optimal image of a target, e.g. an organ such as the heart, can be determined. This way, images of 2D slices of internal tissue or an organ, may be obtained in different view directions, for example, the view directions that a sonographer uses during an examination. To that end, the model may be configured to receive one or more images of internal tissue associated with a first probe position and produce based on the one or more images associated with a first view, an output signal defining information that can be used by the robot controller to move the probe towards a second probe position that can be used to capture images with a second view. This way, based on the information provided by the model, the robot controller can move the probe towards a probe position that is associated with the target view. In an embodiment, the output signal may include direction information, e.g. a vector or information for determining a vector, which can be used by the robot controller to move the probe from a first (current) probe position to a second probe position.
The machine learning model may be trained based on training data that may be stored on a storage medium 126. Sufficient amounts of training data, preferably clinical training data associated with a heterogenous set of patients, is needed to train the model so that it can reliably control the probe to produce images for different predetermined view directions for patients of different size, gender and age. To enable autonomous scanning, a large data set is needed that does not only include images in the different view directions but also positional labels. Thus, to train the model, e.g. a deep neural network, to determine which way to move the probe when an image is provided to its input, each image may be labelled with direction information, e.g. a vector, defining a position relative to an optimal target position that is associated with a target view. This way, the model can learn to determine direction information, e.g. a vector pointing to a target position, based on echocardiographic images.
As no external datasets are available that have such properties, the computer system may be configured to generate labelled clinical training data sets itself. To that end, the computer may include a data collector module 116, that is configured to generate labelled clinical data, i.e. capture and store images of a patient and associate these images with a position and orientation of the probe, e.g. in the format of a vector as described earlier. The computer system may further include a training module 118 configured to train the model based on the training data. Depending on the type of model, e.g. a deep neural network (DNN), such as a deep convolution neural network (CNN), may be trained iteratively wherein labelled images are input to the CNN and the output is evaluated by a loss function and weights of the DNN are adjusted based on gradient descent method, e.g. a stochastic gradient descent method. The computer system may further include an execution module 120 for executing the process of autonomously generating a set of images of a part of a patient, e.g. an organ such as the heart, in different view directions based on the trained model.
It is submitted that computer system in
The data collector module may be configured to control the robotized echography system to generate training data.
Once the target view and associated target position are defined, the data collector module may start data collection based on the target position of the probe (step 204). The data collection process may include the data collector module providing a plurality of position instructions to the robot controller (step 206). A position instruction may move the probe at training position relative to the target position (step 208). When executing the position instructions, the training positions may be selected within a predetermined area around the target position. When the probe moves in a translational and/or angular direction, one or more images may be generated for the training positions of the probe. In an embodiment, the movements of the probe to the different training positions within the predetermined area may be based on a (pseudo)random value. In another embodiment, the movements may be based on a predetermined function.
Each of these images may be labelled with direction information (step 2101), wherein the direction information may define directions and, optionally, distances between training positions and the target position. Hence, the direction information may include vectors defined by a (current) position of the probe and a target position, the smaller the length of the vector, the closer the probe is to the desired target position. Thus, the direction information associated with an image indicates a direction from a current position towards an estimated target position. These labelled images may be stored as clinical training data on a storage medium of the system.
The process of executing a position instruction (step 2082,3, . . . ), i.e. moving the probe to a training position and capturing an image, and associating, e.g. labelling, an image with the direction information (steps 2102,3, . . . ) may be repeated many times so that a large collection of labelled images can be generated. Once a sufficient number of labelled images is generated, these images and associated direction information may be stored as training data and the data collection associated with that target view may be stopped (steps 212, 214). This data generation process may be repeated for different target views, so that multiple sets of labelled image data of different view directions of a target, such as the heart, can be generated in an efficient way. Moreover, the data generation process may be repeated for a heterogenous group of patients of different size, age and gender. Thus, each set of labelled data may define different images that are captured by moving the probe relative to the target position.
During the data generation, the movement of the probe to different training positions may be a random movement within a predetermined area around the target position. Alternatively, the moment of the probe to different training positions within a predetermined area around the target position may be based on a predetermined function. Once one or more target views of a target are provided, the data collection module may autonomously generate training data, i.e. large sets of labelled data, for each respective target view that can be used to train a model. e.g. a deep neural network, for controlling the probe.
Different algorithms may be used to control the generation of image data. For example, in an embodiment, the change in the probe direction may be based on the distance between the probe position and the target position. In another embodiment, a direction may be more preferential if it moves the probe closer to the target position. This preference may be dependent on the basis of a distance of the probe relative to the target position. For example, a preference to move the probe closer to the target position may be made stronger, the further the probe is removed from the target position.
Based on the evaluation by the loss function, training parameters (e.g. weights of the activation functions of the neural network) may be adjusted using well-known algorithms that are based on backpropagation and gradient descent (step 404). The process of providing a labelled image to the input of the deep neural network, evaluating the output of the deep neural network based on the loss function and adjusting the training parameters of the deep neural network may be repeated (step 406) until the estimated vector is sufficiently close to the vector that is associated with the input image (step 408). Here, the decision to stop the training process may be based on the computed loss being smaller than a certain threshold value and/or when the decreasing loss starts to increase due to overfitting. This process results in a trained neural network that is configured to receive an image from a probe at its input and to produce a vector that can be used to move the probe to a further probe position.
In some embodiments, the neural network system used by the robotized echography system may comprise a plurality of deep neural networks, wherein each neural network is trained for a particular target position. In that case, when the system is executed each of the trained neural networks used in finding a target position for an image in a predetermined viewing direction for a certain patient.
As shown in
The one or more first images may be fed to the input of the trained neural network, which generates an output signal which includes first estimated direction information associated with the first probe position (step 504). The first estimated direction information is schematically depicted in
An example of an ultrasound image provided to the input of the deep neural network and an example of direction information that can be derived from the output of the deep neural network is illustrated in
Then, based on the first direction information, the probe may be moved to a second probe position. While moving the probe to the second position, one or more second images may be captured and fed to the input of the trained neural network, which will produce second estimated direction information including a second vector pointing from the second probe position to an estimated proposition of the target view. Again, the second estimated probe position and second direction information may be stored and used to move the probe to a third probe position.
This process of moving the probe to a further probe position, determining further direction information including a further vector pointing from a current probe position towards a further estimated position associated with the target view and storing the probe position with the direction information may be repeated until a set of probe positions and associated estimated direction information are collected (step 508). Based on the collected positions and direction information an optimal probe position of a target view may be determined and the probe may be moved to the optimal probe position to capture one or more images of the target view (step 510).
This process is also shown in
In some embodiments, additional information, in particular vital sign information, such as information from ECG and/or respiration signals, may be used to train the deep neural network.
As described above, the deep neural network may be trained to predict a direction of the probe relative to a target position based on ultrasound images. As will be shown hereunder in greater detail, in some embodiments, the deep neural network may also be trained based on vital sign signals such as ECG and/or respiration signals. In an embodiment, one or more convolutional neural networks (CNNs) may be used CNNs are particular suitable for image processing. Various CNNs architectures may be selected to provide a network with an accuracy prediction. Exemplary embodiments of deep neural networks that may be used with the embodiments in this application are described hereunder in more detail with reference to
As shown in
The input layers of the deep neural network may be configured to receive input data in the form of image data, typically pixelated 2D images representing slices of internal tissue and/or organs of a patient. In one embodiment, the input layer may be configured to receive a sequence of images in time, which may be used in a recurrent model. In another embodiment, the input layer may be configured to receive an individual image.
In an embodiment, in addition to image data, the deep neural network may also be trained on the basis of other data, for example vital sign signals such as ECG signals and/or respiration signals. This data may be added to the one-dimensional array produced by flattening layer 704.
In some embodiment, the deep neural network cannot handle time dependent phenomena like movements of organs, like the hart. The heartbeat may cause parts of the hart to move in different directions. This may cause inaccuracies in the direction information as computed by the trained neural network. This problem may be addressed by adding additional information about the movement to the training data. Hence, one may extract features from an ECG signal that may help the deep neural network to interpreted an ultrasound image. One feature of the ECG signal is the phase of the heartbeat (which is a periodic signal). The phase of the heartbeat may be used as additional information that can be used for training the deep neural network in order to obtain a model that can more accurately determine the direction information based on an input image, e.g. a 2D slice, of the heart.
In a further embodiment, respiratory information may be used as additional information to train the deep neural network. The respiratory information may be beneficial for the deep neural network to recognize effects in images due to respiration. For example, if a patient inhales, the lungs will expand, which will be visible in the image as a “dark spot”. This information can be added to the training data and used during training of the deep neural network as described with reference to
Examples of ECG and respiratory signals are shown in
Subsequently, the features 7381,2 may be processed 740 by a trained deep neural network, which is configured to generate direction information 742 in the form of a set of coordinates which may be regarded as a vector in the space in which the probe can move, wherein the vector has an origin at the position of the probe were the image is captured and wherein the coordinates provide a direction of the vector pointing towards an estimated probe position for the target view.
It is submitted that the figures are only non-limiting examples and many different variants exists without departing from the essence of the invention. For example, in an embodiment, the network architecture depicted in
In further embodiment, the neural network be configured as a classification model. In that case, the translational and rotational axes may be divided in a predetermined number of bins. To do this, the network may classify the situation of each axis (x, y, z, yaw, pitch, roll, and optionally distance). The number of bins may have substantial impact on both the accuracy of the model and the training complexity. For example, two bins may imply a simple decision for one axis: “go left” or “go right”, three bins may allow for a third choice, “no movement”. Increasing the number of bins to five provides more granularity, allowing classes for “far left”, “a little left”, “no movement”, “a little right”, and “far right”. The number of bins is a trade-off between ease of training (few bins), and precision of output (many bins).
Translation axes may need more granularity that the rotational axis. Therefore, the x,y,z-axis may for example be divided in five bins and the yaw, pitch, roll-axis may be divided into a lower number of bins, e.g. three bins. Further a binary cross-entropy may be used for a loss function and to track the total accuracy of the problem. For backpropagation a well-known stochastic gradient descent scheme may be used.
As already discussed with reference to
The probe should be positioned against the skin of the patient such that a force is exerted on the patient. Control of the force exerted onto the patient is important for obtaining good image quality. The force may be controllable between a range starting from a very light contact force up to a strong force, which is still within the comfortable range for the patient.
In order to allow the probe state to make the same movement as a human wrist, the yaw movement should be independent of the roll and pitch movement. Therefore, to achieve yaw, pitch and roll movements the yaw actuator is controlled as the last actuator (before contact with the patient) of the three series connected actuators. The yaw direction provides a movement effect that is similar to the human wrist.
Further, the probe holder may be further rotatably connected to mounting structure 912 which is attached to linear robot as described with reference to
Thus, in contrast to known robotized echography systems, the echography system described with reference to
A further advantage of the system depicted in
In an embodiment, such prohibited areas may be recognized by the system through the force feedback system. For example, when the probe moves around the target spot, a change in vertical z-axis of the probe larger than a certain threshold value may indicate that the probe moves into an area comprising breast tissue. In that case, the probe may move to a position in the data collection area that has a lower z-axis value. In another embodiment, a prohibited area may be marked by a mechanical element connected to the support structure. For example, in an embodiment, a breast may be placed against a suitable curved holder 1002 which may be mechanically mounted to the bed. The holder may prevent the probe from moving into a prohibited probe area.
These embodiments may be especially useful when imaging the apex of the heart, which is located at the left low border of the heart and located just under the breast. To avoid running over breast tissue with the probe, the target area may be approached by the probe from lateral and distal areas only, from where apex is expected, and not from above/from breast tissue. This area (distal or lateral from apex) may be found by using parasternal information, that shows the location of the apex too. This way, the search and data gathering may be limited to a part of the data collection area, instead of the full data collection area around target location.
It is submitted that the embodiments in this application are not limited to the embodiments illustrated in the figures. For example, while the figures illustrate a robotized echography system in which the patient is positioned in prone position on the support structure, in some embodiments, the system may scan body parts while the patient is arranged on the support structure in another pose, e.g. a patient on the back or the side are also, possible. Further, instead of arranging the probe under the support structure, in some embodiments, it is also possible that the robotic system comprising the robot is arranged over the patient. In that case, the support system does not need to have one or more openings or recesses for providing the probe access to one or more body parts of a patient.
The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses. Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2028640 | Jul 2021 | NL | national |
The present application is a § 371 national phase entry of International patent application Serial No. PCT/NL2022/050388, filed Jul. 6, 2022, and published in English, and claims priority from Netherlands application no. 2028640 filed on Jul. 6, 2021.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/NL2022/050388 | 7/6/2022 | WO |
