Patent Application
20240298980
Some applications of the present invention generally relate to medical apparatus and methods. Specifically, some applications of the present invention relate to apparatus and methods for use in procedures that are performed on skeletal anatomy.
Approximately 5 million spine surgeries are performed annually worldwide. Traditional, manual surgery is known as freehand surgery. Typically, for such procedures, a 3D scan (e.g., a CT and/or MRI) scan is performed prior to surgery. A CT scan is typically performed for bony tissue (e.g., vertebra), and an MRI scan is typically performed for soft tissue (e.g., discs).
Reference is made to
A minority of procedures are performed using Computer Aided Surgery (CAS) systems that provide “GPS-like” navigation and/or robotics. Such systems typically make use of CT and/or MRI images that are generated before the patient is in the operating room, or when the patient is within the operating room, but typically before an intervention has commenced. The CT and/or MRI images are registered to the patient's body, and, during surgery, tools are navigated upon the images, the tools being moved manually, robotically or both.
Typically, in CAS procedures, a uniquely-identifiable location sensor is attached to each tool that needs to be tracked by the CAS system. Each tool is typically identified and calibrated at the beginning of the procedure. In addition, a uniquely-identifiable reference sensor is attached, typically rigidly, to the organ. In the case of spinal surgery, the reference sensor is typically drilled into, or fixated onto, the sacrum or spine, and, if surgery is performed along a number of vertebrae, the reference sensor is sometimes moved and drilled into a different portion of the spine, mid-surgery, in order to always be sufficiently close to the surgical site. The images to be navigated upon (e.g., CT, MRI), which are acquired before the patient is in the operating room, or when the patient is within the operating room, but before an intervention has commenced, are registered to the patient's body or a portion thereof. In order to register the images to the patient's body, the current location of the patient's body is brought into the same reference frame of coordinates as the images using the reference sensor. The location sensors on the tools and the reference sensor on the patient's body are then tracked, typically continuously, in order to determine the locations of the tools relative to the patient's body, and a symbolic representation of the tool is displayed upon the images that are navigated upon. Typically, the tool and the patient's body are tracked in 5-6 degrees of freedom.
There are various techniques that are utilized for the tracking of tools, as well as applicable portions of the patient's body, and corresponding location sensors are used for each technique. One technique is infrared (“IR”) tracking, whereby an array of cameras track active IR lights on the tools and the patient's body, or an array of beams and cameras track passive IR reflectors on the tools and the patient's body. In both categories of IR tracking, lines of sight must be maintained at all times between the tracker and the tools. For example, if the line of sight is blocked by the surgeon's hands, this can interfere with the tracking. Another technique is electromagnetic or magnetic tracking, whereby a field generator tracks receivers, typically coils, on the tools and the patient's body. For those latter techniques, environmental interferences from other equipment must be avoided or accounted for. In each of the techniques, the location sensors of the navigation system are tracked using tracking components that would not be present in the operating room in the absence of the navigation system (i.e., the location sensors do not simply rely upon imaging by imaging devices that are typically used in an orthopedic operating room in the absence of the navigation system).
A further technique that can be used with a robotically-driven tool is to start with the tool at a known starting point relative to the patient's body, and to then record motion of the tool from the starting point. Alternatively, such tools can be tracked using the above-described techniques.
Given the nature of CAS procedures, the equipment required for such procedures is typically more expensive than that of non-CAS procedures (non-CAS procedures including open procedures, mini-open procedures, or minimally-invasive procedures that are not computer aided with respect to the guidance of tools). Such procedures typically limit tool selection to those fitted with location sensors as described above, and typically require such tools to be individually identified and calibrated at the beginning of each surgery.
U.S. 2019/0350657 to Tolkowsky, entitled “Apparatus and methods for use with skeletal procedures,” which is assigned to the assignee of the present application, and is incorporated herein by reference, describes apparatus and methods including acquiring 3D image data of a skeletal portion. While a portion of a tool is disposed at a first location with respect to the skeletal portion, 2D x-ray images are acquired from respective views. A computer processor determines the first location with respect to the 3D image data, based upon identifying the first location within the first and second 2D x-ray images. Subsequent to moving the portion of the tool to a second location, an additional 2D x-ray image is acquired from a single image view. The computer processor derives the second location with respect to the 3D image data, based upon identifying the second location of the portion of the tool within the additional 2D x-ray image, and the determined first location of the portion of the tool with respect to the 3D image data. Other applications are also described.
International Patent Application PCT/IL2018/050732 to Tolkowsky, which published as WO 2019/012520, filed Jul. 5, 2018, entitled, “Apparatus and methods for use with image-guided skeletal procedures,” which is assigned to the assignee of the present application, and is incorporated herein by reference, describes apparatus and methods including acquiring 3D image data of a skeletal portion. A computer processor is used to designate a skin-level incision point or a skeletal-portion level entry point within the body of the subject and associate the designated point with the 3D image data. A radiopaque element is positioned on the body of the subject with respect to the skeletal portion and an intraoperative 2D radiographic image is acquired of the skeletal portion, such that the radiopaque element appears in the 2D radiographic image. The computer processor (i) registers the 2D radiographic image to the 3D image data such that the designated point appears in the 2D radiographic image, and (ii) displays a location of the designated point with respect to the radiopaque element on the 2D radiographic image. Other applications are also described.
In accordance with some applications of the present invention, 3D image data of a subject is used in conjunction with 2D images of the subject for planning and performing surgery upon that subject. For some applications, the 3D image data is generated by means of image processing from multiple 2D images of the subject. For some applications, the 3D image data is generated by means of machine learning from 2D images of the subject. For some applications, such machine learning is assisted by data sets collected from other subjects, each data set typically comprising 3D image data, 2D images of a subject, and typically one or more acquisition parameters for the 2D images. For some applications, 2D images are acquired during surgery using one or more acquisition parameters whose values correspond to the values of the same parameters during the acquisition of one or more of the 2D images used for previously generating the 3D image data of the subject.
In accordance with some applications of the present invention, acquired 3D image data may be used for generating 2D projection images that correspond to current positions or views of the 2D imaging device, e.g., an x-ray C-Arm, without necessarily acquiring 2D images from those positions or views. As a result, a surgeon may move the 2D imaging device to different potential image views and, for each potential image view, see a 2D projection image based on the 3D image data. This allows the surgeon to reach a desired view of the applicable skeletal portion, or of one or more tools applied to the skeletal portion, while acquiring fewer x-ray images. Furthermore, planning data related to tool insertion may be associated with the 3D image data and projected onto any of the 2D projection images so as to allow the surgeon to see the planning data with respect to the skeletal portion from each potential image view. Alternatively or additionally, at least a portion of a tool inserted in the skeletal portion may be detected on an acquired 2D image that is registered with the 3D image data, and a simulated view of at least the portion of the tool may be projected onto any of the 2D projection images so as to allow the surgeon to see what the tool would look like from each potential image view.
There is therefore provided, in accordance with some applications of the present invention, a method for generating projection images of a skeletal portion, the method including:
For some applications, the second imaging device is a 2D x-ray imaging device mounted on a moveable C-arm.
For some applications, the method further includes subsequently to step (E), repeating steps (D) and (E).
For some applications, the method further includes iteratively repeating steps (D) and (E).
For some applications, iteratively repeating steps (D) and (E) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor, generating based on the 3D image data a 2D projection image corresponding to each of a plurality of positions of the second imaging device between the beginning and the end of the moving of the second imaging device.
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
For some applications, the method further includes:
For some applications, planning for the tool insertion includes designating at least one locational element selected from the group consisting of: an insertion trajectory, a skin-level entry point, a bone-level entry point, an intermediate point along an insertion trajectory between a skin-level entry point and a target point, a target point, and a simulated tool positioned at any point along an insertion trajectory.
For some applications, the method further includes:
For some applications, planning for the tool insertion includes designating at least one locational element selected from the group consisting of: an insertion trajectory, a skin-level entry point, a bone-level entry point, an intermediate point along an insertion trajectory between a skin-level entry point and a target point, a target point, and a simulated tool positioned at any point along an insertion trajectory.
For some applications, the method further includes iteratively repeating steps (D), (E), and (ii).
For some applications, iteratively repeating steps (D), (E), and (ii) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the planning from step (i) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the planning from step (i) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
For some applications, the method further includes:
For some applications, the method further includes subsequently to step (C), using the at least one computer processor, projecting the planning from step (i) onto the starting image.
For some applications, the method further includes subsequently to step (E), using the at least one computer processor, projecting the planning from step (i) onto the subsequent image.
For some applications, the method further includes subsequently to step (ii), using the at least one computer processor, projecting the planning from step (i) onto the subsequent image generated in the repetition of step (E).
For some applications, the method further includes iteratively repeating steps (D) and (E).
For some applications, iteratively repeating steps (D) and (E) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning from step (i) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning from step (i) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
For some applications, generating the starting image includes:
For some applications, the method further includes subsequently to step (ii), repeating steps (D), (E), and (ii).
For some applications, the method further includes iteratively repeating steps (D), (E), and (ii).
For some applications, iteratively repeating steps (D), (E), and (ii) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (i) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (i) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
For some applications, generating the starting image includes:
For some applications, the method further includes subsequently to step (E), using the at least one computer processor, projecting a simulated view of the at least a portion of the tool detected in step (i) onto the subsequent image.
For some applications, the method further includes subsequently to step (ii), using the at least one computer processor, projecting a simulated view of the at least a portion of the tool detected in step (i) onto the subsequent image generated in the repetition of step (E).
For some applications, the method further includes iteratively repeating steps (D) and (E).
For some applications, iteratively repeating steps (D) and (E) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (i) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (i) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
There is further provided, in accordance with some applications of the present invention apparatus for generating projection images of a skeletal portion, the apparatus for use with:
For some applications, the at least one computer processor is configured to repeat step (B).
For some applications, the at least one computer processor is configured to iteratively repeat step (B).
For some applications, the at least one computer processor is configured to iteratively repeat step (B) by:
For some applications, the at least one computer processor is configured to display as a movie the 2D projection images generated during the moving of the second imaging device.
For some applications, the at least one computer processor is configured to display as a movie the 2D projection images generated during the moving of the second imaging device by displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
There is further provided, in accordance with some applications of the present invention, a method for generating projection images of a skeletal portion, the method including:
For some applications, the second imaging device is a 2D x-ray imaging device mounted on a moveable C-arm.
For some applications, planning for the tool insertion includes designating at least one locational element selected from the group consisting of: an insertion trajectory, a skin-level entry point, a bone-level entry point, an intermediate point along an insertion trajectory between a skin-level entry point and a target point, a target point, and a simulated tool positioned at any point along an insertion trajectory.
For some applications, the method further includes, subsequently to step (E):
For some applications, the method further includes, using the at least one computer processor:
For some applications, the method further includes repeating steps (F), (G), and (H).
For some applications, the method further includes iteratively repeating steps (F), (G), and (H).
For some applications, iteratively repeating steps (F), (G), and (H) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the planning from step (B) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the planning from step (B) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
For some applications, the method further includes repeating steps (F) and (G).
For some applications, the method further includes subsequently to the initial performing of step (G), using the at least one computer processor, projecting the planning from step (i) onto the second 2D projection image.
For some applications the method further includes subsequently to the repetition of step (G), using the at least one computer processor, projecting the planning from step (i) onto the second 2D projection image generated in the repetition of step (G).
For some applications, the method further includes iteratively repeating steps (F) and (G).
For some applications, iteratively repeating steps (F) and (G) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning from step (B) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning from step (B) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
There is further provided, in accordance with some applications of the present invention, apparatus for generating projection images of a skeletal portion, the apparatus for use with:
For some applications, the planning data includes at least one locational element selected from the group consisting of: an insertion trajectory, a skin-level entry point, a bone-level entry point, an intermediate point along an insertion trajectory between a skin-level entry point and a target point, a target point, and a simulated tool positioned at any point along an insertion trajectory.
For some applications, the at least one computer processor is further configured to:
For some applications, the at least one computer processor is configured to repeat step (E).
For some applications, the at least one computer processor is configured to, subsequently to the initial performing of step (E), project the planning data from step (B) onto the second 2D projection image.
For some applications, the at least one computer processor is configured to, subsequently to the repetition of step (E), project the planning data from step (B) onto the second 2D projection image generated in the repetition of step (E).
For some applications, the at least one computer processor is configured to iteratively repeat step (E).
For some applications, the at least one computer processor is configured to iteratively repeat step (E) by:
For some applications the at least one computer processor is configured to display as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning data from step (B) projected thereon.
For some applications, the at least one computer processor is configured to display as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning data from step (B) projected thereon, by displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
There is further provided in accordance with some applications of the present invention, a method for generating projection images of a skeletal portion, the method including:
For some applications, the method further includes repeating steps (E), (F), and (G).
For some applications, the method further includes iteratively repeating steps (E), (F), and (G).
For some applications, iteratively repeating steps (E), (F), and (G) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (C) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (C) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
For some applications, the method further includes repeating steps (E) and (F).
For some applications, the method further includes subsequently to the initial performing of step (F), using the at least one computer processor, projecting a simulated view of the at least a portion of the tool detected in step (C) onto the 2D projection image.
For some applications, the method further includes subsequently to the repetition of step (F), using the at least one computer processor, projecting a simulated view of the at least a portion of the tool detected in step (C) onto the 2D projection image generated in the repetition of step (F).
For some applications, the method further includes iteratively repeating steps (E) and (F).
For some applications, iteratively repeating steps (E) and (F) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (C) projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (C) projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
For some applications, the method further includes:
For some applications, planning for the tool insertion includes designating at least one locational element selected from the group consisting of: an insertion trajectory, a skin-level entry point, a bone-level entry point, an intermediate point along an insertion trajectory between a skin-level entry point and a target point, a target point, and a simulated tool positioned at any point along an insertion trajectory.
For some applications, the method further includes:
For some applications, the method further includes repeating steps (E), (F), (G), and (ii).
For some applications, the method further includes iteratively repeating steps (E), (F), (G), and (ii).
For some applications, iteratively repeating steps (E), (F), (G), and (ii) includes:
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, each of the 2D projection images having projected thereon the planning from step (i) and the simulated view of the at least a portion of the tool detected in step (C).
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device, each of the 2D projection images having projected thereon the planning from step (i) and the simulated view of the at least a portion of the tool detected in step (C).
For some applications, the method further includes:
For some applications, the method further includes subsequently to step (D), using the at least one computer processor, projecting the planning from step (i) onto the acquired 2D image.
For some applications, the method further includes subsequently to step (F), using the at least one computer processor, projecting the planning from step (i) onto the 2D projection image generated in step (F).
For some applications, the method further includes subsequently to step (ii), using the at least one computer processor, projecting the planning from step (i) and a simulated view of the at least a portion of the tool onto the 2D projection image generated in the repetition of step (F).
For some applications, the method further includes iteratively repeating steps (E) and (F).
For some applications, iteratively repeating steps (E) and (F) includes moving the second imaging device and, while moving the second imaging device, using the at least one computer processor:
For some applications, the method further includes, using the at least one computer processor, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning from step (i) and a simulated view of the at least a portion of the tool projected thereon.
For some applications, displaying as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the planning from step (i) and a simulated view of the at least a portion of the tool projected thereon, includes displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
There is further provided in accordance with some applications of the present invention, apparatus for generating projection images of a skeletal portion, the apparatus for use with:
For some applications, the at least one computer processor is configured to repeat step (E).
For some applications, the at least one computer processor is configured to, subsequently to the initial performing of step (E), project a simulated view of the at least a portion of the tool detected in step (C) onto the 2D projection image.
For some applications, the at least one computer processor is configured to, subsequently to the repetition of step (E), project a simulated view of the at least a portion of the tool detected in step (C) onto the 2D projection image generated in the repetition of step (E).
For some applications, the at least one computer processor is configured to iteratively repeat step (E).
For some applications, the at least one computer processor is configured to iteratively repeat step (E) by:
For some applications, the at least one computer processor is configured to display as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (C) projected thereon.
For some applications, the at least one computer processor is configured to display as a movie the 2D projection images generated during the moving of the second imaging device, at least one of the 2D projection images having the simulated view of the at least a portion of the tool detected in step (C) projected thereon, by displaying the 2D projection images as a movie in real-time during the moving of the second imaging device.
There is further provided in accordance with some applications of the present invention, a method for performing a procedure with respect to a skeletal portion within a body of a subject, the method including:
For some applications, the method further includes, using the at least one computer processor, displaying the location of the portion of the tool with respect to the 3D image data.
For some applications, identifying a location of the portion of the tool with respect to the skeletal portion within the first and second 2D x-ray images includes identifying a location of the portion of the tool with respect to the skeletal portion, within the first and second 2D x-ray images, by means of image processing.
For some applications:
For some applications, the method further includes subsequently to step (B):
For some applications, the method further includes, using the at least one computer processor:
For some applications, the method further includes, using the at least one computer processor, displaying the location of the portion of the tool with respect to the at least one designated locational element, with respect to the 3D image data.
For some applications, acquiring the first 2D x-ray image of at least the portion of the tool and the skeletal portion from the first view, the first view being similar to the view of the one of the multiple 2D images used for generating the 3D image data, includes acquiring the first 2D x-ray images using one or more acquisition parameters having a value that is similar to the value of the corresponding one or more acquisition parameters from the acquisition of the one of the multiple 2D x-ray images that were used for generating the 3D image data.
For some applications, acquiring the second 2D x-ray image of at least the portion of the tool and the skeletal portion from the first view, the second view being similar to the view of the other one of the multiple 2D images used for generating the 3D image data, includes acquiring the second 2D x-ray images using one or more acquisition parameters having a value that is similar to the value of the corresponding one or more acquisition parameters from the acquisition of the other one of the multiple 2D x-ray images that were used for generating the 3D image data.
For some applications:
For some applications:
There is further provided in accordance with some applications of the present invention, apparatus for performing a procedure with respect to a skeletal portion within a body of a subject, the apparatus for use with a 2D imaging device configured to acquire 2D x-ray images of at least the skeletal portion, the apparatus including:
There is further provided in accordance with some applications of the present invention, a method for generating 3D image data of a target skeletal portion, the method including:
For some applications, training the machine learning engine includes using training data including a set of 2D training images and corresponding 3D training image data.
For some applications, the method further includes, using the at least one computer processor, subsequently to step (D), reversing the reduction of the visibility of the tool in the at least some of the multiple 2D images such that the tool is once again visible in the at least some of the multiple 2D images.
For some applications, the method further includes:
For some applications, the method further includes, using the at least one computer processor, generating a generally-straight-line representing the tool with respect to the 3D image data.
There is further provided in accordance with some applications of the present invention, apparatus for generating 3D image data of a target skeletal portion, the apparatus for use with a 2D imaging device configured to acquire 2D images of at least the skeletal portion, the apparatus including:
For some applications, the machine learning engine is trained using training data including a set of 2D training images and corresponding 3D training image data.
For some applications, the at least one computer processor is configured to, subsequently to step (B), reverse the reduction of the visibility of the tool in the at least some of the multiple 2D images such that the tool is once again visible in the at least some of the multiple 2D images.
For some applications, the at least one computer processor is further configured to:
For some applications, the at least one computer processor is further configured to generate a generally-straight-line representing the tool with respect to the 3D image data.
There is further provided in accordance with some applications of the present invention, a method for generating 3D image data of a target skeletal portion, the method including:
For some applications, the method further includes, using at least one computer processor, detecting the tool in the 3D image data by means of image processing.
For some applications, the method further includes, using the at least one computer processor, generating a generally-straight-line representing the tool with respect to the 3D image data.
There is further provided in accordance with some applications of the present invention, apparatus for generating 3D image data of a target skeletal portion, the apparatus for use with a 2D imaging device configured to acquire 2D images of at least the skeletal portion, the apparatus including:
For some applications, the apparatus further includes at least one computer processor configured to detect the tool in the 3D image data by means of image processing.
For some applications, the at least one computer processor is further configured to generate a generally-straight-line representing the tool with respect to the 3D image data.
The present invention will be more fully understood from the following detailed description of embodiments thereof, taken together with the drawings, in which:
Reference is now made to
System 20 typically includes a computer processor 22, which interacts with a memory 24, and one or more user interface device 26. Typically, the user interface devices include one or more input devices, such as a keyboard 28 (as shown), and one or more output devices, e.g., a display 30, as shown. Inputs to, and outputs from, the computer processor that are described herein are typically performed via the user interface devices. For some applications, the computer processor as well as the memory and the user interface devices, are incorporated into a single unit, e.g., a tablet device, an all-in-one computer, and/or a laptop computer.
For some applications, the user interface devices include a mouse, a joystick, a touchscreen device (such as a smartphone or a tablet computer) optionally coupled with a stylus, a touchpad, a trackball, a voice-command interface, a hand-motion interface, and/or other types of user interfaces that are known in the art. For some applications, the output device includes a head-up display and/or a head-mounted display, such as Google Glass® or a Microsoft HoloLens®. For some applications, the computer processor generates an output on a different type of visual, text, graphics, tactile, audio, and/or video output device, e.g., speakers, headphones, a smartphone, or a tablet computer. For some applications, a user interface device acts as both an input device and an output device. For some applications, computer processor 22 generates an output on a computer-readable medium (e.g., a non-transitory computer-readable medium), such as a disk or a portable USB drive. For some applications, the computer processor comprises a portion of a picture archiving and communication system (PACS), and is configured to receive inputs from other components of the system, e.g., via memory 24. Alternatively or additionally, the computer processor is configured to receive an input on a computer-readable medium (e.g., a non-transitory computer-readable medium), such as a disk or a portable USB drive. It is noted that, for some applications, more than one computer processor is used to perform the functions described herein as being performed by computer processor 22.
Typically, 3D image data are acquired before the subject is in the operating room for the procedure, or when the subject is in the operating room, but before an intervention has commenced. For example, 3D CT image data of the portion of the skeletal anatomy upon which the procedure is to be performed (and/or neighboring portions of the anatomy) may be acquired using a CT scanner 32. Alternatively or additionally, 3D MRI image data of the portion of the skeletal anatomy upon which the procedure is to be performed (and/or neighboring portions of the anatomy) may be acquired using an MRI scanner. For some applications, 3D x-ray data are acquired. For some applications, 3D image data is generated by means of image processing and image reconstruction from multiple 2D images. For some applications, some acquisition parameters of the 2D images are recorded for later reference or later use. For some applications, the generation utilizes techniques described by U.S. Pat. No. 9,545,233 to Sirpad et al., entitled “On-site verification of implant positioning” and incorporated herein by reference. For some applications, the generation utilizes a series of biplanar or multiplanar 2D image sets acquired along the skeletal portion, including in accordance with techniques used by the EOS Edge and/or EOS System from EOS Imaging of Paris, France.
For some applications, 3D image data of the subject is generated from 2D images of the subject by means of machine learning. For some applications, the 2D images of the subject are acquired before surgery. For some applications, the 2D images of the subject are acquired at the beginning of surgery. For some applications, the 2D images of the subject are acquired during surgery. For some applications, the 2D images are x-ray images. For some applications, some acquisition parameters of the 2D images are recorded for later reference or later use. For some applications, the generated 3D image data is similar, or identical, in its characteristics to CT image data. For some applications, the machine learning is aided by data sets from multiple other subjects, with each such data set typically comprising 3D image data, 2D images, and typically one or more of the acquisition parameters of the 2D images of a specific subject. For some applications, the machine learning is performed using techniques employed by Zebra Medical Vision of Kibbutz Shefayim, Israel. (For example, see https://zebramedblog.wordpress.com/2019/12/19/another-dimension-to-zebras-ai-how-we-impact-the-orthopedic-world/or https://orthospinenews.com/2020/12/08/zebra-medical-vision-secures-a-7th-fda-clearance-for-its-patented-breakthrough-in-orthopedic-surgery-planning/). U.S. Pat. No. 10,867,436 to Oved, entitled “Systems and Methods for Reconstruction of 3D Anatomical Images from 2D Anatomical Images” and incorporated herein by reference, describes a method of training a neural network for reconstructing of a 3D point cloud from 2D image(s). Oved further describes the reconstruction of 3D images depicting a target anatomical structure of a patient from 2D images depicting the target anatomical structure of the patient, including wherein the anatomical structures are skeletal structures, the 2D images are x-ray images, and the reconstructed 3D image data is akin to the data generated by a CT scan.
For some applications, 3D image data of the subject is generated from 2D images of the subject by means of machine learning and without necessarily using training data. A research paper to Chen at al., entitled “Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer,” (hereinafter “Chen et al.”) describes an AI system that can predict 3D properties of 2D images without any 3D training data, and is incorporated herein by reference. Chen et al. describe a framework that, when applied to a neural network, learns to predict shape, texture, and light from 2D images and generate 3D textured shapes.
It should be noted that the terms “deep learning”, “machine learning” and “AI” are used herein interchangeably.
For some applications, the 2D images acquired during surgery are acquired by the same imaging device that was used for acquiring the 2D images that were used for generating the subject's 3D image data. Such arrangement typically improves the accuracy of the subsequent registration of 2D images acquired during surgery with the subject's 3D image data.
Typically, the 3D image data are transferred to memory 24, and are retrieved from the memory by computer processor 22. It is noted that for illustrative purposes,
During the procedure, real time 2D images are acquired by a radiographic imaging device, e.g., a C-arm 34 (as shown), which acquires 2D x-ray images. For some applications, such 2D images are acquired by an imaging device (such as an o-arm or a 3D x-ray c-arm) situated in the operating room and also capable of generating 3D images. For example, such imaging device may be used for generating 3D image data at the beginning of the intervention in order to image the baseline anatomy in 3D, and then again at the latter part of the intervention in order to evaluate its outcomes (such as how well implants were positioned), and in between be used similarly to a regular c-arm in order to generate 2D during the intervention. For some applications, such device fulfils both the roles of the 3D CT and the 2D c-arm, as such roles are described throughout this document with respect to embodiments of the present invention.
For some applications, the 2D images are captured in real time by a frame grabber of system 20 that is connected to an output port of the C-arm. Alternatively or additionally, system 20 and the C-arm are connected to one another via a PACS network (or other networking arrangement, wired or wireless) to which system 20 and C-arm 34 are connected, and the 2D images are transferred, once acquired, to system 20 via the PACS network (e.g., via memory 24). Alternatively or additionally, the C-arm sends image files, for example in DICOM format, directly to system 20 (e.g., via memory 24).
Typically, the interventional part of a procedure that is performed on skeletal anatomy, such as the spine, commences with the insertion of a tool, such as a Jamshidi™ needle 36 which is typical for minimally-invasive (or less-invasive) surgery. A Jamshidi™ needle typically includes an inner tube and an outer tube. The Jamshidi™ needle is typically inserted to or towards a target location, at which point other tools and/or implants are inserted using the Jamshidi™ needle. Typically, in open surgery, for lower-diameter tools and/or implants, the inner tube of the Jamshidi™ needle is removed, and the tool and/or implant is inserted via the outer tube of the Jamshidi™ needle, while for larger-diameter tools and/or implants, the tool and/or implant is inserted by removing the inner tube of the Jamshidi™ needle, inserting a stiff wire through the outer tube, removing the outer tube, and then inserting the tool and/or implant along the stiff wire. For minimally-invasive surgery, the aforementioned steps (or similar steps thereto) are typically performed via small incisions. Alternatively, for more-invasive or open surgery, the tool inserted may be, for example, a pedicle finder and/or a pedicle marker.
It is noted that, in general throughout the specification and the claims of the present application, the term “tool” should be interpreted as including any tool or implant that is inserted into any portion of the skeletal anatomy during a procedure that is performed upon the skeletal anatomy. Such tools may include flexible, rigid and/or semi-rigid probes, and may include diagnostic probes, therapeutic probes, and/or imaging probes. For example, the tools may include Jamshidi™ needles, other needles, k-wires, pedicle finders, pedicle markers, screws, nails, rods, other implants, implant delivery probes, drills, endoscopes, probes inserted through an endoscope, tissue ablation probes, laser probes, balloon probes, injection needles, tissue removal probes, drug delivery probes, stimulation probes, denervation probes, dilators, guides, patient-specific guides, surgical guides, patient-specific surgical guides, a robot, a steerable arm (e.g., a robotic arm or a manually-steerable arm), a tool held by a robot, a tool inserted within a guide, aiming devices, direction-indicating devices, a tool for diagnosing or treating stenosis or for supporting the diagnosis or treatment of stenosis, etc. Typically, such procedures include spinal stabilization procedures, such as vertebroplasty (i.e., injection of synthetic or biological cement in order to stabilize spinal fractures), kyphoplasty (i.e., injection of synthetic or biological cement in order to stabilize spinal fractures, with an additional step of inflating a balloon within the area of the fracture prior to injecting the cement), fixation (e.g., anchoring two or more vertebrae to each other by inserting devices such as screws into each of the vertebrae and connecting the screws with rods), fixation and fusion (i.e., fixation with the additional step of an implant such as a cage placed in between the bodies of the vertebrae), biopsy of suspected tumors, tissue ablation (for example, RF or cryo), injection of drugs, and/or endoscopy (i.e., inserting an endoscope toward a vertebra and/or a disc, for example, in order to remove tissue (e.g., disc tissue, or vertebral bone) that compresses nerves).
Reference is now made to
Reference is now made to
As may be observed, the view of the vertebra that is important for determining the entry point, insertion direction, and insertion depth of the tool is shown in the axial 2D image slice of
In accordance with some applications of the present invention, the intra-procedural location of a tool is determined with respect to 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data), in a non-CAS procedure (e.g., in an open, mini-open and/or minimally-invasive procedure). The techniques described herein are typically practiced without requiring the fitting of location sensors (such as infrared transmitters or reflectors, or magnetic or electromagnetic sensors) to the tool or to the subject, and without requiring identification and/or calibration of tools prior to the procedure. The techniques described herein are typically practiced without requiring the fitting of any radiopaque marker to the tool, rather they rely on the existing radio-opacity of the tool for its identification in the x-ray images. The techniques described herein are typically practiced without requiring knowledge of the precise geometry and/or the dimensions of the tool for its identification in the x-ray images. The techniques described herein typically do not require tracking the location of the subject's body or the applicable portion of the subject's body, and do not assume any knowledge of the location coordinates of the subject's body in some reference frame. The techniques described herein typically do not require location sensors that rely upon tracking technologies (e.g., electromagnetic or IR tracking technologies) that are not typically used in an orthopedic operating room when not using CAS systems. Further typically, the techniques described herein are practiced without requiring knowledge of any precise parameters of any individual pose of the 2D radiographic imaging device (e.g., C-arm 34), and typically without requiring poses of the 2D radiographic imaging device (e.g., C-arm 34) to be tracked relative to each other, and/or relative to the position of the subject. For some applications, 2D radiographic images (e.g., 2D x-ray images) are acquired from two or more views, by moving a radiographic imaging device to respective poses between acquisitions of the images of respective views. Typically, a single x-ray source is used for acquisition of the 2D x-ray images, although, for some applications, multiple sources are used. In general, where views of the 2D radiographic imaging device are described herein as being AP, lateral, oblique, etc., this should not be interpreted as meaning that images must be acquired from precisely such views, rather acquiring images from generally such views is typically sufficient. Typically, the techniques described herein are tool-neutral, i.e., the techniques may be practiced with any applicable tool and typically without any modification and/or addition to the tool.
It is noted that although some applications of the present invention are described with reference to 3D CT imaging, the scope of the present invention includes using any 3D imaging, e.g., MRI, 3D x-ray imaging, 3D ultrasound imaging, and/or other modalities of 3D imaging, mutatis mutandis. Such imaging may be performed prior to, at the commencement of, and/or at some point during, an intervention. For example, the 3D imaging may be performed before the subject has been placed within the operating room, when the subject is first placed within the operating room, or at some point when the subject is in the operating room, but prior to the insertion of a given tool into a given target portion, etc. Similarly, although some applications of the present invention are described with reference to 2D radiographic or x-ray imaging, the scope of the present invention includes using any 2D imaging, e.g., ultrasound and/or other modalities of 2D imaging, mutatis mutandis. Although some applications of the present invention are described with reference to procedures that are performed on skeletal anatomy and/or vertebrae of the spine, the scope of the present invention includes applying the apparatus and methods described herein to other orthopedic interventions (e.g., a joint (e.g., shoulder, knee, hip, and/or ankle) replacement, joint repair, fracture repair (e.g., femur, tibia, and/or fibula), a procedure that is performed on a rib (e.g., rib removal, or rib resection), vascular interventions, cardiovascular interventions, neurovascular interventions, abdominal interventions, diagnostic interventions, therapeutic irradiations, and/or interventions performed on other portions of a subject, including interventions in which 3D image data are acquired prior to the intervention and 2D images are acquired during the intervention, mutatis mutandis.
Reference is now made to
Typically, sets 50 of markers 52 are attached, e.g., by an adhesive disposed on a surface of the marker, e.g., an adhesive disposed on support 53, to a surface of the subject in a vicinity of a site, e.g., skeletal portion, at which an intervention is to be performed, and such that at least some of the markers appear in 2D radiographic images that are acquired of the intervention site from typical imaging views for such an intervention. For example, for a procedure that is performed on the subject's vertebra(e) and particularly within one or more vertebral bodies, the markers are typically placed on the subject's back in a vicinity of the site of the spinal intervention, such that at least some of the markers appear in 2D radiographic images that are acquired of the intervention site from AP imaging views, and potentially from additional imaging views as well. For some applications, the markers are placed on the subject's side in a vicinity of the site of the spinal intervention, such that at least some of the markers appear in 2D radiographic images that are acquired of the intervention site from a lateral imaging view. For some applications, the markers are placed on the subject's back, such that at least some of the markers are level with the subject's sacrum.
For some applications, known dimensions of, or distances between (e.g., markers spaced at 1 cm from other another), radiopaque markers 52 are used in scaling 2D x-ray images comprising portions of the marker set prior to the registration of such 2D images with a 3D data set. Such registration is further described hereinbelow. Typically, and as known in the art, scaling of the images to be registered, when performed prior to the actual registration, facilitates the registration.
For some applications, the set of markers comprises an arrangement wherein portions thereof are visible from different image views. For some applications, such arrangement facilitates for the surgeon the intra-procedural association of elements, including anatomical elements such as a vertebra, seen in a first x-ray image acquired from one view, for example AP, with the same elements as seen in a second x-ray image acquired from a second view, for example lateral. For some applications, such association is performed manually by the surgeon referring to the radiopaque markers and identifying markers that have a known association with one another in the x-ray images, e.g., via matching of alphanumeric characters or distinct shapes. Alternatively or additionally, the association is performed automatically by computer processor 22 of system 20 by means of image processing.
Using known techniques, such association between images, for example of a particular vertebra seen on those images, often requires inserting a tool into or near to, or placing a tool upon, a vertebra of interest such that the tool identifies that vertebra in both images.
According to embodiments of the present invention, association between images acquired from different views (for example AP and lateral, or AP and oblique, or lateral and oblique) is facilitated by any of the following techniques:
For some applications, radiopaque marker set 50 is in the form of a frame-like label, such as is shown in
Reference is now made to
Typically, surgery on skeletal anatomy commences with attaching a sterile surgical drape, typically an incision drape, at and around the surgical site. In the case of spinal surgery, the surgical approach may be anterior, posterior, lateral, oblique, etc., with the surgical drape placed accordingly. For such applications, sets 50 of markers 52 are typically placed above the surgical drape. Alternatively, sets of markers are placed on the subject's skin (e.g., if no surgical drape is used). For some applications, sets of markers are placed under the subject's body, on (e.g., attached to) the surgical table, and/or such that some of the markers are above the surgical table in the vicinity of the subject's body. For some applications, a plurality of sets of markers are used. For example, multiple sets of markers may be placed adjacently to one another. Alternatively or additionally, one or more sets of markers may be placed on the subject's body such that at least some markers are visible in each of a plurality of x-ray image views, e.g., on the back or stomach and/or chest for the AP or PA views, and on the side of the body for the lateral view. For some applications, a single drape with markers disposed thereon extends, for example, from the back to the side, such that markers are visible in both AP and lateral x-ray image views.
For some applications, a first marker set 50a and second marker set 50b are placed on the subject's body such that, at each (or most) imaging view applied during the procedure for the acquisition of images, at least one of the first and second markers (or a portion thereof) is visible in the acquired images. For example, such as is shown in
For some applications, only a first set of markers is placed on the subject's body, typically at a position (e.g., along the spine) that enables it to be visible from each (or most) imaging view applied during the procedure for the acquisition of images.
For some applications, a first marker set 50a and a second marker set 50b are each modular. For example, a marker in the form of a notched ruler, may comprise several ruler-like modules. Typically, the number of modules to be actually applied to the subject's body is related to the overall size of the subject, to the location of the targeted vertebra(e) relative to the anatomical reference point (e.g., sacrum) at which placement of the marker sets begins, or to a combination thereof. For example, a target vertebra in the lumbar spine may require one module, a target vertebra in the lower thoracic spine may require two modules, a target vertebra in the upper thoracic spine may require three modules, etc.
Typically, the sets of markers are positioned on either side of the subject's spine such that even in oblique x-ray image views of the intervention site (and neighboring portions of the spine), at least radiopaque markers belonging to one of the sets of markers are visible. Further typically, the sets of markers are positioned on either side of the subject's spine such that even in zoomed-in views acquired from the direction of the tool insertion, or in views that are oblique (i.e., diagonal) relative to the direction of tool insertion, at least radiopaque markers belonging to one of the sets of markers are visible. Typically, the sets of radiopaque markers are placed on the subject, such that the radiopaque markers do not get in the way of either AP or lateral x-ray images of vertebrae, such that the radiopaque markers do not interfere with the view of the surgeon during the procedure, and do not interfere with registration of 2D and 3D image data with respect to one another (which, as described hereinbelow, is typically based on geometry of the vertebrae).
For some applications, the sets of markers as shown in
Radiopaque markers 52 are typically in the form of markings (e.g., lines, notches, numbers, characters, shapes) that are visible to the naked eye (i.e., the markings are able to be seen without special equipment) as well as to the imaging that is applied. Typically, the markers are radiopaque such that the markers are visible in radiographic images. Further typically, markers that are placed at respective locations with respect to the subject are identifiable. For example, as shown in
For some applications, all markings in the marker set are visible both in the x-ray images (by virtue of being radiopaque) and to the naked eye (or optical camera). For some applications, some elements of the marker set are not radiopaque, such that they are invisible in the x-ray images and yet visible to the naked eye (or camera). For example, a central ruler placed on the subject's body may have notches or markings that correspond directly to those of one or both sets of markers that are to the side(s), and yet unlike the latter sets of markers it is not radiopaque. For some applications, when the marker set is placed dorsally, such a ruler facilitates for the surgeon the localization of specific spinal elements (e.g., vertebrae) when looking at the subject's back and yet does not interfere with the view of those same spinal elements in the x-ray images.
The marker set may include a series of discretely identifiable, e.g., distinct, radiopaque symbols (or discernible arrangements of radio-opaque markers), such as is shown in
For some applications, sets 50 of markers 52, and/or a rigid radiopaque jig are used to facilitate any one of the following functionalities:
For some applications, at least some of the functionalities listed above as being facilitated by use of sets 50 of markers 52, and/or a rigid jig are performed by computer processor 22 even in the absence of sets 50 of markers 52, and/or a rigid jig, e.g., using techniques as described herein. Typically, sets 50 of markers 52, and/or a rigid jig are used for level verification, the determination of a tool entry point or an incision site, performing measurements using rigid markers as a reference, identifying changes in a relative pose of the 2D imaging device (e.g., the x-ray C-arm) and of the subject, and providing general orientation. All other functionalities of system 20 (such as registration of 2D images to 3D image data and other functionalities that are derived therefrom) typically do not necessarily require the use of sets 50 of markers 52, and/or a rigid jig. The above-described functionalities may be performed automatically by computer processor 22, and/or manually.
Applications of the present invention are typically applied, in non-CAS (the term “non-CAS” also refers to not in the current form of CAS at the time of the present invention) spinal surgery, to one or more procedural tasks including, without limitation:
Reference is now made to
For some applications, in step 78 a tool (which in more-invasive surgery is often a pedicle finder) is not yet inserted but rather is positioned relative to a vertebra, wherein such vertebra is often partially exposed at such phase, either manually or using a holder device that is typically fixed to the surgical table. Such holder device typically ensures that the subsequent acquisition in step 80 of two or more 2D radiographic images prior to actual tool insertion are with the tool at a same position relative to the vertebra. For some applications, motion of the applicable portion of the subject in between the acquisition of the two or more images is detected by means of a motion detection sensor as described later in this document. For some applications, if motion is detected then the acquisition of pre-motion images may be repeated.
In a sixth step 80, two or more 2D radiographic images are acquired from respective views that typically differ by at least 10 degrees, e.g., at least 20 degrees (and further typically by 30 degrees or more), and one of which is typically from the direction of insertion of the tool. For some applications, generally-AP and generally-lateral images are acquired. For some applications, two (or more) different generally-AP views, with the second generally-AP view tilted cranially or caudally relative to the first generally-AP view, are acquired. Alternatively or additionally, images from different views are acquired.
As noted hereinabove, for some applications, the 3D image data was generated previously from multiple 2D radiographic images of the subject. For some applications, the generation is by means of image processing and image reconstruction. For some applications, the generation is by means of machine learning. Typically, for enhancing the accuracy, or the speed, or both, of the registration performed in step 82 hereinbelow, it may be advantageous to acquire any of the 2D radiographic images in step 80 from a view that is as similar as possible to the view of a 2D radiographic image used previously for generating the 3D image data:
In a seventh step 82, computer processor 22 of system 20 typically registers the 3D image data to the 2D images, as further described hereinbelow.
As used in the present application, including in the claims, a generally-AP view and a generally-lateral view are views that a person of ordinary skill in the art, e.g., a surgeon, would consider as being, respectively, AP and lateral views even though they deviate from what a surgeon would consider to be, respectively, a true AP view and a true lateral view. Typically, a view of vertebrae is considered to be a true AP view when the targeted vertebra is viewed exactly from an anterior position or a posterior position, so that both of its end plates appear in the image as close as anatomically possible to single lines. Typically, a view of vertebrae is considered to be a true lateral view when the targeted vertebra is viewed exactly laterally, so that both of its end plates appear in the image as close as anatomically possible to single lines. Typically, a view of skeletal anatomy in general is considered to be a true AP view when a skeletal portion is viewed exactly from an anterior position or a posterior position, so that that an imaginary axis between the x-ray source and x-ray detector runs perpendicular, in a vertical direction, relative to the applicable anatomy. Typically, an image of skeletal anatomy in general is considered to be a true lateral view when a skeletal portion is viewed exactly laterally, so that an imaginary axis between the x-ray source and x-ray detector runs perpendicular, in a horizontal direction, relative to the applicable anatomy.
Subsequent to the registration of the 3D image data to the 2D images additional features of system 20 as described in detail hereinbelow may be applied by computer processor 22. For example, in step 84, the computer processor drives display 30 to display a cross-section derived from the 3D image data at a current location of the tip of a tool as identified from a 2D image, and, optionally, to show a vertical line on the cross-sectional image indicating a line within the cross-sectional image somewhere along which the tip of the tool is currently disposed.
It is noted, that, as described in further detail hereinbelow, for some applications, in order to perform step 84, the acquisition of one or more 2D x-ray images of a tool at a first location inside the vertebra is from only a single x-ray image view, and the one or more 2D x-ray images are registered to the 3D image data by generating a plurality of 2D projections from the 3D image data, and identifying a 2D projection that matches the 2D x-ray images of the vertebra. In response to registering the one or more 2D x-ray images acquired from the single x-ray image view to the 3D image data, the computer processor drives a display to display a cross-section derived from the 3D image data at a the first location of a tip of the tool, as identified from the one or more 2D x-ray images, and optionally to show a vertical line on the cross-sectional image indicating a line within the cross-sectional image somewhere along which the first location of the tip of the tool is disposed. Typically, when the tip of the tool is disposed at an additional location with respect to the vertebra, further 2D x-ray images of the tool at the additional location are acquired from the same single x-ray image view, or a different single x-ray image view, and the above-described steps are repeated. Typically, for each location of the tip of the tool to which the above-described technique is applied, 2D x-ray images need only be acquired from a single x-ray image view, which may stay the same for the respective locations of the tip of the tool, or may differ for respective locations of the tip of the tool. Typically, two or more 2D x-rays are acquired from respective views, and the 3D image data and 2D x-ray images are typically registered to the 3D image data (and to each other) by identifying a corresponding number of 2D projections of the 3D image data that match respective 2D x-ray images. In step 86, the computer processor drives display 30 to display the anticipated (i.e., extrapolated) path of the tool with reference to a target location and/or with reference to a desired insertion vector. In step 88, the computer processor simulates tool progress within a secondary 2D imaging view, based upon observed progress of the tool in a primary 2D imaging view. In step 90, the computer processor overlays an image of the tool, a representation thereof, and/or a representation of the tool path upon the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data), the location of the tool or tool path having been derived from current 2D images.
Reference is now made to
For some applications, the computer processor automatically counts the number of vertebrae on the image from an identifiable anatomical reference (e.g., the sacrum) to the marked target vertebra(e). It is then known that the targeted vertebra(e) is vertebra N from the identifiable anatomical reference (even if the anatomical labels of the vertebra(c) are not known). For some applications, the vertebra(e) are counted automatically using image-processing techniques. For example, the image-processing techniques may include shape recognition of anatomical features (of vertebrae as a whole, of traverse processes, and/or of spinous processes, etc.). Or, the image-processing techniques may include outer edge line detection of spine (in a 2D image of the spine) and then counting the number of bulges along the spine (each bulge corresponding to a vertebra). For some applications, the image-processing techniques include techniques described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference. For some applications, the vertebra(c) are counted manually by the operator, starting with the vertebra nearest the anatomical reference and till the targeted vertebra(c).
Referring to step 72 of
For some applications, based upon the combined radiographic images, the computer processor automatically determines a location of the given vertebra (e.g., the previously-marked targeted vertebra) within the combined radiographic images. For some applications, the computer processor automatically determines location of the given vertebra within the combined radiographic images by counting the number of vertebrae on said image from an identifiable anatomical reference (e.g., the sacrum). For some applications, the counting is performed until the aforementioned N. For some applications, the counting is performed until a value that is defined relative to the aforementioned N. For some applications, the vertebra(e) are counted automatically using image-processing techniques. For example, the image-processing techniques may include shape recognition of anatomical features (of vertebrae as a whole, of traverse processes, and/or of spinous processes, etc.). Or, the image-processing techniques may include outer edge line detection of spine (in a 2D image of the spine) and then counting the number of bulges along the spine (each bulge corresponding to a vertebra). For some applications, the image-processing techniques include techniques described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference. For some applications, the computer processor facilitates manual determination of the location of the given vertebra within the combined radiographic images by displaying the combined radiographic images.
For some applications, based upon the combined radiographic images, the operator manually determines, typically by way of counting vertebrae upon the combined images starting at the anatomical reference, a location of the given vertebra (e.g., the previously-marked targeted vertebra) within the combined radiographic images.
For some applications, the marker sets as observed in the stitched x-ray images are overlaid, typically automatically and by means of image processing, upon the corresponding CT images of the spine or of the applicable spinal portions. For some applications, that facilitates subsequent matching by the user between corresponding skeletal elements in the stitched x-ray and in the CT images.
For some applications, and wherein the 3D data comprises two or more 3D data files, each such file relating to a spinal portion, stitching is of two or more 3D data files onto a single 3D data volume.
Reference is now made to
It is noted that in the absence of sets 50 of markers 52, the typical methodology for determining the location of a given vertebra includes acquiring a series of x-rays along the patient's spine from the sacrum, and sticking radiopaque needles into the subject in order to match the x-rays to one another. Typically, in each x-ray spinal image only 3-4 vertebrae are within the field of view, and multiple, overlapping images must be acquired, such as to enable human counting of vertebra using the overlapping images. This technique may also involve switching back and forth between AP and lateral x-ray images. This method is often time-consuming and radiation-intensive.
A known clinical error is wrong-level surgery, as described, for example, in “Wrong-Site Spine Surgery: An Underreported Problem? AAOS Now,” American Association of Orthopedic Surgeons, March 2010. That further increases the desire for facilitating level verification by applications of the present invention, as described herein.
Reference is now made to
(It is noted that in
Typically, the combination of images is similar to stitching of images. However, the images are often not precisely stitched such as to stitch portions of the subject's anatomy in adjacent images to one another. Rather, the images are combined with sufficient accuracy to facilitate counting vertebrae along the spine within the combined image. The physical location of a given vertebra is then known by virtue of it being adjacent to, or in the vicinity of, or observable in the x-ray images relative to, a given one of the identifiable markers. It is noted that in order to combine the radiographic images to one another, there is typically no need to acquire each of the images from an exact view (e.g., an exact AP or an exact lateral view), or for there to be exact replication of a given reference point among consecutive images. Rather, generally maintaining a given imaging direction, and having at least some of the markers generally visible in the images is typically sufficient.
As described hereinabove, for some applications, the computer processor automatically counts (and, for some applications, labels, e.g., anatomically labels, and/or numerically labels) vertebrae within the combined radiographic images in order to determine the location of the previously-marked target vertebra(e), or other vertebra(e) relative to the previously marked vertebra. Alternatively, the computer processor drives the display to display the combined radiographic images such as to facilitate determination of the location of the previously-marked target vertebra(e) by an operator. The operator is able to count to the vertebra within the combined radiographic images, to determine, within the combined images, which of the radiopaque markers are adjacent to or in the vicinity of the vertebra, and to then physically locate the vertebra within the subject by locating the corresponding physical markers.
Reference is now made to
For some applications, a spinal CT image data (in 3D or a 2D slice) matching the viewing direction from which the x-ray images were acquired is displayed concurrently with the stitched x-ray images. For example, in the case of x-ray images acquired from a generally-AP direction, a coronal CT view is displayed. For some applications, the x-ray images, or the stitched x-ray image, are interconnected with the CT image such that when the user (or the system) selects a vertebra on the x-ray, the same vertebra is indicated/highlighted on the CT image, or vice versa. For some applications, such connection is generated by registering one or more Digitally Reconstructed Radiographs (DRRs) of the spine as a whole, or of the corresponding spinal section, or of one or more individual vertebrae, with the x-ray images or stitched image. For some applications, such connection is generated by other means of image processing, including in accordance with techniques described hereinabove in the context of counting vertebrae.
For some applications, generation of the combined image includes blending the edges of individual x-ray images from which the combined image is generated, typically resulting in a more continuous-looking combined image.
Reference is now made to
For some applications, 2D x-ray images of the subject's spine, or of a portion thereof, are stitched into a combined image, or are related spatially to one another without actually stitching them, by using 3D image data of the subject's spine (or of a portion thereof) as a “bridge,” and as described hereinbelow.
For some applications, the 3D image data comprises all of the spinal portions visible in the x-ray images. For some applications, the 3D image data comprises only some of the spinal portions visible in the x-ray images.
For some applications, a plurality of 2D x-ray images are acquired, respective images being of respective locations along at least a portion of the subject's spine. For some applications, all images are acquired from a similar viewing angle, for example an angle that is approximately AP. For some applications, images are acquired from different viewing angles.
For some applications, some or all of the images are acquired with some overlap between consecutive two images with respect to the skeletal portion visible in each of them. For some applications, some or all of the images are acquired with small gaps (typically a portion of a vertebra) between consecutive two images with respect to the skeletal portion visible in each of them.
For some applications, the images are stitched to one another, typically without using radiopaque markers, and while using the subject's 3D image data, to provide a combined image of the spine or of a portion thereof, by a computer processor that performs the following:
Alternatively of additionally, the vertebrae visible in each of the x-ray images are marked as such upon the 3D image data. For some applications, the vertebrae visible in each x-ray image may be related to, or marked on, a sagittal view, or a sagittal cross-section, of the 3D image data. For some applications, the vertebrae visible in each x-ray image may be marked on a coronal view, or a coronal cross-section, of the 3D image data.
For example, if vertebrae L5, LA, L3 and L2 are visible in a first x-ray image, and vertebra L2, L1, T12 and T11 are visible in a second x-ray image:
Alternatively, for example, if vertebrae L5, LA, L3 and L2 are visible in a first x-ray image, and vertebra L1, T12, T11 and T10 are visible in a second x-ray image:
For some applications, the techniques described hereinabove are further applied for level verification, optionally in combination with other techniques described herein.
Thus, reference is now made to
Reference is now made to
It should also be noted that level verification using embodiments of the present invention is also useful for correctly positioning a 3D imaging device (such as an O-arm or a 3D x-ray device), situated within the operating room, relative to the subject's body and prior to an actual 3D scan. A common pre-operative CT or MRI device is, according to the specific scan protocol being used, typically configured to scan along an entire body portion such as a torso. For example, such scan may include the entire lumbar spine, or the entire thoracic spine, or both. In contrast, the aforementioned 3D imaging devices available inside some operating rooms, at the time of the present invention, have a very limited scan area, typically a cubical volume whose edges are each 15-20 cm long. Thus, correct positioning of such 3D imaging device prior to the scan relative to the subject's spine, and in particular relative to the targeted spinal elements, is critical for ensuring that the targeted vertebra(e) are indeed scanned. For some applications, level verification using aforementioned embodiments of the present invention yields an indication to the operator of those visible elements of the marker set, next to which the 3D imaging device should be positioned for scanning the spinal segment desired to be subsequently operated upon, such that an imaging volume of the 3D imaging device at least partially overlaps the targeted vertebra. For some applications, in the operating room, the targeted vertebra(e) are level-verified using embodiments of the present invention and then the 3D imaging device is positioned such that its imaging volume (whose center is often indicated by a red light projected upon the subject's body, or some similar indication) coincides with the targeted vertebra(e). For example, if the marker set is a notched ruler placed on the subject's body along the spine, then using embodiments of the present invention the operator may realize that the 3D imaging device should be positioned such that its red light is projected on the subject's body at a level that is in between notches #7 and #8 of the ruler.
For some applications, when a vertebra is selected in an x-ray image (acquired at any phase of the medical procedure) or a combined x-ray image, a 3D image of the same vertebra is displayed automatically. For some applications, the 3D vertebral image auto-rotates on the display. For some applications, the 3D vertebral image is displayed with some level of transparency, allowing the user to observe tools inserted in the vertebra, prior planning drawn on the vertebra, etc. the selection of the vertebra may be by the user or by the system. The autorotation path (i.e., the path along which the vertebra rotates) may be 2D or 3D, and may be system-defined or user-defined. The level of transparency may be system-defined or user-defined. The same applies not only to vertebrae, but also to other spinal or skeletal elements.
For some applications, based upon counting and/or labeling of the vertebrae in the combined radiographic image, computer processor 22 of system 20 counts and/or labels vertebrae within the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data). For some applications, the computer processor drives the display to display the labeled vertebrae while respective corresponding 2D images are being acquired and displayed. Alternatively or additionally, the computer processor drives the display to display the labeled vertebrae when the combined radiographic image has finished being generated and/or displayed. It is noted that, typically, the computer processor counts, labels, and/or identifies vertebrae on the 3D image data and on the 2D radiographic images without needing to determine relative scales of the 3D image data and 2D images. Rather, it is sufficient for the computer processor to be able to identify individual vertebrae at a level that is sufficient to perform the counting, labeling, and/or identification of vertebrae.
It is noted that the above-described identification of vertebrae that is facilitated by markers 52 is not limited to being performed by the computer processor at the start of an intervention. Rather, the computer processor may perform similar steps at subsequent stages of the procedure. Typically, it is not necessary for the computer processor to repeat the whole series of steps at the subsequent stages, since the computer processor utilizes knowledge of an already-identified vertebra, in order to identify additional vertebrae. For example, after identifying and then performing a procedure with respect to a first vertebra, the computer processor may utilize the combined radiographic image to derive a location of a further target vertebra (which may be separated from the first vertebra by a gap), based upon the already-identified first vertebra. For some applications, in order to derive the location of a further target vertebra, the computer processor first extends the combined radiographic image (typically, using the markers in order to do so, in accordance with the techniques described hereinabove).
Reference is now made to
For some applications, a 2D radiographic image 112 of a portion of the subject's body is acquired in a radiographic imaging modality, using the 2D radiographic imaging device (e.g., C-arm 34), and an optical image 110 of the subject's body is acquired in optical imaging modality, using optical camera 114 (shown in
For some applications, the radiographic image and the optical image are fused with one another and displayed as a joint image. For some applications, any of the images is adjusted (e.g., scaled, distorted, etc.), typically according to elements of the marker set observed in both images, prior to such fusion. For some applications, only the x-ray image is displayed to the operator, with the location of the tool (e.g., knife) positioned upon the subject identified from the optical image and marked upon the x-ray image.
As shown in
Traditionally, in order to determine the location of an incision site, a rigid radiopaque wire (such as a K-wire) is placed on the subject's back at a series of locations, and the x-rays are taken of the wire at the locations, until the incision site is determined. Subsequently, a knife is placed at the determined incision site, and a final x-ray image is acquired for verification. By contrast, in accordance with the technique described herein, initially a single x-ray image may be acquired and bidirectionally mapped to the optical image. Subsequently the wire is placed at a location, and the corresponding location of the wire with respect to the x-ray image can be observed (using the bidirectional mapping) without requiring the acquisition of a new x-ray image. Similarly, when an incision knife is placed at a location, the corresponding location of an applicable portion of the knife (typically, its distal tip) with respect to the x-ray image can be observed (using the bidirectional mapping) without requiring the acquisition of a new x-ray image. Alternatively or additionally, a line can be drawn on the x-ray image (e.g., a vertical line that passes along the vertebral centers, anatomically along the spinous processes of the vertebrae) and the corresponding line can be observed in the optical image overlaid on the patient's back.
It should be noted however that for some applications, and in the absence of an optical camera image of the subject, the marker set that is visible both in the x-ray images and upon the subject's body serves as a joint reference for when identifying insertion points or incision sites by the surgeon. Typically, such identification is superior with respect to time, radiation, iterations, errors, etc., compared with current practices (such as in common non-CAS surgical settings) prior to the present invention.
For some applications, a surgeon places a radiopaque knife 116 (or another radiopaque tool or object) at a prospective incision site (and/or places a tool at a prospective tool insertion location) and verifies the location of the incision site (and/or tool insertion location) by observing the location of the tip of the knife (or portion of another tool) with respect to the x-ray (e.g., via cursor 117), by means of the bi-directional mapping between the optical image and the x-ray image. For some applications, the functionalities described hereinabove with reference to
Reference is now made to
Referring again to step 78 of
Reference is now made to
For some applications, the determination of intended incision/entry site, i.e., designated point 235, includes the following steps for each targeted vertebra, with each step either performed manually by the operator or automatically. (It is noted that some of the steps are optional, and that some of the steps may be performed in a different order to that listed below.)
For some applications and pursuant to the above, in step 76 of
For some applications, such as is shown in
For some applications, a camera image is not available, and the operator estimates, or measures physically, the locations of points 235′ on the subject's back relative to the marker set that is (a) placed on the subject's back and (b) also visible in the x-ray image. For some applications, based on the location of the designated point with respect to the radiopaque element on the 2D radiographic image, the operator labels a location of the designated point on the subject's body.
For some applications, such as is shown in
For some applications, such as is shown in
Reference is now made to
For some applications, and wherein the subject's 3D image data was generated from multiple 2D radiographic images of the subject by means of image reconstruction and/or machine learning as described hereinabove, some or all of the 2D radiographic images referred to in the descriptions of
Reference is now made to
It should be noted that embodiments described hereinbelow are also useful for identifying the insertion point into a vertebra in the case of more-invasive or open surgery, wherein the applicable portion of a vertebra is visible via an incision, or exposed. For some applications, such determination of insertion points is performed according to the following steps for each targeted vertebra, with each step performed manually by the operator or automatically. (It is noted that some of the steps are optional, and that some of the steps may be performed in a different order to that listed below.)
Such steps may be followed by any of the embodiments previously described for skin-level insertion, by which the entry points from the 3D data set are registered to the applicable x-ray image, displayed upon that x-ray image, and used for determining point(s) of entry into the vertebra during surgery.
For some applications, both the incision sites at the skin level, and the entry points into the vertebra at the vertebra's applicable edge, are calculated in the 3D data, then registered to, and displayed upon, the 2D x-ray image, and then used for determining the skin-level incision site and the direction of tool entry through that site, typically in accordance with techniques described hereinabove. For some applications, the distance of the incision site from one or more (typically-nearest) elements of the marker set is measured manually or automatically and displayed to facilitate physical determination of the incision site and/or entry point.
For some applications, planning in its various forms as described hereinabove also comprises marking an out-of-pedicle point along the planned insertion path. An out-of-pedicle point is at or near a location along the planned path where the object being inserted along the path exits the pedicle and enters the vertebral body.
For some applications, one or more of the following points are marked along the planned insertion path: incision at skin level, entry into the vertebra, out-of-pedicle, target, or any other point.
Reference is now made to
Reference is now made to
Reference is made to
For some applications, holder 286 to which the tool is attached also comprises one or more angle gauges, typically digital. In such cases, the aforementioned insertion angles previously measured in the planning phase may be applied when aiming the tool at the vertebra. For some applications, application of the angles is manual by the operator of the holder. For some applications, and when holder 286 is robotic, application of the angles is automated and mechanized. For some applications, it is assumed that the applicable portion of the subject is positioned completely horizontally.
However, it is noted that the registration of the 3D image data and the 2D images to each other may be performed even in the absence of a tool within the images, in accordance with the techniques described hereinbelow.
For some applications and when a tool is present in the 2D images but not present in the 3D images, the visibility of a tool or a portion thereof is reduced (or eliminated altogether) by means of image processing from the 2D images prior to their registration with the 3D image data. After registration is completed, 2D images with the tool present, i.e., as prior to the aforementioned reduction or elimination, are added to (utilizing the then-known registration parameters), or replace, the post-reduction or elimination 2D images, within the registered 2D-3D data, according to the registration already achieved with the post-reduction or elimination 2D images. For some applications, regions in the 2D image comprising a tool or a marker set are excluded when registering the 2D images with the 3D data. For some applications, the aforementioned techniques facilitate registration of the 2D images with the 3D data set because all include at the time of their registration to one another only (or mostly) the subject's anatomy, which is typically the same, and thus their matches to one another need not (or to a lesser extent) account for elements that are included in the 2D images but are absent from the 3D data set. For some applications, the reduction or elimination of the visibility of the tool or a portion thereof is performed using techniques and algorithmic steps as described in US Patent Application 2015-0282889 to Cohen (and Tolkowsky), which is incorporated herein by reference. The same applies to a reduction of elimination of the visibility of previously-placed tools, such as implants (e.g., pedicle screws, rods, cages, etc.), in any of the images, such as prior to image registration.
Typically, the 3D image data and 2D images are registered to each other by generating a plurality of 2D projections from the 3D image data and identifying respective first and second 2D projections that match the first and second 2D x-ray images of the vertebra, as described in further detail hereinbelow. (For some applications, 2D x-ray images from more than two 2D x-ray image views are acquired and the 3D image data and 2D x-ray images are registered to each other by identifying a corresponding number of 2D projections of the 3D image data that match respective 2D x-ray images.) Typically, the first and second 2D x-ray images of the vertebra are acquired using an x-ray imaging device that is unregistered with respect to the subject's body, by (a) acquiring a first 2D x-ray image of the vertebra (and at least a portion of the tool) from a first view, while the x-ray imaging device is disposed at a first pose with respect to the subject's body, (b) moving the x-ray imaging device to a second pose with respect to the subject's body, and (c) while the x-ray imaging device is at the second pose, acquiring a second 2D x-ray image of at least the portion of the tool and the skeletal portion from a second view.
For some applications, the 3D imaging that is used is CT imaging, and the following explanation of the registration of the 3D image data to the 2D images will focus on CT images. However, the scope of the present invention includes applying the techniques describe herein to other 3D imaging modalities, such as MRI and 3D x-ray, mutatis mutandis.
X-ray imaging and CT imaging both apply ionizing radiation to image an object such as a body portion or organ. 2D x-ray imaging generates a projection image of the imaged object, while a CT scan makes use of computer-processed combinations of many x-ray images taken from different angles to produce cross-sectional images (virtual “slices”) of the scanned object, allowing the user to see inside the object without cutting. Digital geometry is used to generate a 3D image of the inside of the object from a large series of 2D images.
Reference is now made to
In the case of 3D CT images, the derived 2D projections are known as Digitally Reconstructed Radiographs (DRRs). If one considers 3D CT data and a 2D x-ray image of the same vertebra, then a simulated x-ray camera position (i.e., viewing angle and viewing distance) can be virtually positioned anywhere in space relative to a 3D image of the vertebra, and the corresponding DRR that this simulated camera view would generate can be determined. At a given simulated x-ray camera position relative to the 3D image of the vertebra, the corresponding DRR that this simulated camera view would generate is the same as the 2D x-ray image. For the purposes of the present application, such a DRR is said to match an x-ray image of the vertebra. Typically, 2D x-ray images of a vertebra from respective views are registered to one another and to 3D image data of the vertebra by generating a plurality of DRRs from 3D CT image data, and identifying respective first and second DRRs (i.e., 2D projections) that match the 2D x-ray images of the vertebra. By identifying respective DRRs that match two or more x-ray images acquired from respective views, the x-ray images are registered to the 3D image data, and, in turn, the x-ray images are registered to one another via their registration to the 3D image data.
For some applications, and due to the summative nature of x-ray imaging, an x-ray image of a given vertebra may also, depending on the x-ray view, comprise elements from a neighboring vertebra. In such case, those elements may be accounted for (by way of elimination or inclusion) during the act of 2D-3D registration, and in accordance with embodiments of the present invention. For some applications, such accounting for is facilitated by 3D segmentation and reconstruction of the given (targeted) vertebra that is the focus of the then-current registration process.
For some applications, 2D x-ray images are enhanced using the corresponding DRRs from the 3D data set. For some applications, one or more of the enhanced images includes only image elements that were already present in the x-ray image, in the corresponding DRR, or in both the x-ray image and the corresponding DRR. For some applications, one or more of the enhanced images additionally includes image elements that were not present in the x-ray image or in the corresponding DRR and were added to the enhanced image. For some applications, the added image elements are generated in an automated manner. For some applications, enhancement is performed online. For some applications, such enhancement results in providing the user, during surgery, with images that are more informative than the original x-ray images with respect to the positions of the surgical tools relative to the patient's anatomy. For some applications, such enhancement results in providing the user, during surgery, with images that are more informative than the original x-ray images with respect to the patient's anatomy. For some applications, the x-ray image, or the corresponding DRR, or both, are inverted (i.e., the colors in the images, which are typically on a grayscale, are inverted) prior to, or in the process of, enhancement.
For some applications, an x-ray image is enhanced by performing an addition of the x-ray image and a corresponding DRR. For some applications, an x-ray image is enhanced by blending it, or combining it by other means of image processing, with a corresponding DRR. For some applications, blending is linear. For some applications, blending is non-linear. For some applications, blending is performed while assigning equal weights to the x-ray image (and/or the pixels thereof), and the corresponding DRR (and/or the pixels thereof). For some applications, blending is performed with different weights assigned to the x-ray image, or to regions thereof, and the corresponding DRR, or to regions thereof. For some applications, the weights are determined by the user. For some applications, the weights are determined by the system. For some applications, the weights are determined pursuant to analysis of the x-ray image, or of the corresponding DRR, or of both. For some applications, analysis is automated in whole or in part.
For some applications, planning data previously generated with respect to the 3D image data (e.g., incision sites, insertion lines, vertebral entry points, or any combination thereof) is projected upon an x-ray image that was enhanced by using the corresponding DRR, wherein such projection is typically performed using techniques described herein.
Reference is now made to
In
For some applications, newly-acquired x-ray images are enhanced by corresponding DRRs that were generated prior to that in the act of registering previously-acquired x-ray images to the same 3D data set. For some applications, the newly-acquired and the previously-acquired x-ray images are acquired from the same poses of the x-ray c-arm relative to the subject. For some applications, the newly-acquired and the previously-acquired x-ray images are combined with one another for the purpose of image enhancement.
For some applications, in order to register the 2D images to the 3D image data, additional registration techniques are used in combination with the techniques described herein. For example, intensity-based methods, feature based methods, similarity measures, transformations, spatial domains, frequency domains, etc., may be used to perform the registration.
For some applications, and wherein the 3D image set was acquired in the operating room, the 3D image set also comprises applicable portions of marker set(s) 50, such that the marker set serves as an additional one-or-more registration fiducial in between the 2D images and the 3D data set.
Typically, by registering the x-ray images to the 3D image data using the above-described technique, the 3D image data and 2D x-ray images are brought into a common reference frame to which they are all aligned and scaled. It is noted that the registration does not require tracking the subject's body or a portion thereof (e.g., by fixing one or more location sensors, such as an IR light, an IR reflector, an optical sensor, or a magnetic or electromagnetic sensor, to the body or body portion, and tracking the location sensors).
Typically, between preprocedural 3D imaging (e.g., 3D imaging performed prior to entering the operating room, or prior to performing a given intervention) and intraprocedural 2D imaging, the position and/or orientation of a vertebra relative to the subject's body and to neighboring vertebrae is likely to change. For example, this may be due to the patient lying on his/her back in preprocedural imaging but on the stomach or on the side for intraprocedural imaging, or the patient's back being straight in preprocedural imaging, but being folded (e.g., on a Wilson frame) in intraprocedural imaging. In addition, in some cases, due to anesthesia the position of the spine changes (e.g., sinks), and once tools are inserted into a vertebra, that may also change its positioning relative to neighboring vertebrae. However, since a vertebra is a piece of bone, its shape typically does not change between the preprocedural 3D imaging and the intraprocedural 2D imaging. Therefore, registration of the 3D image data to the 2D images is typically performed with respect to individual vertebrae. For some applications, registration of the 3D image data to the 2D images is performed on a per-vertebra basis even in cases in which segmentation of a vertebra in the 3D image data leaves some elements, such as portions of the spinous processes of neighboring vertebrae, within the segmented image of the vertebra. In addition, for some applications, registration of the 3D image data to the 2D images is performed with respect to a spinal segment comprising several vertebrae. For example, registration of 3D image data to the 2D images may be performed with respect to a spinal segment in cases in which the 3D image data were acquired when the subject was already in the operating room and positioned upon the surgical table for the intervention.
As described hereinabove, typically, during a planning stage, an operator indicates a target vertebra within the 3D image data of the spine or a portion thereof (e.g., as described hereinabove with reference to
Typically, and since the registration is performed with respect to an individual vertebra, the registration is not affected by motion of the vertebra that occurs between the acquisition of the two x-ray images (e.g., due to movement of the subject upon the surgical table, motion due to respiration, etc.), since both motion of the C-arm and of the vertebra may be assumed to be rigid transformations (and thus, if both motions occur in between the acquisition of the two x-ray images, a chaining of two rigid transformations may be assumed).
As described hereinabove, typically, 2D x-ray images of a vertebra from respective views are registered to one another and to a 3D image data of the vertebra by generating a plurality of DRRs from a 3D CT image, and identifying respective first and second DRRs that match the 2D x-ray images of the vertebra. By identifying respective DRRs that match two or more x-ray images acquired from respective views, the x-ray images are registered to the 3D image data, and, in turn, the x-ray images are registered to one another via their registration to the 3D image data.
For some applications, in order to avoid double solutions when searching for a DRR that matches a given x-ray image, computer processor 22 first determines whether the x-ray image is, for example, AP, PA, left lateral, right lateral, left oblique, or right oblique, and/or from which quadrant a tool is being inserted. The computer processor may determine this automatically, e.g., by means of sets 50 of markers 52, using techniques described herein. Alternatively, such information may be manually inputted into the computer processor.
For some applications, in order to identify a DRR that matches a given x-ray image, computer processor 22 first limits the search space within which it is to search for a matching DRR, by applying the following steps. (It is noted that some of the steps are optional, and that some of the steps may be performed in a different order to that listed below.)
For some applications, in order to identify a DRR that matches a given x-ray image, computer processor 22 first limits the search space within which it is to search for a matching DRR, by identifying the marker set or elements thereof in the x-ray image and applying prior knowledge with which it was provided of what the marker set or its elements look like from different viewing directions, or at different zoom levels, or at different camera openings, or any combination thereof. Typically, the search space is narrowed down to at or near simulated camera positions/values from which the marker set or elements thereof are known to appear in a similar manner to how they appear in the x-ray image.
For some applications, in order to identify a DRR that matches a given x-ray image, some combination of techniques described in the present application is applied.
For some applications, the registration of the 2D (e.g., x-ray) images with the 3D (e.g., CT) data is divided into a pre-processing phase and an online phase, i.e., during a medical procedure. Each of the two phases may be performed locally on a computer, or on a networked computer, or via cloud computing, or by applying any combination thereof.
Reference is now made to
During a medical procedure, i.e., in the online phase, only those characteristics then need to be matched with an x-ray image in the online phase, as follows: (i) a 2D radiographic image is acquired of the skeletal portion (step 298), (ii) computer processor 22 (a) determines at least one specific set of values for the attributes that describe at least a portion of the 2D radiographic image (step 300), (b) searches among the stored N respective sets of attributes for a set that best matches any of the at least one specific set of values (step 302), and (c) uses the set that best matches, to generate an additional 2D projection image from the 3D image data, the additional 2D projection image matching at least the portion of the 2D radiographic image (step 304).
Reference is now made to
For some applications, the pre-processing phase comprises the following steps (some of which are optional and the order of which may vary):
Typically, in the new D-dimensional sub-space, there are M vectors, each corresponding to one of the M DRRs. Each of the M vectors is now reduced to a point with D coordinates in the D-dimensional subspace.
Typically, from M N-dimensional vectors representing DRRs, there has been a reduction to M points in a D-dimensional space. Therefore, the outcome is a great reduction, by several orders of magnitude, the amount of data that we shall need to search in the next phase which is the online phase.
For some applications, the online phase comprises the following steps (some of which being optional and the order of which may vary):
For some applications, the steps of generating a plurality of DRRs from a 3D CT image, and identifying respective first and second DRRs that match the 2D x-ray images of the vertebra are aided by deep-learning algorithms.
For some applications, deep-learning techniques are performed as part of the processing of images of a subject's vertebra, as described in the following paragraphs. By performing the deep-learning techniques, the search space for DRRs of the subject's vertebra that match the x-ray images is limited, which reduces the intraprocedural processing requirement, reduces the time taken to performing the matching, and/or reduces cases of dual solutions to the matching.
For some applications, deep learning may be performed using 3D scan data only of the targeted vertebra, which typically greatly facilitates the task of building the deep-learning dataset. For some applications, during the deep-learning training phase, a large database of DRRs generated from the 3D data of the targeted vertebra, and (at least some of) their known parameters relative to vertebra, are inputted to a deep-learning engine. Such parameters typically include viewing angle, viewing distance, and optionally additional x-ray system and camera parameters. For some applications, the aforementioned parameters are exact. Alternatively, the parameters are approximate parameters. The parameters may be recorded originally when generating the DRRs, or annotated by a radiologist. Thus, the engine learns, given a certain 2D projection image, to suggest simulated camera and x-ray system viewing distances and angles that correspond to that projection image. Subsequently, the deep-learning data is fed as an input to computer processor 22 of system 20. During surgery, in order to register any of the 2D x-ray images to the 3D image data, computer processor uses the deep-learning data by inference in order to limit the search space in which DRRs of the 3D image data that match the x-ray images should be searched for. Computer processor 22 then searches for matching DRRs only within the search space that was prescribed by the deep-learning inference.
The above-described registration steps are summarized in
In a first step 140, the search space for DRRs that match respective x-ray images is limited, for example, using deep-learning data as described hereinabove. Alternatively or additionally, in order to avoid double solutions when searching for a DRR that matches a given x-ray image, the computer processor determines whether the x-ray images are, for example, AP, PA, left lateral, right lateral, left oblique, or right oblique, and/or from which quadrant a tool is being inserted.
In step 141, a plurality of DRRs are generated within the search space.
In step 142, the plurality of DRRs are compared with the x-ray images from respective views of the vertebra.
In step 143, based upon the comparison, the DRR that best matches each of the x-ray images of the vertebra is selected. Typically, for the simulated camera position that would generate the best-matching DRR, the computer processor determines the viewing angle and viewing distance of the camera from the 3D image of the vertebra.
It is noted that the above steps are performed separately for each of the 2D x-ray images that is used for the registration. For some applications, each time one or more new 2D x-ray images are acquired, the image(s) are automatically registered to the 3D image data using the above-described technique. The 2D to 3D registration is thereby updated based upon the new 2D x-ray acquisition(s).
Reference is now made to
As described hereinabove, for each of the x-ray images (denoted X1 and X2), the computer processor determines a corresponding DRR from a simulated camera view (the simulated cameras being denoted C1 for X1 and C2 for X2).
The 3D scan and two 2D images are now co-registered, and the following 3D-2D bi-directional relationship generally exists:
Therefore, referring to
In general, the computer processor identifies the locations of a tool or a portion thereof in the 2D x-ray images, typically, solely by means of image processing. For example, the computer processor may identify the tool by using a filter that detects pixel darkness (the tool typically being dark), using a filter that detects a given shape (e.g., an elongated shape), and/or by using masks. For some applications, the computer processor compares a given region within the image to the same region within a prior image. In response to detecting a change in some pixels within the region, the computer processor identifies these pixels as corresponding to a portion of the tool. For some applications, the aforementioned comparison is performed with respect to a region of interest in which the tool is likely to be inserted, which may be based upon a known approach direction of the tool. For some applications, the computer processor identifies the portion of the tool in the 2D images, solely by means of image processing, using algorithmic steps as described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference. For some applications, the computer processor identifies the portion of the tool in the 2D images, solely by means of image processing, using algorithmic steps as described in US 2012-0230565 to Steinberg, which is incorporated herein by reference. For some applications, the tool or portion thereof is identified manually, and pointed at on one or more of the images, by the operator.
For some applications, identification of the portion of the tool in the 2D images is facilitated, manually or automatically, by defining a region of interest (ROI) in a 2D image around the planned insertion line of the tool, as such line was determined in the planning phase using techniques described by the present application, and then registered to the 2D image using techniques described by the present application. Next, the portion of the tool is searched within the ROI using techniques described by the present application.
For some applications, identification of the portion of the tool in the 2D images is facilitated using a machine-learning engine. The machine-learning engine is trained using a database of 2D training images, each 2D training image being of a tool inserted into or near a skeletal portion. For some applications, the database of 2D training images includes images of a variety of different types of tools, images captured from a variety of different viewing angles, images acquired with a variety of different radiation doses, and/or images of subjects having a variety of different body mass indices. For each 2D training image, the tool is marked, typically manually, such that the machine-learning algorithm learns to identify it. During the marking of the 2D training images, each tool may also be marked to indicate which type of tool it is, such that the machine-learning engine learns to detect a tool within a 2D image and also to characterize the detected tool. It is noted that throughout the present application, including in the claims, identification or detection of a tool or a portion of a tool in a 2D image by means of image processing can be replaced with identification or detection of a tool or a portion of a tool within a 2D image using a machine-learning engine as described hereinabove.
Reference is made to
It is noted that, since the co-registration of the 3D image data to the 2D images is bidirectional, for some applications, the computer processor identifies features that are identifiable within the 3D image data, and determines the locations of such features with respect to the 2D x-rays, as described in further detail hereinbelow. The locations of each such feature with respect to any of the 2D x-rays are typically determined by (a) generating a typically-straight line from the simulated camera that was used to generate the DRR corresponding to such x-ray image and through the feature within the 3D image data and (b) thereby determining the locations of the feature with respect to the x-ray images themselves. For some applications, the locations of such features with respect to the 2D x-ray images are determined by determining the locations of the features within the DRRs that match the respective x-ray images, and assuming that the features will be at corresponding locations within the matching x-ray images.
For some applications, based upon the registration, 3D image data is overlaid upon a 2D image. However, typically, the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data) are displayed alongside 2D images, as described in further detail hereinbelow.
Reference is now made to
For some applications, upon the cross-section, the computer processor drives the display to show a line 166 (e.g., a vertical line), indicating that the location of the tip of the tool is somewhere along that line. For some applications, the line is drawn vertically upon an axial cross-section of the vertebra, as shown. For some applications, the surgeon is able to determine the likely location of the tool along the line based upon their tactile feel. Alternatively or additionally, based on the 3D image data, the computer processor drives the display to display how deep below the skin the vertebra is disposed, which acts as a further aid to the surgeon in determining the location of the tool along the line.
As noted above, typically it is possible to generate an output as shown in
Reference is now made to
Reference is now made to
For some applications, a location within a vertebra is designated within the 3D image data. For example, an operator may designate a target portion (e.g., a fracture, a tumor, a virtual pedicle screw, etc.), and/or a region which the tool should avoid (such as the spinal cord) upon the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data). Alternatively or additionally, the computer processor may identify such a location automatically, e.g., by identifying the portion via image processing. Based upon the registration of the first and second 2D x-ray images to the 3D image data, the computer processor derives a position of the designated location within at least one of the x-ray images, using the techniques described hereinabove. In addition, the computer processor determines an anticipated path of the tool within the x-ray image. Typically, the computer processor determines the anticipated path by determining a direction of an elongate portion of the tool (and/or a center line of the elongate portion) within the x-ray image. Since the tool is typically advanced along a longitudinal insertion path, the computer processor extrapolates the anticipated path by extrapolating a straight line along the determined direction.
For some applications, the computer processor performs a generally similar process, but with respect to a desired approach vector (e.g., for insertion and implantation of a screw) that, for example, is input into the computer processor manually, and/or is automatically derived by the processor. For example, such an approach vector may have been generated during a planning phase, typically upon the 3D image data, and based upon the insertion of a simulated tool into the vertebra. Typically, such an approach vector is one that reaches a desired target, while avoiding the spinal cord or exiting the vertebra sideways.
For some applications, in response to the above steps, the computer processor generates an output indicating a relationship between the anticipated longitudinal insertion path of the tool and the designated location. For some applications, the computer processor generates an output on the display, e.g., as shown in
Referring again to step 90 of
For some applications, the representation of the actual tool (or of a portion thereof) is displayed relative to the planned path of insertion, in accordance with techniques described by the present application. For some applications, the planned path of insertion is generated by embodiments of the present invention. For some applications, the actual tool vs. the planned path is displayed upon a 2D slice or a 2D projection of the 3D data. For some applications, the actual tool vs. the planned path is displayed upon a 3D model generated from the 3D data, with such model typically having some level of transparency allowing to see the representations within it. For some applications, the 3D model is auto-rotated to facilitate the operator's spatial comprehension of actual tool vs. planned path. For some applications, the actual tool vs. the planned path is displayed upon a 2D x-ray image in which the tool can be observed and with the planned path registered from the 3D data, for example by means of matching a DRR generated from the 3D data and comprising the planned path with the 2D x-ray image. For some applications, the planned path comprises one or more points along the path, such as the incision site at skin level, the entry point into the vertebra, the out-of-pedicle point, and the target point, or any combination thereof.
Reference is made to
For some applications, the computer processor generates an output that is indicative of the distance of the tip of the tool from the spinal cord and/or outer vertebral border, e.g., using numbers or colors displayed with respect to the 3D image data. For some applications, the computer processor outputs instructions (e.g., textual, graphical, or audio instructions) indicating that the tool should be redirected. For some applications, as an input to this process, the computer processor determines or receives a manual input indicative of a direction or orientation from which the tool is inserted (e.g., from top or bottom, or left or right).
Reference is now made to
Reference is now made to
Once a correspondence is made in both the first and second x-rays between a tool element in the x-rays and its corresponding planned longitudinal insertion path, computer processor 22 thus identifies which tool in the first x-ray is the same tool in the second x-ray and can then position respective representations of the respective tool elements within a display of the 3D image data.
For some applications, the computer processor matches automatically between a tool in one x-ray image acquired from a first view, and the same tool in a second x-ray image acquired from a second view, by defining a region of interest (ROI) in each x-ray image around the planned insertion line of the tool, as such line was determined in the planning phase using techniques described by the present application and then registered to the 2D image using techniques described by the present application, and then matching between instances of the tool, or portions thereof, that appear in both ROIs.
For some applications, the planned insertion line of each tool is displayed distinctively, e.g., each in a unique color within the 3D image data. The planned respective longitudinal insertion paths may also be distinctively overlaid on the first and second x-ray images, facilitating identification of each insertion path in the x-ray images on which the planning data has been projected (i.e., overlaid), and thus facilitating manual association of each tool with a nearby planned insertion line, e.g., how close the tool is to the planned insertion line, in each of the x-ray images and for each tool among the x-ray images.
For some applications, the planning data (or portions thereof) is, using techniques described by the present application, projected and displayed upon each x-ray image that is acquired and registered with the 3D data. For some applications, a first tool (e.g., needle, wire) seen in an x-ray image is distinguished, typically automatically and typically be means of image processing, from a second tool (e.g., forceps) used to grab the first tool, by the first tool having a single longitudinal shaft and the second tool having a dual longitudinal shaft.
Referring again to step 90 of
For some applications, the computer processor drives the display to display in a semi-transparent format a 3D image of the vertebra with the tool, a representation thereof, and/or a path thereof disposed inside the 3D image. Alternatively or additionally, the computer processor drives the display to rotate the 3D image of the vertebra automatically (e.g., to rotate the 3D image back-and-forth through approximately 30 degrees). For some applications, the computer processor retrieves an image of a tool of the type that is being inserted from a library and overlays the image upon the derived centerline upon the 3D image data. Typically, the tool is placed along the centerline at an appropriate scale with the dimensions being derived from the 3D image data. For some applications, a cylindrical representation of the tool is overlaid upon the derived centerline upon the 3D image data. For some applications, any one of the above representations is displayed relative to a predesignated tool path, as derived automatically by processor 22, or as input manually by the surgeon during a planning stage.
Referring again to
For some applications, the processor allows a 3D image of the vertebra with the tool, a representation of the tool, and/or a path of the tool indicated within the image to be rotated, or the processor rotates the image automatically, in order for the user to better understand the 3D placement of the tool. It is noted that, since the images of the vertebra and the tool were input from different imaging sources, the segmented data of what is the tool (or its representation) and what is the vertebra is in-built (i.e., it is already known to the computer processor). For some applications, the computer processor utilizes this in-built segmentation to allow the operator to virtually manipulate the tools with respect to the vertebra. For example, the operator may virtually advance the tool further along its insertion path, or retract the tool and observe the motion of the tool with respect to the vertebra. For some applications, the computer processor automatically virtually advances the tool further along its insertion path, or retracts the tool with respect to the vertebra in the 3D image data.
For some applications, accuracy of determining the position of the portion of the tool within the 3D image data is enhanced by registering three 2D x-ray images to the 3D image data, the images being acquired from respective, different views from one another. Typically, for such applications, an oblique x-ray image view is used in addition to AP and lateral views. For some applications, accuracy of determining the position of the portion of the tool within the 3D image data is enhanced by using x-ray images in which multiple portions of the tool, or portions of multiple tools, are visible and discernible from one another in the x-ray images. For some applications, the tools are discerned from one another based on a manual input by the operator, or automatically by the computer processor. For some applications, accuracy of determining the position of the portion of the tool within the 3D image data is enhanced by referencing the known shapes and/or dimensions of radiopaque markers 52 as described hereinabove.
Reference is now made to
For some applications, the imaging functionalities described above with reference to the 3D image data are performed with respect to the 2D x-ray images, based upon the co-registration of the 2D images to the 3D image data. For example, the tool may be color-coded in the x-ray images according to how well the tool is placed. For some applications, if the tool is placed incorrectly, the computer processor drives the display to show how the tool should appear when properly placed, within the 2D x-ray images.
Reference is now made to
For some applications of the present invention, images are initially acquired from two poses, which correspond to respective image views. For example,
For some applications, the repeat acquisitions are performed from a 2D x-ray image view that is the same as one of the original 2D x-ray image views, while for some applications the repeat acquisitions are performed from a 2D x-ray image view that is different from both of the original 2D x-ray image views. For some applications, in the subsequent step, the tool within the vertebra is still imaged periodically from one or more additional 2D x-ray image views, in order to verify the accuracy of the position of the tool within the additional views that was derived by the computer processor, and to correct the positioning of the tool within the additional 2D x-ray image views if necessary. For some applications, the C-arm is maintained at a single pose (e.g., AP) for repeat acquisitions during tool insertion and/or manipulation, and the computer processor automatically derives the location of portion of the tool with respect to the 3D image data of the vertebra, and updates the image of the tool (or a representation thereof) within the 3D image data.
Typically, applications as described with reference to
For some applications, the techniques described with reference to
For some applications, computer processor 22 uses one of the following algorithms to perform techniques described by the present invention.
The following algorithm may be implemented by computer processor 22 in cases in which the 3D image data of the subject is generated from 2D images of the subject, typically by means of machine learning, using techniques described herein. It shares some, but not all, of the steps of Algorithm 1.
For some applications, references made hereinbelow to Algorithm 1 also apply to Algorithm 1A.
The following algorithm may be implemented by computer processor 22 in cases in which the 3D image data of the subject is generated from 2D images of the subject, typically by means of machine learning using techniques described herein. It uses some, but not all, of the steps of Algorithm 2.
For some applications, references made hereinbelow to Algorithm 2 also apply to Algorithm 2A.
The following algorithm may be implemented by computer processor 22 in cases in which the 3D image data of the subject is generated from 2D images of the subject, typically by means of machine learning using techniques described herein. Typically, this algorithm is used for generating the position of a tool within the 3D image data, wherein the tool is not generally-straight and may even have a multi-dimensional complex shape.
For some applications, references made hereinbelow to Algorithm 2 also apply to Algorithm 2B.
For some applications, a sensor is coupled with the tool. For some applications, the sensor is a location sensor, or a motion sensor, or a displacement sensor, or a bio-impedance sensor, or an electrical conductivity sensor, or a tissue characterization sensor, or any combination thereof. For some applications, using information received from the sensor, the position of the tool, or of a portion thereof, or of the tip thereof, is calculated in between Step 3 and Step 4 of Algorithm 1 or Algorithm 2, or between iterations of Steps 4 through 8 of Algorithm 1 or Algorithm 2, or between iterations of Steps 4 through 9 of Algorithm 1 or Algorithm 2, and typically while the tool is being moved further or after the tool has been moved further.
The position is typically calculated relative to the prior, already known outcome of Step 3, or of an iteration of Steps 4 through 8, typically by applying the information incrementally to that known outcome. For example:
Subsequently to the calculation and for some applications, the position of the tool, or of a portion thereof, or of the tip thereof, in between steps 3 and 4 of Algorithm 2 or between iterations of Steps 4 through 8 and while the tool is being moved or after it has been moved, is displayed upon the 3D image data, or upon a portion of the 3D image data, or upon 2D cross-sections generated from the 3D image data, or upon 2D images which were used to generate the 3D image data, or upon previously-acquired 2D x-ray images, or any combination thereof.
For some applications when applying Algorithm 1 or Algorithm 2, recommended values (or ranges of values) for C-Arm angles when acquiring images X2 and/or X3 are generated automatically and displayed to the operator. Typically, such recommendation is aimed at providing the optimal set of images for later determining the position of the tool within the 3D image data in accordance with techniques described herein and avoiding a situation known as singularity (or degeneracy). Such singularity may occur when two x-ray images are acquired such that the planes containing the x-ray source and the tool projection on the x-ray detector coincide for both acquisitions. For some applications, and wherein the C-Arm is motorized, the C-Arm is driven automatically to those recommended angles.
For some applications when applying Algorithm 1 or Algorithm 2, the user is alerted, typically automatically, if the position of the tool relative to the skeletal anatomy to which the tool is applied has changed in between the acquisition of image X1 and the acquisition of image X2.
Typically, detecting such change in the position of the tool is possible in two scenarios, (i) immediately after the acquisition of image X2 and (ii) only after the acquisition of further images (e.g., X3). In the first scenario, the planes containing the x-ray source and the tool's projection on the x-ray detector do not coalesce, indicating a tool position change between the acquisition of image X1 and the acquisition of image X2. In the second scenario, the position of the tool within the 3D image data as calculated using images X1 and X3 (for example) is inconsistent with the position of the tool within the 3D image data as calculated using images X1 and X2.
In cases where the tool position change is detected after the acquisition of image X2, image X1 typically needs to be reacquired before advancing the tool further. In cases where the tool position is detected after the acquisition of image X3 or later, both images X1 and X2 typically need to be reacquired before advancing the tool further.
For some applications, and wherein the subject's 3D image data was generated from multiple 2D radiographic images of the subject by means of image reconstruction and/or machine learning as described hereinabove, some or all of the 2D radiographic images (e.g., X1, X2, X3) acquired in accordance with Algorithm 1 or Algorithm 2 are acquired (using techniques described hereinabove) from views that are identical or similar to the views from which the subject's 2D radiographic images that were used in generating the 3D image data were acquired.
For some applications, Algorithm 1 or Algorithm 2 are further facilitated by adding a radio-opaque feature, for example by means of clipping, typically to the out-of-body portion of the tool. In such cases, a feature, or an identifiable sub-feature thereof, serves as a second feature, in addition to the tool's distal tip, for determining the direction of the tool's shaft. For some applications, the clip, or another radiopaque feature attached to the tool, are as shown in
For some applications, for Algorithm 1 or Algorithm 2, a software algorithm is applied for identifying situations of singularity, with respect to the tool, of X-Ray images X1 and X2. For some applications, such algorithm not only identifies the singularity but also recommends which of X1 and/or X2 should be reacquired from a somewhat different pose. For some applications, such algorithm also guides the user as to what such new pose may be. For some applications, the aforementioned clip, or another radiopaque feature attached to the tool, assists in identifying and/or resolving situations of singularity between x-ray images X1 and X2.
For some applications, the use of Algorithm 1 or Algorithm 2 has an additional benefit of reducing the importance that the X-ray images are acquired in what is known as Ferguson views. In a Ferguson view, the end plates appear as flat and as parallel to one another as possible. It is considered advantageous for proper tool insertion into a vertebra. However, once any acquired 2D x-ray image is co-registered with the 3D CT data, as described by applications of the present invention, and furthermore once a tool seen in the 2D x-ray images is registered with the 3D data, again as described by applications of the present invention, the operator can assess in 3D the correctness of the insertion angle and without needing x-ray images acquired specifically in Ferguson view. Typically, it takes multiple trials-and-errors, when manipulating an x-ray c-arm relative to the subject's body, to achieve Ferguson views. Multiple x-ray images are typically acquired in the process till the desired Ferguson view is achieved, with potential adverse implications on procedure time and the amount of radiation to which the subject and medical staff who are present are exposed.
For some applications, the use of Algorithm 1 or Algorithm 2 has an additional benefit of reducing the importance that the X-ray images are acquired in what is known as “bull's-eye” views. In a “bull's-eye” view, the tool being inserted is viewed from the direction of insertion, ideally with the tool seen only as a cross-section, to further facilitate the surgeon's understanding of where the tool is headed relative to the anatomy. However, once any acquired 2D x-ray image is co-registered with the 3D CT data, as described by applications of the present invention, and furthermore once a tool seen in the 2D x-ray images is registered with the 3D data, again as described by applications of the present invention, the operator can assess in 3D the correctness of the insertion angle and without needing x-ray images acquired specifically in “bull's-eye” view. Typically, it takes multiple trials-and-errors, when manipulating an x-ray c-arm relative to the subject's body, to achieve “bull's-eye” views. Multiple x-ray images are typically acquired in the process till the desired “bull's-eye” view is achieved, with potential adverse implications on procedure time and the amount of radiation to which the subject and medical staff who are present are exposed.
For some applications of the present invention, the operator is assisted in manipulating the c-arm to a Ferguson view prior to activating the c-arm for acquiring images. On the system's display, the vertebra in 3D, with the tool depicted upon it, is rotated to a Ferguson view. Next, the operator manipulates the c-arm such that the tool is positioned relative to the detector at a similar angle to the one depicted on the system's display relative to the operator; only then is the c-arm activated to acquire x-ray images.
Reference is now made to
The following algorithm is typically implemented by computer processor 22 even in cases in which the x-ray images are not registered with 3D image data of the vertebra. Typically, this algorithm is for use with a three-dimensional radiopaque jig, such as jig 194, sufficient portions of which are visible in all applicable x-ray images and can be used to relate them to one another. For some applications, the jig includes a 3D array of radiopaque spheres, as shown in
The following algorithm is typically implemented by computer processor 22 even in cases in which the x-ray images are not registered with 3D image data of the vertebra. Typically, this algorithm is for use with a tool that has two or more identifiable points in each 2D x-ray image. For example, this algorithm may be used with a tool to which a clip, or another radiopaque feature is attached as shown in
With reference to
In accordance with some applications, first and second 2D x-ray images are acquired, from respective x-ray image views, of the skeletal portion and a portion of a tool configured to be advanced into the skeletal portion along a longitudinal insertion path, while the portion of the tool is disposed at a first location with respect to the insertion path. The location of a portion of the tool with respect to the skeletal portion is identified within the first and second 2D x-ray images, by computer processor 22 of system 20, by means of image processing, and the computer processor determines a relationship between the first and second 2D x-ray images, e.g., using any one of algorithms 1-4 described hereinabove. Subsequently, the tool is advanced along the longitudinal insertion path with respect to the skeletal portion, such that the portion of the tool is disposed at a second location along the longitudinal insertion path. Subsequent to moving the portion of the tool to the second location along the insertion path, one or more additional 2D x-ray images of at least the portion of the tool and the skeletal portion are acquired from a single image view. In accordance with respective applications, the single image view is the same as one of the original 2D x-ray image views, or is a third, different 2D x-ray image view. Computer processor 22 of system 20 identifies the second location of the portion of the tool within the one or more additional 2D x-ray images by means of image processing, and derives the second location of the portion of the tool with respect to one of the original 2D x-ray image views, based upon the second location of the portion of the tool that was identified within the additional 2D x-ray image, and the determined relationship between the first and second 2D x-ray images. Typically, an output is generated in response thereto (e.g., by displaying the derived location of the tool relative to the x-ray image view with respect to which the location has been derived).
Some examples of the applications of the techniques described with reference to
For some applications, the assumption that the tool, after having been inserted into the vertebra (and typically fixated firmly within the vertebra), has indeed proceeded along an anticipated longitudinal forward path is verified, typically automatically. Consecutive x-ray images acquired from a same pose are overlaid upon one another to check whether, when the images are positioned such that a position of the tool as seen in a second image is longitudinally aligned with a prior position of the same tool in a first image, the observed anatomies in both images indeed overlap with one another. Or, alternatively, when the images are positioned such that the observed anatomies in both images overlap with one another, the position of the tool as seen in a second image is indeed longitudinally aligned with a prior position of the same tool in a first image. For some applications, the motion detection sensor described by the present application is used for verifying that no motion (or no motion above a certain threshold) of the subject has occurred during the acquisition of the subsequent images. For some applications, comparison of the alignment is manual (visual) by the user, or automatic (by means of image processing), or any combination thereof.
Reference is now made to
Reference is now made to
For some applications, the image of the tool (a representation thereof, and/or a path thereof) as derived from the 2D images is overlaid upon the 3D image data of the vertebra as a hologram. As noted hereinabove, since, in accordance with such applications, the images of the vertebra and the tool (or a representation thereof) are input from different imaging sources, the segmented data of what is the tool (or its representation) and what is the vertebra is in-built (i.e., it is already known to the computer processor). For some applications, the computer processor utilizes this in-built segmentation to allow the operator to virtually manipulate the tool with respect to the vertebra, within the hologram. For example, the operator may virtually advance the tool further along its insertion path, or retract the tool and observe the motion of the tool with respect to the vertebra. Or, the computer processor may automatically drive the holographic display to virtually advance the tool further along its insertion path, or retract the tool. For some applications, similar techniques are applied to other tools and bodily organs, mutatis mutandis. For example, such techniques could be applied to a CT image of the heart in combination with 2D angiographic images of a catheter within the heart.
For some applications, an optical camera is used to acquire optical images of a tool. For example, optical camera 114, which is disposed on x-ray C-arm 34, as shown in
For some applications, the location of the tool within the optical image space is determined by using two or more optical cameras, and/or one or more 3D optical cameras. For some applications, even with one 2D optical camera, the 3D image data is overlaid upon the optical image, by aligning two or more tools from each of the imaging modalities. For some applications, even with one 2D optical camera and a single tool, the 3D image data is overlaid upon the optical image, by acquiring additional information regarding the orientation (e.g., rotation) of the tool, and/or the depth of the tool below the skin. For some applications, such information is derived from 3D image data from which the location of the skin surface relative to the vertebra is derived. Alternatively or additionally, such information is derived from an x-ray image in which the tool and the subject's anatomy are visible. Alternatively or additionally, such information is derived from the marker set as seen in an x-ray image in which the tool and the subject's anatomy are visible.
As noted hereinabove, since the images of the vertebra and the tool (or a representation thereof) are input from different imaging sources, the segmented data of what is the tool (or its representation) and what is the vertebra is in-built (i.e., it is already known to the computer processor). For some applications, the computer processor utilizes this in-built segmentation to allow the operator to virtually manipulate the tool with respect to the vertebra, within an augmented reality display. For example, the operator may virtually advance the tool further along its insertion path, or retract the tool and observe the motion of the tool with respect to the vertebra. Or, the computer processor may automatically drive the augmented reality display to virtually advance the tool further along its insertion path, or retract the tool.
Although some applications of the present invention have been described with reference to 3D CT image data, the scope of the present invention includes applying the described techniques to 3D MRI image data. For such applications, 2D projection images (which are geometrically analogous to DRRs that are generated from CT images) are typically generated from the MRI image data and are matched to the 2D images, using the techniques described hereinabove. For some applications, other techniques are used for registering MRI image data to 2D x-ray images. For example, pseudo-CT image data may be generated from the MRI image data (e.g., using techniques as described in “Registration of 2D x-ray images to 3D MRI by generating pseudo-CT data” by van der Bom et al., Physics in Medicine and Biology, Volume 56, Number 4), and the DRRs that are generated from the pseudo-CT data may be matched to the x-ray images, using the techniques described hereinabove.
For some applications, MRI imaging is used during spinal endoscopy, and the techniques described herein (including any one of the steps described with respect to
For some applications, level verification as described hereinabove is applied to a spinal endoscopy procedure in order to determine the location of the vertebra with respect to which the spinal endoscopy is to be performed. Alternatively or additionally, the incision site for the spinal endoscopy may be determined using bidirectional mapping of optical images and x-ray images, as described hereinabove. Alternatively or additionally, planning of the insertion may be performed upon the 3D MRI data as described hereinabove. Alternatively or additionally, actual insertion vs. the planned path may be represented upon the 3D MRI data as described hereinabove. Alternatively or additionally, actual insertion vs. the planned path may be represented upon a 2D x-ray image as described hereinabove. For some applications, MRI image data are registered to intraprocedural 2D x-ray images. Based upon the registration, additional steps which are generally as described hereinabove are performed. For example, the needle, dilator, and/or endoscope (and/or a representation thereof, and/or a path thereof) may be displayed relative to a target within the MRI image data (e.g., a 3D MRI image, a 2D cross-section derived from 3D MRI image data, and/or a 2D projection image derived from 3D MRI image data). For some applications, endoscopic image data are co-registered to intraprocedural 2D x-ray images. For example, respective endoscopic image data points may be co-registered with respective locations within the intraprocedural images. For some applications, the co-registered endoscopic image data are displayed with the intraprocedural images, together with an indication of the co-registration of respective endoscopic image data points with respective locations within the intraprocedural images. Alternatively or additionally, endoscopic image data are co-registered to MRI image data. For example, respective endoscopic image data points may be co-registered with respective locations within the MRI image data. For some applications, the co-registered endoscopic image data are displayed with the MRI image data, together with an indication of the co-registration of respective endoscopic image data points with respective locations within the MRI image data.
For some applications, the techniques described herein are performed in combination with using a steerable arm, e.g., a robotic arm, such as a relatively low-cost robotic arm having 5-6 degrees of freedom or a manually-steerable arm. In accordance with some applications, the robotic arm is used for holding, manipulating, and/or activating a tool, and/or for operating the tool along a pre-programmed path. For some applications, computer processor 22 drives the robotic arm to perform any one of the aforementioned operations responsively to imaging data, as described hereinabove.
For some applications, techniques described herein, including but not limited to Algorithm 1 and Algorithm 2 and the projection of planning data associated with the 3D image data upon the 2D x-ray images, are applied in conjunction in the course of inserting a tool into a vertebra. Reference is now made to
It should be noted that the imaging device generating the 2D x-ray images shown in
For some applications, if the observed direction of tool 416 in
For some applications and alternatively or additionally to
For some applications and in accordance with embodiments of the present invention, including but not limited to the embodiments described in connection with
For some applications and in accordance with embodiments of the present invention, including but not limited to the embodiments described in connection with
For some applications and in accordance with embodiments of the present invention, including but not limited to the embodiments described in connection with
Typically, when the first and second views from which the x-ray images are acquired are two different generally-AP views, as opposed to one AP view and one lateral view for example, the maneuvering of the x-ray device by the surgical staff is notably more efficient in terms of effort, time, or both. In such selection of views, the motion of the x-ray device in between (or, if applicable, during) the acquisition of the x-ray images is typically reduced. Additionally, in the motion of the x-ray device between such views the potential physical interferences between the x-ray device (or sterile sheets covering the x-ray device) and the surgical table (or sterile sheets covering the surgical table or the patient) are also typically reduced.
Additionally or alternatively, it should be noted that for some skeletal portions (for example, the spine) x-ray images acquired from a generally-AP view typically offer better visibility of the skeletal portion, compared with images acquired from other views, because from such view there is typically the least amount of tissue (e.g., fat, muscles, various internal organs) in between the patient's skin and the skeletal portion.
Additionally or alternatively, it should be noted that for some skeletal portions (for example, spinal vertebrae), x-ray images acquired from a generally-AP view typically include a greater number of identifiable anatomical features, compared with images acquired from other views, because of the specific anatomy of that skeletal feature.
Additionally or alternatively, it should be noted that for some skeletal portions (for example, spinal vertebrae), x-ray images acquired from a generally-AP view are typically the most familiar to the surgeon, compared with images acquired from other views.
Additionally or alternatively, it should be noted that for some skeletal portions (for example, spinal vertebrae), x-ray images acquired from a generally-AP view are typically the most intuitive to the surgeon for comprehending tool position with respect to the anatomy, compared with images acquired from other views.
Additionally or alternatively, it should be noted that x-ray images acquired from generally similar views are typically the easiest for the surgeon to compare or relate to one another.
Reference is now made to
For some applications, the notches along a line that represents, with respect to the 3D image data or to a 2D cross-section of such data, the would-be path 422 of tool 420, are distance notches. Typically, the notches are spaced at some fixed interval (e.g., 5 mm), and typically that interval is informed to, or displayed to, or otherwise known to, the user. As a result, typically the user can deduce, from the information added by the existence of the notches, the further distance that the tool still needs to be inserted (or in some cases withdrawn) for reaching a designated target.
Reference is now made to
In step 424, planning of the application of the tool to the targeted anatomy is performed upon the 3D image data and associated with the 3D image data. In step 426, the tool is positioned relative to the targeted anatomy. In step 428, a 2D x-ray image is acquired from a first view and the planning data is projected upon the 2D x-ray image, such that both a portion of the tool and the planning data are visible in the 2D x-ray image. In step 430, in the 2D x-ray image from the first view, a correspondence between a portion of the tool and the planning data is determined. If there is not sufficient correspondence, as depicted by decision diamond 432, then the tool is repositioned and steps 426 through 430 are repeated. If there is sufficient correspondence, as depicted by decision diamond 432, then in step 434, without moving the tool, a 2D x-ray image is acquired from a second view and the planning data is projected upon the 2D x-ray image from the second view, such that both the portion of the tool and the planning data are visible in the 2D x-ray image from the second view. In step 436, the aforementioned 2D x-ray images acquired from the first and second views are registered with the 3D image data by means of image processing. In step 438, the portion of the tool is identified by means of image processing in the 2D x-ray images acquired from the first and second views. In step 440, the position of the tool with respect to the 3D image data is determined and typically displayed with respect to the 3D image data. If the portion of the tool is deemed to be positioned improperly relative to the targeted anatomy, as depicted by decision diamond 442, then the tool is repositioned and steps 426 through 440 are typically repeated. However, if the portion of the tool is deemed to be positioned properly relative to the targeted anatomy, as depicted by decision diamond 442, then in step 444 the medical procedure proceeds further in the knowledge that at the time of acquiring the most recent 2D x-ray images from the first and second views the tools was positioned properly.
For some applications, and additionally or alternatively to steps 438 through 440, the planning data is projected upon the 2D x-ray images acquired from the second view, such that both a portion of the tool and the planning data are visible in the 2D x-ray image, and subsequently the determination whether the tool is positioned properly with respect to the targeted anatomy, and more specifically relative to the planning data or to the applicable portion of that data, is made using the 2D x-ray images acquired from the first and second views and in which both the portion of the tool and the projected planning data are visible.
For some applications, as described hereinabove, and in accordance with embodiments of the present invention, including but not limited to the embodiments described in connection with
Thus, in accordance with embodiments of the present invention, including but not limited to the embodiments described in connection with
Alternatively or additionally, for some applications, using techniques described herein, displaying the position of the tool on the 3-D image data comprises displaying (i) the line along which the longitudinal axis of the tool resides, and (ii) where along the line the tool is positioned, e.g., where along the line the tip of the tool resides. This allows for determining the current depth of the tool with respect to the targeted anatomy, in addition to determining if the tool has deviated, or will deviate if inserted further, to either side of the planned insertion line or of the applicable anatomy.
For some applications, the techniques described herein, including but not limited to Algorithm 1 and Algorithm 2, are applied to determine a current position, within or relative to the 3D image data, of the distal portion of a robotic arm, or of an aiming device held by a robotic arm, or of a directional indicator held by a robotic arm, or of a tool held by a robotic arm, or of a portion of such tool. Typically, the use of 2D x-ray images obviates the need for location sensors and/or tracker, and/or for a navigation system using such sensors and/or tracker. For some applications, the techniques described herein are used, typically for robotic applications, in conjunction with techniques described by any of the following patent applications (or a combination thereof), all of which are incorporated herein by reference: US 20160270853 to Lavallee et al., PCT/EP2018/056608 to Lavallee et al., PCT/EP2017/077370 to Lavallee et al., PCT/EP2017/082041 to Lavallee et al., PCT/EP2017/081803 to Lavallee et al. For some applications, the robot is a handheld robot such as the NAVIO robot from Smith and Nephew PLC (London, UK).
For some applications, the techniques described herein, including but not limited to Algorithm 1 and Algorithm 2, are applied to determine a current position, within or relative to the 3D image data, of a guide for the application of treatment to a skeletal anatomy. For some applications, the guide is for a tool held by a robot. For some applications, the guide is for inserting a rod, for example a femoral rod. For some applications, the guide is for inserting a nail. For some applications, the nail is an interlocking nail, for example for the fixation of a femoral rod. For some applications, the guide is a drill guide. For some applications, the guide is for drilling a screw. For some applications, the guide is for drilling one or more pedicle screws into a vertebra. For some applications, the guide is patient-specific, for example the Firefly technology from Mighty Oaks Medical (Englewood, CO, USA) or the patient-specific guides described by any of the following patent applications (or a combination thereof), all of which are incorporated herein by reference: US 20150073419 to Couture, US 20160274571 to Lavallee et al., US 20160279877 to Lavallee, US 20180049758 to Amis et al., US 20180085133 to Lavallee et al. For some applications, the use of 2D x-ray images obviates, in conjunction with a guide, the need for location sensors and/or tracker, and/or for a navigation system using such sensors and/or tracker.
For some application, indication of the actual direction and/or position of the aforementioned arm, aiming device, direction-indicating device, tool, or guide relative to a skeletal anatomy is displayed in conjunction with the planning data for that anatomy, wherein such planning was typically performed with respect to the 3D image data using techniques described herein. For some applications and typically using techniques described herein, such planning data includes one or more planned insertion lines, skin-level incision sites/entry points, bone-level entry points, or any combination thereof. For some applications, the co-identification and co-indication, within or relative to the 3D image data, of the actual vs. planned direction and/or positions is used by system 20 to generate, typically automatically, directions for how to align the actual direction and/or position of the aforementioned arm, aiming device, direction-indicating device, tool or guide with the planned direction and/or position. For some applications, those directions are delivered by system 20 to the robotic arm and executed, typically automatically, by such arm.
Reference is now made to
In step 446, planning of the application of the tool to the targeted anatomy is performed upon previously-acquired 3D image data of that anatomy and typically in accordance with techniques describe herein. In step 448, the steerable arm, e.g., the robotic arm or a manually-steerable arm, typically holding the tool, is positioned, manually or automatically, such that the tool is typically in the vicinity of the targeted anatomy and typically generally aimed at the targeted anatomy. (It is noted that in this step the tool need not be aimed precisely at the targeted anatomy and/or along the desired insertion line.) In step 450, one or more 2D x-ray image are acquired, typically two 2D x-ray images that are acquired from respective views that typically differ by at least 10 degrees, e.g., at least 20 degrees (and further typically by 30 degrees or more). In step 452, the 2D x-ray images are registered with the 3D image data using techniques described herein. In step 454, which is optional, the planning data or a portion thereof is projected upon one or more of the 2D x-ray images using techniques described herein, such that the planning data is displayed together the current position of the tool (which is visible in the x-ray image) relative to the targeted anatomy. In step 456, the actual position of the tool is displayed in conjunction with the planning data (or a portion thereof) upon the 3D image data of the targeted anatomy using techniques described herein including but not limited to Algorithm 1 and Algorithm 2. In step 458, the differences in 3D space between the actual position of the tool and the planning data are calculated, typically automatically by system 20. Such difference may be, without limitation, between the current direction of the tool and the planned insertion line, or between the current location of the distal tip of the tool and the planned incision site or skin-level entry point, or between the current location of the distal tip of the tool and the planned bone-level entry point, or between the current location of the distal tip of the tool and a target location at the distal end of the planned insertion line, or any combination thereof. Typically, the known scale of the 3D image data is used for calculating the differences not only in angular units but also in distance (e.g., longitudinal) units. In step 460, the differences are translated, by system 20 or by the controller of the robotic arm or any combination thereof, into steering directions for the robotic arm. For some applications, a location, position or displacement sensor, or any combination thereof, is coupled with the applicable portion of the patient's body and detects any motion of such portion in between step 450 and step 460. For some applications, and if such motion was detected, steps 450 through 460 are re-executed in whole or in part. For some applications, one or more cameras are aimed at the applicable portion of the patient's body and detect any motion of such portion in between step 450 and step 460. For some applications, and if such motion was detected, steps 450 through 460 are re-executed in whole or in part. In step 462, the steerable arm, e.g., the robotic arm or a manually-steerable arm, is steered, automatically or manually or in any combination thereof, in accordance with the steering directions. For some applications, progress of the tool is computed and optionally is depicted visually, typically continuously, upon the 3D image data by applying the steering directions concurrently to the robotic arm that is being steered within the skeletal portion and to the depicted tool that is steered virtually within the 3D image data, or by recording the actual values of the joints of the robotic arm that is being steered within the skeletal portion and applying those values to the depicted tool that is steered virtually within the 3D image data. In step 464, one or more 2D x-ray images are acquired to verify, using techniques described herein, that the tool is now directed and/or positioned relative to the 3D image data in accordance with its planned direction and/or position, respectively. If the verification demonstrates that a discrepancy between the actual direction and/or position and the planned one(s) still exists, steps 458 through 464 may be repeated as depicted by the dashed arrow 466 leading back from step 464 to step 458 in
For some applications, as described hereinabove, the steerable arm may be steered using a combination of robotic steering and manual steering. For example, the positional degrees of freedom of a robotic arm (e.g., the x,y,z coordinate position of the robotic arm) may be robotized, and the orientational degrees of freedom (e.g., roll, pitch, and yaw of the robotic arm), may be manually controlled, or vice versa. Thus, for example, in applying the steering directions to the robotic arm, the robotic arm may be robotically positioned and manually oriented, or vice versa.
For some applications, guidance of a tool from a planned position in the targeted anatomy, once such position had been reached and typically in accordance with the steps described by
Reference is now made to
In step 468, which optionally is performed in conjunction with step 446 of
For some applications, the first portion of the skeletal anatomy in which the first insertion is performed, and the second portion of the skeletal anatomy in which the second insertion is performed, may have shifted relative to one another after the 3D image data was acquired. In such case, the calculation of the motion path in step 470 typically accounts for that shift. For example, for a spinal procedure in which the first and second insertions are in different vertebrae, the current intra-procedural positions of the two vertebrae relative to one another are determined with the assistance of registering the two vertebrae, as observed in the x-ray images, with the 3D image data using techniques described herein and optionally in conjunction with segmentation of the 3D image data into individual vertebra.
It should be noted that the imaging device generating the 2D x-ray images referred to in
Reference is now made to
As may be observed in the example shown in
Referring to
For some applications, in order to at least partially correct an x-ray image comprising a radiopaque component that is known to be straight, the computer processor uses techniques for automatically identifying a centerline of an object, for example, as described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference, to generate a centerline of said component. Typically, the computer processor then at least partially corrects the image distortion, in at least a portion of the image in which the component that is known to be straight is disposed, by deforming the portion of the radiographic image, such that the centerline of the radiopaque component of the instrument that is known to be straight appears straight within the radiographic image.
For some applications of the present invention, techniques described hereinabove are combined with a system that determines the location of the tip of a tool with respect to a portion of the subject's body by (a) calculating a location of a proximal portion of the tool that is disposed outside the subject's body, and (b) based upon the calculated position of the proximal portion of the tool, deriving a location of a tip of the tool with respect to the portion of the subject's body with respect to the 3D image data. For example, such techniques may be used with a navigation system that, for example, may include the use of one or more location sensors that are attached to a portion of a tool that is typically disposed outside the subject's body even during the procedure. (It is noted that the location sensors that are disposed upon the tool may be sensors that are tracked by a tracker that is disposed elsewhere, or they may be a tracker that tracks sensors that are disposed elsewhere, and thereby acts as a location sensor of the tool.) For example, a tool may be inserted into the subject's vertebra, such that its distal tip (or a distal portion of the tool) is disposed inside the vertebra, and a location sensor may be disposed on a proximal portion of the tool that is disposed outside the subject's body. The navigation system typically derives the location of the tip of the tool (or a distal portion of the tool), by detecting the location(s) of the location sensor(s) that are disposed on the proximal portion of the tool, and then deriving the location of the tip of the tool (or a distal portion of the tool) based upon an assumed location of the distal tip of the tool (or a distal portion of the tool) relative to the location sensor(s). The navigation system then overlays the derived location of the tip of the tip of the tool (or a distal portion of the tool) with respect to the vertebra upon previously acquired 3D image data (e.g., images acquired prior to the subject being placed in the operating room, or when the subject was in the operating room, but typically prior to the commencement of the intervention). Alternatively or additionally, the location of a proximal portion of the tool that is disposed outside the subject's body may be calculated by video tracking the proximal portion of the tool, and/or by means of tracking motion of a portion of a robot to which the proximal portion of the tool is coupled, relative to a prior known position, e.g., based upon the values of the joints of the robot relative to the corresponding values of the joints of the robot at the prior known position.
In such cases, there may be errors associated with determining the location of the tip of the tool (or a distal portion of the tool), based upon the assumed location of the distal tip of the tool (or a distal portion of the tool) relative to the location sensor(s) being erroneous, e.g., due to slight bending of the tool upon being inserted into the vertebra. Therefore, for some applications, during the procedure, typically periodically, 2D x-ray images are acquired within which the actual tip of tool (or distal portion of the tool) within the vertebra is visible. The location of the tip of the tool (or distal portion of the tool) with respect to the vertebra as observed in the 2D x-ray images is determined with respect to the 3D image data, by registering the 2D x-ray images to the 3D image data. For example, the 2D x-ray images may be registered to the 3D image data using techniques described hereinabove. In this manner, the actual location of the tip of the tool (or distal portion of the tool) with respect to the vertebra is determined with respect to the 3D image data. For some applications, in response thereto, errors in the determination of the location of the tip of the tool (or distal portion of the tool) with respect to the vertebra within the 3D image space resulting from the navigation system, are periodically corrected by system 20. For example, based upon the determined location of at least the tip of the tool (or distal portion of the tool), the computer processor may drive the display to update the indication of the location of the tip of the tool (or distal portion of the tool) with respect to the vertebra with respect to the 3D image data. For some applications, the navigation systems comprise the use of augmented reality, or virtual reality, or robotic manipulation of tools, or any combination thereof.
For some applications, techniques described by the present invention are used in conjunction with one or more sensors that are coupled with a tool to determine an orientation and/or position of the tool, or of a portion thereof, with respect to the 3D scan data, at a time when the tool is moved and without necessitating the acquisition of further 2D images. For some applications, the sensor is a location sensor, or a motion sensor, or a displacement sensor, or a bio-impedance sensor, or an electrical conductivity sensor, or a tissue characterization sensor, or any combination thereof. For some applications, and typically wherein the tool is generally-rigid and the sensor is a location, motion or displacement sensor, the sensor is coupled to the tool's proximal portion. For some applications, and typically wherein the tool is generally-non-rigid and the sensor is a location, motion or displacement sensor, the sensor is coupled to the tool's distal portion. For some applications, and typically wherein the sensor is a bio-impedance, electrical conductivity or tissue characterization sensor, the sensor is coupled to the tool's distal portion. For some applications, the one or more sensors are wireless.
For some applications, embodiments of the present invention described herein are applied in combination with a surgical navigation system.
Reference is now made to
In Step 476, 3D image data (for example, CT) of a skeletal portion of the subject's anatomy is acquired.
In Step 478, a tool coupled with one or more sensors is placed at a first location relative to, or within the skeletal portion.
In Step 480, two 2D images (for example, X-ray images), in which the tool and the skeletal portion are visible, are acquired from different views of the imaging device relative to the skeletal portion.
For some applications, information provided by the one or more sensors is used to determine whether the position of the tool has changed in between the acquisition of the 2D images. Typically, if the position of the tool has changed, the first of the two images is reacquired while the tool remains at the same position as during the previous acquisition of the second of the two 2D images.
In Step 482, the position of the tool relative to the 3D image data is determined, and optionally displayed upon the 3D image data and/or one or more 2D cross-sections thereof, using techniques described previously by the present invention.
In Step 484, the tool is moved relative to the skeletal portion, such that its position and/or orientation is changed relative to the first location. Information about the tool's motion, including changes in orientation, displacement, or a combination thereof, is provided by the one or more sensors, typically wirelessly, to System 20.
In Step 486, which may be concurrent with Step 484, motion of the skeletal portion independently of the motion of the tool, if applicable, is accounted for with respect to the information provided by Step 484, for example by using a reference sensor that is coupled to an applicable portion of the subject's body. For some applications, the net motion, or net change in location, of the tool, relative to the targeted anatomy, is determined by deducting from the motion, or change in location, of the tool as indicated by information from the on-tool sensor(s), the motion, or change in location, of the skeletal portion in which the targeted anatomy resides as indicated by information from the reference sensor. Typically, when the subject is anesthetized such motion of the skeletal portion independently of the motion of the tool does not occur and thus this step is typically not needed.
In Step 488, which may be concurrent with Step 484, a change relative to Step 480 in position of surgical table on which the subject lies, if applicable, is accounted for with respect to the information provided by Step 484. For some applications, information about such motion is received from reference sensor that is coupled to the surgical table, or from a computerized controller of the surgical table, or from the operator of the surgical table, or any combination thereof. Typically, in the course of the insertion of any given tool the position of the surgical table remains unchanged, and thus this step is typically not needed.
In Step 490, the information provided by the one or more sensors with respect to changes in the tool's orientation and/or its displacement is applied to the position of the tool relative to the 3D image data of the skeletal portion that was previously determined in Step 482. As a result, the current position of the tool relative to the 3D image data of the skeletal portion is re-determined. Typically, since as a result of Step 482 the position of the tool within the 3D scan data is already known in the reference frame of coordinates of the 3D image data, the position of the tool is re-determined within that same reference frame of coordinates, by applying the changes in the tool's orientation and/or its displacement that were calculated in Step 484 (and optionally also in Step 486 and/or in Step 488). For example, the change in orientation as reported by the one or more sensors is applied to the tool's prior orientation that was known in that reference frame of coordinates as a result of Step 482. For example, the displacement is applied to the tool's prior position that was known in that reference frame of coordinates as a result of Step 482.
For some applications, the sensor is a location sensor (e.g., magnetic, electromagnetic, optical, radiation-emitting, or any combination thereof) and the current position is calculated by applying to the prior, already-known position the change in location as measured with the sensor. At the time of the present invention, such location sensors are available, for example, from NDI of Ontario, Canada. Such location sensors are commonly used by various surgical navigation systems.
For some applications, the sensor is a displacement and/or motion sensor (e.g., an inertial measurement unit, a gyroscope, an accelerator, or any combination thereof) and the current position is calculated by applying to the prior, already known position the displacement and/or motion measured with the sensor. At the time of the present invention, such sensors are available from multiple suppliers including Xsense Technologies B.V. (Enschede, the Netherlands), MbientLab Inc. (CA, USA) and STT Systems (San Sebastian, Spain).
For some applications, the sensor is capable of distinguishing between cortical bone, cancellous bone, nerves, blood vessels, etc. (e.g., a bio-impedance sensor, an electrical conductivity sensor, some other form of tissue characterization sensor, or any combination thereof) and the current position is calculated by applying the characterization of the tissue in which the sensor is now positioned to the forward trajectory indicated by the prior, already known, position. For example, if the tool is advanced further in a generally-straight line and then the sensor indicates for the first time that it has traversed from cancellous bone to cortical bone, then the location is calculated to be the first occurrence of cortical bone along the path forward from the prior outcome. At the time of the present invention, such sensors or probes incorporating such sensors are available, for example, from SpineGuard, S. A. of Vincennes, France.
In Step 492, the position of the tool relative to the 3D image data, using the calculations performed in Step 490. is displayed upon the 3D image data, or upon a portion of the 3D image data, or upon 2D cross-sections generated from the 3D image data, or upon 2D images which were used to generate the 3D image data, or upon previously-acquired 2D x-ray images, or any combination thereof.
For some applications, any or all of Steps 484 through 492 are repeated one or more times, typically on-line, intermittently or continuously, as depicted by dashed arrow 494.
In Step 496, another one or more 2D images, in which both the tool and the applicable skeletal portion are visible, is acquired.
In Step 498, the position of the tool relative to the 3D image data is re-determined, and optionally displayed upon the 3D image data and/or one or more 2D cross-sections thereof, using techniques described previously by the present invention.
In Step 500, any discrepancy between the position of the tool relative to the skeletal portion as determined in Step 490 and Step 498 is settled, typically by accepting the outcome of Step 498.
For some applications, any or all of Steps 484 through 500 are repeated one or more times, as depicted by dashed arrow 502.
For some applications, if a discrepancy is found in step 500, and the outcome of step 498 is accepted as the location of the portion of the tool, then in the repetition of step 490, the information provided by the one or more sensors with respect to changes in the tool's orientation and/or its displacement is applied to the position of the tool as determined in step 498.
For some applications, the one or more sensors are implemented in accordance with sensors applied by any CAS solutions described in the Background section hereinabove.
For some applications, embodiments of the present invention that are described by
For some applications, techniques described herein for generating projection images (including but not limited to DRRs) from previously-acquired 3D image data (including but not limited to CT) are applied for generating projection images that correspond to current positions or views of the 2D imaging device (including but not limited to an x-ray C-Arm) but without necessarily acquiring corresponding 2D images from those positions or views.
For some applications, the positions or views of the x-ray C-Arm are displayed, typically in the form of Caudal-Cranial (also known as Angular or Secondary) and Left-Right (also known as Orbital, or Primary, or RAO-LAO) C-Arm angles (typically in degrees), upon the x-ray screen and are read by system 20 by means of frame grabbing of the x-ray screens and Optical Character Recognition (OCR) within those screens of the displayed angles. For some applications, C-Arm angles are transferred from the x-ray system to system 20 via a communication protocol. For some applications, C-Arm angles are measured by a sensor that is situated upon the x-ray C-Arm and connected, typically wirelessly, to system 20. For some applications, such sensor comprises one or more gyroscopes, one or more accelerometers, one or more magnetometers, or any combination thereof. For some applications, the sensor is an Inertial Measurement Unit (IMU). For some applications, the C-Arm angles may be known without the use of a sensor, by means of internal measurements by the x-ray system.
For some applications, additionally (or alternatively, in the case where the C-Arm angles have not changed) to tracking the C-Arm angles, the motion of the base of the C-Arm is tracked, such that projection images are generated corresponding to current positions of the base of the C-Arm relative to the subject's body, the operating table, and/or the operating room. (For example, an optical tracker positioned on the base of the C-Arm may track the position of the C-Arm relative to the floor of the operating room, or vice versa.) For some applications, the position of the base of the C-Arm is measured by a sensor that is coupled to the base of the C-Arm and connected, typically wirelessly, to system 20. For some applications, the sensor is a motion sensor. For some applications, the sensor comprises one or more gyroscopes, one or more accelerometers, one or more magnetometers, or any combination thereof. For some applications, the sensor is an Inertial Measurement Unit (IMU).
Reference is now made to
As further demonstrated hereinbelow, benefits typically include, without limitation, a reduction in the radiation induced to patient and staff because, typically, fewer x-ray images need to be acquired prior to reaching a desired view of the applicable skeletal portion, or of one or more tools applied to that portion, or both.
It is noted that some of the steps shown in
In step 504, which for some applications is pre-operative, 3D image data of the applicable skeletal portion is acquired.
In step 506, which is optional, planning for tool insertion is performed upon the 3D image data. Such planning includes, without limitation, an insertion trajectory, a skin-level entry point, a bone-level entry point, an intermediate point along the insertion trajectory between a skin-level entry point and a target point, a target point, a simulated tool or implant positioned at any point along the insertion trajectory, or any combination thereof.
In step 508, which is optional, an x-ray image of the applicable skeletal portion is acquired from a first view, for example an AP view. In step 509, which is also optional, the x-ray image is registered to the 3D image data using techniques described herein.
Assuming that the registration of step 509 was executed, in step 510, which is optional, the planning that was performed previously upon the 3D image data is projected upon the x-ray image acquired in step 508.
In step 512, which is optional, a tool (or a portion thereof) is detected in the x-ray image acquired in Step 508, and marked upon the x-ray image acquired in Step 508.
In step 514, the x-ray C-Arm is positioned at a new view and the angles of the C-Arm are received by system 20, using techniques described herein. For some applications, values of additional parameters of the x-ray camera are received by said techniques. Such parameters may include, without limitation, zoom, rotation, flip, or any combination thereof.
In step 516, a DRR corresponding to the new view is generated from the 3D image data. The DRR corresponds to then-current C-Arm angles, with such angles received by system 20 using techniques described herein. For some applications, the DRR corresponds to the values of the additional known parameters of the x-ray camera. For some applications, the DRR corresponds to the x-ray image acquired in Step 508 and results from the registration of the x-ray image with the 3D image data.
In step 518, which is optional, the planning that was performed previously upon the 3D image data is projected upon the DRR generated in Step 516 using techniques described herein.
In step 520, which is optional, a representation of the tool (or a portion thereof) detected in step 512 is displayed upon the DRR generated in Step 516 using techniques described herein. In step 522, which is optional, (i) a representation of the tool (or a portion thereof) detected in step 512, or (ii) the previously-performed planning or both, are displayed upon cross-sections of the 3D image data or within the 3D image data, using techniques described herein. Such techniques include, without limitation, those that are in accordance with Algorithm 1, Algorithm 1A, Algorithm 2, Algorithm 2A, Algorithm 2B, or any combination thereof. As disclosed by such techniques, step 518 may utilize a prior acquisition of two x-ray images of the skeletal portion, for example by executing Steps 508-510 twice from two different views.
For some applications, steps 508 through 518, or any combination thereof, are repeated.
As illustrated by decision diamond 524, for some applications, subsequently to performing steps 514 and 516 the user may desire to obtain a subsequent DRR from a subsequent view. In this case, system 20 may receive an input to return to step 514, in which system 20 receives C-Arm angles corresponding to the subsequent view and a DRR corresponding to the new view is generated in step 516. As such, steps 514 through 516 may be repeated one or more times, thus providing a selection of subsequent DRRs each corresponding to a subsequent new view. As illustrated by decision diamond 526, if multiple iterations of steps 514-516 are performed, then the user may choose which of the generated DRRs, each corresponding to a subsequent new view from repeated step 514, from which to actually acquire a new x-ray image. This selection of which view to acquire the x-ray image from is typically based upon, or assisted by, the user's observation of the DRRs generated by the one or more repetitions of step 516. It is noted that for each repetition starting from claim 514 any of the optional steps 516-522 may be performed.
For some applications, the first time step 516 is performed, the DRR that is generated corresponding to the new view may correspond to a view from which the user desires to acquire a new x-ray image. In this case steps 514-516 are performed only once, i.e., at decision diamond 524 input is not received to repeat from step 14, and subsequently at decision diamond 526 system 20 determines that multiple iterations of steps 514-516 were not performed, at which point an x-ray image is acquired from the new view in step 530.
Benefits typically include, without limitation, a reduction in the radiation induced to patient and staff because typically fewer x-ray images need to be acquired prior to reaching a desired view of the applicable skeletal portion, or of one or more tools applied to that portion, or both.
For example, such desired view from which an x-ray image will actually be acquired may avoid foreshortening of one or more tools applied to the skeletal portion, or allow good visibility of the skeletal portion, or enable good comprehension of the position of the one or more tools relative to the skeletal portion, or any combination thereof.
For some applications, images (both x-ray images and DRRs) generated throughout the method depicted by
For some applications, steps 504 through 530, or any combination thereof, are applied to an anatomical portion that is, in full or in part, not skeletal.
Reference is now made to
By way of illustration and not limitation, it is noted that the scope of the present invention includes applying the apparatus and methods described herein to any one of the following applications:
For some applications system 20 includes additional functionalities to those described hereinabove. For example, the computer processor may generate an output that is indicative of a current level of accuracy (e.g., of verification of the vertebral level, determination of the insertion site, and/or registration of the 3D image data to the 2D images), e.g., based upon a statistical calculation of the possible error. For some applications, the computer processor generates a prompt indicating that a new x-ray from one or more views should be acquired. For example, the computer processor may generate such a prompt based on the time elapsed since a previous x-ray acquisition from a given view, and/or based on the distance a tool has moved since a previous x-ray acquisition from a given view, and/or based on observed changes in the position of markers 52 relative to the C-arm.
Techniques described herein may be practiced in combination with techniques described in U.S. application Ser. No. 16/083,247 to Tolkowsky, filed Sep. 7, 2018, entitled “Apparatus and methods for use with skeletal procedures,” which is the US national stage of PCT IL/2017/050314, which published as WO 2017/158592.
Applications of the invention described herein can take the form of a computer program product accessible from a computer-usable or computer-readable medium (e.g., a non-transitory computer-readable medium) providing program code for use by or in connection with a computer or any instruction execution system, such as computer processor 22. For the purpose of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Typically, the computer-usable or computer readable medium is a non-transitory computer-usable or computer readable medium.
Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random-access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W) and DVD. For some applications, cloud storage, and/or storage in a remote server is used.
A data processing system suitable for storing and/or executing program code will include at least one processor (e.g., computer processor 22) coupled directly or indirectly to memory elements (such as memory 24) through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. The system can read the inventive instructions on the program storage devices and follow these instructions to execute the methodology of the embodiments of the invention.
Network adapters may be coupled to the processor to enable the processor to become coupled to other processors or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the C programming language or similar programming languages.
It will be understood that blocks of the flowchart shown in
Computer processor 22 and the other computer processors described herein are typically hardware devices programmed with computer program instructions to produce a special purpose computer. For example, when programmed to perform the algorithms described herein, the computer processor typically acts as a special purpose skeletal-surgery-assisting computer processor. Typically, the operations described herein that are performed by computer processors transform the physical state of a memory, which is a real physical article, to have a different magnetic polarity, electrical charge, or the like depending on the technology of the memory that is used.
It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof that are not in the prior art, which would occur to persons skilled in the art upon reading the foregoing description.
The present application is a Continuation of U.S. Ser. No. 18/297,802, filed Apr. 10, 2023, which published as US 2023/0240628 to Cohen et al., and which is a Continuation of PCT/IL2021/051218, filed Oct. 13, 2021, which published as PCT Publication WO 2022/079715 to Tolkowsky et al., and which claims the priority of the following applications; U.S. 63/091,351 to Tolkowsky et al., filed Oct. 14, 2020, entitled, “Apparatus and methods for use with image-guided skeletal procedures,”U.S. 63/130,877 to Tolkowsky et al., filed Dec. 28, 2020, entitled, “Apparatus and methods for use with image-guided skeletal procedures,” andU.S. 63/164,349, to Cohen et al., filed Mar. 22, 2021, entitled, “Apparatus and methods for use with image-guided skeletal procedures.” Each of the abovementioned applications is incorporated herein by reference. The present application is related to the following applications: (A) U.S. Ser. No. 16/901,513, filed Jun. 15, 2020 which issued as U.S. Pat. No. 11,490,967 to Tolkowsky, entitled “Apparatus and methods for use with image-guided skeletal procedures,” which is a Continuation of U.S. Ser. No. 16/083,247, filed Sep. 7, 2018, which issued as U.S. Pat. No. 10,716,631 to Tolkowsky, which is the US national stage application of PCT IL/2017/050314 filed Mar. 13, 2017, which published as PCT Publication WO 2017/158592 to Tolkowsky, and which claims priority from: U.S. Provisional Patent Application No. 62/307,514 to Tolkowsky, filed Mar. 13, 2016, entitled “Freehand Assistant for Spinal Surgery;”U.S. Provisional Patent Application No. 62/334,463 to Tolkowsky, filed May 11, 2016, entitled “Freehand Assistant for Spinal Surgery;”U.S. Provisional Patent Application No. 62/362,607 to Tolkowsky, filed Jul. 15, 2016, entitled “Freehand Assistant for Spinal Surgery;”U.S. Provisional Patent Application No. 62/398,085 to Tolkowsky, filed Sep. 22, 2016, entitled “Freehand Assistant for Spinal Surgery;”U.S. Provisional Patent Application No. 62/439,495 to Tolkowsky, filed Dec. 28, 2016, entitled “Freehand Assistant for Spinal Surgery;” andU.S. Provisional Patent Application No. 62/463,747 to Tolkowsky, filed Feb. 27, 2017, entitled “Freehand Assistant for Spinal Surgery.”(B) U.S. Ser. No. 17/021,324, filed Sep. 15, 2020, which issued as U.S. Pat. No. 11,224,483 to Steinberg et al., entitled “Apparatus and methods for use with image-guided skeletal procedures,” which is a Continuation of U.S. Ser. No. 16/629,449, filed Jan. 8, 2020, which issued as U.S. Pat. No. 11,406,338 to Tolkowsky, which is the US national stage application of PCT/IL2018/050732, filed Jul. 5, 2018, which published as PCT Publication WO 2019/012520 to Tolkowsky et al., and which claims the priority of the following applications: U.S. 62/530,123 to Tolkowsky et al., filed Jul. 8, 2017, entitled, “Apparatus and methods for use with image-guided skeletal procedures,”U.S. 62/556,436 to Tolkowsky et al., filed Sep. 10, 2017, entitled, “Apparatus and methods for use with image-guided skeletal procedures,”U.S. 62/599,802 to Tolkowsky et al., filed Dec. 18, 2017, entitled, “Apparatus and methods for use with image-guided skeletal procedures,” andU.S. 62/641,359 to Tolkowsky et al., filed Mar. 11, 2018, entitled, “Apparatus and methods for use with image-guided skeletal procedures.”(C) International application PCT/IL2019/051272 to Tolkowsky et al., filed Nov. 21, 2019, published as WO 2020/105049 to Tolkowsky et al., entitled “Apparatus and methods for use with image-guided skeletal procedures”, and which claims the priority of the following applications: U.S. 62/770,758 to Tolkowsky, filed Nov. 22, 2018, entitled, “Apparatus and methods for use with image-guided skeletal procedures,”U.S. 62/883,669 to Tolkowsky, filed Aug. 7, 2019, entitled, “Apparatus and methods for use with image-guided skeletal procedures,” andU.S. 62/909,791 to Tolkowsky, filed Oct. 3, 2019, entitled, “Apparatus and methods for use with image-guided skeletal procedures.” Each of the abovementioned applications is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 63091351 | Oct 2020 | US | |
| 63130877 | Dec 2020 | US | |
| 63164349 | Mar 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18297802 | Apr 2023 | US |
| Child | 18440455 | US | |
| Parent | PCT/IL2021/051218 | Oct 2021 | WO |
| Child | 18297802 | US |
