Patent Application
20250200784
The present disclosure relates to object identification and object semantic segmentation systems. The systems can generate a complete sinogram from a plurality of fluoroscopic images containing objects projected with selected angles and, from the complete sinogram, reconstruct cross-sections of the objects. The generation of the complete sinogram may involve determining moments of inertia and principal axes from the plurality of fluoroscopic images and simulating one or more fluoroscopic images projected at other unselected angles using them. When cross-sections of the objects are reconstructed, various geometrical properties may be derived from the reconstructed cross-sections. The geometrical properties can serve to automatically segment the objects in a fluoroscopic image and, in particular, provide initial pose estimation of the objects in the fluoroscopic image.
This Summary is provided to introduce in a simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.
One innovative aspect of the subject matter of this disclosure can be implemented in a system including an imaging device, one or more processors, and a memory. The imaging device is configured to rotate axially about a rotational axis and to capture a plurality of images associated with a sinogram at a plurality of angles, respectively, along the rotational axis. The memory stores instructions that, when executed by the one or more processors, cause the system to detect an object in each image of the plurality of images, determine a moment of inertia associated with the object based on the plurality of images, simulate one or more images for the sinogram based at least in part on the moment of inertia associated with the object, generate a reconstruction of the object based on the sinogram, and determine one or more properties of the object based on the reconstruction.
Another innovative aspect of the subject matter of this disclosure can be implemented in a computer-implemented method. The method includes steps of receiving a plurality of images associated with a sinogram, where the plurality of images is captured by an imaging device at a plurality of angles, respectively, along a rotational axis of the imaging device; detecting an object in each image of the plurality of images; determining a moment of inertia associated with the object based on the plurality of images; simulating one or more images for the sinogram based at least in part on the moment of inertia associated with the object; generating a reconstruction of the object based on the sinogram; and determining one or more properties of the object based on the reconstruction.
Another innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, including one or more processors and a memory. The memory stores instructions that, when executed by the one or more processors, cause the controller to receive a plurality of images associated with a sinogram, where the plurality of images is captured by an imaging device at a plurality of angles, respectively, along a rotational axis of the imaging device; detect an object in each image of the plurality of images; determine a moment of inertia associated with the object based on the plurality of images; simulate one or more images for the sinogram based at least in part on the moment of inertia associated with the object; generate a reconstruction of the object based on the sinogram; and determine one or more properties of the object based on the reconstruction.
Various embodiments are depicted in the accompanying drawings for illustrative purposes and should in no way be interpreted as limiting the scope of the inventions. In addition, various features of different disclosed embodiments can be combined to form additional embodiments, which are part of this disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondence between reference elements.
The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Although certain preferred embodiments and examples are disclosed below, inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and/or uses and to modifications and equivalents thereof. Thus, the scope of the claims that may arise herefrom is not limited by any of the particular embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may be described as multiple discrete operations in turn, in a manner that may be helpful in understanding certain embodiments; however, the order of description should not be construed to imply that these operations are order dependent. Additionally, the structures, systems, and/or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.
Although certain spatially relative terms, such as “outer,” “inner,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” and similar terms, are used herein to describe a spatial relationship of one device/element or anatomical structure to another device/element or anatomical structure, it is understood that these terms are used herein for ease of description to describe the positional relationship between element(s)/structures(s), such as with respect to the illustrated orientations of the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of the element(s)/structures(s), in use or operation, in addition to the orientations depicted in the drawings. For example, an element/structure described as “above” another element/structure may represent a position that is below or beside such other element/structure with respect to alternate orientations of the subject patient or element/structure, and vice-versa. It should be understood that spatially relative terms, including those listed above, may be understood relative to a respective illustrated orientation of a referenced figure.
Certain reference numbers are re-used across different figures of the figure set of the present disclosure as a matter of convenience for devices, components, systems, features, and/or modules having features that may be similar in one or more respects. However, with respect to any of the embodiments disclosed herein, re-use of common reference numbers in the drawings does not necessarily indicate that such features, devices, components, or modules are identical or similar. Rather, one having ordinary skill in the art may be informed by context with respect to the degree to which usage of common reference numbers can imply similarity between referenced subject matter. Use of a particular reference number in the context of the description of a particular figure can be understood to relate to the identified device, component, aspect, feature, module, or system in that particular figure, and not necessarily to any devices, components, aspects, features, modules, or systems identified by the same reference number in another figure. Furthermore, aspects of separate figures identified with common reference numbers can be interpreted to share characteristics or to be entirely independent of one another. In some contexts features associated with separate figures that are identified by common reference numbers are not related and/or similar with respect to at least certain aspects.
The present disclosure provides systems, devices, and methods for a mathematical reconstruction of objects in a projection image, which can be used to derive various geometrical properties of the objects within the image and can serve as an initialization for object semantic segmentation. In particular, systems, devices, and methods in accordance with one or more aspects of the present disclosure can reduce a number of projection images needed to generate a complete sinogram. Additionally, the one or more aspects can serve to identify the objects in projection images and to estimate their initial poses. It is contemplated that the objects can include, but not limited to, a scope and a nodule.
With respect to medical instruments described in the present disclosure, the term “instrument” is used according to its broad and ordinary meaning and may refer to any type of tool, device, assembly, system, subsystem, apparatus, component, or the like. In some contexts herein, the term “device” may be used substantially interchangeably with the term “instrument.”
Robotic surgical systems can be utilized to facilitate instrument navigation to areas within a patient's body. In some embodiments, robotic systems can be configured to provide an interface that allows an operator to navigate robotically-controlled instrumentation by directing the movement of the instrumentation in multiple degrees of freedom. For example, the operator may direct axial translation (i.e., insertion and/or retraction), articulation angle, and/or roll (i.e., articulation angle direction), of endoscopes, access sheaths, guidewires, working instruments (e.g., needles, baskets, lithotripsy devices, etc.). Navigation within organs, branch vessel ostia, or other relatively open three-dimensional space can be challenging due to the need to understand the three-dimensional relationship of the navigated/tracked instrument relative to the anatomical target and/or to determine in which plane the instrument will bend. This task can be difficult, in part, because navigating an instrument through a lumen of the patient from a remote patient access point to the desired site of a procedure requires manipulating the instrument without a direct line of sight of the instrument. A positioning/tracking system may be used to help locate the desired site of the procedure and visualize the navigation of the instrument to the desired site of the procedure. Positioning/tracking systems allow the user to visualize a patient's internal anatomy and the location and/or orientation of the detectable markers of the instrument within the patient's anatomy.
Positioning systems can include imaging systems/modalities, such as positron emission tomography (PET), X-ray computed tomography (CT), X-ray fluoroscopy, magnetic resonance imaging (MRI), camera-based optical systems, and ultrasonic or other sonic imaging systems. Positioning system can further include electromagnetic (EM) tracking systems (e.g., using electromagnetic field generators as described in detail herein), fiber optic tracking systems, and robotic tracking/positioning based on robotic data (e.g., robotic actuator, torque, pose data). Some imaging systems/modalities are not suitable for continuous real-time tracking of instruments, such as PET, CT, and MRI, which generally produce and combine many cross-sectional images of an object to generate a computer-processed image; such an image capture process can be relatively slow, and movement within the image field during the image capture process can produce image artifacts that make such systems unsuitable for real-time tracking of moving instruments in a body. Additionally, some imaging systems/modalities such as X-ray, CT, and fluoroscopy emit potentially harmful ionizing radiation, and it may be desirable to limit the duration of their use.
Electromagnetic (EM) tracking systems and fiber optic tracking systems can provide real-time instrument tracking. EM tracking generally functions by detecting/determining the position/orientation of EM sensing coil(s) (i.e., an EM marker/sensor) in a fluctuating magnetic field. The fluctuating magnetic field induces a current in the coil based on the coil's position and orientation within the field. The coil's position and orientation can thus be determined by measuring the current in the coil. In some cases, a single EM sensor/marker is able to sense its position and orientation in three-dimensional space with five degrees of freedom. That is, the EM sensor can provide data indicating orientation in every direction except around the axial symmetric axis of the coil (i.e., roll). Two EM sensors/markers held in a fixed relative position and orientation on an instrument or other marker device may be used to sense all six degrees of freedom of the instrument. In navigation systems employing EM tracking, an image of an anatomical space can be acquired, wherein the system control circuitry is configured to determine a registration between a frame of reference of the EM sensor(s)/marker(s) associated with a tracked instrument and a frame of reference of an imaging system/modality used to image the anatomical space to depict movement of the tracked instrument within the imaged anatomical space.
Although certain aspects of the present disclosure are described in detail herein in the context of bronchoscopy and ureteroscopy procedures, it should be understood that such context is provided for convenience and clarity, and instrument positioning concepts disclosed herein are applicable to any suitable medical procedures.
With respect to ureteroscopy procedures, surgeons may insert an endoscope (e.g., ureteroscope) into the urinary tract through the urethra to remove urinary stones from the bladder and ureter, such as for the purpose of removing kidney stones. In some procedures, physicians may use a percutaneous nephrolithotomy (“PCNL”) technique that involves inserting a nephroscope through the skin (i.e., percutaneously) and intervening tissue to provide access to the treatment site for breaking-up and/or removing the stone(s). Relatively large kidney stones can be broken into a relatively smaller fragments to facilitate extraction thereof using certain instrumentation, such as laser lithotripsy devices. According to some procedures, a basketing device/system may be used to capture the relatively smaller stone fragment(s) and extract them from the treatment site out of the patient. Any of the instrumentation associated with such ureteroscopy procedures can be robotically-controlled and/or positionally tracked by tracking/detecting marker(s)/sensor(s) associated with the instrumentation using a positioning modality as described in detail herein.
The medical system 100 includes a robotic system 10 (e.g., mobile robotic cart) configured to engage with and/or control a medical instrument (e.g., ureteroscope) including a proximal handle 31 and a shaft 40 coupled to the handle 31 at a proximal portion thereof to perform a procedure on a patient 7. It should be understood that the instrument 40 may be any type of shaft-based medical instrument, including an endoscope (such as a ureteroscope or bronchoscope), catheter (such as a steerable or non-steerable catheter), needle, nephroscope, laparoscope, or other type of medical instrument. The instrument 40 may access the internal patient anatomy through direct access (e.g., through a natural orifice) and/or through percutaneous access via skin/tissue puncture.
The medical system 100 includes a control system 50 configured to interface with the robotic system 10, provide information regarding the procedure, and/or perform a variety of other operations. For example, the control system 50 can include one or more display(s) 56 configured to present certain information to assist the physician 5 and/or other technician(s) or individual(s). The medical system 100 can include a table 15 configured to hold the patient 7. The system 100 may further include an electromagnetic (EM) field generator, such as a robot-mounted EM field generator 80 or an EM field generator 85 mounted to the table 15 or other structure.
Although the various robotic arms 12 are shown in various positions and coupled to various tools/devices, it should be understood that such configurations are shown for convenience and illustration purposes, and that the medical system 100 may include any number of robotic arms 12 which may have different configurations over time and/or at different points during a medical procedure. Furthermore, the robotic arms 12 may be coupled to different devices/instruments than shown in
In some embodiments, the physician 5 can interact with the control system 50 and/or the robotic system 10 to cause/control the robotic system 10 to advance and navigate the medical instrument shaft 40 (e.g., a scope) through the patient anatomy to the target site and/or perform certain operations using the relevant instrumentation. The control system 50 can provide information via the display(s) 56 that is associated with the medical instrument 40, such as real-time endoscopic images captured therewith, and/or other instruments of the system 100, to assist the physician 5 in navigating/controlling such instrumentation. The control system 50 may provide imaging/positional information to the physician 5 that is based on certain positioning modalities, such as fluoroscopy, ultrasound, optical/camera imaging, EM field positioning, or other modality, as described in detail herein.
The various scope/shaft-type instruments disclosed herein, such as the shaft 40 of the system 100, can be configured to navigate within the human anatomy, such as within a natural orifice or lumen of the human anatomy. The terms “scope” and “endoscope” are used herein according to their broad and ordinary meanings, and may refer to any type of elongate (e.g., shaft-type) medical instrument having image generating, viewing, and/or capturing functionality and being configured to be introduced into any type of organ, cavity, lumen, chamber, or space of a body. A scope can include, for example, a ureteroscope (e.g., for accessing the urinary tract), a laparoscope, a nephroscope (e.g., for accessing the kidneys), a bronchoscope (e.g., for accessing an airway, such as the bronchus), a colonoscope (e.g., for accessing the colon), an arthroscope (e.g., for accessing a joint), a cystoscope (e.g., for accessing the bladder), colonoscope (e.g., for accessing the colon and/or rectum), borescope, and so on. Scopes/endoscopes, in some instances, may comprise an at least partially rigid and/or flexible tube, and may be dimensioned to be passed within an outer sheath, catheter, introducer, or other lumen-type device, or may be used without such devices. Endoscopes and other instruments described herein can have associated with distal ends or other portions thereof certain markers/sensors configured to be visible/detectable in a field/space associated with one or more positioning (e.g., imaging) systems/modalities.
The system 100 is illustrated as including a fluoroscopy system 70, which includes an X-ray generator 75 and an image detector 74 (referred to as an “image intensifier” in some contexts; either component 74, 75 may be referred to as a “source” or “emitter” herein), which may both be mounted on a moveable C-arm 71. In some instances, the fluoroscopy system 70 and any portions thereof may be referred as an imaging device. The control system 50 or other system/device may be used to store and/or manipulate images generated using the fluoroscopy system 70. In some embodiments, the bed 15 is radiolucent, such that radiation from the generator 75 may pass through the bed 15 and the target area of the patient's anatomy, wherein the patient 7 is positioned between the ends of the C-arm 71. The structure/arm 71 of the fluoroscopy system 70 may be rotatable or fixed. The fluoroscopy system 70 may be implemented to allow live images to be viewed to facilitate image-guided surgery. The structure/arm 71 can be selectively moveable to permit various images of the patient 7 and/or surgical field to be taken by the fluoroscopy panel source 74.
In the example urology configuration shown in
As with the system 100 of
As shown, the robotic-enabled table system 104 can include a column 144 coupled to one or more carriages 141 (e.g., ring-shaped movable structures), from which the one or more robotic arms 112 may emanate. The carriage(s) 141 may translate along a vertical column interface that runs at least a portion of the length of the column 144 to provide different vantage points from which the robotic arms 112 may be positioned. The carriage(s) 141 may rotate around the column 144 in some embodiments to allow the robotic arms 112 to have access to multiples sides of the table 104. Rotation and/or translation of the carriage(s) 141 can allow the system 102 to align the medical instruments, such as endoscopes and catheters, into different access points on the patient.
With reference to
The robotic system 10 can be physically and/or communicatively coupled to any component of the medical system, such as to the control system 50, the table 15, the EM field generator 80/85, the scope 40, the fluoroscopy system 70, and/or any type of percutaneous-access instrument (e.g., needle, catheter, nephroscope, etc.). For example, the robotic system 10 can include communication interface(s) 214 for communicating with communication interface(s) 254 of the control system 50. The robotic system 10 may be configured to receive control signals from the control system 50 to perform certain operations, such as to position one or more of the robotic arms 12, manipulate the scope 40, and so on. In response, the robotic system 10 can control, using certain control circuitry 211, actuators 217, and/or other components of the robotic system 10 to perform the operations. For example, the control circuitry 211 may control various motors/actuators associated with the various joints of the robotic arms 12 and/or the arm support 17. In some embodiments, the robotic system 10 and/or control system 50 is/are configured to receive images and/or image data from the scope 40 representing internal anatomy of the patient 7 and/or portions of the access sheath or other device components.
The robotic system 10 generally includes an elongated support structure 14 (also referred to as a “column”), a robotic system base 25, and a console 13 at the top of the column 14. The column 14 may include one or more arm supports 17 (also referred to as a “carriage”) for supporting the deployment of the one or more robotic arms 12 (three shown in
The robotic arms 12 may generally comprise robotic arm bases 21 and end effectors 6a-6c, separated by a series of linking arm segments 23 that are connected by a series of joints 24, each joint comprising one or more independent actuators 217. Each actuator may comprise an independently-controllable motor. Each independently-controllable joint 24 can provide or represent an independent degree of freedom available to the robotic arm. In some embodiments, each of the arms 12 has seven joints, and thus provides seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arms 12 to position their respective end effectors 6a-6c at a specific position, orientation, and trajectory in space using different linkage positions and joint angles. This allows for the system to position and direct a medical instrument from a desired point in space while allowing the physician to move the arm joints into a clinically advantageous position away from the patient to create greater access, while avoiding arm collisions. Positioned at the upper end of column 14, the console 13 can provide both a user interface for receiving user input and a display screen 56 (or a dual-purpose device such as, for example, a touchscreen) to provide the physician/user with both pre-operative and intra-operative data. The robotic cart 10 can further include a handle 27, as well as one or more wheels 28.
The end effectors 6a-6c of each of the robotic arms 12 may comprise, or be configured to have coupled thereto, an instrument device manipulator (IDM; e.g., scope handle 31), which may be attached using a sterile adapter component in some instances. The combination of the end effectors 6a-6c and associated IDM, as well as any intervening mechanics or couplings (e.g., sterile adapter), can be referred to as a manipulator assembly. An IDM can provide power and control interfaces. For example, the interfaces can include connectors to transfer pneumatic pressure, electrical power, electrical signals, and/or optical signals from the robotic arm 12 to the IDM. The IDMs may be configured to manipulate medical instruments (e.g., surgical tools/instruments), such as the scope 40, using techniques including, for example, direct drives, harmonic drives, geared drives, belts and pulleys, magnetic drives, and the like. The robotic system 10 also may include one or more power supply interface(s) 219.
As referenced above, the system 100 can include certain control circuitry configured to perform certain of the functionality described herein, including the control circuitry 211 of the robotic system 10 and the control circuitry 251 of the control system 50. That is, the control circuitry of the systems 100, 101, 102 may be part of the robotic system 10, the control system 50, or some combination thereof. Therefore, any reference herein to control circuitry may refer to circuitry embodied in a robotic system, a control system, or any other component of a medical system, such as the medical systems 100, 101, and 102 shown in
The control circuitry 211, 251 may comprise computer-readable media storing, and/or configured to store, hard-coded and/or operational instructions corresponding to at least some of the steps and/or functions illustrated in one or more of the present figures and/or described herein. Such computer-readable media can be included in an article of manufacture in some instances. The control circuitry 211/251 may be entirely locally maintained/disposed or may be remotely located at least in part (e.g., communicatively coupled indirectly via a local area network and/or a wide area network). Any of the control circuitry 211, 251 may be configured to perform any aspect(s) of the various processes disclosed herein.
With further reference to
The control system 50 and/or robotic system 10 can include certain user controls (e.g., controls 55), which may comprise any type of user input (and/or output) devices or device interfaces, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video-game-type controllers), computer mice, trackpads, trackballs, control pads, and/or sensors (e.g., motion sensors or cameras) that capture hand gestures and finger gestures, touchscreens, and/or interfaces/connectors therefore. Such user controls are communicatively and/or physically coupled to respective control circuitry. The control system can include a structural tower 51, as well as one or more wheels 58 that support the tower 51. The control system 50 can further include certain communication interface(s) 254 and/or power supply interface(s) 259.
In some embodiments, the endoscope assembly 30 includes a handle or base 31 coupled to an endoscope shaft 40 (referred to simply as an “endoscope,” or “scope” in certain contexts herein). For example, the endoscope 40 can include an elongate shaft including one or more lights 49 and one or more cameras or other imaging devices 48. The scope 40 can further include one or more working channels 44, which may run a length of the scope 40.
The scope assembly 30 can further comprise one or more positioning markers and/or sensors 63, which may be configured to generate signals indicating a position of the marker(s)/sensor(s) 63 within an electromagnetic field. Such markers 63 may comprise, for example, one or more conductive coils (or other embodiment of an antenna), which may be disposed at a known, fixed orientation relative to one another to allow for the determination of multiple degrees of freedom with respect to position determination. The marker(s) 63 can be configured to generate and/or send sensor position data to another device and/or produce a detectable distortion or signature in an electromagnetic field. The sensor/marker position data can indicate a position and/or orientation of the medical instrument 40 (e.g., the distal end 42 thereof) and/or can be used to determine/infer a position/orientation of the medical instrument.
The scope 40 can be articulable, such as with respect to at least a distal portion 42 of the scope 40, so that the scope 40 can be steered within the human anatomy. In some embodiments, the scope 40 is configured to be articulated with, for example, six degrees of freedom, including XYZ coordinate movement, as well as pitch, yaw, and roll. Certain position sensor(s) (e.g., electromagnetic sensors) of the scope 40, where implemented, may likewise have similar degrees of freedom with respect to the positional information they generate/provide.
For robotic implementations, robotic arms of a robotic system can be configured/configurable to manipulate the scope 40. For example, an instrument device manipulator (e.g., scope handle) can be coupled to an end effector of a robot arm and can manipulate the scope 40 using elongate movement members. The elongate movement members may include one or more pull wires (e.g., pull or push wires), cables, fibers, and/or flexible shafts. For example, the robotic end effector may be configured to actuate multiple pull wires (not shown) coupled to the scope 40 to deflect the tip 42 of the scope 40.
In various embodiments, the anatomical space in which the scope 40 or other instrument may be localized (i.e., where position of the scope/instrument is determined/estimated) is a three-dimensional portion of a patient's vasculature, tracheobronchial airways, urinary tract, gastrointestinal tract, or any organ or space accessed via such lumens. Various positioning/imaging modalities may be implemented to provide images/representations of the anatomical space. Suitable imaging subsystems include, for example, X-ray, fluoroscopy, CT, PET, PET-CT, CT angiography, Cone-Beam CT, 3DRA, single-photon emission computed tomography (SPECT), MRI, Optical Coherence Tomography (OCT), and ultrasound. One or both of pre-procedural and intra-procedural images may be acquired. In some embodiments, the pre-procedural and/or intra-procedural images are acquired using a C-arm fluoroscope. In connection with some embodiments, particular positioning and imaging systems/modalities are described; it should be understood that such description may relate to any type of positioning system/modality.
In the fluoroscopy system 70, the source/emitter 75 and the detector/receiver 74 may be positioned at opposing ends of the C-arm 71 to maintain a fixed relationship in position and orientation. The source/emitter 75 can generate and the detector/receiver 74 can receive a fluoroscopy field (e.g., X-ray) 601 to record intensity of the fluoroscopic field that pass through an object 78 positioned on the bed 15. Areas of the object 78 that are denser attenuates more of the fluoroscopy field and results in lower intensity on the detector/receiver 74, and vice versa. As shown, the C-arm 71 may be axially rotated about a center of rotation along one or more axes. For example, the illustrated C-arm 71 may be rotated along the ‘X’ axis (e.g., to provide horizontally rotated images), the ‘Y’ axis (e.g., to provide vertically rotated images), or both. The object 78, which can be any semantically segmentable object, can be positioned at or near the center of rotation of the C-arm 71. Some example semantically segmentable objects include, without limitation, a scope, a medical tool (e.g., a biopsy needle, basket, forceps, etc.), and an anatomical feature.
Axially rotating the C-arm 71 to capture multiple projected images at various angles can provide a more complete two-dimensional or three-dimensional understanding of various geometrical properties of the object 78 than a single image may provide. However, capturing multiple projected images can be time and resource intensive. Accordingly, an improved technique that can efficiently and reliably derive various geometrical properties using a minimum number of projected images is desired.
In some instances, a conventional angle frequently used for a C-arm or environment setup may be selected to serve as the neutral configuration 70b. For instance, either of 0° or 90° configurations, referred as “anterior posterior” or “lateral” configurations, may be such conventional angles used as the neutral configuration 70b. The conventional angles may be clinically preferred for their ease of identification and reference.
The angular offset θ between the positive configuration 70a and neutral configuration 70b or the negative configuration 70c and neutral configuration 70b can be any angular separation as long as they are substantially the same in size. For instance, when a positive angular offset +θ is selected to be +5°, +10°, +15°, +30°, +45°, or the like, a corresponding negative angular offset −θ may be, respectively, −5°, −10°, −15°, −30°, −45°, or the like. Those two angles are to be positioned along the same rotational axis defined by the positive angle and be symmetric in relation to the neutral configuration 70b. While any angles may be used, for computational efficiency, angles that are different enough (>15°) to show some variability in projected cross-sections but not too variable to show too little similarity (<30°) are typically preferred.
It is noted that while
In some embodiments, objects appearing in the fluoroscopic images may be segmented to include some identified objects while excluding other objects or features. For instance, the fluoroscopic images of
Fluoroscopic sources, such as the source/emitter 75, emit a beam through the object at various angles. Detectors, such as the receiver 74, measure the intensity of the fluoroscopic field after they pass through the object. The data collected at each angle is represented in a sinogram, a two-dimensional array where one axis represents the angle of the beam, and the other represents the position of the detector/receivers 74.
The projected image 718a-c can reflect a first object 712a-c and a second object 714a-c, each object having different physical properties that reflect how bright the projected image 718a-c depicts the objects 712, 714. The projected images 718a-c can be appended as slices to another projected image at a neighboring angle to gradually populate the sinogram 720a-c. It is noted that the region of interest 710 is a two-dimensional area and that the projected images 718 is a one-dimensional line. However, the general idea described here in relation to generation of a sinogram can be expanded to a three-dimensional space of interest and a two-dimensional projected image.
Traditionally, the imaging field 716 is applied at a fixed angular interval (e.g., 0.5° interval, 1° interval, 2° interval, etc.) to project images 718 at various angles starting from 0° toward 180° and combine each of the projected images 718 for a complete sinogram.
If a reconstruction of a cross-section of an object is desired, the complete sinogram can be filtered and back-projected to generate the reconstruction. The reconstruction is equivalent of generating a cross-section of the region of interest 710 for a given angle from the completed sinogram 720. One common method for reconstructing cross-sections from sinograms is filtered back projection (FBP). The process involves applying a transform, inverse or otherwise, to the sinogram, applying a filter in the frequency domain to enhance certain features or suppress artifacts, and then back-projecting the filtered data to obtain a two-dimensional image. In addition to FBP, there are other reconstruction algorithms, such as iterative methods. Iterative reconstruction algorithms can iteratively refine an initial estimate of the image to fit the measured sinogram data. Examples include the Maximum Likelihood Expectation Maximization (MLEM) algorithm and the Algebraic Reconstruction Technique (ART).
In some embodiments, it is contemplated that a first cross-section of the first object 712 may be separately reconstructed from a second cross-section of the second object 714 which may also be separately reconstructed for a given angle. Further, the first cross-section and the second cross-section may be merged, overlayed, or otherwise combined to provide a third cross-section depicting both the first object 712 and the second object 714. In contrast with the traditional method, the techniques presented herein can efficiently remove the need to project the object at every angle. That is, unlike the traditional method of generating the sinogram, the techniques may rely on a select few (e.g., three or more) projected images and, from the projected images, simulate a sinogram that is complete for all angles.
At block 802, the process 800 may involve acquiring or otherwise accessing at least three projected images. In some embodiments, the projected images can be the fluoroscopic images in
At block 804, centers of masses in the fluoroscopic images may be determined. In each of the fluoroscopic images, the center of mass (xc) in the can be computed based on the following formula:
where Δp is the pixel spacing along the profile (e.g., an axis in the projected image, such as an axis along the linear projected images 718a-c of
At block 806, having computed the center of masses for the projected images, the second moments of area (Ix) for planes orthogonal to a given projected image can be determined. In some instances, Ix can be computed based on the following formula:
The present disclosure relies on at least three images that are projected from three-dimensional space (3D) onto a two-dimensional plane (2D). A major limitation of using a single 2D projection is that an object cross-sections, such as the scope cross-sections, may not be rotationally symmetric and, hence, moments of inertia (Ix) may vary with the angle of projection about the object axis (e.g., scope axis). The operational concept of the mathematical reconstruction is based on the fact that the principal moments of inertia (Imax, Imin) of an object can be completely determined with only three angularly displaced projections. That is, any two orthogonal projections (Ix, Iy) can yield a constant defined as the polar moment of inertia (Ip) that is equivalent to the sum of the principal moments of inertia (Imax, Imin). In other words, the polar moment of inertia (Ip), which remains constant, is equal to any sum of the two orthogonal moments of inertia (Ix, Iy) and the sum of the principal moments of inertia (Imax, Imin). This relationship can be summarized with the following formula:
At block 808, the process 800 may involve determining the moment of inertia orthogonal to Ix, the orthogonal moment of inertia referenced as Iy. As described in relation to
where Ixy is the product of inertia, and Iy is the moment of inertia orthogonal to Ix. Computing Iθ+ and Iθ− individually using the Equation (4) for θ+ and θ−, summing Iθ+ and Iθz, and finally solving for Iy provides:
Accordingly, Iy can be determined based on the two additional projected images, the known Ix from Equation (2), and Equation (4).
At block 810, the process 800 may involve determining the product of inertia (Ixy). Using either of Iθ+ or Iθ− to solve for Ixy with Equation (4) provides:
It is noted that the two orthogonal projections (Ix, Iy) are known. Angle (θ) and the moment of inertia (Iθ) at the angle (θ) is known. Thus, Ixy can be determined by substituting the known values into Equation (6).
At block 812, the process 800 may involve determining the principal moments of inertia (Imax, Imin). In three-dimensional space, for any arbitrary shape assuming uniform mass distribution, there are always three perpendicular axes through any point, usually the center of mass for convenience, for which the moments of inertia are maximized or minimized. These axes are known as the principal axes. The moments of inertia about these principal axes are referred to as the principal moments of inertia which are, here, Imax and Imin. The principal moments of inertia, Imax and Imin, can be determined based on the previously determined moments of inertia (Ix, Iy, and Iθ) as follows:
At block 814, the process 800 may involve determining the principal angle (φ) of Imax to the reference coordinate system. When it is assumed that Ix and Iy are about the reference coordinate system, φ can be computed by:
At block 816, the process 800 may involve generating a sinogram. In some embodiments, the principal moments of inertia (Imax, Imax) and the product of inertia (Ixy) can be used to generate the sinogram. Specifically, Imax, Imax, and Ixy can be used to compute second moments of area for all angles that were not measured (e.g., not involved with the projected images) until a complete ensemble of projections can be produced at a fixed interval (e.g., a 1° interval) for projections from 0° toward 180°. The complete ensemble can be the sinogram of a particular object (e.g., only a scope, only a nodule, etc.) for all projection angles φ from 0° toward 180°.
Complete data will consist of an object thickness distribution, such as the scope thickness distribution having uniform density, at each angle. That is, the thickness distribution may be generated based on an assumption that the object has a uniform density profile. The object thickness distribution may be characterized by total area (A, generally fixed for all projections of the same imaging device), center of mass position (cφ), and the second moment of area at angle φ (Iφ). From the math above, everything is known about each projection except the actual distribution. Subsequent procedures will begin with generating or simulating a distribution for all of the missing projections corresponding to the known A, Iφ, and cφ.
The generated scope thickness profiles are first fitted to a function with a sufficient number of vertices so that fit metric may provide a substantial fit, such as the least squares fit of at least R2≥99% or other sufficient fit value, while retaining the original A, Iφ, and cφ to an acceptable error. The function can include a polynomial function, a spline function, or the like. It is contemplated that non-uniform rational basis spline (NURBS) is a model using basis splines (B-splines) that can be utilized as a fitting function for the purpose, although various other functions are contemplated.
The generation of the distribution can begin from the projected image at the neutral configuration, which is the image with known (e.g., measured) properties at angle φ=0°. Beginning with the fitted distribution of φ=0°, vertices of the fitting function can be iteratively adjusted for the simulated projected image at angle φ=1° while retaining the original A and cφ until Iφ satisfies a condition associated with a threshold value, such as rising above, falling below, or reaching the threshold value. Projected images at φ=2° can be simulated with the similar iterative adjustment beginning with the fitted distribution of φ=1°, and so forth. Similar process can be repeated for each missing projected images at other angles and also, toward negative angles, until the sinogram is complete.
Thus, in contrast with the traditional method, the techniques of the present disclosure can utilize Equations (1)-(9) with three projected images to simulate other projected images and complete a sinogram. For example,
It is noted that the example sinogram 900 and the reconstructed cross-section 950 of the object 952 are of a bone but, as described previously, the object 952 can be any semantically segmentable object including without limitation, a scope, a scope tip or any portion of the scope, a nodule tissue, or any other anatomical features. The reconstruction 950 showing the object 952 at a zero angle confirms that the reconstruction 950 closely matches the actual projected image of the object 952 at zero angle (e.g., a neutral configuration 70b of
In some embodiments, the reconstruction 950 can involve filtering and back-projection performed on the simulated sinogram 900. For example,
The reconstructed cross-sectional images can include a mathematical reconstruction of the basis objects/materials (e.g., a scope vs. a nodule) in the images and the images can be used to derive various geometrical properties of the objects and materials within the image(s). The derived properties, in turn, can serve to initialize object semantic segmentation and indicate to a user viewing the projected and simulated images a more accurate identification of the objects. For example, derived geometrical properties including principal moments of inertia, second moments of area, an object's center of mass (which may be close to a centroid of the object in instances where the object is homogeneous), axis of rotation, or other properties may be displayed to a user in association with the image and help identify objects and their properties. As another example, some derived properties such as a center of mass of a scope tip can help estimate pose of the scope tip in the image.
The presentation 1000 can include at least one reconstructed cross-section. The reconstructed cross-section can be displayed on a computer monitor or printed for analysis. In some implementations, modern medical imaging systems may provide detailed three-dimensional reconstructions as well.
The reconstructed object can be a scope 1002 or any portions thereof (e.g., a scope tip, a sheath tip, a biopsy needle, basket, forceps, etc.) as well as various anatomical feature(s) 1004 (e.g., a nodule, a lesion, a tissue, etc.). Additionally, the presentation 1000 can include an indicator referencing the reconstructed object and present the indicator in association with the object. For example, the indicator can be a location indicator referencing the center of mass or locations of other physical properties regarding the object. As another example, the indicator can reference one or more principal axes and provide a navigation software with valuable information regarding orientation of the object, such as the scope orientation. In some embodiments, the accurate reconstruction of the objects can help segmentation and detection of one or more objects.
At block 1102, the process 1100 may involve accessing at least three projected images. As described, the projected images can be axially rotated images generated by a fluoroscopic imaging device attached to a C-arm at known, symmetric, angular offset. For instance, the projected images can be the cross-sections shown in
At block 1104, the process 1100 may involve determining at least one principal moment of inertia using the three projected images. In some implementations, an object of interest may be left included while other objects or features may be excluded from the three projected images. Specifically, the principal moments of inertia (Imax, Imin) can be determined from the three projected images based on the rotational axis theorem.
At block 1106, the process 1100 may involve generating a sinogram. Here, for remaining angles other than the three angles associated with the three projected images, projected images of the remaining angles can be simulated based on the at least one principal moment of inertia. In some implementations, the projected images can be simulated at fixed intervals for 0° toward 180° to complete the sinogram such that the completed sinogram may consist of a thickness distribution of an object at each angle. In some implementations, simulating the projected images for each adjacent angle may involve iteratively adjusting a number of vertices of a function fitted to a projected image of the object projected at a previous angle. The projected image at the previous angle may be based on an original projected image or a simulated projected image.
At block 1108, the process 1100 may involve reconstructing an object as seen from a specific angle based on the generated sinogram. In some implementations, the reconstruction may involve some post-processing steps. As an example of a post-processing step, the reconstruction may involve an inverse transformation. The actual reconstruction process can be quite complex based on various post-processing algorithms and steps employed, the specific imaging technique, and equipment used. The choice of reconstruction method can affect image quality, spatial resolution, and the ability to distinguish different structures within the imaged object.
At block 1110, the process 1100 may involve deriving a geometrical property from the reconstructed object. Some example derivable geometrical property includes a centroid of the object, moments of inertia, second moment of area, or the like. In some implementations, the object can be annotated in the original projected image or the simulated projected image based on the derived geometrical property.
Optionally, the process 1100 may involve determining a three-dimensional position of the object, for example, the distal tip of a scope or a nodule centroid. The three-dimensional position of the object may later be used as an initialization for pose estimation algorithms. When determining such three-dimensional position of the object, the process 1100 may involve performance of some additional blocks.
At block 1112, the process 1100 may involve performing triangulation. Triangulation is a common method of projecting points from a two-dimensional plane to a three-dimensional space using, in some implementations, point-of-capture parameters. In fluoroscopic applications, the point-of-capture parameters may be determined based on a C-arm position or the fluoroscopic imaging device attached to the C-arm. Accordingly, such parameters determined during installation and/or calibration may be used to perform triangulation.
At block 1114, the process 1100 may involve initializing pose estimation based on the triangulation. For example, a three-dimensional position of the object may be determined based on the triangulation and can serve as an initial estimated pose of the object. The initial estimated pose may be presented to a user on a display in association with a projected image.
The controller 1200 includes a communication interface 1210, a processing system 1220, and a memory 1230. The communication interface 1210 is configured to communicate with one or more components of the medical system. For example, the communication interface 1210 may include an image source interface 1212 for communicating with one or more image sources (such as the fluoroscopy imaging system 70 of
The memory 1230 includes a non-transitory computer-readable medium (including one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, or a hard drive, among other examples) that stores the following software (SW) modules: an object detection SW module 1231 to detect an object in each image of the plurality of images; a moment of inertia SW module 1232 to determine a moment of inertia associated with the object based on the plurality of images; a sinogram simulation SW module 1233 to simulate one or more images for the sinogram based at least in part on the moment of inertia associated with the object; an object reconstruction SW module 1234 to generate a reconstruction of the object based on the sinogram; and an object analysis SW module 1235 to determine one or more properties of the object based on the reconstruction. Each of the software modules 1231-1235 includes instructions that, when executed by the processing system 1220, causes the controller 1200 to perform the corresponding functions.
The processing system 1220 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the controller 1200 (such as in the memory 1230). For example, the processing system 1220 may execute the object detection SW module 1231 to detect an object in each image of the plurality of images. The processing system 1220 also may execute the moment of inertia SW module 1232 to determine a moment of inertia associated with the object based on the plurality of images. The processing system 1220 may execute the sinogram simulation SW module 1233 to simulate one or more images for the sinogram based at least in part on the moment of inertia associated with the object. The processing system 1220 may further execute the object reconstruction SW module 1234 to generate a reconstruction of the object based on the sinogram. Still further, the processing system 1220 may execute the object analysis SW module 1235 to determine one or more properties of the object based on the reconstruction.
The controller receives a plurality of images associated with a sinogram, where the plurality of images is captured by an imaging device at a plurality of angles, respectively, along a rotational axis of the imaging device (1302). In some implementations, the imaging device may include a fluoroscopy system. The controller detects an object in each image of the plurality of images (1304). In some implementations, the object may include an instrument, configured to be driven through a luminal network, or an anatomical feature. The controller further determines a moment of inertia associated with the object based on the plurality of images (1306).
The controller simulates one or more images for the sinogram based at least in part on the moment of inertia associated with the object (1308). The controller generates a reconstruction of the object based on the sinogram (1310). In some implementations, the reconstruction may include a cross-section of the object. The controller further determines one or more properties of the object based on the reconstruction (1312). In some implementations, the controller may further present a location indicator referencing the object on a display.
In some implementations, the plurality of images may include a first image captured at a first angle of the plurality of angles, a second image captured at a second angle of the plurality of angles, and a third image captured at a third angle of the plurality of angles, where the first angle and the second angle are separated by the same degree of angular rotation as between the second angle and the third angle. In some implementations, adjacent angles of the plurality of angles may be separated by a fixed interval.
In some aspects, the sinogram may include a thickness distribution for the object over the plurality of angles. In some implementations, the controller may further determine a fitting function, having a plurality of vertices, that fits a known thickness profile to the thickness distribution for the object. In some implementations, the simulating of each image of the one or more images for the sinogram may include iteratively adjusting the plurality of vertices of the fitting function until a second moment of area at the angle associated with the simulated image satisfies a threshold condition.
In some implementations, the controller may further perform a semantic segmentation operation based at least in part on the one or more properties of the object determined based on the reconstruction. In some implementations, the controller may further determine an axis associated with the moment of inertia, determine a second moment of area associated with an axis orthogonal to the axis associated with the moment of inertia, and determine a principal angle associated with the object based on the moment of inertia and the second moment of area.
Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein can be performed in a different sequence, may be added, merged, or left out altogether. Thus, in certain embodiments, not all described acts or events are necessary for the practice of the processes.
Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is intended in its ordinary sense and is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous, are used in their ordinary sense, and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Conjunctive language such as the phrase “at least one of X, Y and Z,” unless specifically stated otherwise, is understood with the context as used in general to convey that an item, term, element, etc. may be either X, Y or Z. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y and at least one of Z to each be present.
It should be appreciated that in the above description of embodiments, various features are sometimes grouped together in a single embodiment, Figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that any claim require more features than are expressly recited in that claim. Moreover, any components, features, or steps illustrated and/or described in a particular embodiment herein can be applied to or used with any other embodiment(s). Further, no component, feature, step, or group of components, features, or steps are necessary or indispensable for each embodiment. Thus, it is intended that the scope of the inventions herein disclosed and claimed below should not be limited by the particular embodiments described above, but should be determined only by a fair reading of the claims that follow.
It should be understood that certain ordinal terms (e.g., “first” or “second”) may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not necessarily indicate priority or order of the element with respect to any other element, but rather may generally distinguish the element from another element having a similar or identical name (but for use of the ordinal term). In addition, as used herein, indefinite articles (“a” and “an”) may indicate “one or more” rather than “one.” Further, an operation performed “based on” a condition or event may also be performed based on one or more other conditions or events not explicitly recited.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example embodiments belong. It be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The spatially relative terms “outer,” “inner,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms, may be used herein for ease of description to describe the relations between one element or component and another element or component as illustrated in the drawings. It be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation, in addition to the orientation depicted in the drawings. For example, in the case where a device shown in the drawing is turned over, the device positioned “below” or “beneath” another device may be placed “above” another device. Accordingly, the illustrative term “below” may include both the lower and upper positions. The device may also be oriented in the other direction, and thus the spatially relative terms may be interpreted differently depending on the orientations.
Unless otherwise expressly stated, comparative and/or quantitative terms, such as “less,” “more,” “greater,” and the like, are intended to encompass the concepts of equality. For example, “less” can mean not only “less” in the strictest mathematical sense, but also, “less than or equal to.”
This application claims priority and benefit under 35 U.S.C. § 119 (e) to U.S. Provisional Patent Application No. 63/609,452, filed Dec. 13, 2023, which is incorporated herein by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63609452 | Dec 2023 | US |
