Patent Application
20240081773
Aspects of the present disclosure relate to systems, devices and methods for performing a surgical step or surgical procedure with visual guidance using one or more head mounted displays and with display of one or more imaging studies. Aspects of the present disclosure relate to systems, devices and methods for operating one or more head mounted displays with real-time wireless display of tracking information of tools or instruments registered with the patient's anatomy. Aspects of the present disclosure relate to systems, devices and methods for operating an imaging system with augmented reality display of an image acquisition area or volume prior to image acquisition.
With computer assisted surgery, e.g. surgical navigation or robotics, pre-operative and/or intra-operative imaging studies of the patient can be used. The imaging studies can be displayed in the operating room on an external computer monitor and the patient's anatomy, e.g. landmarks, can be registered in relationship to the information displayed on the monitor. Since the surgical field is in a different location and has a different view coordinate system for the surgeon's eyes than the external computer monitor, hand-eye coordination can be challenging for the surgeon. Image acquisition for pre-operative and/or intra-operative imaging studies frequently requires the acquisition of imaging data prior to acquiring the definitive imaging studies, for example used for diagnostic purposes or image guidance purposes.
Aspects of the disclosure relate to a system comprising at least one head mounted display, a robot, wherein the robot comprises an end effector, a first computing system comprising one or more computer processors, and a second computing system comprising one or more computer processors, wherein the first computing system is in communication with the robot, wherein the second computing system is in communication with the at least one head mounted display, wherein the second computing system is configured to display, by the at least one head mounted display, a virtual user interface comprising at least one virtual object, wherein the second computing system is configured to generate a command based at least in part on at least one interaction with the at least one virtual object displayed in the virtual user interface, wherein the second computing system is configured to transmit the command to the first computing system using wireless transmission, wherein the command is configured to cause the first computing system to control the robot for movement, activation, operation, de-activation, or any combination thereof, of a robot component, a robot motor, a robot actuator, a robot drive, a robot controller, a robot hydraulic system, a robot piezoelectric system, a robot switch, the end effector, or any combination thereof.
In some embodiments, the command is configured to control the end effector within a predetermined operating boundary, a predetermined operating range, a predetermined operating zone, or a predetermined operating volume.
In some embodiments, the first computing system is connected to the robot by wire, or wherein the first computing system is connected to the robot by wireless connection.
In some embodiments, the second computing system is connected to the at least one head mounted display by wire, or wherein the second computing system is connected to the at least one head mounted display by wireless connection.
In some embodiments, the second computing system is configured to display, by the at least one head mounted display, a representation of a predetermined operating boundary, a predetermined operating range, a predetermined operating zone, or a predetermined operating volume of the end effector or an expected outcome following the movement, activation, operation, de-activation or a combination thereof of the robot component, robot motor, robot actuator, robot drive, robot controller, robot hydraulic system, robot piezoelectric system, robot switch, the end effector, or any combination thereof.
In some embodiments, the end effector comprises a physical surgical tool or a physical surgical instrument.
In some embodiments, the first computing system is configured to obtain real-time tracking information of a component of the robot, the end effector, a target object, a target anatomic structure of a patient, the at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof.
In some embodiments, the second computing system is configured to obtain real-time tracking information of a component of the robot, the end effector, a target object, a target anatomic structure of a patient, the at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof.
In some embodiments, the first computing system is configured to obtain real-time tracking information of a physical tool, a physical instrument, or any combination thereof coupled to the robot.
In some embodiments, the second computing system is configured to obtain real-time tracking information of a physical tool, a physical instrument, or any combination thereof coupled to the robot. In some embodiments, the first computing system is configured to wirelessly transmit the real-time tracking information of the component of the robot, the end effector, a target object, a target anatomic structure of a patient, the at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof. In some embodiments, the second computing system is configured to wirelessly transmit the real-time tracking information of the component of the robot, the end effector, a target object, a target anatomic structure of a patient, the at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof.
In some embodiments, the second computing system is configured for displaying, by the at least one head mounted display, a 3D stereoscopic view. In some embodiments, the 3D stereoscopic view is superimposed onto an anatomic structure of a patient. In some embodiments, the 3D stereoscopic view comprises a predetermined trajectory of the end effector, a representation of a predetermined operating boundary of the end effector, a representation of a predetermined operating range of the end effector, a representation of a predetermined operating zone of the end effector, a representation of a predetermined operating volume of the end effector, or a combination thereof. In some embodiments, the 3D stereoscopic view comprises a predetermined trajectory of the end effector, a representation of a predetermined operating boundary of the end effector, a representation of a predetermined operating range of the end effector, a representation of a predetermined operating zone of the end effector, a representation of a predetermined operating volume of the end effector or a combination thereof following the movement, activation, operation, de-activation or combination thereof of the robot component, robot motor, robot actuator, robot drive, robot controller, robot hydraulic system, robot piezoelectric system, robot switch, the end effector or any combination thereof.
In some embodiments, the first computing system, the second computing systems or both are configured to turn on or turn off the display of the virtual user interface. In some embodiments, the 3D stereoscopic view comprises a predetermined trajectory of the end effector, a representation of a predetermined operating boundary of the end effector, a representation of a predetermined operating range of the end effector, a representation of a predetermined operating zone of the end effector, a representation of a predetermined operating volume of the end effector or a combination thereof prior to executing the command.
In some embodiments, the wireless transmission comprises a Bluetooth signal, WiFi signal, LiFi signal, a radiofrequency signal, a microwave signal, an ultrasound signal, an infrared signal, an electromagnetic wave or any combination thereof.
In some embodiments, the system comprises two or more head mounted displays, wherein the wireless transmission is a multicast, broadcast transmission or any combination thereof.
In some embodiments, the at least one virtual object comprises one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof.
In some embodiments, the interaction is a collision detection between a physical object and the at least one virtual object. In some embodiments, the interaction is a collision detection between a user's finger and the at least one virtual object. In some embodiments, the interaction is a collision detection between a tracked pointer, tracked tool, tracked instrument, or a combination thereof and the at least one virtual object.
In some embodiments, the interaction with the at least one virtual object comprises a gaze tracking.
Aspects of the disclosure relate to a system comprising at least one head mounted display, a robot, wherein the robot comprises an end effector, a first computing system comprising one or more computer processors, wherein the first computing system is in communication with the robot, a second computing system comprising one or more computer processors, wherein the second computing system is in communication with the at least one head mounted display, wherein the second computing system is configured to display, by the at least one head mounted display, a virtual user interface comprising at least one virtual object, wherein the second computing system is configured to generate an event message based at least in part on at least one interaction with the at least one virtual object displayed in the virtual user interface, wherein the second computing system is configured to transmit the event message to the first computing system using wireless transmission, wherein the second computing system is configured to generate a command based on the event message, and wherein the command is configured to cause the first computing system to control the robot for movement, activation, operation, de-activation, or any combination thereof, of a robot component, a robot motor, a robot actuator, a robot drive, a robot controller, a robot hydraulic system, a robot piezoelectric system, a robot switch, the end effector, or any combination thereof.
In some embodiments, the end effector comprises a scalpel, a saw, a cutting tool, a wire, a needle, a pin, a drill, a burr, a mill, a reamer, an impactor, a broach, a laser, a radiofrequency device, a thermocoagulation device, a cryoablation device, a radioactive probe, a radioactivity emitting device, a pulsed energy emitting device, an ultrasonic energy emitting device, a microwave energy emitting device or a combination thereof.
In some embodiments, the command comprises a subcommand, wherein the subcommand is configured to execute an accept or cancel function of the command.
Aspects of the disclosure relate to system, comprising: (a) at least one head mounted display, at least one camera or scanning device, wherein the at least one camera or scanning device is configured to track real-time information of the at least one head mounted display, of at least one anatomic structure of a patient, and of at least one physical surgical tool or physical surgical instrument, (b) a first computing system comprising one or more computer processors, wherein the first computing system is configured to obtain the real-time tracking information of the at least one head mounted display, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, wherein the first computing system is configured for wireless transmission of the real-time tracking information of the at least one head mounted display, the at least one anatomic structure of the patient, and the at least one physical surgical tool or physical surgical instrument, (c) a second computing system comprising one or more computer processors, wherein the second computing system is configured for wireless reception of the real-time tracking information of the at least one head mounted display, the at least one anatomic structure of the patient, and the at least one physical surgical tool or physical surgical instrument, wherein the second computing system is configured to generate a 3D stereoscopic view, wherein the stereoscopic view comprises a 3D representation of the at least one tracked physical surgical tool or physical surgical instrument, and wherein the at least one head mounted display is configured to display the 3D stereoscopic view.
In some embodiments, the one or more computer processors of the second computing system generate the 3D stereoscopic view for a view angle of the head mounted display relative to the at least one anatomic structure of the patient using the real-time tracking information of the at least one head mounted display.
In some embodiments, the real-time tracking information comprises tracking information of multiple head mounted displays. In some embodiments, the real-time tracking information comprises a head mounted display specific label or tag for each head mounted display, or wherein the real-time tracking information is labeled for each tracked head mounted display. In some embodiments, the wireless transmission is a multicast or broadcast transmission to the multiple head mounted displays.
In some embodiments, the real-time tracking information comprises tracking information of two or more head mounted displays. In some embodiments, the two or more head mounted displays are located in different locations. In some embodiments, the real-time tracking information comprises a head mounted display label for each head mounted display, wherein each heard mounted display has a different label. In some embodiments, the real-time tracking information is labeled for each tracked head mounted display.
In some embodiments, the one or more computer processors of the second computing system generate the 3D stereoscopic view for an interpupillary distance adjusted for a user wearing the head mounted display.
In some embodiments, the second computing system is communicatively coupled to the at least one head mounted display.
In some embodiments, the second computing system is integrated with the at least one head mounted display.
In some embodiments, the second computing system is separate from the at least one head mounted display and is connected to a display unit of the at least one head mounted display using at least one cable.
In some embodiments, the the wireless transmission, the wireless reception, or both comprise a WiFi signal, a LiFi signal, a Bluetooth signal or a combination thereof.
In some embodiments, the camera or scanning device is separate from the at least one head mounted display.
In some embodiments, the camera or scanning device is integrated or attached to the at least one head mounted display.
In some embodiments, the wireless transmission comprises sending data packets comprising the real-time tracking information of the at least one head mounted display, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, at a rate of 20 Hz or greater.
In some embodiments, the wireless reception comprises receiving data packets comprising the real-time tracking information of the at least one head mounted display, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, at a rate of 20 Hz or greater.
In some embodiments, the system further comprises a third computing system, wherein the third computing system is configured for wireless reception of the real-time tracking information from the first computing system and wherein the third computing system is configured for wireless transmission of the real-time tracking information to the second computing system. In some embodiments, the third computing system comprises a chain of computing systems configured for wireless reception and wireless transmission of the real-time tracking information.
In some embodiments, the system further comprises a third computing system, wherein the third computing system is communicatively coupled to a second head mounted display, wherein the third computing system is configured for wireless reception of the real-time tracking information of the second head mounted display, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, wherein the third computing system is configured to generate a 3D stereoscopic view by the second head mounted display using the tracking information of the second head mounted display.
In some embodiments, the tracking information of the second head mounted comprises a label specific to the second head mounted display for identifying the tracking information of the second head mounted display by the third computing system.
In some embodiments, the system further comprises a fourth computing system, wherein the fourth computing system is communicatively coupled to a third head mounted display, wherein the fourth computing system is configured for wireless reception of the real-time tracking information of the third head mounted display, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, wherein the fourth computing system is configured to generate a 3D stereoscopic view by the third head mounted display using the tracking information of the third head mounted display.
In some embodiments, the tracking information of the third head mounted comprises a label specific to the third head mounted display for identifying the tracking information of the third head mounted display by the fourth computing system.
In some embodiments, the system further comprises a fifth computing system, wherein the fifth computing system is communicatively coupled to a fourth head mounted display, wherein the fifth computing system is configured for wireless reception of the real-time tracking information of the fourth head mounted display, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, wherein the fifth computing system is configured to generate a 3D stereoscopic view by the fourth head mounted display using the tracking information of the fourth head mounted display.
In some embodiments, the tracking information of the fourth head mounted comprises a label specific to the fourth head mounted display for identifying the tracking information of the fourth head mounted display by the fifth computing system.
In some embodiments, the real-time tracking information comprises one or more coordinates. In some embodiments, the one or more coordinates comprise coordinates of the at least one anatomic structure of the patient. In some embodiments, the one or more coordinates comprise coordinates of the at least one physical surgical tool or physical surgical instrument. In some embodiments, the one or more coordinates comprise coordinates of the at least one head mounted display.
In some embodiments, the at least one head mounted display comprises at least one optical see-through head mounted display.
In some embodiments, the at least one head mounted display comprises at least one video see-through head mounted display.
In some embodiments, the at least one camera or scanning device comprises a laser scanner, a time-of-flight 3D laser scanner, a structured-light 3D scanner, a hand-held laser scanner, a LIDAR scanner, a time-of-flight camera, a depth camera, a video system, a stereoscopic camera system, a camera array, or a combination thereof.
In some embodiments, the system comprises at least one inertial measurement unit. In some embodiments, the at least one inertial measurement unit is integrated or attached to the at least one physical surgical tool or physical surgical instrument. In some embodiments, the at least one inertial measurement unit is integrated or attached to the at least one anatomic structure of the patient. In some embodiments, the at least one inertial measurement unit is integrated or attached to the at least one head mounted display. In some embodiments, the real-time tracking information of the at least one head mounted display comprises information from the at least one inertial measurement unit.
Aspects of the disclosure relate to a system, comprising (a) two or more head mounted displays, (b) at least one camera or scanning device, wherein the at least one camera or scanning device is configured to track real-time information of the two or more head mounted displays, of at least one anatomic structure of a patient, and of at least one physical surgical tool or physical surgical instrument, (c) a first computing system comprising one or more computer processors, wherein the first computing system is configured to obtain real-time tracking information of at least one anatomic structure of a patient, of at least one physical surgical tool or physical surgical instrument, and of the two or more head mounted displays, wherein the tracking information of the two or more head mounted displays is labeled for each of the two or more head mounted displays, wherein the first computing system is configured for wireless transmission of the real-time tracking information of the at least one anatomic structure of the patient, the tracking information of the at least one physical surgical tool or physical surgical instrument, and the labeled tracking information of the two or more head mounted displays, (d) a second computing system, wherein the second computing system is configured for wireless reception of the real-time tracking information of the at least one anatomic structure of the patient, the tracking information of the at least one physical surgical tool or physical surgical instrument, and the labeled tracking information of the first of the two or more head mounted displays, wherein the second computing system is configured to generate a first 3D stereoscopic display specific for a first viewing perspective of the first head mounted display using the labeled tracking information of the first head mounted display, wherein the first head mounted display is configured to display the 3D stereoscopic display, (e) a third computing system, wherein the third computing system is configured for wireless reception of the real-time tracking information of the at least one anatomic structure of the patient, the tracking information of the at least one physical surgical tool or physical surgical instrument, and the labeled tracking information of the second of the two or more head mounted displays, wherein the third computing system is configured to generate a second 3D stereoscopic display specific for a second viewing perspective of the second head mounted display using the labeled tracking information of the second head mounted display, wherein the first and second stereoscopic displays comprise a 3D representation of the at least one physical surgical tool or physical surgical instrument.
In some embodiments, the second computing system is communicatively coupled to a first of the two or more head mounted displays, and wherein the third computing system is communicatively coupled to a second of the two or more head mounted displays.
Aspects of the disclosure relate to a system of preparing an imaging data acquisition associated with a patient comprising at least one computer processor, an augmented reality display device, an imaging system, wherein the at least one computer processor is configured to obtain real-time tracking information of one or more components of the imaging system, wherein the at least one computer processor is configured to generate a 3D representation of a surface, a volume or combination thereof, wherein the 3D representation of the surface, volume or combination thereof is at least in part derived from information about a geometry of the one or more components of the imaging system, information about a geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof, wherein the at least one computer processor is configured to generate an augmented view, the augmented view comprising the 3D representation of the surface, volume or combination thereof, wherein the at least one computer processor is configured to display, by the augmented reality display device, the augmented view at a defined position and orientation relative to the one or more components of the imaging system, and wherein the position and orientation of the augmented view is updated based on the real time tracking information of the one or more components of the imaging system.
In some embodiments, the 3D representation of the surface, volume or combination thereof does not contain imaging data from a patient.
In some embodiments, the imaging system is configured to acquire 2D, 3D, or 2D and 3D imaging data of the patient within the 3D representation of the surface, volume or combination thereof.
In some embodiments, the at least one computer processor is configured to generate the 3D representation of the surface, volume or combination thereof before acquisition of 2D, 3D, or 2D and 3D imaging data of the patient, or wherein the at least one computer processor is configured to display the 3D representation of the surface, volume or combination thereof before acquisition of 2D, 3D, or 2D and 3D imaging data of the patient.
In some embodiments, the surface, volume or combination thereof comprises information about a limit, an edge, a margin, a boundary, a circumference, a perimeter, an envelope or a combination thereof of a 2D, 3D, or 2D and 3D imaging data acquisition.
In some embodiments, the at least one computer processor is configured to generate the surface, volume or combination thereof at least in part from information about a geometry of the imaging system, information about a geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof.
In some embodiments, the system is configured to facilitate determining a desired position and orientation of the augmented view, wherein the desired position and orientation comprises a target anatomic structure of the patient.
In some embodiments, the at least one computer processor is configured to adjust the augmented view responsive to movement of the one or more tracked components of the imaging system, wherein the adjustment is configured to maintain the augmented view at the defined position and orientation relative to the one or more components of the imaging system.
In some embodiments, the information about the geometry of the imaging system, information about the geometry of the image acquisition, information about one or more image acquisition parameter, or a combination thereof comprises information about one or more imaging system components, a geometric relationship between one or more imaging system components, a collimator, a grid, an image intensifier, a detector resolution, an x-ray source, an x-ray tube setting, a kVp setting, an mA setting, an mAs setting, a collimation, a tube—detector distance, a tube—patient distance, patient—detector distance, a patient—image intensifier distance, a table height relative to a tube, a detector, a table position relative to a tube, a detector, or combination thereof, a patient position, a C-arm position, orientation, or combination thereof, a gantry position, orientation or combination thereof, a grid height, a grid width, a grid ratio, a field of view, a center of a field of view, a periphery of a field of view, a matrix, a pixel size, a voxel size, an image size, an image volume, an imaging plane, an image dimension in x, y, z and/or oblique direction, an image location, an image volume location, a scan coverage, a pitch, an in-plane resolution, a slice thickness, an increment, a detector configuration, a detector resolution, a detector density, a tube current, a tube potential, a reconstruction algorithm, a scan range, a scan boundary, a scan limit, a rotational axis of the imaging system, a rotational center of the imaging system, a reconstructed slice thickness, a segmentation algorithm, a window, a level, a brightness, a contrast, a display resolution, or a combination thereof.
In some embodiments, the imaging system comprises an x-ray system, a fluoroscopy system, a C-arm, a 3D C-arm, a digital tomosynthesis imaging system, an angiography system, a bi-planar angiography system, a 3D angiography system, a CT scanner, an MRI scanner, a PET scanner, a SPECT scanner, a nuclear scintigraphy system, a 2D ultrasound imaging system, a 3D ultrasound imaging system, or a combination thereof.
In some embodiments, the at least one computer processor is configured to obtain real-time tracking information of the augmented reality display device, an anatomic structure of the patient, a patient table used with the imaging system, the imaging system, the one or more components of the imaging system, or a combination thereof.
In some embodiments, the system further comprises a camera or scanner configured to acquire the real-time tracking information of the augmented reality display device, the anatomic structure of the patient, the patient table used with the imaging system, the imaging system, the one or more components of the imaging system, or a combination thereof. In some embodiments, the camera or scanner comprises a navigation system, a 3D scanner, a LIDAR system, a depth sensor, an IMU or a combination thereof. In some embodiments, the real-time tracking information comprises coordinate information of the augmented reality display device, the anatomic structure of the patient, the patient table used with the imaging system, the imaging system, the one or more components of the imaging system, or a combination thereof.
In some embodiments, the real-time tracking information comprises location information of the augmented reality display device, the anatomic structure of the patient, the patient table used with the imaging system, the imaging system, one or more components of the imaging system components, or a combination thereof. In some embodiments, the camera or scanner comprises a laser scanner, time-of-flight 3D scanner, structured-light 3D scanner, hand-held laser scanner, a time-of-flight camera or a combination thereof.
In some embodiments, the system is configured to obtain real-time tracking information of the imaging system using intrinsic information from the imaging system, wherein the intrinsic information comprises pose data, sensor data, camera data, 3D scanner data, controller data, drive data, actuator data, end effector data, data from one or more potentiometers, data from one or more video systems, data from one or more LIDAR systems, data from one or more depth sensors, data from one or more inertial measurement units, data from one or more accelerometers, data from one or more magnetometers, data from one or more gyroscopes, data from one or more force sensors, data from one or more pressure sensors, data from one or more position sensors, data from one or more orientation sensors, data from one or more motion sensors, position and/or orientation data from step motors, position and/or orientation data from electric motors, position and/or orientation data from hydraulic motors, position and/or orientation data from electric and/or mechanical actuators, position and/or orientation data from drives, position and/or orientation data from robotic controllers, position and/or orientation data from one or more robotic computer processors, or a combination thereof.
In some embodiments, the imaging system is configured to generate an x-ray beam. In some embodiments, the x-ray beam of the imaging system is cone shaped or cylindrical. In some embodiments, the x-ray beam of the imaging system originates from one or more point sources.
In some embodiments, the x-ray beam of the imaging system is collimated.
In some embodiments, the imaging system is configured to generate an x-ray beam, wherein the 3D representation of the surface, volume or combination thereof comprises information about a limit, an edge, a margin, a boundary, a circumference, a perimeter, an envelope or a combination thereof of the x-ray beam.
In some embodiments, the system further comprising a user interface. In some embodiments, the user interface comprises a virtual user interface, wherein the virtual interface comprises at least one virtual object. In some embodiments, the at least one virtual object comprises one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof. In some embodiments, the virtual user interface comprises a gesture recognition, gaze recognition, gaze lock, eye tracking, hand tracking, pointer tracking, instrument tracking, tool tracking, or a combination thereof. In some embodiments, the at least one computer processor is configured to generate a command based at least in part on at least one interaction of a user with the at least one virtual object displayed in the virtual user interface. In some embodiments, the command is configured to move, tilt, or rotate one or more components of the imaging system, one or more components of a patient table or a combination thereof. In some embodiments, the command is configured to activate, operate, de-activate or a combination thereof a motor, an actuator, a drive, a controller, a hydraulic system, a switch, an electronic circuit, a computer chip, an x-ray tube, an image intensifier, a functional unit of an imaging system, or a combination thereof. In some embodiments, the command is configured to move or modify a geometry of the imaging system, a patient table, a geometric relationship between one or more imaging system components, a collimator, a grid, an image intensifier, a detector resolution, a setting of the imaging system, a parameter of the imaging system, a parameter of the imaging data acquisition, a display parameter, an x-ray source setting, an x-ray tube setting, a kVp setting, an mA setting, an mAs setting, a collimation, a tube—detector distance, a tube—patient distance, patient—detector distance, a patient—image intensifier distance, a table height relative to a tube, a detector, a table position relative to a tube, a detector, a patient position, a C-arm position, orientation, or combination thereof, a gantry position, orientation or combination thereof, a grid height, a grid width, a grid ratio, a field of view, a matrix, a pixel size, a voxel size, an image size, an image volume, an imaging plane, an image dimension in x, y, z and/or oblique direction, an image location, an image volume location, a scan coverage, a pitch, an in-plane resolution, a slice thickness, an increment, a detector configuration, a detector resolution, a detector density, a tube current, a tube potential, a reconstruction algorithm, a scan range, a scan boundary, a scan limit, a reconstructed slice thickness, a segmentation algorithm, a window, a level, a brightness, a contrast, a display resolution, or a combination thereof. In some embodiments, the command is configured to set and/or modify one or more image acquisition parameters of the imaging system. In some embodiments, the command is configured to set, move, and/or modify a position, orientation, size, area, volume, or combination thereof of a 2D, 3D or 2D and 3D imaging data acquisition. In some embodiments, the command is configured to set, move, and/or modify one or more coordinates of the 3D representation. In some embodiments, the command is configured to set, move and/or modify a dimension, a size, an area, a volume or a combination thereof of the 3D representation. In some embodiments, the setting, moving, and/or modifying of the dimension, size, area, volume or a combination thereof of the 3D representation is configured to set, move and/or modify a 2D, 3D or 2D and 3D imaging data acquisition to remain at the location of the 3D representation. In some embodiments, the command is configured to activate, operate, de-activate or a combination thereof a sensor, a camera, a video system, a 3D scanner, a Lidar system, a navigation system, a potentiometer, a piezoelectric system, a piezoelectric mechanism, a piezoelectric lock or release system, a controller, a drive, a motor, a hydraulic system, an actuator, or a combination thereof of the imaging system, an imaging system component, a patient table or a combination thereof. In some embodiments, the sensor comprises a depth sensor, inertial measurement unit, accelerometer, magnetometer, gyroscope, force sensor, pressure sensor, position sensor, orientation sensor, motion sensor, or a combination thereof.
In some embodiments, one or more components of the imaging system are attached to or integrated into a robot. In some embodiments, the robot is configured to move one or more components of the imaging system.
In some embodiments, the virtual user interface is configured to generate an event message triggered by a collision detection. In some embodiments, the system further comprises an event handler configured to process the event message. In some embodiments, the event handler is configured to generate a command.
In some embodiments, the computing system is configured to generate a command, wherein the command is triggered by the virtual user interface.
In some embodiments, the system is configured to determine a desired location of the augmented view associated with the imaging system to acquire 2D, 3D, or 2D and 3D imaging data at the desired location.
In some embodiments, the augmented reality display device is a head mounted display, and the augmented view comprises a 3D stereoscopic view.
In some embodiments, the at least one computer processor is configured to project the 3D stereoscopic view at the coordinates of intended 2D, 3D or 2D and 3D imaging data acquisition of the patient. In some embodiments, the location of the 2D, 3D, or 2D and 3D imaging data acquisition comprises one or more target anatomic structures of the patient.
Aspects of the disclosure relate to a method of preparing an image acquisition by an imaging system in a patient comprising: tracking one or more components of the imaging system in real time; obtaining, by the at least one computer processor, information about a geometry of one or more components of the imaging system, information about a geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof; generating, by the at least one computer processor, a 3D representation of a surface, a volume or combination thereof, wherein the 3D representation of the surface, the volume or combination thereof is at least in part derived from the information about the geometry of the one or more components of the imaging system, information about the geometry of the image acquisition, information about the one or more image acquisition parameters, or combination thereof; generating, by the at least one computer processor, an augmented view, the augmented view comprising the 3D representation of the surface, volume or combination thereof; and displaying, by an augmented reality display device, the augmented view, wherein the position and orientation of the augmented view is defined relative to the one or more components of the imaging system and is updated based on real time tracking information of the one or more components of the imaging system.
In some embodiments, the 3D representation of the surface, volume or combination thereof does not contain imaging data from the patient.
In some embodiments, the imaging system is configured to acquire 2D, 3D, or 2D and 3D imaging data of the patient, and wherein the 2D, 3D, or 2D and 3D imaging data of the patient are acquired within the 3D representation of the surface, volume or combination thereof.
In some embodiments, the augmented view at the defined position relative to the one or more components of the imaging system moves in relation with the tracked one or more components of the imaging system, wherein the moving facilitates superimposing or aligning the 3D representation with a target anatomic structure of the patient.
In some embodiments, the position, orientation, position and orientation of the augmented view is adjusted in response to movement of the tracked one or more components of the imaging system.
In some embodiments, the step of generating the augmented view is before the step of acquiring 2D, 3D, or 2D and 3D imaging data of the patient, or wherein the step of displaying the augmented view is before the step of acquiring 2D, 3D, or 2D and 3D imaging data of the patient.
In some embodiments, the augmented reality display device is a head mounted display, and wherein the augmented view comprises a 3D stereoscopic view.
Illustrative, non-limiting example embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.
The following description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the presently disclosed embodiments.
Aspects of the present disclosure provide, among other things, systems, devices and methods for a simultaneous visualization of live data of the patient and digital representations of virtual data such as virtual operating ranges, virtual operating areas, virtual operating volumes, e.g. for robots and/or imaging system, virtual image acquisition ranges, virtual image acquisition areas, virtual image acquisition volumes, e.g. for imaging systems, virtual cuts and/or virtual surgical guides including cut blocks, drilling guides, one or more virtual axes, one or more virtual planes or a combination thereof through a head mounted display (HMD) or other augmented reality display system. In some embodiments, the system can include one or more HMDs or other augmented reality display systems, one or more processors and one or more user interfaces. In some embodiments, the surgical site including live data of the patient, the HMD, and the virtual data are registered in a common coordinate system. In some embodiments, the virtual data are superimposed onto and aligned with the live data of the patient. In some embodiments, the head mounted display is a see-through HMD, e.g. a video see-through HMD or an optical see-through HMD. Unlike virtual reality head systems that blend out live data, the HMD can allow the surgeon to see the live data of the patient through the HMD, e.g. the surgical field, while at the same time observing virtual data of the patient and/or virtual surgical instruments or implants with a predetermined position and/or orientation using the display of the HMD unit.
In any of the embodiments of the disclosure, a video see-through HMD or an optical see-through HMD can be used. In any of the embodiments of the disclosure, other augmented reality display devices can be used, e.g. in conjunction with an HMD or instead of an HMD, for example a tablet, e.g. an iPad (Apple, Cupertino, CA) or Surface (Microsoft, Redmond, WI), or a smart phone, e.g. an iPhone (Apple Cupertino, CA). When other augmented reality display devices are used, they can comprise an optional video camera or scanner, e.g. 3D scanner, including, for example, a LIDAR system, for scanning physical objects, such as an imaging system, a surgical robot (also referred herein as robotic system or surgical robotic system), an OR table, a patient on the OR table, an imaging system table, a patient on the imaging system table, a physical tool, a physical instrument, an end effector etc. The augmented reality display can comprise a composite or mixed reality or augmented reality display of the video feed and virtual devices or virtual objects, e.g. a virtual end effector or a 3D representation of an x-ray beam or intended image acquisition. Any virtual devices, virtual surgical guide, virtual tool or instrument, or virtual object known in the art or described in the specification can be co-displayed with the video feed or video images. The virtual devices, virtual surgical guide, virtual tool or instrument, or virtual object known in the art or described in the specification can be displayed in conjunction with the video feed and can optionally be registered with the physical objects, devices (e.g. an imaging system or a surgical robot) or a physical patient or target anatomic structure included in the video feed or video images. Any of the registration techniques described in the specification or known in the art can be used. The terms mixed reality and augmented reality as used throughout the disclosure can be used interchangeably. In any of the illustrations, the term HMD (i.e. head mounted display) can be used interchangeably with an augmented reality display device, mixed reality display device, e.g. a tablet or smart phone. In some embodiments, the terms head mounted display, HMD, augmented reality display device, mixed reality display device can be used interchangeably.
In some embodiments, an operator such as a surgeon can look through an HMD observing physical data or information on a patient, e.g. a surgical site or changes induced on a surgical site, while pre-existing data of the patient are superimposed onto the physical visual representation of the live patient. Systems, methods and techniques to improve the accuracy of the display of the virtual data superimposed onto the live data of the patient are described in International Patent Application No. PCT/US2018/012459, which is incorporated herein by reference in its entirety.
Methods and systems of registration and cross-referencing including registration and cross-referencing surgical sites and one or more HMDs or other augmented reality display systems (e.g. using inside-out tracking techniques, outside-in tracking techniques, and combinations thereof) such as the ones described in PCT International Application Serial Nos. PCT/US2017/021859, PCT/US2018/013774 and PCT/US2019/015522 can be used. Methods, techniques, and systems of displaying virtual data in various surgical, medical or dental applications using one or more HMDs or other augmented reality display systems such as the ones described in PCT International Application Serial Nos. PCT/US2017/021859, PCT/US2018/013774, PCT/US2019/61698, PCT/US2019/015522, and U.S. Pat. No. 9,861,446 can be used. These applications are hereby incorporated by reference in their entireties.
Aspects of the present disclosure relate to systems, devices and methods for performing a surgical step or surgical procedure with visual guidance using a head mounted display. In some embodiments, the head mounted display can be a see-through head mounted display, e.g. an optical see-through head mounted display, for example for augmented reality applications. In some embodiments, the head mounted display can be a non-see through head mounted display, e.g. video-see through type, for virtual reality applications, optionally with video display including video streaming of live data from the patient, e.g. video feed from a camera integrated into, attached to, or separate from the head mounted display. The head mounted display can provide surgical guidance in a mixed reality environment.
Some aspects of the disclosure relate to a system for performing a surgical procedure, the system comprising: a processor; a see-through head mounted display or other augmented reality display device; and a marker attached to a patient, wherein, the system is configured to generate a 3D stereoscopic view or augmented view of a virtual surgical guide, wherein the virtual surgical guide is a placement indicator at one or more predetermined coordinates indicating a predetermined position, predetermined orientation or combination thereof for aligning a physical surgical tool or a physical surgical instrument, wherein the system is configured to display the 3D stereoscopic view by the see through head mounted display onto the patient, e.g. a patient's spine, a patient's joint, a patient's tooth, gum, dental structure or combination thereof. The processor can be configured to determine a distance between one or more predetermined coordinates of the virtual surgical guide and the see through head mounted display, wherein the one or more predetermined coordinates of the virtual surgical guide can be referenced to or based on the marker. In some embodiments, the processor can be configured to adjust at least one focal plane, focal point, convergence or combination thereof of the display of the 3D stereoscopic view based on a determined distance, e.g. using inside-out or outside-in tracking or a combination thereof.
In some embodiments, the system can be configured to track, e.g. in real-time, a robot component, an end effector, an imaging system component, or a combination thereof. Inside-out tracking can comprise tracking, for example, a head mounted display, an augmented reality display device, an anatomic structure of the patient, a patient table used with the imaging system, an imaging system, one or more components of the imaging system, a surgical instrument, a surgical tool, an implant, a surgical robot, a robot integrated with or part of the imaging system, a physical object or any combination thereof using at least one camera, scanner (including navigation systems, LIDAR systems etc.) or combination thereof integrated into a head mounted display or augmented reality display device. Outside-in tracking can comprise tracking, for example, a head mounted display, an augmented reality display device, an anatomic structure of the patient, a patient table used with the imaging system, an imaging system, one or more components of the imaging system, a surgical instrument, a surgical tool, an implant, a surgical robot, a robot integrated with or part of the imaging system, a physical object or any combination thereof using at least one camera, scanner (including navigation systems, LIDAR systems etc.) or combination thereof separate from a head mounted display or augmented reality display device.
In some embodiments, the system comprises one or more markers. In some embodiments, the marker can be configured to reflect or emit light with a wavelength between 380 nm and 700 nm.
In some embodiments, the marker can be configured to reflect or emit light with a wavelength greater than 700 nm. In some embodiments, the marker can be a radiofrequency marker, or the marker can be an optical marker, wherein the optical marker can include a geometric pattern.
In some embodiments, the one or more markers can comprise at least one marker attached to the patient, at least one marker attached to a see through head mounted display, at least one marker attached to a structure in the operating room or any combination thereof.
In some embodiments, the system can be configured to determine one or more coordinates using one or more cameras.
In some embodiments, the one or more cameras detect light with a wavelength between 380 nm and 700 nm. In some embodiments, the one or more cameras detect light with a wavelength above 700 nm. In some embodiments, the one or more cameras detect light with a wavelength between 380 nm and 700 nm, above 700 nm or a combination thereof.
In some embodiments, the system comprises at least one camera, scanner, 3D scanner, LIDAR system, depth sensor, inertial measurement unit (IMU), oscilloscope, gyroscope, or a combination thereof integrated into or attached to the head mounted display. In some embodiments, at least one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or a combination thereof is separate from the head mounted display. In some embodiments, the one or more camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or a combination thereof are configured to determine the position, orientation, or position and orientation of the marker. In some embodiments, the one or more camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or a combination thereof are configured to determine one or more coordinates of the marker. In some embodiments, the one or more camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or a combination thereof are configured to track the one or more coordinates of the marker during movement of the marker. In some embodiments, the one or more camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or a combination thereof are configured to determine one or more coordinates of the patient directly (e.g. markerless), e.g. by detecting select anatomic landmarks and/or structures and/or surfaces, e.g. a spinal structure and/or surface, articular structure and/or surface, tooth and/or surface, gum and/or surface, dental structure and/or surface, other structure and/or surface or body tissues. In some embodiments, the one or more camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof are configured to determine one or more coordinates of the see through head mounted display.
In some embodiments, the system is configured to track the one or more coordinates of the see through head mounted display during movement of the patient, the see through head mounted display, or the patient and the see through head mounted display. The movement of the patient can be, for example, the movement of a spine, one or more spinal elements, a head, a joint, one or more articular surfaces, a mandible, a maxilla, a tooth.
In some embodiments, the system comprises one or more processors. In some embodiments, the one or more processors are configured to generate the 3D stereoscopic view of the virtual surgical guide, virtual display, e.g. virtual axis, virtual plane, virtual operating range, area or volume (e.g. of a robot and/or imaging system), virtual image acquisition range, area or volume (e.g. of an imaging system). In some embodiments, the one or more processors are configured to determine the distance between the one or more predetermined coordinates of the virtual surgical guide, virtual display, e.g. virtual axis, virtual plane, virtual operating range, area or volume (e.g. of a robot and/or imaging system), virtual image acquisition range, area or volume (e.g. of an imaging system) and the see through head mounted display. In some embodiments, the one or more processors are configured to track one or more coordinates of at least one or more markers, one or more anatomic structures, one or more see through head mounted displays, or combinations thereof during movement of the patient, the see through head mounted display or the patient and the see through head mounted display. In some embodiments, the one or more processors are configured to determine the distance between the one or more predetermined coordinates of the virtual surgical guide, virtual display, e.g. virtual axis, virtual plane, virtual operating boundary, virtual operating range, area or volume (e.g. of a robot and/or imaging system), virtual image acquisition range, area or volume (e.g. of an imaging system) and the see through head mounted display during movement of the marker and/or the anatomic structure, movement of the see through head mounted display, or movement of the marker and/or the anatomic structure and the see through head mounted display. In some embodiments, one or more processors are configured to adjust at least one focal plane, focal point, convergence or combination thereof based on a change in a determined distance, e.g. from an HMD to a surgical site and/or anatomic structure. In some embodiments, one or more computer processors and/or computing systems, e.g. a first, second, third, fourth, etc. computer processor and/or computing systems are configured to display, e.g. by a computer monitor and/or one or more head mounted displays, a virtual surgical guide, e.g. a virtual axis, virtual plane, a virtual operating range, virtual operating area or virtual operating volume (e.g. of a robot and/or imaging system), a virtual image acquisition range, virtual image acquisition area or virtual image acquisition volume (e.g. of an imaging system). A first computer processor and/or computing system can be configured to communicate (e.g. via direct cable connection or wireless connection) to a robot and/or an imaging system or to be communicatively coupled to the robot and/or imaging system. A second computer processor and/or computing system can be configured to communicate (e.g. via direct cable connection or wireless connection) to one or more head mounted displays or to be communicatively coupled to the head mounted display(s).
In some embodiments, the physical surgical tool or physical surgical instrument can be configured to effect a tissue removal in the patient. A tissue removal can be, for example, an osteotomy of a bone (e.g. using an osteotome, as used in spinal deformity operations or in articular procedures), a pinning, drilling, milling, reaming, broaching, impacting and/or cutting of a bone using, for example, a pin, drill, mill, reamer, broach, impactor, and/or saw or sawblade, optionally attached to or integrated into a robot (e.g. hand-held or attached to an OR table) or a robotic arm. In some embodiments, a robotic end effector can be configured to effect a tissue removal or tissue alteration in the patient. The tissue removal or tissue alteration can be a removal of bone or a removal of cartilage or a removal of bone and cartilage, or a removal of a tooth and/or dental tissue, a tissue ablation, a tissue coagulation, a cell transfer, an implantation etc. Examples include a thermocoagulation, a cryoablation, a cutting with a scalpel or other cutting device. A tissue removal or alteration can be a removal or addition/supplementation of bone, bone tissue, cartilage, dental tissue, gum, gum tissue, brain, brain tissue, organ (e.g. liver, spleen, kidneys, bowel, stomach, heart, lung, thyroid, parathyroid tissue), skin, dermal tissue, subcutaneous tissue, or any combination thereof.
Aspects of the present disclosure relate to devices and methods for performing a surgical step or surgical procedure with visual guidance using one or more head mounted displays and with display of one or more imaging studies, e.g. x-rays, Panorex views, CT scan (for example, spiral CT, cone beam CT), MRI scan, ultrasound scan, PET scan, SPECT scan or a combination thereof.
In some embodiments, the device can comprise a Bluetooth transmitter and/or receiver.
Bluetooth can be a packet-based protocol with a master/slave architecture. One master can communicate with multiple slaves in a piconet. A master Bluetooth device can communicate with multiple devices in a piconet. The devices can switch roles, by agreement, and the slave can become the master (for example, a headset initiating a connection to a phone can begin as master—as an initiator of the connection—but may subsequently operate as the slave).
Bluetooth can be a layer protocol architecture comprising core protocols, cable replacement protocols, telephony control protocols, and adopted protocols. The device can, in some embodiments, employ high-speed Bluetooth protocols. The device can comprise an interface between a server and the device using a Bluetooth device. The interface can be HCI (Host Controller Interface).
The Host Controller Interface can provide a command interface for the controller and for the link manager, which can allow access to the hardware status and control certain registers. This interface can provide an access layer for all Bluetooth devices. The HCI layer of the machine can exchange commands and data with the HCI firmware present in the Bluetooth device. The HCI can, in some embodiments, automatically discover other Bluetooth devices that are within the coverage radius.
The hardware that constitutes a Bluetooth device, including the Bluetooth device that can optionally be within the device, can include two parts: a radio device, responsible for modulating and transmitting the signal and a digital controller. These specific parts can, in some embodiments be physically separate and can in other embodiments be physically together.
The digital controller can, in some embodiments, be a computer processor or a central processing unit (CPU). In some embodiments, the computer processor CPU can run a Link Controller; and interfaces with the host device, such as the Host Controller Interface. The Link Controller can be responsible for the processing of the baseband and the management of ARQ and physical layer FEC protocols. The computer processor or the CPU can, in some embodiments, handle the transfer functions (both asynchronous and synchronous), audio coding, and data encryption. The computer processor or CPU of the device can, in some embodiments, be responsible for performing the instructions related to the Bluetooth of the host device, in order to simplify its operation. For the performance of specific instructions related to the Bluetooth of the host device, the computer processor or the CPU can run software called Link Manager that has the function of communicating with other devices through the LMP protocol.
The Link Manager can, in some embodiments, establish the connection between devices. For example, the Link Manager can establish the connection between the devices. The Link Manager can be responsible for the establishment, authentication and configuration of the link. The Link Manager can furthermore find other managers and communicates with them due to the management protocol of the LMP link.
The Link Manager Protocol can comprise a number of PDUs (Protocol Data Units) that can be sent from one device to another. The following is a list of supported services:
The system, when in discoverable mode, can transmit the following information on demand:
The system can have a unique 48-bit address. The system can have a friendly Bluetooth name, which can be set by the user. This name can appear when another user scans for devices and in lists of paired devices.
During pairing between the server and the system the two can establish a relationship by creating a shared secret or a link key. If both devices store the same link key, they can be paired or bonded.
The following are pairing mechanisms that can be used in some embodiments of the disclosure:
In some embodiments, the device comprises a Bluetooth transmitter and/or receiver wherein the Bluetooth transmitter and/or receiver is configured to work in conjunction with an augmented reality surgical guidance system, a surgical navigation system, a robot, a robotic system, and/or a handheld robot.
In some embodiments, the Bluetooth transmitter and/or receiver and the established connection between the Bluetooth transmitter and/or receiver and the augmented reality surgical guidance system, surgical navigation system, robot, robotic system, and/or handheld robot can work in conjunction with one or more on/off switches and/or one or more potentiometers, e.g. digital potentiometers, and/or one or more rheostats and/or one or more actuators to regulate the speed of the movement of the saw blade or movement of the drill bit or to provide haptic feedback.
For example, in cases where the augmented reality surgical guidance system, surgical navigation system, robot, robotic system, and/or handheld robot detects a movement of a surgical instrument or tool deviating from an intended surgical axis, target, target area, target volume, tissue resection target, area, volume (e.g. bone or tissue removal or resection, e.g. with a bone drill or bone saw) by a specific distance in any direction in one or more dimensions, the augmented reality surgical guidance system, surgical navigation system, robot, robotic system, and/or handheld robot can transmit information to the Bluetooth receiver which can regulate the Bluetooth switch, including both a transmitter and receiver, to activate an on/off switch and/or a potentiometer, e.g. digital, and/or a rheostat and/or a specific actuator for haptic feedback. In cases where the augmented reality surgical guidance system, surgical navigation system, robot, robotic system, and/or handheld robot detects a movement of a drill or saw or other power tool that approaches, for example, a specific anatomical structure or safe zone, the augmented reality surgical guidance system, surgical navigation system, robot, robotic system, and/or handheld robot can similarly work in conjunction with the Bluetooth switch within the device attached to the drill or saw to adjust, control, and/or regulate an on/off switch and/or a potentiometer and/or a rheostat and/or a specific actuator for haptic feedback. The same concept can similarly work for turning on or increasing the speed of the movement of the saw blade or the drill bit or other power tool or instrument when approaching certain anatomic structures.
The Bluetooth switch, Bluetooth receiver, and/or Bluetooth transmitter can, in some embodiments, employ low latency Bluetooth in order to provide instant saw or drill speed regulation or instant haptic feedback.
In some embodiments, the device comprises a WiFi transmitter and/or receiver. In some embodiments, the device can comprise WiFi capability. Different versions of WiFi can be used including but not limited to: 802.11a, 802.11b, 802.11g, 802.11n (Wi-Fi 4[40]), 802.11h, 802.11i, 802.11-2007, 802.11-2012, 802.11ac (Wi-Fi 5[40]), 802.11ad, 802.11af, 802.11-2016, 802.11ah, 802.11ai, 802.11aj, 802.11aq, 802.11ax (Wi-Fi 6[40]), and 802.11ay. In some embodiments, the device comprises a WiFi transmitter and/or receiver wherein the WiFi transmitter and/or receiver is configured to work in conjunction with a surgical guidance system. In some embodiments, the system can include routers that can be configured for intranet and internet connections.
In some embodiments, the system can utilize several distinct radio frequency ranges. For example, the system utilizes the 802.11 standard, it can include distinct radio frequencies ranges for use in Wi-FI communications such as: 900 MHz, 2.4 GHz, 5 GHz, 5.9 GHz, and 60 GHz bands. Each frequency or range can have a multitude of channels.
In some embodiments, the system and/or device's Wi-Fi can be part of the IEEE 802 protocol family. In some embodiments, the system and/or device can comprise one or more transmitters. WiFi transmitters are low power devices.
In some embodiments, the system and/or device can comprise one or more antennas. The system and/or device can comprise an access point compliant with 802.11b and/or 802.11g.
Using the stock omnidirectional antenna can have a range of 100 m (0.062 mi). The identical radio with an external semi parabolic antenna (15 dB gain) with a similarly equipped receiver at the far end can have a range over 20 miles.
In some embodiments, the system and/or device can comprise multiple-input and multiple-output. The system and/or device including but not limited to standards such as IEEE 802.11n and IEEE 802.11ac, can comprise multiple antennas for extended range and higher speeds.
In some embodiments, the WiFi can comprise Local Area Networks (LAN). In some embodiments, the device can include one or more access points. A wireless access point can connect a group of wireless devices to an adjacent wired LAN. In some embodiments, the device can include one or more wireless adapters. Wireless adapters can allow devices to connect to a wireless network
In some embodiments, the device can include one or more routers. Wireless routers can integrate a Wireless Access Point, Ethernet switch, and internal router firmware application that provides IP routing, NAT, and DNS forwarding through an integrated WAN-interface.
In some embodiments, the device can include one or more wireless network bridges.
Wireless network bridges can act to connect two networks to form a single network at the data-link layer over Wi-Fi. The main standard is the wireless distribution system (WDS).
Wireless bridging can connect a wired network to a wireless network.
In some embodiments, the device can include one or more security features. Security features can be any security standard known in the art.
In some embodiments, the WiFi transmitter and/or receiver and the established connection between the WiFi transmitter and/or receiver and the augmented reality surgical guidance system can work in conjunction with one or more on/off switches and/or one or more potentiometers and/or one or more rheostats and/or one or more actuators to regulate the oscillation of a saw blade or movement of a drill bit or to provide haptic feedback.
For example, in cases where the augmented reality surgical guidance system detects a movement of a drill or saw or other power tool or instrument deviating from the intended cut/drill surgical axis, the surgical guidance system can regulate a WiFi switch, including both a transmitter and receiver, to activate an on/off switch and/or a potentiometer, e.g. digital, and/or a rheostat and/or a specific actuator for haptic feedback. In cases where the surgical guidance system detects a movement of a drill or saw or other power tool or instrument that approaches, for example, a specific anatomical structure or safe zone, the surgical guidance system can similarly work in conjunction with a WiFi switch within the device attached to a drill or saw or other power tool or instrument to activate an on/off switch and/or a potentiometer and/or a rheostat and/or a specific actuator for haptic feedback. The same concept can similarly work for turning on or increasing the speed of the movement of a saw blade or a drill bit or other power tool or instrument when approaching certain anatomic structures.
In some embodiments, the device can comprise a LiFi transmitter and/or receiver. In some embodiments, the device can comprise LiFi capability. LiFi can use light from light-emitting diodes (LEDs) as a medium to deliver networked, mobile, high-speed communication.
In some embodiments, the system can comprise visible light communications (VLC). VLC works by switching the current to the LEDs off and on at very high speeds.
In some embodiments, the system can comprise Bg-Fi. Bg-Fi can be a Li-Fi system consisting of an application for a mobile device, and a simple consumer product device, with color sensor, microcontroller, and embedded software. Light from the mobile device display communicates to the color sensor on the consumer product, which converts the light into digital information. Light emitting diodes enable the consumer product to communicate synchronously with the mobile device.
In some embodiments, the Li-Fi system can be wireless and can use 802.11 protocols. In some embodiments, the LiFi system can use ultraviolet, infrared and visible light communication. One part of the visible light communication can be designed from communication protocols established by the IEEE 802 workgroup. The IEEE 802.15.7 standard can, in some embodiments, define the physical layer (PHY) and media access control (MAC) layer.
The modulation formats recognized for PHY I and PHY II are on-off keying (OOK) and variable pulse position modulation (VPPM). The Manchester coding used for the PHY I and PHY II layers can include the clock inside the transmitted data by representing a logic 0 with an OOK symbol “01” and a logic 1 with an OOK symbol “10”, all with a DC component. The DC component avoids light extinction in case of an extended run of logic 0's.
The use of LiFi provides additional benefits as the light waves are unlikely to affect or hinder the efficiency of a medical procedure or medical devices.
In some embodiments, the device can comprise a LiFi transmitter and/or receiver wherein the LiFi transmitter and/or receiver is configured to work in conjunction with a surgical guidance system. In some embodiments, the LiFi transmitter and/or receiver and the established connection between the LiFi transmitter and/or receiver and the augmented reality surgical guidance system can work in conjunction with one or more on/off switches and/or one or more potentiometers and/or one or more rheostats and/or one or more actuators to regulate the oscillation of a saw blade or movement of a drill bit or to provide haptic feedback.
For example, in cases where an augmented reality surgical guidance system detects a movement of a drill or saw deviating from the intended cut/drill surgical axis, the surgical guidance system can regulate the LiFi switch, including both a transmitter and receiver, to activate an on/off switch and/or a potentiometer and/or a rheostat and/or a specific actuator for haptic feedback. In cases where the surgical guidance system detects a movement of a drill or saw that approaches, for example, a specific anatomical structure or safe zone, the surgical guidance system can similarly work in conjunction with the LiFi switch within a device attached to or integrated into a drill or saw to activate an on/off switch and/or a potentiometer and/or a rheostat and/or a specific actuator for haptic feedback. The same concept can similarly work for turning on or increasing the speed of the movement of a saw blade or a drill bit when approaching certain anatomic structures.
In some embodiments, other forms of wireless data transmission known in the art can be used, not only Bluetooth, Wifi, Lifi, but also, but not limited to, a radiofrequency signal, a microwave signal, an ultrasound signal, an infrared signal, an electromagnetic wave or a combination thereof. Any form of wireless data transmission known in the art can be used in any of the embodiments.
In some embodiments, the system comprises at least one camera, video system and/or scanner (e.g. a 3D scanner, a laser scanner, a LIDAR system or LIDAR scanner), a depth sensor, an IMU or a combination thereof integrated into or attached to the see through head mounted display. In some embodiments, at least one camera, video system and/or scanner (e.g. a 3D scanner, a laser scanner, a LIDAR system or LIDAR scanner), a depth sensor, an IMU or a combination thereof is/are separate from the head mounted display. In some embodiments, one or more camera, video system, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof are configured to determine the position, orientation, or position and orientation of a marker, a surface, and/or a tissue. In some embodiments, one or more camera, video system, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof are configured to determine one or more coordinates of a marker, a surface, and/or a tissue. In some embodiments, one or more camera, video system, scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or a combination thereof are configured to track one or more coordinates of a marker, a surface, and/or a tissue during movement of the marker, the surface, and/or the tissue. In some embodiments, one or more camera, video system, scanner, 3D scanner, LIDAR system, depth sensor, IMU, oscilloscope, gyroscope or a combination thereof are configured to determine one or more coordinates of a see through head mounted display. In some embodiments, one or more markers can be attached to or integrated into a physical instrument, a physical tool, a physical trial implant, a physical implant, a physical device, one or more HMDs or other augmented reality display systems, a robot, a robotic arm, a handheld robot, an end effector, an imaging system, one or more components of an imaging system or a combination thereof.
The term imaging system as used throughout the specification can comprise any imaging system using ionizing or non-ionizing radiation. The term imaging system as used throughout the specification can comprise any imaging system utilizing x-rays, e.g. a radiography system, a projection radiography system, a fluoroscopy system, a 2D fluoroscopy system, a 3D fluoroscopy system (e.g. using a 3D C-arm system), a cone beam CT system, a spiral CT system, CT imaging systems using pencil beam geometry, fan beam geometry, open beam geometry, a CT imaging system using a single detector array, a CT imaging system using multiple detector arrays, an electron beam CT imaging system, a conventional radiography system, a digital radiography system, a digital tomosynthesis system, a dual energy imaging system, a dual energy subtraction imaging system, a subtraction imaging system, an angiography imaging system, a uni-planar angiography system, a bi-planar angiography system, a 3D angiography system; the term imaging system as used throughout the specification can comprise a magnetic resonance imaging (MRI) system, an ultrasound imaging system; the term imaging system as used throughout the specification can comprise a radionuclide imaging system, a scintillation detector imaging system for radionuclide imaging, a semiconductor detector imaging system for radionuclide imaging, a pulse height spectroscopy imaging system for radionuclide imaging, a planar nuclear imaging system, a cardiac radionuclide imaging system, a single photon emission computed tomography (SPECT) imaging system, a positron emission tomography (PET) imaging system. The term imaging system as used throughout the specification can comprise or any combination of the foregoing imaging systems, e.g. a combined x-ray—ultrasound imaging system, a SPECT MRI imaging system, a PET MRI imaging system, a 2D radiography/fluoroscopy—3D cone beam CT imaging system etc.
In any of the embodiments, the term imaging parameter can be used interchangeably with the terms image acquisition parameter, acquisition parameter, acquisition setting, image acquisition setting.
The term imaging parameter or imaging parameters as used throughout the specification can comprise one or more of the above parameters and/or other parameters known in the art. Any of the foregoing parameters can be set, defined, determined, adjusted, modified using a user interface, including a graphical user interface. The graphical user interface can comprise virtual user interface, e.g. using a head mounted display or other augmented reality display device. The virtual interface can, for example, use a collision detection, e.g. for generating and/or enabling one or more commands. Any of the above imaging parameters and/or other imaging parameters known in the art can be set, defined, determined, adjusted, modified using a virtual user interface, using any of the embodiments described in the specification, and/or any combination of embodiments described in the specification.
The term virtual interface can be used interchangeably with the term virtual user interface. A virtual user interface can be a graphical user interface. A virtual user interface can be displayed by one or more head mounted displays or other augmented reality display devices. The virtual user interface displayed by a first, second, third, fourth etc. head mounted display can be the same or different. A virtual user interface can comprise at least one virtual object. A virtual user interface or the at least one virtual object can comprise one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof. One or more commands can be generated by an interaction, e.g. of a user, with a virtual user interface. The interaction can be, for example, a collision detection, e.g. between a user's finger and a virtual object, e.g. a virtual button, or between a tracked pointer or tool or surgical instrument and a virtual object.
In one embodiment, as shown in non-limiting, strictly exemplary fashion in
In some embodiments, a first computing system comprising one or more processor can be configured to transmit data to a second computing system configured to generate a display by a head mounted display or other augmented reality display device based on the transmitted data.
In some embodiments, a first computing system comprising one or more processor can be configured to transmit data to a second computing system configured to generate a display by a head mounted display or other augmented reality display device. In some embodiments, a second computing system configured to generate a display by a head mounted display or other augmented reality display device can be configured to transmit data to a first computing system separate from the head mounted display or other augmented reality display device. In some embodiments, the first computing system and the second computing system can be configured to transmit and/or receive data from each other, e.g. for updating a display by a head mounted display or other augmented reality display device.
The data or data packets that can be received and/or transmitted can comprise, for example, any of the data listed in Table 2:
The data or data packets can comprise stereoscopic and/or non-stereoscopic views or data prepared for stereoscopic and/or non-stereoscopic views or displays by one or more HMDs or other augmented reality display systems. Stereoscopic and/or non-stereoscopic views or data prepared for stereoscopic and/or non-stereoscopic views or displays by one or more HMDs or other augmented reality display systems can be updated in near real-time or real-time, e.g. with a rate of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz, 100 Hz, or any other rate or frequency including higher rates or frequencies.
Position, orientation, position and orientation, direction of movement and/or tracking data for one or more HMDs or other augmented reality display systems can be obtained, measured and/or generated using, for example, one or more cameras or scanners (e.g. video systems, 3D scanners, LIDAR systems, depth sensors; using visible light and/or infrared and/or any other wavelength) integrated into or attached to the one or more HMDs or other augmented reality display systems, one or more cameras and/or scanners (e.g. video systems, 3D scanners, LIDAR systems, depth sensors; using visible light and/or infrared and/or any other wavelength) separate from the one or more HMDs or other augmented reality display systems, one or more lasers, e.g. integrated or attached to the one or more HMDs or other augmented reality display systems and/or separate from the one or more HMDs or other augmented reality display systems, one or more inertial measurement units integrated or attached to the one or more HMDs or other augmented reality display systems or a surgeon's head, one or more markers, e.g. active markers (e.g. RF markers), passive markers (e.g. infrared markers), optical markers (e.g. with geometric patterns, QR codes, bar codes, defined shapes, e.g. triangles, squares, rectangles etc.) integrated or attached to the one or more HMDs or other augmented reality display systems, any combination thereof optionally visible to the camera(s) and/or scanner(s), or any combination thereof.
Position, orientation, position and orientation, direction of movement and/or tracking data for one or more HMDs or other augmented reality display systems can be generated, transmitted and/or received in near real time or in real time, e.g. with a rate of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz, 100 Hz, or any other rate or frequency including higher rates or frequencies.
Position, orientation, position and orientation, direction of movement and/or tracking data for one or more HMDs or other augmented reality display systems can be transmitted by one or more computer processors integrated into or connected to the one or more HMDs or other augmented reality display systems via a wireless access point or router wirelessly to a separate computing system with one or more computer processors. The separate computing system can process the data about the position, orientation, position and orientation, direction of movement and/or tracking data of the one or more HMDs or other augmented reality display systems received and package them with other data for corresponding time points or time intervals, e.g. patient tracking data and/or instrument tracking data, for transmission, optionally back to the one or more HMDs or other augmented reality display systems.
Position, orientation, position and orientation, direction of movement and/or tracking data for one or more HMDs or other augmented reality display systems can be obtained, acquired and/or generated by one or more cameras or scanners separate from the one or more HMDs or other augmented reality display systems and can be processed by one or more computer processors connected to or integrated into the camera and/or scanner and/or connected to or integrated into a separate computing system, e.g. a server, optionally connected directly or wirelessly to the camera or scanner. The separate computing system can process the data about the position, orientation, position and orientation, direction of movement and/or tracking data of the one or more HMDs or other augmented reality display systems received and package them with other data for corresponding time points or time intervals, e.g. patient tracking data and/or instrument tracking data, for transmission (and/or reception), for example back to the one or more HMDs or other augmented reality display systems.
Any of the data listed in Table 2 and any additional data can be transmitted and/or received in real-time, or near real-time, with transmitting and/or receiving rates of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz, 100 Hz, or any other rate or frequency including higher rates or frequencies.
When multiple data sets, e.g. different types of data such as instrument tracking data and/or HMD or other augmented reality display system tracking data and/or patient or surgical site (e.g. a spine, joint, tooth or vascular structure) tracking data and/or virtual user interface and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) are transmitted and/or received, they can be transmitted and/or received simultaneously or non-simultaneously. Data sets including any of the data listed in Table 2 can optionally be labelled, e.g. with a time stamp, time point, time interval (e.g. within 1 transmission or data reception, for example, for a rate of 60 Hz, within 16.66 ms or less or, for example, any other value within the time allocated for transmission and reception), a time label, a time tag or any combination thereof.
In some embodiments, coordinate information, registration data, tracking data or a combination thereof of one or more HMDs or other augmented reality display systems can optionally be labeled or coded for each specific HMD or other augmented reality display system, e.g. by a computer processor integrated into, attached to, or connected to a camera or scanner (optionally part of a first or second computing unit), a computer processor integrated into, attached to, or connected to a first computing unit (e.g. in a server), and/or a computer processor integrated into, attached to, or connected to a second computing unit (e.g. in a client, for example integrated into an HMD or other augmented reality display system or connected to an HMD or other augmented reality display system).
In some embodiments, data packets (for example, as listed in Table 2) can comprise multiple types of data, e.g. data comprising instrument tracking data, data comprising HMD or other augmented reality display system tracking data and/or a data comprising patient or surgical site (e.g. a spine, joint, dental, vascular, organ or neural structure) tracking data and/or virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument), all packaged within the same data packet. As the data packet(s) is/are transmitted or received, e.g. data comprising instrument tracking data and/or data comprising HMD or other augmented reality display system tracking data and/or a data comprising patient or surgical site (e.g. a spine, joint, dental, vascular, organ or neural structure) tracking data and/or virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) can be transmitted or received together, e.g. simultaneously.
In some embodiments, transmission and/or reception can be processed by one or more computer processors, e.g. in a first and/or a second computing system, and/or integrated or attached to a camera or scanner (e.g. integrated or attached to an HMD or other augmented reality display system, integrated or attached to a robot, separate from an HMD or other augmented reality display system or robot etc.), so that data, for example, instrument tracking data and/or HMD or other augmented reality display system tracking data and/or patient or surgical site (e.g. a spine or joint) tracking data, acquired with the same time stamp, time point, time label, time tag, or within the same time interval or any combination thereof are transmitted and/or received in the same data packet.
In some embodiments, transmission and/or reception can be processed by one or more computer processors so that data, for example, instrument tracking data and/or HMD or other augmented reality display system tracking data and/or patient or surgical site (e.g. a spine or joint) tracking data and/or virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument), acquired within the same time interval (and optionally labelled with the same time interval) are transmitted and/or received by one or more computer processors, e.g. in a first and/or a second computing system, within a defined time period in multiple data packets; the defined time period can be corresponding to, matching, or overlapping with the time interval. Optionally the defined time period for transmitting and/or receiving data packets can be a time period bounded or defined by or derived from the transmission and/or reception rate, e.g. <0.16666666 sec for a transmission and/or reception rate of 60 Hz, or <0.0333333 sec for a transmission and/or reception rate of 30 Hz, <0.04 sec for a transmission and/or reception rate of 25 Hz, or any other value.
Data packets (for example, as listed in Table 2), e.g. a first data packet comprising instrument tracking data, a second data packet comprising HMD or other augmented reality display system tracking data and/or a third data packet comprising patient or surgical site (e.g. a spine, joint, dental, vascular, organ or neural structure) tracking data and/or a fourth data packet comprising virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) can be transmitted and/or received simultaneously, for example using different frequencies. Data packets, e.g. a first data packet comprising instrument tracking data, a second data packet comprising HMD or other augmented reality display system tracking data and/or a third data packet comprising patient or surgical site (e.g. a spine or joint) tracking data and/or a fourth data packet comprising virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) can be transmitted and/or received sequentially (e.g. using the same or different frequencies). Data packets, e.g. a first data packet comprising instrument tracking data, a second data packet comprising HMD or other augmented reality display system tracking data and/or a third data packet comprising patient or surgical site (e.g. a spine or joint) tracking data and/or a fourth data packet comprising virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) can be transmitted and/or received in an offset manner, e.g. with a pause or lag interval spaced in between. Data packets, e.g. a first data packet comprising instrument tracking data, a second data packet comprising HMD or other augmented reality display system tracking data and/or a third data packet comprising patient or surgical site (e.g. a spine or joint) tracking data and/or a fourth data packet comprising virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) can be transmitted and/or received in an interleaved manner. Data packets, e.g. a first data packet comprising instrument tracking data, a second data packet comprising HMD or other augmented reality display system tracking data and/or a third data packet comprising patient or surgical site (e.g. a spine or joint) tracking data and/or a fourth data packet comprising virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) can be transmitted and/or received in a non-overlapping manner during the transmission and/or the reception. Data packets, e.g. a first data packet comprising instrument tracking data, a second data packet comprising HMD or other augmented reality display system tracking data and/or a third data packet comprising patient or surgical site (e.g. a spine or joint) tracking data and/or a fourth data packet comprising virtual user interface data and/or interaction with virtual user interface data (e.g. with a tracked physical tool or instrument) can be transmitted and/or received in an overlapping manner during the transmission and/or the reception.
Thus, data packets, e.g. comprising one or more of data comprising instrument tracking data, data comprising HMD or other augmented reality display system tracking data, or data comprising patient or surgical site data can be transmitted or received in a simultaneous, synchronous fashion and/or alternatively in an non-synchronous or asynchronous fashion.
The data can be transmitted in near real time or in real time, e.g. with a rate of 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 35 Hz, 40 Hz, 45 Hz, 50 Hz, 55 Hz, 60 Hz, 65 Hz, 70 Hz, 80 Hz, 90 Hz, 100 Hz, or any other rate or frequency including higher rates or frequencies.
Data transmission can be performed using wireless data transmission protocols known in the art, e.g. Bluetooth, Wifi, LiFi, etc. and, as described, for example in PCT International Application Serial Nos. PCT/US2017/021859, PCT/US2018/013774, PCT/US2019/061698 and PCT/US2019/015522, which are hereby incorporated by reference in their entirety.
In some embodiments, the system can comprise a tracking system or sensor, e.g. optical tracking systems, for example using infrared and/or visible light cameras, video systems, 3D scanners, LIDAR systems, depth sensors, radiofrequency tracking systems, or combinations thereof, e.g. for outside-in or inside-out tracking. Multiple tracking systems can be used at the same time or, optionally combined, e.g. inside-out and outside-in tracking. Any tracking, sensor, or registration system known in the art (e.g. optical tracking systems, for example using infrared and/or visible light cameras, video systems, 3D scanners, LIDAR systems, depth sensors, radiofrequency tracking systems, or combinations thereof), can be used, for example as described in a non-limiting fashion in PCT International Application Serial Nos. PCT/US2017/021859, PCT/US2018/013774, PCT/US2019/061698 and PCT/US2019/015522, which are hereby incorporated by reference in their entirety.
A first computing system can comprise one or more computer processors. The second computing system can comprise one or more computer processors. The second computing system can be part of one or more mobile, wireless HMD or other augmented reality display system units. The second computing system can be part of a first mobile, wireless HMD or other augmented reality display system unit. A third computing system can be part of a second mobile wireless HMD or other augmented reality display system unit. A fourth computing system can be part of a third mobile wireless HMD or other augmented reality display system unit. A fifth computing system can be part of a third mobile wireless HMD or other augmented reality display system unit, etc. A first computing system can comprise one or more computer processors. A first computing system can, for example, be a server or controller or computing unit as shown in
One or more of the modules 1100-1160 can be integrated or combined. For example, the AR visualization module 1150 can be integrated or combined with the AR display module 1160.
One or more of the modules 1100-1160 can be run by the same computer processor or the same group of computer processors. One or more of the modules 1100-1160 can be run by different computer processors.
One or more of the computer processors operating a tracking engine or tracking module 1100, one or more computer processors operating an instrument calibration module 1110, one or more of the computer processors operating a headset calibration module 1120, one or more of the computer processors operating an imaging and navigation module 1130, one or more of the computer processors operating an AR wireless networking module 1140, one or more of the computer processors operating an AR visualization module 1150, and/or one or more of the computer processors operating an AR display module 1160 can be the same.
One or more of the computer processors operating a tracking engine or tracking module 1100, one or more computer processors operating an instrument calibration module 1110, one or more of the computer processors operating a headset calibration module 1120, one or more of the computer processors operating an imaging and navigation module 1130, one or more of the computer processors operating an AR wireless networking module 1140, one or more of the computer processors operating an AR visualization module 1150, and/or one or more of the computer processors operating an AR display module 1160 can be different.
The first computing system can, for example, be stationary, e.g. on a cart or stand. In some embodiments, the first computing system can also be mobile, e.g. part of a mobile, wireless HMD system or other augmented reality display system.
A second, third, fourth, fifth or more computing systems can comprise one or more computer processors. The one or more computer processors can be configured to run different operating modules as shown, for example, in
The second computing system can be part of one or more mobile, wireless HMD or other augmented reality display system units. A second computing system can be part of a first mobile, wireless HMD or other augmented reality display system unit. A third computing system can be part of a second mobile wireless HMD or other augmented reality display system unit. A fourth computing system can be part of a third mobile wireless HMD or other augmented reality display system unit. A fifth computing system can be part of a third mobile wireless HMD or other augmented reality display system unit, etc. The first, second, third, fourth etc. mobile wireless HMD unit can be worn by a user, e.g. a physician, a surgeon, a dentist, a physician or dental assistant etc. The first, second, third, fourth etc. mobile wireless HMD unit can be a video see-through HMD. The first, second, third, fourth etc. mobile wireless HMD unit can be an optical see-through HMD.
One or more modules can be combined or integrated and can, for example, be operated by the same one or more computer processors or, optionally, by different one or more computer processors.
One or more of the computer processors operating a tracking engine or tracking module 1100, one or more computer processors operating an instrument calibration module 1110, one or more of the computer processors operating a headset calibration module 1120, one or more of the computer processors operating an imaging and navigation module 1130, one or more of the computer processors operating an AR wireless networking module 1140, one or more of the computer processors operating an AR visualization module 1150, and/or one or more of the computer processors operating an AR display module 1160 can be the same.
One or more of the computer processors operating a tracking engine or tracking module 1100, one or more computer processors operating an instrument calibration module 1110, one or more of the computer processors operating a headset calibration module 1120, one or more of the computer processors operating an imaging and navigation module 1130, one or more of the computer processors operating an AR wireless networking module 1140, one or more of the computer processors operating an AR visualization module 1150, and/or one or more of the computer processors operating an AR display module 1160 can be different.
One or more modules can be operated by a first computing system, while one or more different modules can be operated by a second, or third, or fourth, etc. computing system. One or more modules can be operated by a first computing system, while one or more different modules can be operated by a second, and third, and fourth, etc. computing system.
One or more modules can be operated by a first computing system, while one or more same modules can be operated by a second, or third, or fourth, etc. computing system. One or more modules can be operated by a first computing system, while one or more same modules can be operated by a second, and third, and fourth, etc. computing system.
In one example, a first computing system can comprise a tracking module or tracking engine 1100, an instrument calibration module 1110, a headset calibration module 1120, an imaging and navigation module 1130, a AR wireless networking module 1140, and an AR visualization module 1150; a second computing system can comprise an AR display module 1160.
In another example, a first computing system can comprise a tracking module or tracking engine 1100, an instrument calibration module 1110, a headset calibration module 1120, an imaging and navigation module 1130, and a AR wireless networking module 1140; a second computing system can comprise an AR visualization module 1150 and an AR display module 1160.
In another example, a first and a second computing system can comprise one or more of the same modules, for example dedicated to the same and/or different functions. For example, a first computing system can comprise an AR wireless networking module 1140, for example for data transmission; a second computing system can also comprise an AR wireless networking module 1140, for example for data reception.
In another example, a first computing system can comprise a tracking module or tracking engine 1100, an instrument calibration module 1110, an imaging and navigation module 1130, and an AR wireless networking module 1140; a second computing system can comprise a headset calibration module 1120, an AR wireless networking module 1140, an AR visualization module 1150 and an AR display module 1160.
Any combination of same and/or different modules, including duplication of modules on different (first, second, third, fourth, fifth) computing systems is possible and within the scope of this disclosure.
Using one or more computer processors, e.g. in a second computing system, the AR display 1160 module can generate the stereoscopic or non-stereoscopic view of a first person for the first person's respective view angle in relationship to one or more anatomic landmarks or anatomic structures of a patient. Using one or more computer processors, e.g. in a third computing system, the AR display 1160 module can generate the stereoscopic or non-stereoscopic view of a second person for the second person's respective view angle in relationship to one or more anatomic landmarks or anatomic structures of the patient. Using one or more computer processors, e.g. in a fourth computing system, the AR display 1160 module can generate the stereoscopic or non-stereoscopic view of a third person for the third person's respective view angle in relationship to one or more anatomic landmarks or anatomic structures of the patient. Using one or more computer processors, e.g. in a fifth computing system, the AR display 1160 module can generate the stereoscopic or non-stereoscopic view of a fourth person for the fourth's person's respective view angle in relationship to one or more anatomic landmarks or anatomic structures of the patient, etc.
The second, third, fourth, fifth or more computing systems can be the same. The second, third, fourth, fifth or more computing systems can be different, e.g. integrated or connected to different mobile units and/or different HMDs or other augmented reality display systems.
A first, second, third, fourth, fifth or more computing systems can be the same. A first, second, third, fourth, fifth or more computing systems can be different. A first, second, third, fourth, fifth or more computer processors can be the same. A first, second, third, fourth, fifth or more computer processors can be different.
A first, second, third, fourth, fifth or more computer processor can have the same processing speed. At least one of a first, second, third, fourth, fifth or more or more computer processor can have a different processing speed. For example, a computer processor can have a processing speed of 1 GHz, 1.5 GHz, 2.0 GHz, 2.1 GHz, 2.2 GHz, 2.3 GHz, 2.4 GHz, 2.5 GHz, 2.6 GHz, 2.7 GHz, 2.8 GHz, 2.9 Ghz, 3.0 GHz or greater. Any value is possible. Some applications of the disclosure can benefit from higher processing speeds, e.g. above 1.5 GHz or 2.0 GHz, for example when data intense, complex data packets are being acquired, generated, transmitted and/or received (see Table 2 also). A computer processor can, for example, be a Qualcomm Snapdragon 845 or later (Qualcomm, San Diego, CA 92121).
Unicast, Multicast, or Broadcast Transmission and/or Reception
In some embodiments, data or data packets, e.g. as listed in Table 2, can be transmitted and/or received with unicast transmission and/or reception, for example between a first computing system or server and a second computing system or client; the second computing system can be configured to generate the stereoscopic or non-stereoscopic 2D or 3D display 1160 by the HMD or other augmented reality display system.
In some embodiments, a first unicast transmission can be transmitted from a first computing system and received by a second computing system, e.g. integrated into or connected to a first HMD or other augmented reality display system, with the first unicast transmission comprising the specific tracking information for the first HMD or other augmented reality display system and, optionally, instrument and/or surgical site tracking data. A second, third, fourth, fifth or more unicast transmission can be transmitted from a first computing system to a third, fourth, fifth, sixth or more computing system, e.g. integrated into or connected to a second, third, fourth, fifth or more HMD or other augmented reality display system, respectively. The second, third, fourth, fifth or more unicast transmission can be sequential, e.g. overlapping or non-overlapping.
A second unicast transmission can be transmitted from a first computing system and received by a third computing system, e.g. integrated into or connected to a second HMD or other augmented reality display system, with the second unicast transmission comprising the specific tracking information for the second HMD or other augmented reality display system and, optionally, instrument and/or surgical site tracking data.
A third unicast transmission can be transmitted from a first computing system and received by a fourth computing system, e.g. integrated into or connected to a third HMD or other augmented reality display system, with the third unicast transmission comprising the specific tracking information for the third HMD or other augmented reality display system and, optionally, instrument and/or surgical site tracking data.
A fourth unicast transmission can be transmitted from a first computing system and received by a fifth computing system, e.g. integrated into or connected to a fourth HMD or other augmented reality display system, with the fourth unicast transmission comprising the specific tracking information for the fourth HMD or other augmented reality display system and, optionally, instrument and/or surgical site tracking data.
A fifth unicast transmission can be transmitted from a first computing system and received by a sixth computing system, e.g. integrated into or connected to a fifth HMD or other augmented reality display system, with the fifth unicast transmission comprising the specific tracking information for the fifth HMD or other augmented reality display system and, optionally, instrument and/or surgical site tracking data.
Any number of unicast transmissions can be transmitted from a first computing system and received by a corresponding number of HMDs or other augmented reality display systems. If an overall transmission and reception rate of 30 Hz, 40 Hz, 50 Hz, 60 Hz, or 70 Hz is desired, the sequential unicast transmissions can be completed within 0.0333 sec, 0.025 sec, 0.02 sec, 0.0166 sec, 0.01428 sec, or any other value. The next round of sequential unicast transmissions and/or receptions can then start in order to achieve near real-time or real-time transmission and/or reception of specific tracking information for the different headsets, along with, optionally, instrument tracking and/or patient tracking data for stereoscopic and/or non-stereoscopic display of instrument and/or patient data by the one or more HMDs or other augmented reality display systems.
In some embodiments, data or data packets, e.g. as listed in Table 2, can be transmitted and/or received with multicast transmission and/or reception, for example between a first computing system or server and multiple clients, e.g. a second computing system, third computing system, fourth computing system, fifth computing system, etc., optionally each with one or more computer processors; the second computing system, third computing system, fourth computing system, fifth computing system, etc. can be configured to generate the stereoscopic or non-stereoscopic 2D or 3D display 1160 by the corresponding first, second, third, and fourth etc. HMDs or other augmented reality display systems.
In some embodiments, data or data packets, e.g. as listed in Table 2, can be transmitted and/or received with broadcast transmission and/or reception, for example between a first computing system or server and multiple clients (for example all available clients), e.g. a second computing system, third computing system, fourth computing system, fifth computing system, etc., optionally each with one or more computer processors; the second computing system, third computing system, fourth computing system, fifth computing system, etc. can be configured to generate the stereoscopic or non-stereoscopic 2D or 3D display 1160 by the corresponding first, second, third, and fourth etc. HMDs or other augmented reality display systems.
With multicast or broadcast transmission and/or reception, the position, orientation, position and orientation, direction of movement and/or tracking data for each HMD or other augmented reality display system can be labelled for each HMD or other augmented reality display system, e.g. corresponding to the HMD or other augmented reality display system number, for example label “1” for the first HMD or other augmented reality display system, label “2” for the second HMD or other augmented reality display system, label “3” for the third HMD or other augmented reality display system, label “4” for the fourth HMD or other augmented reality display system, label “5” for the fifth HMD or other augmented reality display system, etc. The second computing system, third computing system, fourth computing system, fifth computing system, etc. can be configured to generate the stereoscopic or non-stereoscopic 2D or 3D display 1160 for the corresponding first, second, third, and fourth etc. HMDs or other augmented reality display systems based on the labels corresponding to each HMD or other augmented reality display system and tracking data for each respective HMD or other augmented reality display system. For example, the second computing system can identify the label for the first HMD or other augmented reality display system, e.g. “1”, in the received data and generate the stereoscopic or non-stereoscopic 2D or 3D display for the first HMD or other augmented reality display system using the HMD or other augmented reality display system tracking data labeled for the first HMD or other augmented reality display system; the third computing system can identify the label for the second HMD or other augmented reality display system, e.g. “2”, in the received data and generate the stereoscopic or non-stereoscopic 2D or 3D display for the second HMD or other augmented reality display system using the HMD or other augmented reality display system tracking data labeled for the second HMD or other augmented reality display system; the fourth computing system can identify the label for the third HMD or other augmented reality display system, e.g. “3”, in the received data and generate the stereoscopic or non-stereoscopic 2D or 3D display for the third HMD or other augmented reality display system using the HMD or other augmented reality display system tracking data labeled for the third HMD or other augmented reality display system; the fifth computing system can identify the label for the fourth HMD or other augmented reality display system, e.g. “4”, in the received data and generate the stereoscopic or non-stereoscopic 2D or 3D display for the fourth HMD or other augmented reality display system using the HMD or other augmented reality display system tracking data labeled for the fourth HMD or other augmented reality display system; and so forth for any number of HMDs or other augmented reality display systems used. In this manner, each client or second, third, fourth, fifth etc. computing system, optionally with one or more computer processors, can generate the stereoscopic or non-stereoscopic 2D or 3D display or augmented view for each HMD or other augmented reality display system with the correct view angle and viewing perspective for each specific HMD or other augmented reality display system, for example in relationship to one or more tracked anatomic structures of the patient and/or one or more tracked physical tools, instruments and/or implants and/or one or more markers attached to a patient, e.g. a fiducial array attached to a bone. For example, each client or second, third, fourth, fifth etc. computing system, optionally with one or more computer processors, can generate the stereoscopic or non-stereoscopic 2D or 3D display for each HMD or other augmented reality display system with the correct view angle and viewing perspective for each specific HMD or other augmented reality display system for a virtual display, e.g. a virtual user interface or display of one or more interactions of a tracked physical surgical tool or instrument with a virtual user interface, for example in relationship to one or more tracked anatomic structures of the patient and/or one or more tracked physical tools, instruments and/or implants and/or one or more markers attached to a patient, e.g. a fiducial array attached to a bone. In this manner, a display of a virtual user interface or of one or more interactions therewith can, for example, be displayed in the display plane of the physical HMD unit, e.g. a waveguide display or mirror based display. In this manner, for example, a display of a virtual user interface or of one or more interactions therewith can be displayed in a predetermined display plane for a first, second, third, fourth, fifth and/or sixth HMDs, for example a display plane substantially parallel to the user's retina or a display plane substantially perpendicular to one or more pupillary axes of the user's eyes. In other embodiments, a display of a virtual user interface or of one or more interactions therewith can be displayed in a predetermined position and/or orientation for a first, second, third, fourth, fifth and/or sixth HMDs, for example a display plane at a predetermined position and/or orientation in relationship to a patient, a surgical site, one or more markers attached to the patient, e.g. a fiducial array attached to a bone, one or more markers attached to a structure in an operating room, e.g. an OR table, OR light etc.
In some embodiments, a network of HMDs or other augmented reality display systems can be used. One or more HMDs or other augmented reality display systems can comprise at least one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof integrated into or attached to the HMD or other augmented reality display system. The at least one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof integrated into or attached to the one or more HMDs or other augmented reality display systems can be used to generate coordinate and/or tracking information of one or more HMDs or other augmented reality display systems, a patient, an anatomic structure of a patient, one or more physical surgical tools, one or more physical surgical instruments, one or more robot (e.g. a robot with a robotic arm, a handheld robot, or a combination thereof) or a combination thereof.
Two or more of the HMDs or other augmented reality display systems can optionally interconnect and create a network, e.g. for a shared experience of the augmented views and/or for multi-directional generation of coordinate information and/or tracking information. The use of multi-directional generation of coordinate information and/or tracking information can be helpful to reduce or avoid line of sight issues. For example, when the line of sight is blocked for a first camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof integrated into or attached to a first HMD, the line of sight can be intact or maintained for a second, third, fourth, fifth, etc. or combination thereof camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof integrated into or attached to a first, second, third, fourth of fifth, etc. HMD.
The HMDs or other augmented reality display systems can be organized in a client-server network where multiple HMD clients can centralized around a single server, e.g. a first computing unit. Thus, HMD devices can be relieved of computing power when outsourcing tasks which are computational intensive (image processing) to the server. Moreover, battery life of the HMD's can be significantly prolonged which can make the approach attractive even in case of a single HMD client. The server can be accessible in the OR.
In case of multiple clients, different data inputs from the various perspectives (e.g. from a first, second, third, fourth, fifth etc. HMD) can be used by the server to increase the accuracy of the calculations (e.g. by averaging out errors). In some embodiments, coordinate information and/or tracking information, e.g. from spatial maps from two or more HMD clients, can be obtained and processed by the server. For example, spatial maps can consist of triangular meshes built from each HMD's depth sensor information. Once spatial maps have been transferred from a first, second, third, fourth, fifth or combination there of HMD to the server, the different meshes can be combined into a combined, more accurate mesh using, for example, an averaging algorithm: For example, the data from a first HMD can be used as the baseline. From each face in the baseline mesh, a ray can be cast along the surface normal of the face. Intersection points between the ray and all other meshes can be calculated. A new vertex for the combined mesh can be derived as the average of all intersection points along the ray. The new vertices from adjacent triangles in the baseline mesh can be connected to form the faces in the combined mesh. The combined mesh can then be transferred back to the individual HMD's for refinement of the registration, coordinate or tracking information and/or for refinement of the real-time or near real-time updating of the stereoscopic or non-stereoscopic HMD display, e.g. superimposed and/or aligned with an anatomic structure or anatomic landmark of a patient.
In some embodiments, once coordinate information, registration information, tracking information, surface information, e.g. of one or more HMDs or other augmented reality display systems, a patient, an anatomic structure of a patient, one or more physical surgical tools, one or more physical surgical instruments, one or more robot or a combination thereof has been obtained using at least one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof integrated into or attached to the HMDs or other augmented reality display systems and the information has been transferred from a first, second, third, fourth, fifth or combination there of HMD to the server, the data generated by the at least one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof integrated into or attached to the two or more HMDs or other augmented reality display systems can be combined, e.g. into a combined, more accurate surface or surface mesh using, for example, an averaging algorithm. Optionally, a weighting can be applied to the data transferred by different HMDs or other augmented reality display systems, e.g. with a higher weight for HMDs or other augmented reality display systems located closer to the patient and/or closer to the at least one camera, scanner, 3D scanner, LIDAR system, depth sensor, IMU or a combination thereof.
Intrinsic and/or Extrinsic Tracking of Surgical Robots
In some embodiments, surgical robots can comprise a robotic arm, a handheld robot, handheld portions, or a combination thereof. In some embodiments, surgical robots can comprise one or more sensor, camera, video system, scanner, e.g. 3D scanner, LIDAR system, depth sensor, controller, electric controller, mechanical controller, drive, actuator, end effector, attachment mechanism, potentiometer, inertial measurement unit, accelerometer, magnetometer, gyroscope, force sensor, pressure sensor, position sensor, orientation sensor, motion sensor, wire, step motor, electric motors, hydraulic motor, electric and/or mechanical actuator, switch, display unit, computer processor, or a combination thereof.
In some embodiments, coordinate information, tracking data or a combination thereof of one or more end effectors, physical tools or instruments integrated or attached to a or part of a robot and/or of one or more physical implants, physical implant components, or physical trial implants attached to a robot can be generated with use of pose data, sensor data, camera data, 3D scanner data, controller data, drive data, actuator data, end effector data or a combination thereof of the robot or one or more robot components, for example obtained using intrinsic or internal data generated by or including intrinsic or internal, integrated or attached sensors, potentiometers, cameras, video systems, 3D scanners, LIDAR systems, depth sensors, inertial measurement units, accelerometers, magnetometers, gyroscopes, force sensors, pressure sensors, position sensors, orientation sensors, motion sensors, position and/or orientation feedback from robot step motors, position and/or orientation feedback from robot electric motors, position and/or orientation feedback from robot hydraulic motors, position and/or orientation feedback from robot electric and/or mechanical actuators, position and/or orientation feedback from robot drives, position and/or orientation feedback from robotic controllers, position and/or orientation feedback from one or more robotic computer processors, or a combination thereof. If one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof is used for generating intrinsic or internal robot data, the data can optionally be corrected for any distance and/or angular offset between the one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof and an end effector, a surgical tool or instrument attached to or integrated into or part of the robot, e.g. a cutting tool, tissue removal tool (e.g. a drill, saw, reamer, impactor), or an ablation tool. Alternatively and/or additionally, If one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof is used for generating intrinsic or internal robot data, the data can optionally be corrected for any distance and/or angular offset between the one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof and an anatomic structure, surface and/or landmark of a patient. Any combination of offset, e.g. distance and/or angle, correction is possible. In some embodiments, one or more cameras, video systems, 3D scanners, LIDAR systems, depth sensors external to a robot (e.g. on a stand, in an OR light and/or one or more HMDs or other augmented reality display systems) can be used for determining the distance and/or angle offset.
In some embodiments, coordinate information, tracking data or a combination thereof of one or more end effectors, physical tools or instruments integrated or attached to or part of a robot and/or of one or more physical implants, physical implant components, or physical trial implants attached to a robot can be obtained or generated with use of one or more cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or combination thereof extrinsic or external to the robot and, for example, integrated or attached to one or more HMDs or other augmented reality display systems, separate from one or more HMDs or other augmented reality display systems (e.g. on a stand, tripod, attached to or integrated into OR lighting, OR fixtures, an imaging system (e.g. x-ray, cone beam CT, CT)), or a combination thereof. One or more computer processors can be configured, for example, using the one or more cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or combination thereof extrinsic or external to the robot, to determine the position, orientation, direction of movement, one or more coordinates, or combination thereof of at least a portion of the one or more end effectors, physical surgical tools, at least a portion of the robot, or a combination thereof (e.g. using image processing and/or pattern recognition and/or an artificial neural network) or of one or more markers, e.g. active markers (e.g. RF markers), passive markers (e.g. infrared markers), optical markers (e.g. with geometric patterns, QR codes, bar codes, defined shapes, e.g. triangles, squares, rectangles etc.), LEDs or a combination thereof integrated or attached to the one or more end effectors, physical tools or instruments, integrated or attached to at least portions of the robot, or a combination thereof (extrinsic or external data).
In some embodiments, one or more displays by one or more computer monitors or by one or more HMDs or other augmented reality display systems can be generated, wherein the display can be non-stereoscopic (e.g. by the computer monitor, other augmented reality display device(s) and/or the HMD) or stereoscopic (e.g. by the HMD).
In some embodiments, one or more computer processors can generate a display, e.g. by a computer monitor and/or one or more HMDs or other augmented reality display systems, of virtual data, e.g. a virtual surgical plan, one or more virtual surgical guides (e.g. a virtual axis, virtual plane, virtual gut guide) and/or one or more patient surface(s) using intrinsic or internal robot data, e.g. registration data, coordinate data, and/or tracking data of one or more HMDs or other augmented reality display systems, a patient, an anatomic structure of a patient, one or more physical surgical tools, one or more physical surgical instruments, one or more robot, or a combination thereof.
In some embodiments, one or more computer processors can generate a display, e.g. by a computer monitor and/or one or more HMDs or other augmented reality display systems, of virtual data, e.g. a virtual surgical plan, one or more virtual surgical guides (e.g. a virtual axis, virtual plane, virtual gut guide) and/or one or more patient surface(s) using extrinsic or external robot data, e.g. registration data, coordinate data, and/or tracking data of end effectors, one or more physical surgical tools, one or more physical surgical instruments, one or more robot, or a combination thereof.
In some embodiments, one or more computer processors can generate a display, e.g. by a computer monitor and/or one or more HMDs or other augmented reality display systems, of virtual data, e.g. a virtual surgical plan, one or more virtual surgical guides (e.g. a virtual axis, virtual plane, virtual gut guide) and/or one or more patient surface(s) using intrinsic or internal and extrinsic or external robot data, e.g. registration data, coordinate data, and/or tracking data of one or more HMDs or other augmented reality display systems, a patient, an anatomic structure of a patient, one or more physical surgical tools, one or more physical surgical instruments, one or more robot, or a combination thereof. In this example, intrinsic and or internal robot data can, optionally, be displayed using a different color or display pattern than extrinsic or external robot data, thereby highlighting potential differences and/or deviations. In some embodiments, one or more computer processors can be used to compute any differences and/or deviations between intrinsic or internal and extrinsic or external robot data, e.g. a difference in a projected instrument or tool path, e.g. a drill path, a saw path, a difference in a planned or executed tissue resection. One or more computer processors can be configured to generate a difference display, for example by a computer monitor and/or one or more HMDs or other augmented reality display systems, e.g. using color coding, line or bar charts or any other chart known in the art, and/or alphanumeric display. The difference between intrinsic or internal and extrinsic or external robot data, e.g. registration data, coordinate data, and/or tracking data of one or more HMDs or other augmented reality display systems, a patient, an anatomic structure of a patient, one or more physical surgical tools, one or more physical surgical instruments, one or more robot, or a combination thereof can be used to highlight any potential deviation of a robot from a predetermined plan, e.g. a predetermined tissue resection (for example a predetermined tissue resection volume, tissue resection area, tissue resection surface, bone cut, drilling, reaming, milling, impacting).
Intrinsic and/or Extrinsic Tracking of Imaging Systems and Imaging System Components
In some embodiments, an imaging system can comprise one or more imaging system components. In some embodiments, one or more imaging system components can comprise one or more sensor, camera, video system, scanner, e.g. 3D scanner, LIDAR system, depth sensor, controller, electric controller, mechanical controller, drive, actuator, end effector, attachment mechanism, potentiometer, inertial measurement unit, accelerometer, magnetometer, gyroscope, force sensor, pressure sensor, position sensor, orientation sensor, motion sensor, wire, step motor, electric motors, hydraulic motor, electric and/or mechanical actuator, switch, display unit, computer processor, or a combination thereof.
In some embodiments, coordinate information, tracking data or a combination thereof of one or more imaging system components can be generated with use of pose data, sensor data, camera data, 3D scanner data, controller data, drive data, actuator data, end effector data or a combination thereof of the one or more imaging system components, for example obtained using intrinsic or internal data generated by or including intrinsic or internal, integrated or attached sensors, potentiometers, cameras, video systems, 3D scanners, LIDAR systems, depth sensors, inertial measurement units, accelerometers, magnetometers, gyroscopes, force sensors, pressure sensors, position sensors, orientation sensors, motion sensors, position and/or orientation feedback from imaging system component step motors, position and/or orientation feedback from imaging system component electric motors, position and/or orientation feedback from imaging system component hydraulic motors, position and/or orientation feedback from system component electric and/or mechanical actuators, position and/or orientation feedback from imaging system component drives, position and/or orientation feedback from imaging system component controllers, position and/or orientation feedback from imaging system component computer processors, or a combination thereof. If one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof is used for generating intrinsic or internal imaging system component data, the data can optionally be corrected for any distance and/or angular offset between the one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof and one or more imaging system components. Alternatively and/or additionally, If one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof is used for generating intrinsic or internal imaging system component data, the data can optionally be corrected for any distance and/or angular offset between the one or more integrated or attached cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or a combination thereof and an anatomic structure, surface and/or landmark of a patient. Any combination of offset, e.g. distance and/or angle, correction is possible. In some embodiments, one or more cameras, video systems, 3D scanners, LIDAR systems, depth sensors external to one or more imaging system components (e.g. on a stand, in an OR light and/or one or more HMDs or other augmented reality display systems) can be used for determining the distance and/or angle offset.
In some embodiments, coordinate information, tracking data or a combination thereof of one or more imaging system components can be obtained or generated with use of one or more cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or combination thereof extrinsic or external to the imaging system components and, for example, integrated or attached to one or more HMDs or other augmented reality display systems, separate from one or more HMDs or other augmented reality display systems (e.g. on a stand, tripod, attached to or integrated into OR lighting, OR fixtures, an imaging system (e.g. x-ray, cone beam CT, CT)), or a combination thereof. One or more computer processors can be configured, for example, using the one or more cameras, video systems, 3D scanners, LIDAR systems, depth sensors, or combination thereof extrinsic or external to the one or more imaging system components, to determine the position, orientation, direction of movement, one or more coordinates, or combination thereof of at least a portion of the one or more imaging system components (e.g. using image processing and/or pattern recognition and/or an artificial neural network) or of one or more markers, e.g. active markers (e.g. RF markers), passive markers (e.g. infrared markers), optical markers (e.g. with geometric patterns, QR codes, bar codes, defined shapes, e.g. triangles, squares, rectangles etc.), LEDs or a combination thereof integrated or attached to the one or more imaging system components (extrinsic or external data).
In some embodiments, one or more displays by one or more computer monitors or by one or more HMDs or other augmented reality display systems can be generated, wherein the display can be non-stereoscopic (e.g. by the computer monitor, other augmented reality display device(s) and/or the HMD) or stereoscopic (e.g. by the HMD).
In some embodiments, one or more computer processors can generate a display, e.g. by a computer monitor and/or one or more HMDs or other augmented reality display systems, of virtual data, e.g. a virtual surgical plan, one or more virtual surgical guides (e.g. a virtual axis, virtual plane, virtual gut guide), one or more pre-operative or intra-operative imaging data and/or one or more patient surface(s) using intrinsic or internal imaging system data, e.g. registration data, coordinate data, and/or tracking data of one or more imaging system components.
In some embodiments, one or more computer processors can generate a display, e.g. by a computer monitor and/or one or more HMDs or other augmented reality display systems, of virtual data, e.g. a virtual surgical plan, one or more virtual surgical guides (e.g. a virtual axis, virtual plane, virtual gut guide) one or more pre-operative or intra-operative imaging data and/or one or more patient surface(s) using extrinsic or external tracking data, e.g. registration data, coordinate data, and/or tracking data of one or more HMDs or other augmented reality display systems, a patient, an anatomic structure of a patient, one or more physical surgical tools, one or more physical surgical instruments, one or more imaging system components, or a combination thereof.
In some embodiments, one or more computer processors can generate a display, e.g. by a computer monitor and/or one or more HMDs or other augmented reality display systems, of virtual data, e.g. a virtual surgical plan, one or more virtual surgical guides (e.g. a virtual axis, virtual plane, virtual gut guide) and/or one or more patient surface(s) using intrinsic or internal and extrinsic or external data, e.g. registration data, coordinate data, and/or tracking data of one or more HMDs or other augmented reality display systems, a patient, an anatomic structure of a patient, one or more physical surgical tools, one or more physical surgical instruments, one or more imaging system components, or a combination thereof. In this example, intrinsic and or internal imaging system data can, optionally, be displayed using a different color or display pattern than extrinsic or external imaging system data, thereby highlighting potential differences and/or deviations. In some embodiments, one or more computer processors can be used to compute any differences and/or deviations between intrinsic or internal and extrinsic or external imaging system data. One or more computer processors can be configured to generate a difference display, for example by a computer monitor and/or one or more HMDs or other augmented reality display systems, e.g. using color coding, line or bar charts or any other chart known in the art, and/or alphanumeric display.
Aspects of the disclosure relate to a system comprising at least one head mounted display or other augmented reality display device, at least one camera or scanning device, a first computing system comprising one or more computer processors and a second computing system comprising one or more computer processors,
In some embodiments, the one or more computer processors of the second computing system generate the 3D stereoscopic view for the view angle of the head mounted display or other augmented reality display device relative to the at least one anatomic structure of the patient using the real-time tracking information of the at least one head mounted display or other augmented reality display device.
In some embodiments, the real-time tracking information comprises tracking information of two or more head mounted display or other augmented reality display devices.
In some embodiments, the real-time tracking information comprises a head mounted display or other augmented reality display device specific label for each head mounted display or other augmented reality display device. In some embodiments, the real-time tracking information is labeled for each tracked head mounted display or other augmented reality display device.
In some embodiments, the real-time tracking information comprises tracking information of two or more head mounted display or other augmented reality display devices. In some embodiments, the two or more head mounted display or other augmented reality display devices are located in different locations. In some embodiments, the real-time tracking information comprises a head mounted display or other augmented reality display device specific label for each head mounted display or other augmented reality display device. In some embodiments, the real-time tracking information is labeled for each tracked head mounted display or other augmented reality display device.
In some embodiments, the one or more computer processors of the second computing system generate the 3D stereoscopic view for an interpupillary distance adjusted for a person wearing the head mounted display or other augmented reality display device.
In some embodiments, the second computing system is integrated with the at least one head mounted display or other augmented reality display device.
In some embodiments, the second computing system is separate from the at least one head mounted display or other augmented reality display device and is connected to the display unit of the at least one head mounted display or other augmented reality display device using at least one cable.
In some embodiments, the wireless transmission or reception or transmission and reception comprises a WiFi signal, a LiFi signal, a Bluetooth signal or a combination thereof.
In some embodiments, the camera or scanning device is separate from the at least one head mounted display or other augmented reality display device.
In some embodiments, the camera or scanning device is integrated or attached to the at least one head mounted display or other augmented reality display device.
In some embodiments, the wireless transmission comprises sending data packets comprising the real-time tracking information of the at least one head mounted display or other augmented reality display device, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, at a rate of 20 Hz or greater.
In some embodiments, the wireless reception comprises receiving data packets comprising the real-time tracking information of the at least one head mounted display or other augmented reality display device, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, at a rate of 20 Hz or greater.
In some embodiments, the system comprising a third computing system, wherein the third computing system is configured for wireless reception of the real-time tracking information from the first computing system and wherein the third computing system is configured for wireless transmission of the real-time tracking information to the second computing system.
In some embodiments, the third computing system comprises a chain of computing systems configured for wireless reception and wireless transmission of the real-time tracking information.
In some embodiments, the system comprises a third computing system, wherein the third computing system is connected to or integrated into a second head mounted display or other augmented reality display device, wherein the third computing system is configured for wireless reception of the real-time tracking information of the second head mounted display or other augmented reality display device, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, wherein the third computing system is configured to generate a 3D stereoscopic view by the second head mounted display or other augmented reality display device using the tracking information of the second head mounted display or other augmented reality display device.
In some embodiments, the tracking information of the second head mounted comprises a label specific to the second head mounted display or other augmented reality display device for identifying the tracking information of the second head mounted display or other augmented reality display device by the third computing system.
In some embodiments, the system comprising a fourth computing system, wherein the fourth computing system is connected to or integrated into a third head mounted display or other augmented reality display device, wherein the fourth computing system is configured for wireless reception of the real-time tracking information of the third head mounted display or other augmented reality display device, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, and wherein the fourth computing system is configured to generate a 3D stereoscopic view by the third head mounted display or other augmented reality display device using the tracking information of the third head mounted display or other augmented reality display device.
In some embodiments, the tracking information of the third head mounted comprises a label specific to the third head mounted display or other augmented reality display device for identifying the tracking information of the third head mounted display or other augmented reality display device by the fourth computing system.
In some embodiments, the system comprises a fifth computing system, wherein the fifth computing system is connected to or integrated into a fourth head mounted display or other augmented reality display device, wherein the fifth computing system is configured for wireless reception of the real-time tracking information of the fourth head mounted display or other augmented reality display device, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, and wherein the fifth computing system is configured to generate a 3D stereoscopic view by the fourth head mounted display or other augmented reality display device using the tracking information of the fourth head mounted display or other augmented reality display device.
In some embodiments, the tracking information of the fourth head mounted comprises a label specific to the fourth head mounted display or other augmented reality display device for identifying the tracking information of the fourth head mounted display or other augmented reality display device by the fifth computing system.
In some embodiments, the real-time tracking information comprises one or more coordinates.
In some embodiments, the one or more coordinates comprise coordinates of the at least one anatomic structure of the patient.
In some embodiments, the one or more coordinates comprise coordinates of the at least one physical surgical tool or physical surgical instrument.
In some embodiments, the one or more coordinates comprise coordinates of the at least one head mounted display or other augmented reality display device.
In some embodiments, the at least one head mounted display or other augmented reality display device comprises at least one optical see-through head mounted display or other augmented reality display device.
In some embodiments, the at least one head mounted display or other augmented reality display device comprises at least one video see-through head mounted display or other augmented reality display device.
In some embodiments, the at least one camera, the at least one scanning device or the at least one camera and the at least one scanning device comprises a laser scanner, a time-of-flight 3D laser scanner, a structured-light 3D scanner, a hand-held laser scanner, a LIDAR scanner, a time-of-flight camera, a depth camera, a video system, a stereoscopic camera system, a camera array, or a combination thereof.
In some embodiments, the system comprises at least one inertial measurement unit. In some embodiments, the at least one inertial measurement unit is integrated or attached to the at least one physical surgical tool or physical surgical instrument. In some embodiments, the at least one inertial measurement unit is integrated or attached to the at least one anatomic structure of the patient. In some embodiments, the at least one inertial measurement unit is integrated or attached to the at least one head mounted display or other augmented reality display device.
In some embodiments, the real-time tracking information of the at least one head mounted display or other augmented reality display device comprises information from the at least one inertial measurement unit.
Aspects of the disclosure relate to a system comprising two or more head mounted display or other augmented reality display devices, at least one camera or scanning device, a first computing system comprising one or more computer processors,
In some embodiments, a physical tool or instrument (see Table 2), e.g. a tracked pointer, a tracked stylus, a tracked tool, a tracked instrument or a combination thereof, can be used for interacting with a virtual interface display by an HMD. Any tracking technique known in the art can be used, e.g. inside-out tracking, outside-in tracking or a combination thereof, as described, for example in PCT International Application Serial Nos. PCT/US2017/021859, PCT/US2018/013774, PCT/US2019/61698 and PCT/US2019/015522, which are hereby incorporated in their entirety.
A tracked pointer, a tracked stylus, or another tracked tool or tracked instrument or a combination thereof can comprise one or more markers. In some embodiments, the marker can be configured to reflect or emit light with a wavelength between 380 nm and 700 nm or any value or range or subrange therebetween. In some embodiments, the marker can be configured to reflect or emit light with a wavelength greater than 700 nm. For example, the marker can be configured to reflect or emit light with a wavelength between 700 nm and 1 mm or any value or range or subrange therebetween. In some embodiments, the marker can be configured to reflect or emit light with a wavelength less than 380 nm. For example, the marker can be configured to reflect or emit light with a wavelength between 50 nm and 380 nm or any value or range or subrange therebetween. In some embodiments, the marker can be a radiofrequency marker, e.g. an active marker, an infrared marker, e.g. a retroreflective or passive marker. The marker can be an optical marker, e.g. an optical marker that comprises a geometric pattern. One, two or more markers can be attached to or integrated into a tracked pointer, a tracked stylus, other tracked tool, other tracked instrument or a combination thereof. A tracked pointer, a tracked stylus, other tracked tool, other tracked instrument or a combination thereof can also comprise one or more integrated or attached IMUs.
In some embodiments, the system comprises at least one camera, scanner (e.g. 3D scanner, laser scanner), LIDAR system, depth sensor, IMU or a combination thereof integrated into or attached to the head mounted display or other augmented reality display device. In some embodiments, at least one camera, scanner (e.g. 3D scanner, laser scanner), LIDAR system, depth sensor, IMU or a combination thereof can be separate from the head mounted display or other augmented reality display device. The at least one camera, scanner (e.g. 3D scanner, laser scanner), LIDAR system, depth sensor, IMU or a combination thereof integrated or attached to the one or more HMDs or other augmented reality display systems and/or separate from the one or more HMDs or other augmented reality display systems can be configured to scan and/or detect a pointer, stylus, tool, instrument or a combination thereof; the pointer, stylus, tool, instrument or a combination thereof can be tracked in 3D space, e.g. as they are being moved by a user. The tracking can be direct, e.g. by directly recognizing the instrument, for example utilizing a stored shape, surface, and/or 2D or 3D outline data of the pointer, stylus, tool, instrument or combination thereof or a library of shapes, surfaces, and/or 2D or 3D outline data of one or more pointer, stylus, tool, instrument or combination thereof, using one or more computer processors.
Direct Tracking of Tool and/or Instrument for Interaction with Virtual Interface
One or more computer processors can be configured to detect a physical tool or instrument, e.g. a tracked pointer, a tracked stylus, other tracked tool or tracked instrument, or a combination thereof, using a camera and/or scanner (e.g. a video camera, infrared camera, 3D scanner, laser scanner, LIDAR, imaging system etc.), e.g. in the image or scanner data, and, optionally, follow and/or track the pointer, stylus, tool, instrument, or combination thereof in real-time or near real-time, for example within the 3D space included in the image or scanner data. The one or more camera and/or scanner (e.g. a video camera, infrared camera, 3D scanner, laser scanner, LIDAR, imaging system etc.) can be integrated into or attached to one or more HMD units or can be separate from one or more HMD units or a combination thereof. Optionally, a pointer, stylus, tool, instrument, or combination thereof included in the image or scanner data can be compared against a database or library of stored shapes, surfaces, and/or 2D or 3D outline data of one or more (optionally different) pointer(s), stylus(s), other tool(s), other instrument(s) or combination thereof and can be identified using the database of stored shapes, surfaces, and/or 2D or 3D outline data of the one or more (optionally different) pointer(s), stylus(s), other tool(s), other instrument(s) or combination thereof. Identification of the pointer, stylus, tool, instrument, or combination thereof included in the image or scanner data can, optionally, facilitate tracking of the pointer, stylus, tool, instrument or combination thereof. The database can also comprise information about one or more optical markers, fiducials, fiducial arrays, and/or marker or array configurations.
In some embodiments, a pointer, a surgical tool or instrument can comprise a unique marker, fiducial array or marker/array configuration. In this manner, one or more computer processors can be configured to detect and/or identify the unique marker, fiducial array or marker/array configuration associated with that pointer, surgical tool or instrument. If a pointer, tool or instrument has been identified by at least one computer processor using any of the foregoing techniques, the at least one computer processor can associate and/or activate and/or display certain functions, e.g. display functions and/or system functions associated with that tool.
For example, if a pointer has been identified by the at least one computer processor, e.g. based on identification of its unique shape (e.g. relative to a database of tool shapes) or based on identification of a unique marker, fiducial array or marker/array configuration, the identification by the at least one computer processor can trigger or initiate a specific function. For example, during set-up of an augmented reality system for a surgical procedure, a calibration procedure can be activated. When a pointer (or other tool or instrument) is identified the system can, for example, automatically display, by an HMD or other augmented reality device, a virtual object, e.g. one associated with the calibration procedure. In this example of an AR display calibration, the virtual object can be moved by moving the tracked physical pointer. The movement of the virtual object can be corresponding to the movement of the tracked physical pointer, or it can be at a different movement ratio between movement of the virtual object and movement of the tracked physical pointer, e.g. 1:1, 1.5:1, 2.0:1, 0.5:1.0, or any other ratio, for example expressed in mm, cm, and/or angular degrees. By moving the tracked physical pointer so that the virtual object, e.g. a virtual marker ball, is superimposed and/or aligned with a corresponding physical object, e.g. a physical marker ball, the system can determine the coordinate difference and/or coordinate transfer and/or distance, angular movement required to superimpose the virtual object onto the physical object; the information can be used to move a virtual display, facilitated by at least one computer processor, as a means of optimizing the superimposition and/or alignment for a user between virtual display of any virtual objects (e.g. a virtual spine and/or virtual instrument and/or virtual implant) and the corresponding physical objects and/or structures (e.g. a corresponding physical spine and/or a corresponding physical instrument and/or a corresponding physical implant). The data/information related to the coordinate difference and/or coordinate transfer and/or distance and/or angular movement required to superimpose a virtual object onto a corresponding physical object can be used, by the at least one computer processor, as a means of AR system calibration to more closely match a virtual AR display, axis of an AR display, center of an AR display with the optical axis of the user's eye(s).
The data can optionally be wirelessly transmitted and/or received by a first computing system (e.g. communicably connected to a navigation system, a robot, and/or an imaging system) and a second (or more) computing system(s) (communicably connected to one or more HMDs or other augmented reality display devices).
In another example, a system can detect another unique marker, fiducial array or marker/array configuration associated, for example, with another instrument, e.g. an awl. The identification of the unique marker, fiducial array or marker/array configuration associated with the awl can trigger a display, by an HMD or other augmented reality display device, of a targeting tool for targeting the awl, e.g. superimposed onto a target anatomic structure of the patient.
One or more computer processors can be configured to detect and/or identify the tip and/or axis and/or direction of movement of a pointer, stylus, tool, instrument or combination thereof, for example by detecting a shape, contour and/or outline, optional identification of the stylus, tool, instrument or combination thereof used, and optional use of known shape data and/or dimensions of the stylus, tool, instrument or combination thereof.
The one or more computer processors configured to detect a pointer, stylus, other tool, other instrument, or combination thereof using a camera and/or scanner, e.g. in the image or scanner data, and, optionally, configured to follow and/or track the pointer, stylus, other tool, other instrument, or combination thereof in real-time or near real-time can be part of a first computing system, for example, a server or controller or computing unit as shown in
Tracking of Tool and/or Instrument Using Markers for Interaction with Virtual Interface
One or more computer processors can be configured to detect one or more markers integrated or attached to one or more physical tool or instrument, e.g. a tracked pointer, a tracked stylus, other tracked tool or tracked instrument, or a combination thereof, using a camera and/or scanner, e.g. a video camera, infrared camera, 3D scanner, laser scanner, LIDAR, imaging system etc., and, optionally, follow and/or track the one or more pointer, stylus, tool, or instrument in real-time or near real-time, using the one or more markers. The one or more camera and/or scanner (e.g. a video camera, infrared camera, 3D scanner, laser scanner, LIDAR, imaging system etc.) can be integrated into or attached to one or more HMD units or can be separate from one or more HMD units or a combination thereof.
One or more computer processors can be configured to detect and/or identify the tip and/or axis and/or direction of movement of a pointer, stylus, other tool, other instrument or combination thereof, using one or more integrated or attached markers.
The one or more computer processors configured to detect and/or track one or more markers integrated or attached to one or more pointer, stylus, other tool, other instrument, or combination thereof, using a camera and/or scanner in real-time or near real-time can be part of a first computing system, for example, a server or controller or computing unit as shown in
One or more HMDs or other augmented reality display systems can optionally generate a 2D or 3D stereoscopic or non-stereoscopic virtual display or augmented view comprising a virtual interface, e.g. superimposed on a physical patient, physical anatomic structure, physical anatomic landmark and/or physical anatomic surface and/or near or adjacent to a physical patient, physical anatomic structure, physical anatomic landmark and/or physical anatomic surface. One or more computer processors, e.g. in a first (e.g. a server) or second (e.g. a client) computing system can be configured to generate a 2D or 3D stereoscopic or non-stereoscopic virtual display comprising a virtual interface at a predetermined location and/or orientation relative to one or more anatomic structures and/or the patient. One or more computer processors, e.g. in a first (e.g. a server) or second (e.g. a client) computing system can be configured to generate a 2D or 3D stereoscopic or non-stereoscopic virtual display comprising a virtual interface, for example at a predetermined location and/or orientation relative to one or more markers (e.g. infrared markers, radiofrequency markers, active markers, passive markers, optical markers [e.g. with geometric patterns, bar codes, QR codes etc.], LED's etc.) attached to one or more anatomic structures and/or the patient and/or a fixed structure in the operating room. The one or more markers can, for example, be a fiducial array attached to one or more bones. One or more computer processors, e.g. in a first (e.g. a server) or second (e.g. a client) computing system can be configured to generate a 2D or 3D stereoscopic or non-stereoscopic virtual display comprising a virtual interface at a predetermined location and/or orientation relative to one or more structures in the operating room, e.g. an OR table, an OR light, an external computer monitor etc. One or more computer processors, e.g. in a first (e.g. a server) or second (e.g. a client) computing system can be configured to generate a 2D or 3D stereoscopic or non-stereoscopic virtual display comprising a virtual interface at a predetermined location and/or orientation relative to the user's eyes and/or face and/or relative to the physical HMD or other augmented reality display unit, e.g. the housing of the HMD unit or the physical display (e.g. combiner, waveguide and/or mirror) of the HMD.
A 2D or 3D stereoscopic or non-stereoscopic virtual display comprising a virtual interface displayed by one or more HMDs or other augmented reality display systems can comprise, for example, one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof. The one or more one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof can be displayed, by the one or more HMDs or other augmented reality display systems, using one or more computer processors, in 2D, in 3D, or a combination thereof. For example, a virtual slider can be in 2D and/or in 3D. A 3D virtual slider can, for example, comprise an activation or sliding field oriented in x-direction, an activation or sliding field oriented in y-direction, and an activation or sliding field oriented in z-direction. The system can be configured for detecting various interactions by a user with the one or more virtual objects of a virtual interface, for example an interaction via gesture recognition, gaze recognition, gaze lock, eye tracking, hand tracking, pointer tracking, instrument tracking, tool tracking, or a combination thereof. For example, a tracked finger, tracked hand, tracked pointer, tracked instrument, tracked tool can interact with the virtual interface. The interaction can trigger an event message, optionally managed by an event handler, and/or a command. The interaction, event message, command or combination thereof can optionally be transmitted and/or received between a first and a second computing system, for example a first computing system (e.g. communicably connected to a navigation system, a robot, and/or an imaging system) and a second (or more) computing system(s) (communicably connected to one or more HMDs or other augmented reality display devices.
In some embodiments, the system can comprise a collision detection module or other interaction module. The collision detection module or other interaction module can be a module separate from other modules, such as an AR visualization module 1150 or an AR display module 1160. The collision detection module or other interaction module can be part of another module, e.g. a submodule of another module, such as an AR visualization module 1150 or an AR display module 1160.
One or more of the computer processors operating a collision detection module or other interaction module, one or more computer processors operating a tracking engine or tracking module 1100, one or more computer processors operating an instrument calibration module 1110, one or more of the computer processors operating a headset calibration module 1120, one or more of the computer processors operating an imaging and navigation module 1130, one or more of the computer processors operating an AR wireless networking module 1140, one or more of the computer processors operating an AR visualization module 1150, and/or one or more of the computer processors operating an AR display module 1160 can be the same.
One or more of the computer processors operating a collision detection module or other interaction module, one or more computer processors operating a tracking engine or tracking module 1100, one or more computer processors operating an instrument calibration module 1110, one or more of the computer processors operating a headset calibration module 1120, one or more of the computer processors operating an imaging and navigation module 1130, one or more of the computer processors operating an AR wireless networking module 1140, one or more of the computer processors operating an AR visualization module 1150, and/or one or more of the computer processors operating an AR display module 1160 can be different.
In some embodiments, one or more HMDs or other augmented reality display systems and one or more physical tools or physical instruments, e.g. a pointer, a stylus, other tools, other instruments, can be tracked, e.g. using inside-out or outside-in tracking. The coordinates, position and/or orientation of a virtual display comprising a virtual interface displayed by one or more HMDs or other augmented reality display systems can also be tracked. One or more computer processors can be configured, using one or more collision detection modules, to detect collisions between a gaze (e.g. using gaze tracking, gaze lock), a finger (e.g. using finger/hand tracking), a hand (e.g. using hand tracking), an eye (e.g. using eye tracking), one or more tracked physical tools or physical instruments, e.g. a tracked pointer, a tracked stylus, other tracked physical tools, other tracked physical instruments, or a combination thereof and a virtual display comprising the virtual interface, e.g. one or more virtual objects such as virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof.
One or more computer processors can use polygon based collision detection or detection of other interactions. One or more computer processors can use volume based collision detection or detection of other interactions. One or more computer processors can be configured with a predetermined tolerance for a collision detection or detection of other interactions, e.g. <0.1, <0.5, <1.0, <1.5, <2.0, <3.0, <4.0, <5.0, <10.0 mm, or any other value, and/or <0.1, <0.5, <1.0, <1.5, <2.0, <3.0, <4.0, <5.0, <10.0, <15.0, <20.0 degrees, or any other value. In some embodiments, the tolerance for a collision detection or detection of other interactions can be selected and/or predetermined, for example, to enable a particular application, e.g. activating or executing a command, moving a virtual slider, selecting a virtual button, etc. The tolerance for a collision detection or detection of other interactions can be the same or different for different applications, e.g. an HMD calibration, an instrument calibration, an AR visualization module, e.g. comprising selection of a predetermined path for a physical tool or instrument.
Different collision detection modules or packages known in the art include, for example I-collide [Cohen et al. “I—COLLIDE: An Interactive and Exact Collision Detection System for Large-Scale Environments,” in The 1995 ACM International 3D Graphics Conference], V-clip [Mirtich et al. “V-Clip: Fast and Robust Polyhedral Collision Detection,” ACM Trans. Graphics, 17, 3, pp. 177-208], SWIFT [Ehrmann et al. “SWIFT: Accelerated Proximity Queries Between Convex Polyhedra by Multi-Level Voronoi Marching,” Technical report, Computer Science Department, University of North Carolina at Chapel Hill], RAPID [Gottschalk et al. OBB-Tree: A Hierarchical Structure for Rapid Interference Detection,” Computer Graphics SIGGRAPH '96 Proceedings 30, pp. 171-180], V-collide [Hudson et al. “V-COLLIDE: Accelerated Collision Detection for VRML”, in Proceedings of the Second Symposium on Virtual Reality Modeling Language. California, United States, ACM Press], PQP [Larsen et al. “Fast Proximity Queries With Swept Sphere Volumes” Technical Report TR99-018, Department of Computer Science, University of North Carolina. SOLID [Bergen et al. “User's Guide to the SOLID Interference Detection Library”; SWIFT [Ehrmann et al. “Accurate and Fast Proximity Queries Between Polyhedra Using Surface Decomposition”, Eurographics. Computer Graphics Forum, 20, 3, or VPS [McNeely et al. “Six Degree-of-Freedom Haptic Rendering Using Voxel Sampling”, SIGGRAPH 99 Conference Proceedings, Annual Conference Series, pp. 401-408].
In some embodiments, a collision detection module, e.g. I-collide, can utilize convex polyhedra for multi-body collision detection. In some embodiments, a collision detection module such as RAPID can utilize non-convex models, detecting, for example, detects pair-wise collisions. Some collision detection modules, e.g. V-collide, can be configured to detect multiple body collisions.
Some collision detection modules, e.g. PQP, can support non-convex modes and/or can optionally perform distance computation and/or tolerance verification queries. Some collision detection modules, e.g. SWIFT, can comprise intersection detection, tolerance verification, exact and approximate distance computation, contact determination, or a combination thereof. Some collision detection methods, e.g. I-collide, RAPID, PQP and/or SWIFT, can be based on polygon intersection. Some collision detection packages, e.g. VPS, can utilize voxels and can, for example, detect collisions, perform tolerance verification, approximate distances, and determine contact normal, center of mass, or a combination thereof.
One or more computer processors can be configured to utilize sweep-based continuous collision detection. Sweep-based continuous collision detection can use a Time Of Impact (TOI) algorithm to compute potential collisions, e.g. for a gaze (e.g. using gaze tracking, gaze lock), a finger (e.g. using finger/hand tracking), a hand (e.g. using hand tracking), an eye (e.g. using eye tracking), for one or more tracked physical pointer, tracked physical tool, tracked physical instrument, or a combination thereof by sweeping its forward trajectory using its current velocity based on the tracking data. If there are contacts with the virtual display, comprising, for example a virtual interface, e.g. along the moving direction of the one or more tracked physical pointer, tracked physical tool, tracked physical instrument, or a combination thereof, one or more computer processors can be configured to detect the collision. In some embodiments, one or more computer processors can be configured to compute a time of impact, e.g. for a given moving direction and/or speed of a tracked physical pointer, physical tool, tracked physical instrument, or combination thereof. The one or more computer processors can perform sub steps from that time onwards, computing the velocity after TOI then re-sweep.
One or more computer processors can be configured to utilize speculative continuous collision detection. Speculative continuous collision detection can operate by increasing a broad-phase axis-aligned minimum bounding box of a tracked physical surgical tool, tracked physical instrument or combination thereof, based on the linear and angular motion of the tracked physical surgical tool, tracked physical instrument or combination thereof. The algorithm can be speculative since it can pick all potential contacts during the next physical step. The contacts can then be fed into a solving program operated by one or more computer processors, which can ensure that applicable or predetermined contact constraints can be satisfied. One or more computer processors, e.g. on a first computing unit (e.g. a server) and/or a second computing unit (e.g. a client integrated or connected to an HMD or other augmented reality display device) can operate commercially available software with one or more integrated collision detection modules or programs, e.g. Unity software (Unity Software, Inc.).
Commands and/or Executable Actions Triggered Using Virtual Interface
In some embodiments, a first computing unit, e.g. a server or controller, can comprise a collision detection module or a module for detection of other interactions (e.g. software program). In some embodiments, a second computing unit, e.g. a mobile client (for example communicably connected to or part of one or more HMD or other augmented reality display units) can comprise a collision detection module or a module for detection of other interactions, program or software. In some embodiments, for example when a system comprises multiple HMDs or other augmented reality display systems, a second, third, fourth, fifth, sixth, etc. computing unit, e.g. a second, third, fourth, fifth, sixth, etc. wireless, mobile client, can comprise a collision detection module or a module for detection of other interactions, program or software, for example a collision detection module or a module for detection of other interactions, program or software for each HMD unit and associated computing unit. In some embodiments, one or more computer processors of a first computing unit and a second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. computing unit can operate the same collision detection module or module for detection of other interactions. In some embodiments, one or more computer processors of a first computing unit and a second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. computing unit can operate different collision detection modules or modules for detection of other interactions.
In some embodiments, one or more computer processors of a first computing unit and a second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. computing unit can operate the same collision detection module or module for detection of other interactions, which can be used for the same functions and/or interactions and/or commands of/with a virtual object displayed as part of a virtual interface. In some embodiments, one or more computer processors of a first computing unit and a second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. computing unit can operate the same collision detection module or module for detection of other interactions, which can be used for the different functions and/or interactions and/or commands of/with a virtual interface.
In some embodiments, one or more data packets, for example as described in Table 2, e.g. tracking data of one or more HMDs or other augmented reality display systems, one or more virtual displays, one or more physical pointers, physical tools, or physical instruments, one or more physical implants, one or more robots or robotic arms, can be transmitted from a first computing system wirelessly to a second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. computing system, e.g. a second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. wireless, mobile client integrated into or connected to a first, and/or second, and/or third, and/or fourth, and/or fifth, etc. HMD or other augmented reality display unit. One or more computer processors of the second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. wireless, mobile client can operate one or more collision detection modules, e.g. using Unity software (Unity Software, Inc., 30 3rd Street, San Francisco, CA 94103), to detect collisions of one or more tracked physical tools or instruments, e.g. a physical pointer or a physical stylus or other physical tool or instrument, with a virtual interface displayed by the one or more HMDs or other augmented reality display systems; the collision(s) of the tracked physical tool or instrument, or a gaze (e.g. using gaze tracking, gaze lock), a finger (e.g. using finger/hand tracking), a hand (e.g. using hand tracking), an eye (e.g. using eye tracking) with different portions, aspects, fields or displays of the virtual interface, e.g. a virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof, can be used to trigger one or more actions and/or one or more commands, which can, optionally be transmitted from the second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. wireless, mobile client and be received by the first computing unit for further processing, e.g. execution of commands by one or more computer processors.
In some embodiments, coordinate and/or tracking data of one or more HMDs or other augmented reality display systems and/or one or more virtual displays by one or more HMDs or other augmented reality display systems (see Table 2) can be received by a first computing unit, along with tracking data of one or more physical tools or physical instruments, one or more physical implants, one or more robots or robotic arms. The first computing unit or system can operate one or more collision detection modules to detect collisions of one or more tracked physical tools or instruments, e.g. a physical pointer or a physical stylus or other physical tool or instrument, with the virtual display, e.g. a virtual interface, displayed by the one or more HMDs or other augmented reality display systems. The collision(s) of the tracked physical tool or instrument with different portions, aspects, fields or displays of the virtual interface, e.g. a virtual object such as a virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof, can be used to trigger one or more actions and/or one or more commands, which can be processed by one or more computer processors of the first computing system and which can, optionally be transmitted to a second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. wireless, mobile client connected to or integrated into a first, second, third, fourth, and/or fifth etc. HMD or other augmented reality display device, optionally triggering commands and/or actions by one or more computer processors of the second, and/or third, and/or fourth, and/or fifth, and/or sixth, etc. wireless, mobile client(s) and one or more connected HMDs or other augmented reality display systems.
In some embodiments, a physical tool or instrument (see Table 2), e.g. a tracked pointer, a tracked stylus, a tracked tool, a tracked instrument or a combination thereof, can be used for interacting with a virtual interface display by an HMD or other augmented reality display device. In some embodiments, a gaze (e.g. using gaze tracking, gaze lock), a finger (e.g. using finger/hand tracking), a hand (e.g. using hand tracking), an eye (e.g. using eye tracking) or a combination thereof can be used for interacting with a virtual interface display by an HMD or other augmented reality display device. In some embodiments, a collision or other interaction, e.g. of a tracked physical tool or instrument, with a virtual display, e.g. a virtual object in a virtual user interface, displayed by a first HMD can be detected by one or more computer processors in a first computing system, e.g. a server, and/or a second computing system, e.g. a client integrated, attached to or connected to the first HMD (and/or optionally a second, third, fourth, fifth, sixth or more HMD). The collision and/or other interaction can optionally be used to execute a function and/or to change the appearance of the virtual display, e.g. virtual interface, e.g. with a color change, display of different buttons and/or functions, etc.
Data and/or execution functions related to or triggered by a collision and/or other interaction of a tracked physical tool or instrument with a virtual display, e.g. a virtual interface, and/or changes in a virtual display, e.g. a virtual interface, triggered by the collision or other interaction with the tracked physical tool or instrument for display by one or more HMDs or other augmented reality display systems can be generated by one or more computer processors in a first computing unit, e.g. a server, and can be transmitted to one or more additional computing units, for example a second, third, fourth, fifth or more computing unit, e.g. a client integrated, attached to or connected to one or more HMDs or other augmented reality display systems.
Data and/or execution functions related to or triggered by a collision and/or other interaction of a tracked physical tool or instrument with a virtual display, e.g. a virtual interface, and/or changes in a virtual display, e.g. a virtual interface, triggered by the collision or other interaction with the tracked physical tool or instrument for display by one or more HMDs or other augmented reality display systems can be generated by one or more computer processors in a second (or first) computing unit, e.g. a client integrated, attached to or connected to a first HMD, and can be transmitted to a first (or second) computing unit, e.g. a server (e.g. separate from the one or more HMDs or other augmented reality display systems). One or more computer processors in the second (or first) computing unit, e.g. the server, can be configured to process and/or transmit the data and/or execution functions related to or triggered by the collision and/or other interaction of the tracked physical tool or instrument with the virtual display, e.g. a virtual interface, and/or changes in the virtual display, e.g. a virtual interface, triggered by the collision or other interaction with the tracked physical tool or instrument to one or more additional computing units, for example a second, third, fourth, fifth or more computing unit, e.g. one or more clients integrated, attached to or connected to one or more additional HMDs or other augmented reality display systems for display by the one or more additional HMDs or other augmented reality display systems.
A first computer system CS #1 1310 can, for example, reside in a server or computer, for example located in the operating room. The one or more first computer systems CS #1 1310 can also be located in a remote location, e.g. outside the operating room, and/or can comprise a cloud computing system and/or communicate through a cloud computing system, e.g. through a wired or a wireless connection. One or more second CS #2 1320, third CS #3 1330, or more computer systems can be integrated into, connected to or attached to one or more head mounted displays HMD A 1360 and/or HMD B 1370 or other augmented reality display devices. The tracking system 1300 can be separate from the one or more head mounted displays HMD A 1360 and/or HMD B 1370 or other augmented reality display devices. In some embodiments, the tracking system 1300 can be integrated or attached to the one or more head mounted displays HMD A 1360 and/or HMD B 1370 or other augmented reality display devices.
The tracking system 1300 can be configured to track 1410 optionally a hand and/or finger 1380, a pointer, tool, instrument and/or implant 1390, optionally a first 1360, second 1370, third, fourth, fifth etc. head mounted display, a patient (not shown), and/or a robot 1340 and/or an imaging system 1350 or a combination thereof, for example with use of the first computer system CS #1 1310 and/or optionally with use of a second computer system CS #2 1320, a third computer CS #3 1330, and/or additional computer systems.
The tracking system 1300 can be configured to track 1410 optionally a hand and/or finger 1380, a pointer, tool, instrument and/or implant 1390, optionally a first 1360, second 1370, third, fourth, fifth etc. head mounted display, a patient (not shown), and/or a robot 1340 and/or an imaging system 1350 or a combination thereof, and to transmit the tracking information 1415 to a first computer system CS #1 1310 and/or optionally a second computer system CS #2 1320, a third computer CS #3 1330, and/or additional computer systems.
The second CS #2 1320, third CS #3 1330, and/or additional computer systems can be integrated into, connected to or attached to one or more HMDs 13601370 or other augmented reality display systems. The one or more additional computer systems can be attached to, connected to or located inside a robotic system 1340 and/or an imaging system 1350. The tracking 1410 can comprise recording one or more coordinates of and/or tracking a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or a first 1360, second 1370, third, fourth, fifth etc. head mounted display or other augmented reality display device, and/or a patient (not shown) by the first computer system CS #1 1310 and/or the second computer system CS #2 1320, and/or the third computer CS #3 1330, and/or additional computer systems, e.g. in a robot 1340 or imaging system 1350, for example using the camera or scanner 1300. The tracking 1410 can comprise recording one or more coordinates of a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or a first 1360, second 1370, third, fourth, fifth etc. head mounted display, and/or the patient (not shown) by the first computer system CS #1 1310 and/or the second computer system CS #2 1320, and/or the third computer CS #3 1330, and/or additional computer systems, e.g. in a robot 1340 or imaging system 1350, for example using the camera or scanner 1300. The system can track a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or a first 1360, second 1370, third, fourth, fifth etc. head mounted display, and/or the patient (not shown) by the first computer system CS #1 1310 and/or the second computer system CS #2 1320, and/or the third computer CS #3 1330, and/or additional computer systems, e.g. in a robot 1340 or imaging system 1350, for example using the camera or scanner 1300.
One or more computer processors of the one or more first 1310, second 1320, third 1330 etc. computer systems, can be configured to perform one or more coordinate transformations 1420 and/or to determine a pose 1420 of the hand or finger 1380, the one or more pointer, tool, instrument, and/or implant 1390, the first 1360, second 1370, third, fourth, fifth etc. head mounted display, and/or the patient (not shown).
The one or more computer processors of the one or more first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to transmit and/or receive 1430 information about the finger or hand 1380, or the one or more pointer, tool, instrument, and/or implant 1390, e.g. their pose, information about or for the first 1360, second 1370, third, fourth, fifth etc. head mounted display or other augmented reality display devices, e.g. the pose of the head mounted display(s) or other augmented reality display devices or display data for the head mounted display(s) 13601370 or other augmented reality display devices, and/or the patient (not shown), for example using Bluetooth or a WiFi or LiFi wireless access point, e.g. including a transmitter and/or receiver, or a wired connection. The transmission and/or reception 1430 can comprise data about gaze direction of the wearer of a headset, e.g. a first 1360, second 1370, third, fourth, fifth etc. head mounted display. The transmission and/or reception 1430 can be wireless, e.g. using a broadcast or multiple unicast transmissions to and/or from the first 1360, second 1370, third, fourth, fifth etc. head mounted display, as described in the specification. The transmission can comprise positional data, display data and/or other data. The transmission 1430 can be to or from a first computer system 1310 to or from a second computer system 1320, to or from a third computer system 1330, to or from a fourth, fifth or more computer system. The transmission can be to or from a second computer system 1320 to or from a first computer system 1310, to or from a third computer system 1330, to or from a fourth, fifth or more computer system. The transmission can be to or from a third computer system 1330 to or from a first computer system 1310, to or from a second computer system 1320, to or from a fourth, fifth or more computer system, and so forth.
In any of the embodiments throughout the specification, the transmission 1430 can be unidirectional and/or bi-directional, simultaneous uni-directional and/or bi-directional, and/or non-simultaneous, e.g. sequential uni-directional and/or bi-directional, or a combination thereof between the one or more first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also integrated into or attached to a robot and/or an imaging system.
In any of the embodiments throughout the specification, a second 1320, third 1330 etc. or additional computer systems, can be integrated into, connected to, and/or attached to and/or separate from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1340, and/or one or more imaging systems 1350. One or more computer systems 1310 can be separate from, e.g. standalone, from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1340, and/or one or more imaging systems 1350.
One or more computer processors of one or more first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to generate one or more user interface controls 1440. The one or more user interface controls 1440 can, for example, be virtual interface controls. The one or more user interface controls 1440 can comprise, for example, one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof.
One or more computer processors of the one or more first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to perform one or more collision detections 1445, for example between a hand or finger 1380 and a virtual interface displayed by one or more head mounted displays 13601370 or other augmented reality display devices, or a tracked physical pointer, physical tool, physical instrument, physical implant component 1390 or a combination thereof and a virtual interface displayed by one or more head mounted displays 13601370 or other augmented reality display devices; the collision detection can be performed, for example, using the tracking system 1300 (or a camera or scanner integrated or attached to an HMD or other augmented reality display devices). One or more computer processors of the one or more first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to perform a gesture recognition, for example between a hand or finger 1380 and a virtual interface using one or more cameras, optionally integrated or attached to one or more head mounted displays 13601370. The first computer system 1310 can, for example, be a standalone computer system or a cloud computing system. The second 1320, third 1330 etc. or additional computer systems, can be, for example, integrated into, connected to, and/or attached to and/or separate from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1340, and/or one or more imaging systems 1350.
One or more computer systems, e.g. a first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to generate and/or transmit, optionally wirelessly, one or more event message 1450. The one or more event message can, for example, be an event message relative to a collision detection 1445 between a hand or finger 1380, a tracked physical pointer, physical tool, physical instrument, physical implant component 1390 or a combination thereof and a virtual interface, e.g. using the tracking system 1300. The one or more event message 1450 can optionally comprise an event message related to a robot 1340 control, interface, force, and/or position and/or orientation and/or an imaging system 1350 control, interface, force, and/or position and/or orientation.
One or more computer systems, e.g. a first 1310, second 1320, third 1330, fourth, fifth etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can comprise an optional event handler 1460 configured to handle, manage and/or process one or more optional event message 1450.
One or more computer systems, e.g. a first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to initiate and/or process an optional event action 1470, e.g. executing a command, e.g. based on the event message 1450 and, optionally, information or data received from the event handler 1460. The command can be transmitted through a wireless or a wired connection, e.g. to a robot 1340 or an imaging system 1350.
In some embodiments, the one or more computer systems configured to generate an event message 1450 can be the same or different computer system comprising the event handler 1460 and/or can be the same or different computer system initiating and/or processing the event action 1470, e.g. executing a command. In some embodiments, one or more computer processors configured to generate an event message 1450 can be the same or different computer processors comprising the event handler 1460 and/or can be the same or different computer processors initiating and/or processing the event action 1470, e.g. executing a command. In some embodiments, a first, second, third, fourth, fifth, sixth or more computer system can be the same for managing different tasks, or can be different for managing different tasks.
A first, optional computer system CS #1 1310 can, for example, reside in a server or computer, for example located in the operating room. The one or more optional first computer systems CS #1 1310 can also be located in a remote location, e.g. outside the operating room, and/or can comprise a cloud computing system and/or communicate through a cloud computing system, e.g. through a wired or a wireless connection. One or more second CS #2 1320, third CS #3 1330, or more computer systems can be integrated into, connected to or attached to one or more head mounted displays HMD A 1360 and/or HMD B 1370 or other augmented reality display devices.
The one or more tracking system(s) 1300 can be integrated or attached to the one or more head mounted displays HMD A 1360 and/or HMD B 1370 or other augmented reality display devices.
The one or more tracking systems 1300 can be configured to track 1410 optionally a hand and/or finger 1380, e.g. of a user, a pointer, tool, instrument and/or implant 1390, a patient (not shown), and/or a robot 1340 and/or an imaging system 1350 or a combination thereof, for example with use of a second computer system CS #2 1320, a third computer CS #3 1330, and/or additional computer systems. An optional first computer system CS #1 1310 can also be used for tracking, including for processing tracking information.
The tracking system 1300 can be configured to track 1410 optionally a hand and/or finger 1380, a pointer, tool, instrument and/or implant 1390, a patient (not shown), and/or a robot 1340 and/or an imaging system 1350 or a combination thereof, and to transmit the tracking information 1415 to a second computer system CS #2 1320, a third computer CS #3 1330, and/or additional computer systems; the transmission can be wired or wireless. The one or more tracking systems 1300 can be integrated into or attached to one or more head mounted displays 13601370 or other augmented reality display devices. The one or more computer systems 13201330, including one or more computer processors, can be integrated into or connected to the one or more head mounted displays 13601370 or other augmented reality display devices. The second CS #2 1320, third CS #3 1330, and/or additional computer systems can be integrated into, connected to or attached to one or more HMDs 13601370 or other augmented reality display systems. The one or more additional computer systems can be attached to, connected to or located inside a robotic system 1340 and/or an imaging system 1350.
The tracking 1410 can comprise recording one or more coordinates of and/or tracking a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or a patient (not shown) by a second computer system CS #2 1320, and/or a third computer CS #3 1330, and/or additional computer systems, e.g. in a robot 1340 or imaging system 1350, for example using the camera(s) or scanner(s) 1300. The tracking 1410 can comprise recording one or more coordinates of a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or the patient (not shown) by the second computer system CS #2 1320, and/or the third computer CS #3 1330, and/or additional computer systems, e.g. in a robot 1340 or imaging system 1350, for example using the camera or scanner 1300. The system can track a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or the patient (not shown) by the second computer system CS #2 1320, and/or the third computer CS #3 1330, and/or additional computer systems, e.g. in a robot 1340 or imaging system 1350, for example using tracking system, for example with a camera or scanner 1300.
One or more computer processors of the one or more first 1310, second 1320, third 1330 etc. computer systems, can be configured to perform one or more coordinate transformations and/or to determine a pose 1420 of the hand or finger 1380, the one or more pointer, tool, instrument, and/or implant 1390, and/or optionally the first 1360, second 1370, third, fourth, fifth etc. head mounted display or other augmented reality display device (e.g. in relationship to a marker attached to a patient), and/or the patient (not shown).
The one or more computer processors of the one or more first, optional, 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to transmit and/or receive 1430 information about the finger or hand 1380, or the one or more pointer, tool, instrument, and/or implant 1390, e.g. their pose, and/or optionally information about or for the first 1360, second 1370, third, fourth, fifth etc. head mounted display, e.g. the pose of the head mounted display(s) (or other augmented reality display device) (e.g. relative to a marker on a patient) or display data for the head mounted display(s) 13601370 or other augmented reality display devices, and/or the patient (not shown), for example using a wireless connection, e.g. Bluetooth or a wireless WiFi or LiFi access point, optionally including a transmitter and/or receiver, or using a wired connection. The transmission and/or reception 1430 can comprise data about gaze direction of the wearer of a headset (for example for use with a gaze cursor) (or data about gaze direction of the user of another augmented reality display devices), e.g. a first 1360, second 1370, third, fourth, fifth etc. head mounted display. The transmission and/or reception 1430 can be wired or wireless, e.g. using a broadcast or multiple unicast transmissions to and/or from the first 1360, second 1370, third, fourth, fifth etc. head mounted display, as described in the specification. The transmission can comprise positional data, display data and/or other data. The transmission 1430 can be to or from an optional first computer system 1310 to or from a second computer system 1320, to or from a third computer system 1330, to or from a fourth, fifth or more computer system. The transmission can be to or from a second computer system 1320 to or from an optional first computer system 1310, to or from a third computer system 1330, to or from a fourth, fifth or more computer system. The transmission can be to or from a third computer system 1330 to or from an optional first computer system 1310, to or from a second computer system 1320, to or from a fourth, fifth or more computer system, and so forth.
In any of the embodiments throughout the specification, the transmission 1430 can be unidirectional and/or bi-directional, simultaneous uni-directional and/or bi-directional, and/or non-simultaneous, e.g. sequential uni-directional and/or bi-directional, or a combination thereof between the one or more first (optional) 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also integrated into or attached to a robot and/or an imaging system.
In any of the embodiments throughout the specification, a second 1320, third 1330 etc. or additional computer systems, can be integrated into, connected to, and/or attached to and/or separate from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1340, and/or one or more imaging systems 1350. One or more computer systems 1310 can be separate from, e.g. standalone, from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1340, and/or one or more imaging systems 1350.
One or more computer processors of one or more first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to generate one or more user interface controls 1440. The one or more user interface controls 1440 can, for example, be virtual interface controls. The one or more user interface controls 1440 can comprise, for example, one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof.
One or more computer processors of the one or more first (optional) 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to perform one or more collision detections 1445, for example between a hand or finger 1380 and a virtual interface displayed by one or more head mounted displays 13601370 or other augmented reality display devices, or a tracked physical pointer, physical tool, physical instrument, physical implant component 1390 or a combination thereof and a virtual interface displayed by one or more head mounted displays 13601370 or other augmented reality display devices; the collision detection can be performed, for example, using the tracking system 1300. One or more computer processors of the one or more first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to perform a gesture recognition, for example between a hand or finger 1380 and a virtual interface, displayed by one or more head mounted displays 13601370, using one or more cameras, optionally integrated or attached to one or more head mounted displays 13601370. The one or more cameras can be part of the one or more tracking systems 1300. The optional first computer system 1310 can, for example, be a standalone computer system or a cloud computing system. The second 1320, third 1330 etc. or additional computer systems, can be, for example, integrated into, connected to, and/or attached to and/or separate from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1340, and/or one or more imaging systems 1350.
One or more computer systems, e.g. a first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to generate and/or transmit, optionally wirelessly, one or more event message 1450. The one or more event message can, for example, be an event message relative to a collision detection 1445 between a hand or finger 1380, a tracked physical pointer, physical tool, physical instrument, physical implant component 1390 or a combination thereof and a virtual interface, e.g. detected using the tracking system 1300. The one or more event message 1450 can optionally comprise an event message related to a robot 1340 control, interface, force, and/or position and/or orientation and/or an imaging system 1350 control, interface, force, and/or position and/or orientation.
One or more computer systems, e.g. a first 1310, second 1320, third 1330, fourth, fifth etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can comprise an optional event handler 1460 configured to handle, manage and/or process one or more optional event message 1450.
One or more computer systems, e.g. a first 1310, second 1320, third 1330 etc. or additional computer systems, e.g. also in a robot 1340 and/or an imaging system 1350, can be configured to initiate and/or process an optional event action 1470, e.g. executing a command, e.g. based on the event message 1450 and, optionally, information or data received from the event handler 1460. The command can be transmitted through a wireless or a wired connection, e.g. to a robot 1340 or an imaging system 1350.
In some embodiments, the one or more computer systems configured to generate an event message 1450 can be the same or different computer system comprising the event handler 1460 and/or can be the same or different computer system initiating and/or processing the event action 1470, e.g. executing a command. In some embodiments, one or more computer processors configured to generate an event message 1450 can be the same or different computer processors comprising the event handler 1460 and/or can be the same or different computer processors initiating and/or processing the event action 1470, e.g. executing a command. In some embodiments, a first, second, third, fourth, fifth, sixth or more computer system can be the same for managing different tasks, or can be different for managing different tasks.
One or more first CS #1 1510, second CS #2 1520, or more computer systems can be integrated into, connected to or attached to one or more head mounted displays HMD A 1360 and/or HMD B 1370. The one or more tracking system(s) 1300 can be integrated or attached to the one or more head mounted displays HMD A 1360 and/or HMD B 1370 or other augmented reality display devices.
The one or more tracking systems 1300 can be configured to track 1410 optionally a hand and/or finger 1380, e.g. of a user a pointer, tool, instrument and/or implant 1390, a patient (not shown), and/or a robot 1530 and/or an imaging system 1540 or a combination thereof, for example with use of a first computer system 1510, a second computer 1520, and/or additional computer systems.
The tracking system 1300 can be configured to track 1410 optionally a hand and/or finger (e.g. of a user) 1380, a pointer, tool, instrument and/or implant 1390, a patient (not shown), and/or a robot 1530 and/or an imaging system 1540 or a combination thereof, and to transmit the tracking information 1415 to a first computer system CS #1 1510, a second computer CS #2 1520, and/or additional computer systems; the transmission can be wired or wireless. The one or more tracking systems 1300 can be integrated into or attached to one or more head mounted displays 13601370 or other augmented reality display devices. The one or more computer systems 15101520, including one or more computer processors, can be integrated into or connected to the one or more head mounted displays 13601370 or other augmented reality display devices. The first CS #1 1510, second CS #2 1520, and/or additional computer systems can be integrated into, connected to or attached to one or more HMDs 13601370 or other augmented reality display systems. The one or more additional computer systems can be attached to, connected to or located inside a robotic system 1530 and/or an imaging system 1540.
The tracking 1410 can comprise recording one or more coordinates of and/or tracking a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or a patient (not shown) by a first computer system CS #1 1510, and/or a second computer CS #2 1520, and/or additional computer systems, e.g. in a robot 1530 or imaging system 1540, for example using the camera(s) or scanner(s) 1300. The tracking 1410 can comprise recording one or more coordinates of a hand and/or finger 1380, e.g. of a user, a pointer, tool, instrument, and/or implant 1390, and/or the patient (not shown) by the first computer system CS #1 1510, and/or the second computer CS #2 1520, and/or additional computer systems, e.g. in a robot 1530 or imaging system 1540, for example using the camera or scanner 1300. The system can track a hand and/or finger 1380, a pointer, tool, instrument, and/or implant 1390, and/or the patient (not shown) by the first computer system CS #1 1510, and/or the second computer CS #2 1520, and/or additional computer systems, e.g. in a robot 1530 or imaging system 1540, for example using the tracking system with, for example, a camera or scanner 1300.
One or more computer processors of the one or more first 1510, second 1520, etc. computer systems, can be configured to perform one or more coordinate transformations 1420 and/or to determine a pose 1420 of the hand or finger 1380, the one or more pointer, tool, instrument, and/or implant 1390, and/or optionally the first 1360, second 1370, third, fourth, fifth etc. head mounted display (e.g. in relationship to a marker attached to a patient), and/or the patient (not shown).
The one or more computer processors of the one or more first 1510, second 1520, and/or additional computer systems, e.g. also in a robot 1530 and/or an imaging system 1540, can be configured to transmit and/or receive 1430 information about the finger or hand 1380, e.g. of a user, or the one or more pointer, tool, instrument, and/or implant 1390, e.g. their pose, and/or optionally information about or for the first 1360, second 1370, third, fourth, fifth etc. head mounted display, e.g. the pose of the head mounted display(s) (e.g. relative to a marker on a patient) or display data for the head mounted display(s) 13601370 or other augmented reality display devices, and/or the patient (not shown), for example using a wireless connection, e.g.
Bluetooth or a wireless WiFi or LiFi access point, optionally including a transmitter and/or receiver, or using a wired connection. The transmission and/or reception 1430 can comprise data about gaze direction of the wearer of a headset (for example for use with a gaze cursor), e.g. a first 1360, second 1370, third, fourth, fifth etc. head mounted display or other augmented reality display devices. The transmission and/or reception 1430 can be wired or wireless, e.g. using a broadcast or multiple unicast transmissions to and/or from the first 1360, second 1370, third, fourth, fifth etc. head mounted display or other augmented reality display devices, as described in the specification. The transmission can comprise positional data, display data and/or other data. The transmission 1430 can be to or from a first computer system 1510 to or from a second computer system 1520, to or from a third computer system, e.g. part of or coupled to a robot 1530, to or from a fourth, fifth or more computer system. The transmission can be to or from a second computer system 1520 to or from a first computer system 1510, to or from a third computer system 1530, to or from a fourth, fifth or more computer system. The transmission can be to or from a third computer system 1530 to or from a first computer system 1510, to or from a second computer system 1520, to or from a fourth, fifth or more computer system, and so forth.
In any of the embodiments throughout the specification, the transmission (and/or reception) 1430 can be unidirectional and/or bi-directional, simultaneous uni-directional and/or bi-directional, and/or non-simultaneous, e.g. sequential uni-directional and/or bi-directional, or a combination thereof between the one or more first 1510, second 1320, third 1330 etc. or additional computer systems, e.g. also integrated into or attached to a robot and/or an imaging system.
In any of the embodiments throughout the specification, a first 1510, second 1520 etc. or additional computer systems, can be integrated into, connected to, and/or attached to and/or separate from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1530, and/or one or more imaging systems 1540. One or more computer systems can be separate from, e.g. standalone, from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1530, and/or one or more imaging systems 1540.
One or more computer processors of one or more first 1510, second 1520, or additional computer systems, e.g. also in a robot 1530 and/or an imaging system 1540, can be configured to generate one or more user interface controls 1440. The one or more user interface controls 1440 can, for example, be virtual interface controls. The one or more user interface controls 1440 can comprise, for example, one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof.
One or more computer processors of the one or more first 1510, second 1520, or additional computer systems, e.g. also in a robot 1530 and/or an imaging system 1540, can be configured to perform one or more collision detections 1445, for example between a hand or finger 1380 and a virtual interface displayed by one or more head mounted displays 13601370 or other augmented reality display devices, or a tracked physical pointer, physical tool, physical instrument, physical implant component 1390 or a combination thereof and a virtual interface displayed by one or more head mounted displays 13601370; the collision detection can be performed, for example, using the tracking system 1300. One or more computer processors of the one or more first 1510, second 1520, or additional computer systems, e.g. also in a robot 1530 and/or an imaging system 1540, can be configured to perform a gesture recognition, for example between a hand or finger 1380 and a virtual interface, displayed by one or more head mounted displays 13601370, using one or more cameras, optionally integrated or attached to one or more head mounted displays 13601370. The one or more cameras can be part of the one or more tracking systems 1300. The first 1510, second 1520 or additional computer systems, can be, for example, integrated into, connected to, and/or attached to and/or separate from one or more HMDs 1360, 1370 or other augmented reality display systems, one or more robotic systems 1530, and/or one or more imaging systems 1540.
One or more computer systems, e.g. a first 1510, second 1520, or additional computer systems, e.g. also in a robot 1530 and/or an imaging system 1540, can be configured to generate and/or transmit, optionally wirelessly, one or more event message 1450. The one or more event message can, for example, be an event message relative to a collision detection 1445 between a hand or finger 1380, a tracked physical pointer, physical tool, physical instrument, physical implant component 1390 or a combination thereof and a virtual interface, e.g. detected using the tracking system 1300. The one or more event message 1450 can optionally comprise an event message related to a robot 1530 control, interface, force, and/or position and/or orientation and/or an imaging system 1540 control, interface, force, and/or position and/or orientation.
One or more computer systems, e.g. a first 1510, second 1520, third, fourth, fifth etc. or additional computer systems, e.g. also in a robot 1530 and/or an imaging system 1540, can comprise an optional event handler 1460 configured to handle, manage and/or process one or more optional event message 1450.
One or more computer systems, e.g. a first 1510, second 1520, third or additional computer systems, e.g. also in a robot 1530 and/or an imaging system 1540, can be configured to initiate and/or process an optional event action 1470, e.g. executing a command, e.g. based on the event message 1450 and, optionally, information or data received from the event handler 1460. The command can be transmitted through a wireless or a wired connection, e.g. to a robot 1530 or an imaging system 1540.
In some embodiments, the one or more computer systems configured to generate an event message 1450 can be the same or different computer system comprising the event handler 1460 and/or can be the same or different computer system initiating and/or processing the event action 1470, e.g. executing a command. In some embodiments, one or more computer processors configured to generate an event message 1450 can be the same or different computer processors comprising the event handler 1460 and/or can be the same or different computer processors initiating and/or processing the event action 1470, e.g. executing a command. In some embodiments, a first, second, third, fourth, fifth, sixth or more computer system can be the same for managing different tasks, or can be different for managing different tasks.
AR System Architecture for Use with Multiple Devices Including One or More HMD
In some embodiments, a system can comprise at least one head mounted display or augmented reality display device, at least one camera or scanning device, wherein the at least one camera or scanning device can be configured to track real-time information of the at least one head mounted display or augmented reality display device, of at least one anatomic structure of a patient, and of at least one physical surgical tool or physical surgical instrument, a first computing system comprising one or more computer processors, wherein the first computing system can be configured to obtain the real-time tracking information of the at least one head mounted display or augmented reality display device, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, wherein the first computing system can be configured for wireless transmission of the real-time tracking information of the at least one head mounted display, the at least one anatomic structure of the patient, and the at least one physical surgical tool or physical surgical instrument, a second computing system comprising one or more computer processors, wherein the second computing system can be configured for wireless reception of the real-time tracking information of the at least one head mounted display or augmented reality display device, the at least one anatomic structure of the patient, and the at least one physical surgical tool or physical surgical instrument, wherein the second computing system can be configured to generate an augmented or 3D stereoscopic view, wherein the augmented or stereoscopic view can comprise a 3D representation of the at least one tracked physical surgical tool or physical surgical instrument, and wherein the at least one head mounted display or augmented reality display device can be configured to display the 3D stereoscopic view or augmented view. In some embodiments, the one or more computer processors of the second computing system can generate the 3D stereoscopic view for a view angle of the head mounted display relative to the at least one anatomic structure of the patient using the real-time tracking information of the at least one head mounted display. In some embodiments, the real-time tracking information can comprise tracking information of multiple head mounted displays. In some embodiments, the real-time tracking information can comprise a head mounted display or augmented reality display device specific label or tag for each head mounted display, or the real-time tracking information can be labeled for each tracked head mounted display or augmented reality display device. In some embodiments, the wireless transmission can be a multicast or broadcast transmission to the multiple head mounted displays.
In some embodiments, the real-time tracking information can comprise tracking information of two or more head mounted displays or augmented reality display devices. In some embodiments, the two or more head mounted displays or augmented reality display devices are located in different locations. In some embodiments, the real-time tracking information can comprise a head mounted display or augmented reality display device label for each head mounted display or augmented reality display device, wherein each head mounted display or augmented reality display device has a different label. In some embodiments, the real-time tracking information can be labeled for each tracked head mounted display or augmented reality display device. In some embodiments, one or more computer processors of a second computing system can generate a 3D stereoscopic view for an interpupillary distance adjusted for a user wearing the head mounted display. In some embodiments, the second computing system can be integrated with the at least one head mounted display. In some embodiments, the second computing system can be separate from the at least one head mounted display and is connected to a display unit of the at least one head mounted display using at least one cable. In some embodiments, the wireless transmission, the wireless reception, or both comprise a WiFi signal, a LiFi signal, a Bluetooth signal, a radiofrequency signal or a combination thereof. In some embodiments, a camera or scanning device is separate from at least one head mounted display. In some embodiments, a camera or scanning device can be integrated or attached to at least one head mounted display. In some embodiments, the wireless transmission can comprise sending data packets comprising the real-time tracking information of at least one head mounted display, at least one augmented reality display device, at least one anatomic structure of a patient, and at least one physical surgical tool or physical surgical instrument, at a rate of 20 Hz or greater. In some embodiments, a wireless reception can comprise receiving data packets comprising the real-time tracking information of the at least one head mounted display, the at least one augmented reality display device, the at least one anatomic structure of a patient, and the at least one physical surgical tool or physical surgical instrument, at a rate of 20 Hz or greater. In some embodiments, the real-time tracking information comprises one or more coordinates, e.g. for wireless transmission and/or reception, e.g. to and from a first and second computing system. In some embodiments, the one or more coordinates can comprise coordinates of the at least one anatomic structure of the patient. In some embodiments, the one or more coordinates can comprise coordinates of the at least one physical surgical tool or physical surgical instrument. In some embodiments, the one or more coordinates can comprise coordinates of the at least one head mounted display or augmented reality display device. In some embodiments, at least one head mounted display can comprise at least one optical see-through head mounted display. In some embodiments, at least one head mounted display can comprise at least one video see-through head mounted display. In some embodiments, at least one camera or scanning device can comprise a laser scanner, a time-of-flight 3D laser scanner, a structured-light 3D scanner, a hand-held laser scanner, a LIDAR scanner, a time-of-flight camera, a depth camera, a video system, a stereoscopic camera system, a camera array, or a combination thereof. In some embodiments, a system can comprise at least one inertial measurement unit. In some embodiments, the at least one inertial measurement unit can be integrated or attached to the at least one physical surgical tool or physical surgical instrument.
In some embodiments, the at least one inertial measurement unit can be integrated or attached to the at least one anatomic structure of the patient. In some embodiments, the at least one inertial measurement unit can be integrated or attached to the at least one head mounted display or augmented reality display device. In some embodiments, the real-time tracking information of the at least one head mounted display or augmented reality display device can comprises information from the at least one inertial measurement unit. In some embodiments, a second computing system is communicatively coupled to the at least one head mounted display.
In some embodiments, a system can comprise two or more head mounted displays or augmented reality display devices, at least one camera or scanning device, wherein the at least one camera or scanning device can be configured to track real-time information of the at least two or more head mounted displays or augmented reality display devices, of at least one anatomic structure of a patient, and of at least one physical surgical tool or physical surgical instrument, a first computing system comprising one or more computer processors, wherein the first computing system can be configured to obtain real-time tracking information of at least one anatomic structure of a patient, of at least one physical surgical tool or physical surgical instrument, and of the two or more head mounted displays or augmented reality devices, wherein the tracking information of the two or more head mounted displays or augmented reality display devices can be labeled for each of the two or more head mounted displays or augmented reality display devices, wherein the first computing system can be configured for wireless transmission of the real-time tracking information of the at least one anatomic structure of the patient, the tracking information of the at least one physical surgical tool or physical surgical instrument, and the labeled tracking information of the two or more head mounted displays or augmented reality display devices, a second computing system, wherein the second computing system can be configured for wireless reception of the real-time tracking information of the at least one anatomic structure of the patient, the tracking information of the at least one physical surgical tool or physical surgical instrument, and the labeled tracking information of the first of the two or more head mounted displays or augmented reality display devices, wherein the second computing system can be configured to generate a first 3D stereoscopic display specific for a first viewing perspective of the first head mounted display or augmented reality display device using the labeled tracking information of the first head mounted display or augmented reality display device, wherein the first head mounted display or augmented reality display device can be configured to display the 3D stereoscopic display or augmented view, a third computing system, wherein the third computing system can be configured for wireless reception of the real-time tracking information of the at least one anatomic structure of the patient, the tracking information of the at least one physical surgical tool or physical surgical instrument, and the labeled tracking information of the second of the two or more head mounted displays or augmented reality display devices, wherein the third computing system can be configured to generate a second 3D stereoscopic display or augmented view specific for a second viewing perspective of the second head mounted display or augmented reality display device using the labeled tracking information of the second head mounted display or augmented reality display devices, and wherein the first and second stereoscopic displays or augmented reality display devices can comprise a 3D representation of the at least one physical surgical tool or physical surgical instrument.
In some embodiments of the disclosure, a system can comprise at least one head mounted display, a robot, wherein the robot can comprise an end effector, a first computing system comprising one or more computer processors, wherein the first computing system can be in communication with the robot, a second computing system comprising one or more computer processors, wherein the second computing system can be in communication with the at least one head mounted display, wherein the second computing system can configured to display, by the at least one head mounted display, a virtual user interface comprising at least one virtual object, wherein the second computing system can be configured to generate a command based at least in part on at least one interaction with the at least one virtual object displayed in the virtual user interface, wherein the second computing system can be configured to transmit the command to the first computing system using wireless transmission, wherein the command can be configured to cause the first computing system to control the robot for movement, activation, operation, de-activation, or any combination thereof, of a robot component, a robot motor, a robot actuator, a robot drive, a robot controller, a robot hydraulic system, a robot piezoelectric system, a robot switch, the end effector, or any combination thereof. In some embodiments, the command can be configured to control the end effector within a predetermined operating boundary, a predetermined operating range, a predetermined operating zone, or a predetermined operating volume. In some embodiments, the first computing system can be connected to the robot by wire, or the first computing system can be connected to the robot by wireless connection. In some embodiments, the second computing system can be connected to the at least one head mounted display by wire, or the second computing system can be connected to the at least one head mounted display by wireless connection. In some embodiments, the second computing system can be configured to display, by the at least one head mounted display, a representation of a predetermined operating boundary, a predetermined operating range, a predetermined operating zone, or a predetermined operating volume of the end effector or an expected outcome following the movement, activation, operation, de-activation or a combination thereof of the robot component, robot motor, robot actuator, robot drive, robot controller, robot hydraulic system, robot piezoelectric system, robot switch, the end effector, or any combination thereof. In some embodiments, the end effector can comprise a physical surgical tool or a physical surgical instrument.
In some embodiments, a first computing system can be configured to obtain real-time tracking information of a component of the robot, an end effector, a target object, a target anatomic structure of a patient, at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof. In some embodiments, a second computing system can be configured to obtain real-time tracking information of a component of a robot, an end effector, a target object, a target anatomic structure of a patient, at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof. In some embodiments, the first computing system can be configured to obtain real-time tracking information of a physical tool, a physical instrument, or any combination thereof coupled to the robot. In some embodiments, the second computing system can be configured to obtain real-time tracking information of a physical tool, a physical instrument, or any combination thereof coupled to the robot. In some embodiments, the first computing system can be configured to wirelessly transmit the real-time tracking information of the component of the robot, the end effector, a target object, a target anatomic structure of a patient, the at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof. In some embodiments, the second computing system can be configured to wirelessly transmit the real-time tracking information of the component of the robot, the end effector, a target object, a target anatomic structure of a patient, the at least one head mounted display, a physical tool, a physical instrument, a physical implant, a physical object, or any combination thereof.
In some embodiments, a second computing system can be configured for displaying, by the at least one head mounted display, a 3D stereoscopic view. In some embodiments, the 3D stereoscopic view can be superimposed onto an anatomic structure of a patient. In some embodiments, the 3D stereoscopic view can comprise a predetermined trajectory of the end effector, a representation of a predetermined operating boundary of the end effector, a representation of a predetermined operating range of the end effector, a representation of a predetermined operating zone of the end effector, a representation of a predetermined operating volume of the end effector, or a combination thereof. In some embodiments, the 3D stereoscopic view can comprise a predetermined trajectory of the end effector, a representation of a predetermined operating boundary of the end effector, a representation of a predetermined operating range of the end effector, a representation of a predetermined operating zone of the end effector, a representation of a predetermined operating volume of the end effector or a combination thereof following the movement, activation, operation, de-activation or combination thereof of the robot component, robot motor, robot actuator, robot drive, robot controller, robot hydraulic system, robot piezoelectric system, robot switch, the end effector or any combination thereof.
In some embodiments, a first computing system, a second computing system, or both can be configured to turn on or turn off the display of the virtual user interface. In some embodiments, a wireless transmission can comprise a Bluetooth signal, WiFi signal, LiFi signal, a radiofrequency signal, a microwave signal, an ultrasound signal, an infrared signal, an electromagnetic wave or any combination thereof.
In some embodiments, a 3D stereoscopic view can comprise a predetermined trajectory of an end effector, a representation of a predetermined operating boundary of the end effector, a representation of a predetermined operating range of the end effector, a representation of a predetermined operating zone of the end effector, a representation of a predetermined operating volume of the end effector or a combination thereof prior to executing a command.
In some embodiments, a the system can comprise two or more head mounted displays, wherein the wireless transmission can be a multicast, broadcast transmission or any combination thereof.
In some embodiments, a virtual object displayed by the HMD can comprise one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof. In some embodiments, an interaction with a virtual interface can comprise a collision detection between a physical object and the at least one virtual object. In some embodiments, an interaction can be a collision detection between a user's finger and at least one virtual object displayed by an HMD.
In some embodiments, an interaction can comprise a collision detection between a tracked pointer, tracked tool, tracked instrument, or a combination thereof and at least one virtual object displayed by the HMD. In some embodiments, an interaction with a virtual object displayed by the HMD can comprise a gesture recognition, gaze recognition, gaze lock, hand tracking, eye tracking or a combination thereof. In some embodiments, hand tracking, eye tracking and voice control can be used, e.g. also without interaction with a virtual object displayed by the HMD.
In some embodiments, an end effector can comprise a scalpel, a saw, a cutting tool, a wire, a needle, a pin, a drill, a burr, a mill, a reamer, an impactor, a broach, a laser, a radiofrequency device, a thermocoagulation device, a cryoablation device, a radioactive probe, a radioactivity emitting device, a pulsed energy emitting device, an ultrasonic energy emitting device, a microwave energy emitting device or a combination thereof. In some embodiments, a command can comprise a subcommand, wherein the subcommand is configured to execute an accept or cancel function of the command.
In some embodiments, a system can comprise at least one head mounted display, a robot, wherein the robot can comprise an end effector, a first computing system comprising one or more computer processors, wherein the first computing system can be in communication with the robot, a second computing system comprising one or more computer processors, wherein the second computing system can be in communication with the at least one head mounted display, wherein the second computing system can be configured to display, by the at least one head mounted display, a virtual user interface comprising at least one virtual object, wherein the second computing system can be configured to generate an event message based at least in part on at least one interaction with the at least one virtual object displayed in the virtual user interface, wherein the second computing system can be configured to transmit the event message to the first computing system using wireless transmission, wherein the second computing system can be configured to generate a command based on the event message, wherein the command can be configured to cause the first computing system to control the robot for movement, activation, operation, de-activation, or any combination thereof, of a robot component, a robot motor, a robot actuator, a robot drive, a robot controller, a robot hydraulic system, a robot piezoelectric system, a robot switch, the end effector, or a combination thereof.
AR Guidance of Imaging Systems and/or Image Acquisitions
In some embodiments, preparing an image acquisition by an imaging system in a patient (prior to the actual image acquisition) can comprise tracking, by at least one computer processor, one or more components of the imaging system in real time, optionally obtaining, by the at least one computer processor, information about a geometry of one or more components of the imaging system, information about a position, orientation and/or geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof, generating, by the at least one computer processor, a 3D representation of a surface, a volume or combination thereof, wherein the 3D representation of the surface, the volume or combination thereof can be at least in part derived from or based on the information about the geometry of the one or more components of the imaging system, information about the position, orientation and/or geometry of the image acquisition, information about the one or more image acquisition parameters, or a combination thereof, generating, by the at least one computer processor, a 3D stereoscopic view or an augmented view, wherein the 3D stereoscopic view or augmented view can comprise the 3D representation of the surface, volume or combination thereof, optionally displaying, by an augmented reality display device, e.g. a head mounted display, a tablet, an iPad, an iPhone or other AR display device, the 3D stereoscopic view or augmented view at a position defined relative to the one or more components of the imaging system or at a position defined relative to the patient, wherein the position and orientation of the 3D stereoscopic view or augmented view can be determined based on the real time tracking information of the imaging system, and then acquiring, by the imaging system, 2D, 3D, or 2D and 3D imaging data of the patient within the 3D representation of the surface, volume or combination thereof or at the location of the 3D representation of the surface, volume or combination thereof.
In some embodiments, the 3D representation of the surface, volume or combination thereof does not contain imaging data from the patient. In some embodiments, the 3D representation of the surface, volume or combination thereof can comprise imaging data from the patient from a prior image acquisition, preceding the current, planned, intended image acquisition. In some embodiments, the 3D representation of the surface, volume or combination thereof can comprise model data, e.g. a generic 3D model of patient anatomy, an avatar of patient anatomy, etc. In some embodiments, the 3D representation can comprise a 2D outline, 3D outline, a mesh, a group of surface points, or a combination thereof at least in part derived from or based on the information about the geometry of the one or more components of the imaging system, information about the position, orientation and/or geometry of the image acquisition, information about the one or more image acquisition parameters, or a combination thereof.
The shape of the 3D representation of the surface, volume or combination thereof, optionally displayed, by a head mounted display or another augmented reality display device can change responsive to changes in a geometry of one or more components of the imaging system (e.g. a collimator), changes in a position, orientation and/or geometry of the image acquisition (e.g. a change in tube location, detector location, image intensifier location, field of view, collimation), and/or changes in one or more image acquisition parameters.
In some embodiments, the imaging system or one or more of its components can be moved, e.g. by a user, optionally assisted by one or more motors, controllers, drives, hydraulic and/or electric system, and/or robotic components and/or arms, and the augmented view or 3D stereoscopic view, in some embodiments optionally superimposed onto the patient, can be configured by the system and/or one or more computer processor to move in relation with the tracked one or more components of the imaging system. When the imaging system or one or more of its components are moved or when one or more parameters determining the location and/or orientation of the image acquisition are changed, the system can be configured to adjusting the position, orientation, position and orientation of the augmented view or 3D stereoscopic view in response to the movement of the tracked imaging system, the movement of the one or more of its components and/or the changes in the one or more parameters determining the location and/or orientation of the image acquisition. In some embodiments, the adjusting or adjustment is configured to maintain the augmented view at the defined position and orientation relative to the one or more components of the imaging system. In some embodiments, the augmented view at the defined position relative to the one or more components of the imaging system moves in relation with the tracked one or more components of the imaging system, wherein the moving facilitates superimposing or aligning the 3D representation with a target anatomic structure of the patient. In some embodiments, the imaging system can be configured to acquire 2D, 3D, or 2D and 3D imaging data of a patient, and wherein the 2D, 3D, or 2D and 3D imaging data of the patient are acquired within the 3D representation of the surface, volume or combination thereof.
In some embodiments, the step of generating the augmented view is before the step of acquiring 2D, 3D, or 2D and 3D imaging data of the patient, or wherein the step of displaying the augmented view is before the step of acquiring 2D, 3D, or 2D and 3D imaging data of the patient.
In some embodiments, at least one computer processor can be configured to generate a 3D representation of a surface, a volume or combination thereof for display by an HMD or an augmented reality device, wherein the 3D representation of the surface, volume or combination thereof can comprise information about a geometry of the one or more components of the imaging system, information about a position, orientation and/or geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof, wherein the 3D representation of the surface, volume or combination thereof can, for example, not contain imaging data from a patient. A user can move the 3D representation of the surface or volume superimposed onto the patient, for example by interaction with a virtual user interface displayed by the HMD or an augmented reality or mixed reality display device. The movement of the 3D representation can, for example, be intended to move the 3D representation into a desired anatomic area or volume (e.g. of a target anatomic area) to be included and/or for inclusion in an imaging data acquisition. The system can then optionally be configured to move the imaging system, one or more components of the imaging system, a patient table used in conjunction with the imaging system, or any combination thereof, for example using one or more motors, controllers, drives, hydraulic and/or electric system, and/or robotic components and/or arms, responsive to the movement of the 3D representation. In this fashion, a subsequent imaging data acquisition can be obtained in the desired area or volume of the subsequent imaging data acquisition.
In some embodiments, the step of acquiring the 2D or 3D or 2D and 3D images can be after the step of generating the augmented view or 3D stereoscopic view. In some embodiments, the step of acquiring the 2D or 3D or 2D and 3D images can be after the step of displaying the augmented view or 3D stereoscopic view.
In some embodiments, at least one computer processor can be configured to generate a 3D representation of the surface, volume or combination thereof before acquisition of 2D, 3D, or 2D and 3D imaging data of the patient, or at least one computer processor can be configured to display the 3D representation of the surface, volume or combination thereof before acquisition of 2D, 3D, or 2D and 3D imaging data of the patient.
In some embodiments, a system of preparing an imaging data acquisition associated with a patient can comprise at least one computer processor, a head mounted display, e.g. a video see-through head mounted display or an optical see-through head mounted display, an augmented reality display device, e.g. a tablet or smart phone, an imaging system, wherein the at least one computer processor can be configured to obtain real-time tracking information of one or more components of the imaging system (and optionally a patient table, the HMD, the augmented reality display device, an anatomic structure of a patient, a robot), wherein the at least one computer processor can be configured to generate a 3D representation of a surface, a volume or combination thereof, wherein the 3D representation of the surface, volume or combination thereof can comprise information about a geometry of the one or more components of the imaging system, information about a position, orientation and/or geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof, wherein the 3D representation of the surface, volume or combination thereof can, for example, not contain imaging data from a patient, or can contain imaging data from a prior imaging data acquisition but not the current, planned, intended image acquisition, of the patient, wherein the at least one computer processor can be configured to generate an augmented view or 3D stereoscopic view, the augmented view or 3D stereoscopic view comprising the 3D representation of the surface, volume or combination thereof, wherein the at least one computer processor can be configured to display, by the head mounted display or augmented reality device, the augmented view or 3D stereoscopic view, e.g. superimposed onto the patient or combined with a video feed of the patient or the imaging system (if an augmented reality display device such as a table or smart phone is used), wherein the position and/or orientation of the augmented view or 3D stereoscopic view (including, optionally, an intended position and/or orientation) can be determined based on the real time tracking information of the one or more components of the imaging system, and wherein the imaging system can be configured to acquire 2D, 3D, or 2D and 3D imaging data of the patient within the or at the location of the 3D representation of the surface, volume or combination thereof. In some embodiments, the imaging system can also acquire imaging studies with more than 3 dimensions, four example when sequential scanning is performed with time being the 4th dimension. Representative examples comprise, for example, ultrasound flow studies, Doppler ultrasound, angiography including vascular run-offs, spiral CT imaging after contrast bolus for vascular imaging (e.g. through or during a cardiac cycle) or MR angiography studies or imaging combination studies, e.g. SPECT-CT or PET-MRI.
In some embodiments, the at least one computer processor can be configured to generate the 3D representation of the surface, volume or combination thereof before acquisition of the 2D, 3D, or 2D and 3D imaging data of the patient and/or in some embodiments, the at least one computer processor can be configured to display the 3D representation of the surface, volume or combination thereof before acquisition of the 2D, 3D, or 2D and 3D imaging data of the patient.
In some embodiments, at least one computer processor can be configured to generate a 3D representation of a surface, volume or combination thereof, which can comprise information about a limit, an edge, a margin, a boundary, a circumference, a perimeter, or a combination thereof of a planned or intended or upcoming imaging data acquisition, e.g. a 2D, 3D, 4D, 5D or combination thereof imaging acquisition. The limit, edge, margin, boundary, circumference, perimeter, or a combination thereof of a planned or intended or upcoming imaging data acquisition can not contain or comprise any imaging data of the patient or can comprise imaging data of the patient from a prior imaging data acquisition, but not from the planned or intended or upcoming imaging data acquisition. The limit, edge, margin, boundary, circumference, perimeter, or a combination thereof of a planned or intended or upcoming imaging data acquisition can comprise information about a geometry of the one or more components of the imaging system, information about a position, orientation and/or geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof. The limit, edge, margin, boundary, circumference, perimeter, or a combination thereof of a planned or intended or upcoming imaging data acquisition can be based on or can be derived from information about a geometry of the one or more components of the imaging system, information about a position, orientation and/or geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof.
The location, position, and/or orientation of the limit, edge, margin, boundary, circumference, perimeter, or a combination thereof, the location, position, and/or orientation of the surface or volume of the 3D representation of a planned or intended or upcoming imaging data acquisition can be based on the tracked location, position, orientation, and/or coordinates of the imaging system, and/or the tracked location, position, orientation, and/or coordinates of one or more components of the imaging system.
In some embodiments, the at least one computer processor can be configured to generate the surface, volume or combination thereof at least in part from information about a geometry of the imaging system, information about a position, orientation and/or geometry of one or more components of the imaging system, information about a geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof. In some embodiments, the at least one computer processor can be configured to determine a desired location and orientation of the augmented view or 3D stereoscopic view associated with the imaging system, wherein the at least one computer processor can be configured to generate the 3D representation of the surface, volume or combination thereof before acquisition of the 2D, 3D, or 2D and 3D imaging data of the patient, or wherein the at least one computer processor can be configured to display the 3D representation of the surface, volume or combination thereof before acquisition of the 2D, 3D, or 2D and 3D imaging data of the patient.
In some embodiments, the system can be configured to acquire the 2D, 3D, or 2D and 3D imaging data from the patient by the imaging system at the location and orientation of the 3D representation of the surface, volume or combination thereof and/or at the location and orientation of the stereoscopic view or augmented view, displayed by an HMD or an augmented reality display device such as a tablet or smart phone, of the 3D representation of the surface, volume or combination thereof.
In some embodiments, the at least one computer processor can be configured to project the augmented view or 3D stereoscopic view of the 3D representation of the surface, volume or combination thereof (based at least in part on information about a geometry of the imaging system, information about a position, orientation and/or a geometry of one or more components of the imaging system, information about a geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof) at the coordinates of the planned imaging data acquisition of the patient, which can be a 2D, 3D, 4D, 5D or more imaging data acquisition (accounting, for example, also for temporal elements, subtraction imaging etc.).
In some embodiments, the location of the imaging data acquisition, e.g. a 2D, 3D, 4D or combination thereof imaging data acquisition, can comprise one or more anatomic structures of the patient. The anatomic structures can optionally be tracked using any of the techniques described in the specification or known in the art.
In some embodiments, the system can be configured to facilitate determining a desired position and orientation of the augmented view, wherein the desired position and orientation comprises a target anatomic structure of the patient. In some embodiments, the at least one computer processor can be configured to adjust the position, orientation, or position and orientation of the augmented view or 3D stereoscopic view in response to movement of one or more tracked components of the imaging system and/or movement of the patient table. The adjustment can be configured to maintain the augmented view at the defined position and orientation relative to the one or more components of the imaging system.
In some embodiments, the information (for example used for generating the representation of the surface or volume of the planned imaging data acquisition) about the geometry of the imaging system, information about the position, orientation and/or geometry of one or more imaging system components, information about the geometry of the image acquisition, information about one or more image acquisition parameter, or a combination thereof can comprise information about one or more imaging system components, a geometric relationship between one or more imaging system components, a collimator, a grid, an image intensifier, a detector resolution, an x-ray source, an x-ray tube setting, a kVp setting, an mA setting, an mAs setting, a collimation, a tube—detector distance, a tube—patient distance, patient—detector distance, a patient—image intensifier distance, a table height relative to a tube, a detector, a table position relative to a tube, a detector, or combination thereof, a patient position, a C-arm position, orientation, or combination thereof, a gantry position, orientation or combination thereof, a grid height, a grid width, a grid ratio, a field of view, a center of a field of view, a margin or periphery of a field of view, a matrix, a pixel size, a voxel size, an image size, an image volume, an imaging plane, an image dimension in x, y, z and/or oblique direction, an image location, an image volume location, a scan coverage, a pitch, an in-plane resolution, a slice thickness, an increment, a detector configuration, a detector resolution, a detector density, a tube current, a tube potential, a reconstruction algorithm, a scan range, a scan boundary, a scan limit, a rotational center of a scan acquisition, a rotational axis of a scan acquisition, a reconstructed slice thickness, a segmentation algorithm, a window, a level, a brightness, a contrast, a display resolution, or any other setting parameter tabulated in Table 1, or any combination thereof.
The geometric relationship between one or more imaging system components can, for example, comprise a tube—detector distance, a tube—image intensifier distance, a rotational axis (e.g. in a spiral CT or cone beam CT or 3D C-arm), a center of rotation, an MRI coil dimension, a sensitivity profile (e.g. for an MRI profile) in relationship to one or more dimensions/geometries etc. The information about a collimator can comprise, for example, a grid spacing, a grid depth, a collimator dimension and any changes thereto, e.g. induced by movement of one or more motors, controllers and or drives.
In some embodiments, an imaging system can comprise an x-ray system, a fluoroscopy system, a C-arm, a 3D C-arm, a digital tomosynthesis imaging system, an angiography system, a bi-planar angiography system, a 3D angiography system, a CT scanner, an MRI scanner, a PET scanner, a SPECT scanner, a nuclear scintigraphy system, a 2D ultrasound imaging system, a 3D ultrasound imaging system, or a combination thereof.
In some embodiments, at least one computer processor can be configured to obtain real-time tracking information of a head mounted display, an augmented reality display device, an anatomic structure of the patient, a patient table used with the imaging system, an imaging system, one or more components of the imaging system, a surgical instrument, a surgical tool, an implant, a surgical robot, a robot integrated with or part of the imaging system, or any combination thereof. Any of the tracking techniques described in the specification using extrinsic and intrinsic tracking can be used. For example, a surgical navigation system, camera, 3D scanner can be used. Inside out or outside in tracking can be used. Intrinsic information can be used for tracking as described throughout the specification.
In some embodiments, the system can comprise a camera or scanner configured to acquire the real-time tracking information of the head mounted display, the anatomic structure of the patient, the patient table used with the imaging system, the imaging system, the one or more components of the imaging system, a surgical instrument, a surgical tool, an implant or a combination thereof. In some embodiments, a camera or scanner comprise at least one of a navigation system, a 3D scanner, a LIDAR system, a depth sensor, an IMU or a combination thereof. In some embodiments, tracking information can comprise coordinate information of a head mounted display, an augmented or mixed reality display device, an anatomic structure of the patient, a patient table used with the imaging system, an imaging system, one or more components of the imaging system, or a combination thereof. In some embodiments, tracking information can comprise location information of a head mounted display, an augmented or mixed reality display device, an anatomic structure of the patient, a patient table used with the imaging system, an imaging system, one or more components of the imaging system components, or a combination thereof. In some embodiments, a camera or scanner can comprise a laser scanner, time-of-flight 3D scanner, structured-light 3D scanner, hand-held laser scanner, a time-of-flight camera or a combination thereof.
In some embodiments, a system can be configured to obtain real time tracking information of an imaging system using intrinsic information from the imaging system. Intrinsic information can comprise pose data, sensor data, camera data, 3D scanner data, controller data, drive data, actuator data, end effector data, data from one or more potentiometers, data from one or more video systems, data from one or more LIDAR systems, data from one or more depth sensors, data from one or more inertial measurement units, data from one or more accelerometers, data from one or more magnetometers, data from one or more gyroscopes, data from one or more force sensors, data from one or more pressure sensors, data from one or more position sensors, data from one or more orientation sensors, data from one or more motion sensors, position and/or orientation data from step motors, position and/or orientation data from electric motors, position and/or orientation data from hydraulic motors, position and/or orientation data from electric and/or mechanical actuators, position and/or orientation data from drives, position and/or orientation data from robotic controllers, position and/or orientation data from one or more robotic computer processors, or a combination thereof.
In some embodiments, an imaging system, e.g. an x-ray system, fluoroscopy system, C-arm, 3D C-arm, cone beam CT, CT scanner, SPECT CT scanner, PET CT scanner can be configured to generate an x-ray beam. In some embodiments, an x-ray beam of an imaging system can be cone shaped or cylindrical, pencil shaped, or fan shaped. In some embodiments, an x-ray beam of an imaging system can originate from one or more point sources. In some embodiments, an x-ray beam of can imaging system can be collimated. The collimation can be adjustable. The adjustment of the collimation can be performed using a graphical user interface, e.g. a virtual user interface displayed by an HMD. The collimation can determine the shape, limit, margin, perimeter of a 3D representation of a surface or volume of an image acquisition. The 3D representation can be displayed in stereoscopic form by an HMD, superimposed onto a patient lying on a patient table in the imaging system.
In some embodiments, the system can comprise a user interface. In some embodiments, a user interface can comprise a virtual user interface, wherein the virtual interface can comprise at least one virtual object. In some embodiments, a virtual object, e.g. displayed as part of a virtual interface by an HMD or other augmented reality display device, can comprise one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof. In some embodiments, a virtual user interface can comprise a gesture recognition, gaze recognition, gaze lock or a combination thereof. Other interfaces can comprise a voice recognition, eye tracking, hand tracking, pointer tracking, instrument tracking, tool tracking, or a combination thereof, which can optionally also be used to “touch” and/or activate one or more virtual objects displayed by an virtual interface. In some embodiments, at least one computer processor can be configured to generate a command based at least in part on at least one interaction of a user with at least one virtual object displayed by an HMD or other augmented reality display device in a virtual user interface. In some embodiments, a command can be configured to move, tilt, or rotate one or more components of an imaging system, one or more components of a patient table or a combination thereof. In some embodiments, a command can be configured to activate, operate, de-activate or a combination thereof a motor, an actuator, a drive, a controller, a hydraulic system, a switch, an x-ray tube, an image intensifier, or an imaging system or an ultrasound transducer, an ultrasound receiver, a robot (e.g. a surgical robot) or a combination thereof.
In some embodiments, a command, e.g. from a virtual interface, can be configured to move one or more components of a robot and/or one or more end effectors, to move or modify a geometry of am imaging system, a patient table, a geometric relationship between one or more imaging system components, a collimator, a grid, an image intensifier, a detector resolution, a setting of the imaging system, a parameter of the imaging system, a parameter of the imaging data acquisition, a display parameter, an x-ray source setting, an x-ray tube setting, a kVp setting, an mA setting, an mAs setting, a collimation, a tube—detector distance, a tube—patient distance, patient—detector distance, a patient—image intensifier distance, a center of rotation, e.g. of an image acquisition, a center of rotation of a rotational movement of a tube and/or detector system, a center of rotation of a rotational movement of a C-arm, a center of rotation of a rotational movement of a cone beam CT scanner, a center of a spiral acquisition of a spiral CT scanner, a table height relative to a tube, a detector, a table position relative to a tube, a detector, a patient position, a C-arm position, orientation, or combination thereof, a gantry position, orientation or combination thereof, a grid height, a grid width, a grid ratio, a field of view, a matrix, a pixel size, a voxel size, an image size, an image volume, an imaging plane, an image dimension in x, y, z and/or oblique direction, an image location, an image volume location, a scan coverage, a pitch, an in-plane resolution, a slice thickness, an increment, a detector configuration, a detector resolution, a detector density, a tube current, a tube potential, a reconstruction algorithm, a scan range, a scan boundary, a scan limit, a reconstructed slice thickness, a segmentation algorithm, a window, a level, a brightness, a contrast, a display resolution, or a combination thereof.
In some embodiments, a command can be configured to set and/or modify one or more image acquisition parameters of an imaging system, e.g. an x-ray system, C-arm, 3D C-arm, cone beam CT system, CT scanner, spiral CT scanner, MRI scanner, ultrasound system, radionuclide imaging device, SPECT scanner, PET system, cardiac imaging system, angiography system, or any combination thereof.
In some embodiments, a command can be configured to set, move, and/or modify a position, orientation, size, area, volume, or combination thereof of am imaging data acquisition, e.g. a 2D, 3D, 4D, 5D imaging data acquisition or any combination thereof. In some embodiments, a command can be configured to set, move, and/or modify one or more coordinates of the 3D representation, e.g. of a surface or volume representing the limit, perimeter, outer margin of an upcoming, planned, intended image acquisition, e.g. for stereoscopic display by an HMD or augmented display, e.g. of a video feed from an imaging system, a patient etc., by another augmented reality display device. In some embodiments, a command can be configured to set, move, and/or modify one or more coordinates of an upcoming, planned, intended imaging data acquisition, e.g. a 2D, 3D, 4D, 5D imaging data acquisition or any combination thereof.
In some embodiments, a command can be configured to set, move and/or modify a dimension, a size, an area, a volume or a combination thereof of a 3D representation, e.g. of a surface or volume representing the limit, perimeter, outer margin of an upcoming, planned, intended image acquisition, e.g. for stereoscopic display by an HMD.
In some embodiments, a command can be configured to set, move and/or modify a dimension, a size, an area, a volume or a combination thereof of an upcoming, planned, intended imaging data acquisition, e.g. a 2D, 3D, 4D, 5D imaging data acquisition or any combination thereof.
In any embodiments throughout the specification, a command can be configured to activate, operate, de-activate or a combination thereof a sensor, a camera, a video system, a 3D scanner, a Lidar system, a navigation system, a potentiometer, an electronic circuit, a computer chip, a piezoelectric system, a piezoelectric mechanism, a piezoelectric lock or release system, a controller, a drive, a motor, a hydraulic system, an actuator, a functional unit or a combination thereof of an imaging system, an imaging system component, a patient table or a combination thereof.
In any embodiments throughout the specification, a command can be configured to activate, operate, de-activate or a combination thereof a sensor, a camera, a video system, a 3D scanner, a Lidar system, a navigation system, a potentiometer, an electronic circuit, a computer chip a piezoelectric system, a piezoelectric mechanism, a piezoelectric lock or release system, a controller, a drive, a motor, a hydraulic system, an actuator, or a combination thereof of a surgical robot, a component of a surgical robot, a functional unit of a surgical robot, a patient table used in conjunction with a surgical robot or a combination thereof.
In any embodiments throughout the specification, a sensor can comprise a depth sensor, inertial measurement unit, accelerometer, magnetometer, gyroscope, force sensor, pressure sensor, position sensor, orientation sensor, motion sensor, or a combination thereof.
In some embodiments, one or more components of an imaging system can be attached to, integrated with, or part a robot. A robot can be configured to move one or more components of the imaging system.
In some embodiments, a virtual user interface can be configured to generate an event message triggered by a user interaction, e.g. a collision detection. The system can comprise an event handler configured to process the event message. An event handler can be configured to generate a command. In some embodiments, a computing system can be configured to generate a command, wherein the command can be triggered by a virtual user interface.
In some embodiments, a system can be configured to determine/identify a desired location/orientation of an augmented view or 3D stereoscopic view, optionally associated with the imaging system, to acquire imaging data, e.g. 2D, 3D, 4D or any combination thereof, at a desired location/orientation.
In some embodiments, a system can be configured to determine a desired location of an augmented view associated with the imaging system to acquire 2D, 3D, or 2D and 3D imaging data at the desired location. In some embodiments, an augmented reality display device can be a head mounted display, and wherein the augmented view can comprise a 2D, 3D and/or a 3D stereoscopic view. In some embodiments, at least one computer processor can be configured to project the 3D stereoscopic view at the coordinates of intended 2D, 3D or 2D and 3D imaging data acquisition of the patient. In some embodiments, the location of the 2D, 3D, or 2D and 3D imaging data acquisition comprises one or more target anatomic structures of the patient.
The surface of volume can describe an outer envelope, margin, perimeter, of the energy beam 1640. The 3D representation and the stereoscopic display by the HMD or other augmented reality display device can be generated prior to activating/turning on the energy beam. The 3D representation 1630 of the surface or volume can be based on/derived from a geometry of the one or more components of the imaging system, information about a geometry of the image acquisition, information about one or more image acquisition parameters, or a combination thereof. The 3D representation 1630 of the surface, volume or combination thereof can, for example, not contain imaging data from a patient 1650, or can contain imaging data from a prior imaging data acquisition but not the current, planned, intended, desired image acquisition, of the patient 1650. Also shown are an x-ray tube 1660 and an image intensifier 1670 of a C-arm system. The position, orientation, position and orientation of the 3D representation 1630 can be defined in relationship to one or more components of the imaging system, the patient table, or a combination thereof. The position, orientation, position and orientation of the 3D representation 1630 can be fixed in relationship to one or more components of the imaging system; the system can be configured that the 3D representation moves with movement of the one or more components of the imaging system, and/or any changes in the position, orientation, and/or geometry of the imaging system, any changes in the position, orientation, and/or geometry of one or more imaging system components (including, for example, also a collimator), any changes in the position, orientation, and/or geometry of the image acquisition, any changes in one or more image acquisition parameters or a combination thereof.
The imaging system, in this non-limiting example a 3D C-arm 1680, and/or one or more of its components, and/or the patient table 1690 can be moved, e.g. by a user, optionally assisted by one or more motors, controllers, drives, hydraulic and/or electric system, and/or robotic components and/or arms. Optionally, the imaging system, in this non-limiting example a 3D C-arm 1680, and/or one or more of its components, and/or the patient table 1690 can be tracked, e.g. using any of the techniques described throughout the specification. In addition, the patient 1650 and/or the target organ or target anatomic structure, such as a portion of a spine 1710 can also optionally be tracked, e.g. using any of the techniques described throughout the specification. The system, including one or more of its computing systems and/or computer processors, can be configured so that the 3D stereoscopic view or augmented view of the 3D representation 1700 of the surface or volume of the envelope, perimeter, limits, margin, or combination thereof of the intended or desired imaging data acquisition, for example, superimposed, by the HMD or other augmented reality display device, onto the patient (prior to the actual imaging data acquisition) can move in relation with the tracked imaging system 1680, tracked one or more components of the imaging system 16601670, tracked patient table 1690, and/or, optionally, the tracked anatomic target structure, such as a spine 1710 (e.g. optionally with one or more attached markers or fiducials or fiducial arrays, not shown). When the imaging system 1680 or one or more of its components 16601670 are moved, or when the patient table 1690 is moved, or when one or more parameters determining and/or influencing the location and/or orientation of the image acquisition are changed, or any combination thereof, the system can be configured to adjusting the position, orientation, position and orientation of the 3D stereoscopic view or augmented view of the 3D representation 1700 in response to the movement of the tracked imaging system 1680, the movement of the one or more of its components 16601670, the movement of the tracked patient table 1690, and/or the changes in the one or more parameters determining and/or influencing the location and/or orientation of the image acquisition.
In
In
An exploded view 1720, shows what the images will look like after the actual image data acquisition. Only a small lateral portion of the spine 1730 will be included in the image data volume acquisition and the resultant images.
In
The imaging system, in this non-limiting example a radiography (e.g. 3D) 1750, angiography (e.g. 3D, 4D) 1750 or other x-ray based imaging system 1750, and/or one or more of its components, and/or the patient table 1690 can be moved, e.g. by a user, optionally assisted by one or more motors, controllers, drives, hydraulic and/or electric system, and/or robotic components and/or arms. Optionally, the imaging system, in this non-limiting example a radiography (e.g. 3D) 1750, angiography (e.g. 3D, 4D) 1750 or other x-ray based imaging system 1750, and/or optionally one or more of its components, and/or optionally the patient table 1690 can be tracked, e.g. using any of the techniques described throughout the specification. In addition, the patient 1650 and/or the target organ or target anatomic structure, such as a portion of a spine 1710, can optionally also be tracked, e.g. using any of the techniques described throughout the specification. The system, including one or more of its computing systems and/or computer processors, can be configured so that the 3D stereoscopic view of the 3D representation 1700 of the surface or volume of the envelope, perimeter, limits, margin, or combination thereof of the intended or desired imaging data acquisition displayed by the HMD or other augmented reality device at a defined position, orientation, coordinates or combination thereof in relationship to the one or more components of the imaging system (and, as in this example, onto the patient) prior to the actual imaging data acquisition can move in relation with the tracked imaging system 1750, tracked one or more components of the imaging system, optionally tracked patient table 1690, and/or the (optionally tracked) anatomic target structure, such as a spine 1710 (e.g optionally with one or more attached markers or fiducials or fiducial arrays, not shown). When the imaging system 1750 or one or more of its components are moved, or, optionally, when the patient table 1690 is moved, or when one or more parameters determining and/or influencing the location and/or orientation of the image acquisition are changed, or any combination thereof, the system can be configured to adjusting the position, orientation, position and orientation of the 3D stereoscopic view or augmented view of the 3D representation 1700 in response to the movement of the tracked imaging system 1750, the movement of the one or more of its components, the movement of the tracked patient table 1690, and/or the changes in the one or more parameters determining and/or influencing the location and/or orientation of the image acquisition.
In
In
An exploded view 1720, shows what the images will look like after actual the image data acquisition. The entire target portion of the spine 1730 will be included in the image data volume acquisition and the resultant images.
Thus, by moving one or more of the tracked imaging system 1750, tracked one or more components of the imaging system, optionally tracked patient table 1690, and/or the patient 1650, and/or the anatomic target structure, such as a spine 1710 prior to the actual image data acquisition and by optimizing the position, orientation, or position and orientation and/or coordinates of the 3D representation 1700, displayed by the HMD or other augmented reality display device, of a surface or volume of the envelope, margin, perimeter and/or limits of the 3D imaging data volume acquisition intended for a patient 1650, and/or the target anatomic target structure, such as a spine 1710, the actual image data acquisition can be optimized with regard to the coverage and/or inclusion of the target anatomic target structure, such as a spine 1710. This approach can obviate the need for repeat scout images and/or x-ray images (e.g. AP, lateral) prior to the actual 3D volume acquisition.
The imaging system, in this non-limiting example a CT, cone beam CT, spiral CT, MRI system, SPECT system, PET system 1780 or a combination thereof, and/or one or more of its components, and/or the patient table 1690 can be moved, e.g. by a user, optionally assisted by one or more motors, controllers, drives, hydraulic and/or electric system, and/or robotic components and/or arms. Optionally, the imaging system, in this non-limiting example a CT, cone beam CT, spiral CT, MRI system, SPECT system, PET system 1780 or a combination thereof, and/or one or more of its components, and/or optionally the patient table 1690 can be tracked, e.g. using any of the techniques described throughout the specification. In addition, the patient 1650 and/or the target organ or target anatomic structure, such as a portion of a spine 1710, can also optionally be tracked, e.g. using any of the techniques described throughout the specification. The system, including one or more of its computing systems and/or computer processors, can be configured so that the 3D stereoscopic view or augmented view of the 3D representation 1700 of the surface or volume of the envelope, perimeter, limits, margin, or combination thereof of the intended or desired imaging data acquisition displayed by the HMD or other augmented reality device at a defined position/orientation relative to one or more components of the imaging system, optionally superimposed onto the patient, prior to the actual imaging data acquisition can move in relation with the tracked imaging system 1780, tracked one or more components of the imaging system, (optionally tracked) patient table 1690, and/or the anatomic target structure, such as a spine 1710 (optionally also tracked, e.g. with one or more attached markers or fiducials or fiducial arrays, not shown). When the imaging system 1780 or one or more of its components are moved, or when the patient table 1690 is moved, or when one or more parameters determining and/or influencing the location and/or orientation of the image acquisition are changed, or any combination thereof, the system can be configured to adjusting the position, orientation, position and orientation of the 3D stereoscopic view or augmented view of the 3D representation 1700 in response to the movement of the tracked imaging system 1750, the movement of the one or more of its components, the movement of the optionally tracked patient table 1690, and/or the changes in the one or more parameters determining and/or influencing the location and/or orientation of the image acquisition.
In
In
Only a small posterior portion of the spine 1730 will be included in the image data volume acquisition and the resultant images.
In
Thus, by moving one or more of the tracked imaging system 1780, tracked one or more components of the imaging system, (optionally tracked) patient table 1690, and/or the patient 1650, and/or the (optionally tracked) anatomic target structure, such as a spine 1710 prior to the actual image data acquisition and by optimizing the position, orientation, or position and orientation and/or coordinates of the 3D representation 1700 of a surface or volume of the envelope, margin, perimeter and/or limits of the 3D imaging data volume acquisition intended or desired for a patient 1650, and/or the target anatomic target structure, such as a spine 1710, displayed by the HMD or other augmented reality device prior to the actual image data acquisition, the actual image data acquisition can be optimized with regard to the coverage and/or inclusion of the target anatomic target structure, such as a spine 1710. This approach can obviate the need for repeat scout images, e.g. MRI scout images, CT scout images, and/or x-ray images (e.g. AP, lateral) prior to the actual 3D volume acquisition.
In some embodiments, an imaging calibration phantom can be used for determining the location of a 3D representation of a surface or volume containing information about an envelope, limit, perimeter, and/or boundary of an intended imaging acquisition in relationship to one or more components of an imaging system. The imaging calibration phantom can, for example, comprise one or more image visible markers, e.g. spheres, components, or compartments, containing various type of materials depending on the imaging modality used. For example, for imaging systems using x-rays or x-ray beams, the materials can comprise a metal, e.g. aluminum, titanium, steel, lead or any other metal. For imaging systems utilizing radionuclides, the phantom can comprise image visible markers, e.g. spheres, compartments or containers holding, containing one or more radioisotopes or radionuclides. The phantom can comprise spheres, compartments or containers holding, containing contrast media suitable for a particular imaging modality or imaging system. For example, for MRI, the phantom can comprise spheres, compartments or containers containing gadolinium-DTPA-doped water; alternatively, oil based fluids can also be used.
The phantom can comprise defined geometric patterns or arrangements of image visible markers within a single plane or within multiple planes. For example, for imaging modality using x-rays, one or more metal spheres or beads, i.e. image visible markers, can be arranged in a 3D layout, with multiple different spheres or beads located in different locations, each with different x, y, and z-coordinates, for example in multiple layers.
The phantom and the image visible markers, e.g. spheres, beads etc. contained therein can be smaller, similar in size, or larger than the surface, volume, envelope, limit, perimeter of an intended imaging acquisition. For example, the phantom can be larger than the maximum envelope, limit, perimeter and/or boundary of the imaging acquisition. The image visible markers can be closely spaced in 3 dimensions, e.g. at 20 mm, 15 mm, 10 mm, 5 mm, 3 mm, 2 mm, 1 mm intervals or any other interval. The phantom can be tracked using any of the tracking or registration techniques known in the art or described in the specification. For example, the phantom can comprise one or more fiducial markers, optical markers, fiducial arrays etc.
In some embodiments, the phantom can be placed on a table associated with the imaging system, e.g. a patient table, located in the general area and/or opening of the imaging system or imaging system components, e.g. a C-arm or O-arm, CT scanner, cone beam CT or other imaging system.
The system can be configured to track a head mounted display, an augmented or mixed reality display device, a patient table used with the imaging system, an imaging system, one or more components of the imaging system components, or a combination thereof and the phantom.
The geometry of the image visible markers within the phantom or integrated or attached to the phantom can be known based on the manufacturing specification and/or an optional post-manufacturing imaging test.
A computer processor can be configured to detect the image visible markers of the phantom automatically, for example using automated image segmentation software. The phantom with the image visible markers can be placed on the patient table of the imaging system. The location, position, and/or orientation of the phantom can be determined using the tracking system in relationship to the tracked imaging system or tracked one or more components of the imaging system. An image acquisition can be performed; the computer processor can detect the image visible markers included in the image acquisition. Thus, the computer processor can determine which image visible markers were included in the image acquisition and which of the image visible markers of the phantom were not included; the boundary of the image visible markers included in the image(s)/image volume and the image visible markers not included in the image(s)/image volume can be used by the computer processor to determine the boundary, limit, perimeter, envelope, surface or volume of the image acquisition, e.g. in relationship to the tracked phantom and, with that, in relationship to the tracked one or more components of the imaging system. The one or more components of the imaging system can be tracked using extrinsic or intrinsic tracking information and/or techniques, as described throughout the specification. Someone skilled in the art can recognize that the definition of the boundary, limit, perimeter, envelope of the image acquisition, used for the generation of the 3D representation of the surface or volume of the image acquisition, can be improved the more closely the image visible markers are spaced together within the phantom. In some embodiments, the phantom can be larger than the largest image acquisition volume provided by the imaging system.
With the location, position, and/or orientation of the tracked phantom in relationship to the tracked imaging system or tracked one or more components of the imaging system known based on the tracking information, and the geometry of the image visible markers of the phantom known, and/or the image visible markers detected in the image acquisition data/volume known, a computer processor can generate an estimate of a surface or volume of the boundary, limit, perimeter, and/or envelope of the image acquisition.
In some embodiments, the image acquisition can be repeated for different geometries of the imaging system (e.g. different tube detector distances), different patient table heights, different positions, orientations and/or geometries of one or more components of the imaging system (e.g. a collimator), different geometries of the image acquisition, and/or different image acquisition parameters. Using the foregoing techniques for determining the surface or volume of the boundary, limit, perimeter, and/or envelope of the image acquisition, the data and corresponding 3D representations of the surface or volume of the boundary, limit, perimeter, and/or envelope of the image acquisitions can be stored for a given imaging system and different geometries, positions, orientations, and/or image acquisition parameters.
When a patient is subsequently placed on the imaging system table, a 3D representation of the surface or volume of the boundary, limit, perimeter, and/or envelope of the image acquisition can be generated and/or displayed corresponding to or reflecting a set of geometries, positions, orientations, and/or image acquisition parameters selected for that patient and the intended scan. The system can then display, by an HMD or other augmented reality display device, the 3D representation in relationship to the one or more imaging components; the imaging system can be moved to the target anatomic area of the patient, while maintaining the display of the 3D representation in relationship to the one or more imaging components. The imaging system can be moved to an optimized position for a subsequent image acquisition relative to the target anatomic area, as seen in the stereoscopic view or augmented view by the position of the 3D representation in relationship to the target anatomic area.
Someone skilled in the art can recognize that data or information related to the surface or volume of the boundary, limit, perimeter, and/or envelope of am image acquisition can also be retrieved from an imaging system profile or database provided by an imaging system manufacturer; the surface or volume of the boundary, limit, perimeter, and/or envelope of am image acquisition and the 3D representations and can be automatically updated and/or adjusted based on intrinsic imaging system information, e.g. different geometric settings of the imaging system (e.g. different tube detector distances), different positions, orientations and/or geometric settings of one or more components of the imaging system (e.g. a collimator), different geometric settings of the image acquisition, and/or different image acquisition parameters.
In some embodiments, multiple head mounted displays can be used. Head mounted displays (HMD) can be video see-through head mounted displays or optical see-through head mounted displays. Referring to
Virtual data of the patient can be projected superimposed onto live data of the patient for each individual viewer by each individual HMD for their respective view angle or perspective by registering live data of the patient, e.g. the surgical field, and virtual data of the patient as well as each HMD in a common, shared coordinate system. Thus, virtual data of the patient including aspects of a virtual surgical plan can remain superimposed and/or aligned with live data of the patient irrespective of the view angle or perspective of the viewer and alignment and/or superimposition can be maintained as the viewer moves his or her head or body.
When images of the patient are superimposed onto live data seen through the optical head mounted display, in many embodiments image segmentation can be desirable. Any known algorithm in the art can be used for this purpose, for example thresholding, seed point techniques, live wire, deformable models, statistical models, active shape models, level set methods, marching cubes algorithms, artificial neural networks, deep learning techniques, or combinations thereof and the like. Many of these algorithms are available is part of open-source or commercial libraries, for instance the Insight Segmentation and Registration Toolkit (ITK), the Open Source Computer Vision Library OpenCV, G'MIC (GREYC's Magic for Image Computing), Caffe, or MATLAB (MathWorks, Natick, Mass.). A representative workflow for segmentation and subsequent is provided in
An optional intra-operative imaging study 41 can be obtained. The pre-operative 40 or intra-operative 41 imaging study can be segmented 42, extracting, for example, surfaces, volumes or key features. An optional 3D reconstruction or 3D rendering 43 can be generated. The pre-operative 40 or intra-operative 41 imaging study and any 3D reconstruction or 3D rendering 43 can be registered in a common coordinate system 44. The pre-operative 40 or intra-operative 41 imaging study and any 3D reconstruction or 3D rendering 43 can be used for generating a virtual surgical plan 45. The virtual surgical plan 45 can be registered in the common coordinate system 44. The surgical site 46 can be registered in the common coordinate system 44. Intra-operative measurements 47 can be obtained and can be used for generating a virtual surgical plan 45. A head mounted display 48 can project or display stereoscopic images of virtual data or virtual data 49 superimposed onto and aligned with the surgical site. The HMD 48 can be configured to use a built-in camera or image capture or video capture system 50 to optionally detect and/or measure the position and/or orientation and/or alignment of one or more optical markers 51, which can be used for the coordinate measurements 52, which can be part of the intra-operative measurements 47.
In some embodiments, the HMD can display a virtual object, e.g. an arbitrary virtual plane over the surgical field. The virtual object/arbitrary virtual plane can be moveable using a virtual or other interface. For example, the virtual object/arbitrary virtual plane can include a “touch area”, wherein gesture recognition software, for example the one provided by Microsoft with the Microsoft Hololens including, for example, the integrated virtual “drag function” for holograms can be used to move the arbitrary virtual plane. For example, one or more cameras integrated or attached to the HMD can capture the movement of the surgeon's finger(s) in relationship to the touch area; using gesture tracking software, the virtual object/virtual plane can then be moved by advancing the finger towards the touch area in a desired direction. The movement of the virtual object/virtual plane via the user interaction, e.g. with gesture recognition, gaze tracking, pointer tracking etc., can be used to generate a command by a computer processor. The command can trigger a corresponding movement of one or more components of a surgical robot and/or an imaging system.
The HMD can display the virtual object/arbitrary virtual plane in any location initially, e.g. projected onto or outside the surgical field, e.g. a hip joint, knee joint, shoulder joint, ankle joint, or a spine. The HMD can optionally display the virtual object/arbitrary virtual plane at a defined angle, e.g. orthogonal or parallel, relative to a fixed structure in the operating room, which can, for example, be recognized using one or more cameras, image capture or video capture systems and/or a 3D scanner integrated into the HMD and spatial recognition software such as the one provided by Microsoft with the Microsoft Hololens or which can be recognized using one or more attached optical markers or navigation markers including infrared or RF markers. For example, one or more optical markers can be attached to an extension of the operating table. The HMD can detect these one or more optical markers and determine their coordinates and, with that, the horizontal plane of the operating room table. The virtual object/arbitrary virtual plane can then be displayed perpendicular or at another angle relative to the operating room table.
For example, in a hip replacement, the HMD can display a virtual arbitrary plane over the surgical site. The virtual arbitrary plane can be perpendicular to the operating table or at another predefined or predetermined angle relative to the OR table. Using a virtual interface, e.g. a touch area on the virtual surgical plane and gesture tracking, the HMD can detect how the surgeon is moving the virtual arbitrary plane. Optionally, the virtual arbitrary plane can maintain its perpendicular (or of desired other angle) orientation relative to the OR table while the surgeon is moving and/or re-orienting the plane; a perpendicular orientation can be desirable when the surgeon intends to make a perpendicular femoral neck cut. A different angle can be desirable, when the surgeon intends to make the femoral neck cut with another orientation. The position and/or orientation of the virtual object can be transmitted from a second computing system in communication with the HMD to a first computing system, e.g. in communication with a surgical robot and/or an imaging system. The position and/or orientation of the virtual object can be used to set coordinates for an image acquisition, e.g. in an area or volume defined in relationship to the virtual object. The position and/or orientation of the virtual object can be used to set coordinates for a bone resection by a surgical robot, e.g. with an end effector, wherein the end effector can comprise a pin, drill, mill, saw, saw blade, reamer, impactor or a combination thereof.
Using the touch area or other virtual interface, the surgeon can move the virtual object, e.g. arbitrary virtual plane, into a desired position, orientation and/or alignment. The moving of the arbitrary virtual plane can include translation and rotation or combinations thereof in any desired direction using any desired angle or vector, which can be transmitted wirelessly from the second to the first computing system and which can be used to generate one or more commands, for example for moving, aligning, positioning and/or orienting an end effector, a surgical robot, an imaging system or a combination thereof. The surgeon can move the virtual object, e.g. arbitrary virtual plane to intersect with select anatomic landmarks or to intersect with select anatomic or biomechanical axes. The surgeon can move the virtual object, e.g. arbitrary virtual plane to be tangent with select anatomic landmarks or select anatomic or biomechanical axes.
For example, in a hip replacement, the surgeon can move the arbitrary virtual plane to be tangent with the most superior aspect of the greater trochanter and the most superior aspect of the lesser trochanter.
Optionally, for example with a pointer with an attached optical marker or an attached navigation marker, or with his finger detected using an image or video capture system integrated into the HMD and gesture recognition software such as the one provided by Microsoft with the Hololens, or with his finger with an attached optical marker or navigation marker, the surgeon can point at and identify the sulcus point, e.g. the lowest point between the greater trochanter and the femoral neck, which can be an additional reference. The line connecting the most superior aspect of the greater trochanter and the most superior aspect of the lesser trochanter can then be determined on a pre-operative or intra-operative AP radiograph of the hip; optionally, the sulcus point can also be detected on the AP radiograph. The AP radiograph can include a template used by the surgeon for selecting and sizing, for example, the femoral and acetabular component, as well as the liner and/or femoral heads. The radiographic template can include an indication for the femoral neck cut. The angle between the line connecting the most superior aspect of the greater trochanter and the most superior aspect of the lesser trochanter and the indication for the femoral neck cut can be determined.
The virtual femoral neck cut plane prescribed and projected or displayed in this manner can also be a virtual guide, e.g. a virtual cut block that projects, for example, a virtual slot for guiding a physical saw. The virtual guide or virtual cut block can have one or more dimensions identical to a physical guide or cut block, so that the physical guide or cut block can be aligned with the virtual guide or cut block. The virtual guide or cut block can be an outline, 2D or 3D, partial or complete, of the physical guide or cut block, with one or more identical dimensions, so that the surgeon can align the physical guide or cut block with the virtual guide or cut block. The virtual guide or cut block can include placement indicia for the physical guide or cut block.
In a spine, a joint, e.g. a hip joint, one or more HMDs or other augmented reality display systems, one or more virtual data sets or virtual data can be registered in a common coordinate system. In a joint, e.g. a hip joint, two opposing articular surfaces, e.g. with opposing cartilage surfaces and underlying subchondral bone, can be registered separately and/or optionally jointly in a coordinate system, e.g. a common coordinate system. A first articular surface can be located on the pelvic side, i.e. on the acetabulum, a second articular surface can be located on the proximal femur. Registering the first articular surface and/or or associated bones and/or structures and the second articular surface and/or or associated bones and/or structures separately in a common coordinate system can have the benefit of allowing movement, e.g. flexion and/or extension and/or rotation and/or abduction, and/or adduction, and/or elevation and/or other movements, e.g. translation, of the first articular surface and/or or associated bones and/or structures, e.g. on the acetabular side, in relationship to the second articular surface and/or or associated bones and/or structures, e.g. on the proximal femoral side, while maintaining registration of the first articular surface and/or associated bones and/or structures, e.g. on the acetabular side, and/or the second articular surface and/or or associated bones and/or structures, e.g. on the proximal femoral side, e.g. in a common coordinate system or a sub-coordinate system, for example optionally along with one or more HMDs or other augmented reality display systems and/or optionally fixed structures in the operating room, e.g. the OR table, and/or other structures or anatomic landmarks of the patient, e.g. irrespective movement of the individual portions of the joint; the foregoing applies to any joint in the human body, e.g. a shoulder, elbow, wrist, finger, knee, ankle, foot or toe joint or a temporomandibular joint. In this manner, the hip joint or any other joint can be placed in different positions, e.g. flexion, extension, rotation, abduction, adduction, e.g. a degree of hip abduction, e.g. 20, 30, 40 or other degrees, e.g. during placement of a femoral component, and a degree of hip abduction, e.g. 30, 40, or 50 or other degrees, during placement of the acetabular component, or any other degrees for either component placement depending on surgical technique and surgeon preference, while the registration of the acetabular and/or the registration of the proximal femoral side and the display of any virtual data, e.g. a virtual surgical guide, a virtual cut plane, a virtual implant component on the acetabular side and/or the proximal femoral side can be maintained and superimposed onto the corresponding anatomic area, e.g. the area intended for implant component placement, irrespective of the movement of individual portions of the joint, thereby allowing the one or more HMDs or other augmented reality display systems to maintain anatomically registered displays of virtual data superimposed onto the corresponding portions of the physical joint anatomy, e.g. an articular surface, including a normal, damaged and/or diseased cartilage and/or subchondral bone and/or cortical bone, e.g. in a tangent, intersecting and/or offset manner, e.g. external and/or internal to the normal, damaged and/or diseased cartilage and/or subchondral bone and/or cortical bone.
The surgeon can align or substantially superimpose the physical femoral AP cut block with the digital hologram of the virtual femoral AP cut block or its 2D or 3D outline or one or more placement indicators projected by the HMD. Once adequate alignment or superimposition of the physical AP cut block with the virtual AP cut block or its 2D or 3D outline or one or more placement indicators displayed by the HMD has been achieved, the surgeon can pin the physical AP cut block and perform the cuts. By utilizing preoperative 3D data information or intra-operative information, e.g. from optical marker and image or video capture or scanner measurements (including stereoscopic imaging), for the position, alignment and rotation of the physical femoral AP cut block with the assistance of the HMD, the surgeon can perform the anterior and posterior femoral cuts in a highly accurate manner, thereby achieving accurate rotational alignment of the femoral component. The same approaches and display options, e.g. virtual cut blocks, 2D or 3D outline or one or more placement indicators, can be applied to all subsequent femoral preparation steps including chamfer cuts and chamfer cut blocks.
Optionally, a virtual interface, e.g. with one or more virtual objects, can be displayed by the HMD(s), for interaction by the surgeon(s). For example, the virtual object can be a virtual surgical guide, e.g. the virtual AP cut block. The interaction can comprise moving the virtual object, e.g. the virtual surgical guide, for example using a tracked pointer, a gesture recognition, a finger tracking, an object tracking, etc. The interaction can be used to generate, by a computer processor, a command which can be configured, for example, to initiate, start, stop, activate, de-activate, move, adjust the position/orientation of the virtual object, e.g. virtual surgical guide, and, optionally correspondingly, one or more components, controllers, drivers, motors, sensors, relays, processors or combination thereof of a surgical robot and/or an imaging system. The command can, optionally, be transmitted from a computing system/computer processor in communication with the one or more HMDs or other augmented reality display systems to a computing system/computer processor in communication with a robot and/or an imaging system.
Of note, similar steps and HMD guided femoral procedures are also possible using the HMD with any of the other registration and cross-referencing techniques described in the present disclosure or known in the art, for example intraoperative image guidance.
In
Optionally, a virtual interface, e.g. with one or more virtual objects, can be displayed by the HMD(s), for interaction by the surgeon(s). For example, the virtual object can be a virtual surgical guide, e.g. a virtual cut plane, a virtual pin, a virtual drill, a virtual axis, a virtual tool, a virtual instrument, a virtual implant. The interaction can comprise moving the virtual object, e.g. the virtual surgical guide, for example using a tracked pointer, a gesture recognition, a finger tracking, an object tracking, etc. The interaction can be used to generate, by a computer processor, a command which can be configured, for example, to initiate, start, stop, activate, de-activate, move, adjust the position/orientation of the virtual object, e.g. virtual surgical guide, e.g. the virtual cut plane, virtual pin, virtual drill, virtual axis, virtual tool, virtual instrument, virtual implant and, optionally correspondingly, one or more components, controllers, drivers, motors, sensors, relays, processors or combination thereof of a surgical robot and/or an imaging system. The command can, optionally, be transmitted from a computing system/computer processor in communication with the one or more HMDs or other augmented reality display systems to a computing system/computer processor in communication with a robot and/or an imaging system.
In some embodiments, data can be obtained using an HMD, for example one manufactured by Microsoft, the Microsoft Hololens or Hololens 2 (Microsoft, Redmond, WI). The Hololens can use, for example, Windows holographic APIs including Unity (Unity Technologies, San Francisco, CA) and Vuforia 6.2 (PTC, Inc., Needham, MA).
Markers can be mounted on a wooden board with a size of 2.8×2.8 cm and arranged at a distance of 3.0×3.0 cm and static measurements of displacement of optically detected marker positions vs. actual marker positions can be obtained at an angle of approximately 40 degrees between the Hololens and the board at a distance of approximately 32.5 cm to the center of the board.
A virtual interface for the path of a a pedicle screw, an interbody device, or other types of implant, e.g. for joint replacement, can use, for example, the Unity for HoloLens engine (Unity Technologies, San Francisco, CA). Unity's GestureRecognizer interface allows for recognition of different hold, navigation and manipulation functions. Additionally, the Gaze functionality can be available for implementation of a cursor controlled by the user's view direction. Thus, in select applications, the user's gaze can control the cursor including cursor movement. Closure of the eye lid can, for example, also be used to generate a command to execute a function. With the virtual interface, the planning can be performed, for example, on fluoroscopic images displayed by an HMD using gesture commands which are mapped to entry points and vectors. A vector corresponding to the intended path of the surgical instrument(s), e.g. an awl or the pedicle screw, can be placed by the surgeon using gesture commands, e.g. a closed position of the thumb and index finger or an open position of a thumb and index finger as shown in
The user interaction, e.g. via gesture recognition or a virtual user interface, for example displaying virtual objects such as virtual buttons, sliders, keyboards, etc., can be used to generate, by a computer processor, a command which can be configured, for example, to initiate, start, stop, activate, de-activate, move, adjust the position/orientation of the virtual object, e.g. virtual surgical guide, e.g. the virtual cut plane, virtual trajectory, virtual pin, virtual drill, virtual axis, virtual tool, virtual instrument, virtual implant and, optionally correspondingly, one or more components, controllers, drivers, motors, sensors, relays, processors or combination thereof of a surgical robot and/or an imaging system. The command can, optionally, be transmitted from a computing system/computer processor in communication with the one or more HMDs or other augmented reality display systems to a computing system/computer processor in communication with a robot and/or an imaging system, which can then effect or cause the initiating, starting, stopping, activating, de-activating, moving, adjusting of the one or more components, controllers, drivers, motors, sensors, relays, processors or combination thereof of a surgical robot and/or an imaging system. For example, the command can cause the computing system/computer processor in communication with a robot to move a robotic arm and end effector and/or a drill guide to align with a trajectory defined by the surgeon using the virtual user interface and/or gesture recognition.
The user interaction, e.g. via gesture recognition or a virtual user interface, for example displaying virtual objects such as for example one or more virtual button, virtual field, virtual cursor, virtual pointer, virtual slider, virtual trackball, virtual node, virtual numeric display, virtual touchpad, virtual keyboard, or a combination thereof, can be used to generate, by a computer processor, a command which can be configured, for example, to initiate, start, stop, activate, de-activate, move, adjust the position/orientation of the virtual object, e.g. virtual surgical guide, e.g. the virtual cut plane, virtual trajectory, virtual pin, virtual drill, virtual axis, virtual tool, virtual instrument, virtual implant (e.g. a virtual screw or virtual cage or other interbody device) and, optionally correspondingly, one or more components, controllers, drivers, motors, sensors, relays, processors or combination thereof of a surgical robot and/or an imaging system. The command can, optionally, be transmitted from a computing system/computer processor in communication with the one or more HMDs or other augmented reality display systems to a computing system/computer processor in communication with a robot and/or an imaging system, which can then effect or cause the initiating, starting, stopping, activating, de-activating, moving, adjusting of the one or more components, controllers, drivers, motors, sensors, relays, processors or combination thereof of a surgical robot and/or an imaging system. For example, the command can cause the computing system/computer processor in communication with a robot to move a robotic arm and end effector and/or a drill guide to align with a trajectory defined by the surgeon using the virtual user interface and/or gesture recognition.
All publications, patents mentioned herein are hereby incorporated by reference in their entirety as if each individual publication or patent was specifically and individually indicated to be incorporated by reference.
This application is a continuation of U.S. patent application Ser. No. 17/691,923, filed Mar. 10, 2022, which claims the benefit of and the priority to U.S. Provisional Application Ser. No. 63/158,941, filed Mar. 10, 2021, U.S. Provisional Application Ser. No. 63/163,156, filed Mar. 19, 2021, U.S. Provisional Application Ser. No. 63/173,565, filed Apr. 12, 2021, and U.S. Provisional Application Ser. No. 63/232,376, filed Aug. 12, 2021, the entire contents of each of which is hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| 63158941 | Mar 2021 | US | |
| 63163156 | Mar 2021 | US | |
| 63173565 | Apr 2021 | US | |
| 63232376 | Aug 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17691923 | Mar 2022 | US |
| Child | 18465388 | US |
