Patent Application
20220013223
A majority of surgical procedures are performed using a technique called minimally invasive surgery (MIS) wherein a video camera and the surgical instruments are inserted into the body cavity through small openings called portals. MIS is less traumatic on the patient and results in quicker recovery times.
Advances in video technology, computer enhanced vision, and artificial intelligence enable better and faster procedures. However, MIS techniques such as endoscopic and laparoscopic surgeries, where there is only an indirect view of the operative field, present new challenges. For example, unlike open surgeries, there is no easy way for the surgeon to point by gesture to an observed tissue anomaly or physical artifact. The ability to direct and focus viewer attention by gesture is one of the most intuitive and important means for a surgeon to convey accurate information to support staff, consulting surgeons and specialists who are witnessing a live procedure over video. Hand gestures are also a convenient means of interacting with supporting equipment.
New video assistive tools and methods continue to evolve and are enabled by ever more sophisticated graphical user interfaces and software using mathematical algorithms, artificial intelligence, and augmented reality. Previously described, the Software Tools Platform for Medical Environments (U.S. Pat. No. 9,526,586B2), a.k.a. Surgeons Video Tool Kit (SVTK) is an example that provides a continuously expanding set of software tools. Many of these tools were pioneered in sophisticated graphical user interface software and video games. The tools provide enhanced vision, instant analysis, and extend human perception and analysis beyond human capability.
Many of these innovations depend upon the hand-eye coordination capabilities of the surgeon. Most surgical procedures require a surgeon to use and coordinate two instruments simultaneously, controlled by the left and right hands. Because both hands are busy, user interfaces such as voice commands, foot pedals, body gestures, even computer monitored eye movement are used to direct assistance from computers, electronic monitoring devices, and supporting staff.
Surgical video is playing an increasing role in modern medicine. Indeed, most surgeons reported using videos to prepare for surgery and indicated that YouTube was the preferred source. There is growing support for intraoperative video recording. In the past, the widespread adoption of intraoperative video has been hampered by legal concerns relating to healthcare provider liability, and patient privacy.
Video analysis allows studying both surgical technique (i.e. the details of how an operation is conducted) and surgical skill (i.e. how well a surgeon performs a procedure). There is growing enthusiasm to tackle the challenges of directly evaluating and improving surgeon performance using intraoperative video.
Surgical video in combination with intraoperative computer vision opens the door to real-time, automated surgical analysis. It enables artificial intelligence (AI) to analyze and interpret videos during an operation. By teaching the AI to understand what is happening during surgeries, the AI will develop capabilities to assist surgeons in assessing the risk for a postoperative complication or even provide surgeons with additional data to improve operating room decisions.
The present invention seeks to address the need to precisely identify and communicate a specific position on tissue or an organ with a Virtual Pointing Device (VPD) software tool. The VPD uses a combination of audio key words and the surgeon's hand movements to invoke various functionality. For example, the VPD can be used to overlay a synthetic visual dotted line starting from the end of the selected instrument and extending a specified distance in the direction the instrument is pointed. The VPD accomplishes this by analyzing the live endoscopic video and calculating the direction that the surgical instrument is pointing. Depending on the instrument employed, the system determines whether to use edge detection algorithms for resolving direction, or artificial intelligence, or a model derived from physical specifications and/or artificial intelligence neural networks trained with visual observations of the instrument.
To fully realize the potential benefits surgical video requires a secure and highly scalable compute infrastructure that networks various sources of surgical video and supports saving, storing, managing processing and distributing vast volumes of video.
The present invention, referred to herein as Cloudcapture, is a foundational video architecture and a framework for collecting surgical video at an enterprise scale. It enables hospitals to ingest, manage, and fully utilize patient surgical video within the hospital and to share video with designated users outside of the hospital. It is a passive solution that automatically records video during surgery and provides key clips post surgery. It stores video to the cloud and integrates videos to the patient's Electronic Health Record (EHR).
Other features and aspects of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the features in accordance with embodiments of the invention. The summary is not intended to limit the scope of the invention, which is defined solely by the claims attached hereto.
The process flow for a surgeon in the operating room before a procedure can be described as follows:
The bottom Display stripe 504 is available to software functions for a variety of purposes. Examples may include:
The main menu panel 506 is always located at the upper right corner of both the Primary and Assistant displays. The menu 506 is illustrated in an open mode with the submenu selections displayed as an ordered list. It may also be collapsed into a small single stripe. The screen pixels allocated for the Menu Panel 506 is 600 across×1890 (20×70 Visio template cells). Each submenu horizontal stripe, when closed, is 600 pixels wide (20×30). When opened, the submenu panel extends downward as needed.
The preferences submenu 602 consists of defaults selected for the current surgeon or procedure. Once the preferences are selected, the submenu 602 can be closed until a new procedure or a change is needed. Selections are saved and recalled when the Surgeon/Procedure is used again. For example, the Grid Tool. The Pop ups submenu consists of functions that remain active until no longer needed. For example, the Annotation Tools. The built-in submenu consists of functions that were selected to be part of the default screen layout configured for a specific surgeon or procedure. These functions have a dedicated display area. For example, the SnapShots, Main video window, DVR tools.
Submenu functions that are not part of the default screen layout still need a dedicated display area (“scratchpad”) 604 to execute their tasks. Two types of “scratchpad” display panels 604 are available. The Drop Down can be expanded in place within the main menu panel using same width and as much depth as needed, subject to availability. The Grid tool is shown opened as a Drop Down. The Pop Up can be used for functions that require a large scratchpad area can pop up an arbitrary size scratchpad window anchored to a specified X,Y location, possibly beside or on top of the main video or other windows, for example, the Video Enhancement window. Words in highlighted color are commands and appear in the bottom stripe when the menu selection is active. Commands can be activated by voice, by mouse click, or keyboard of underlined character.
A Snapshot frame 902 is displayed with 3 components: the top stripe 904 (240×27 pixels); the scaled image 906 (240×135 pixels); and the bottom stripe 908 (240×27 pixels). The top stripe 902 uses an Alpha Identifier (A, B, C, D, . . . ) and can be selected (clicked) to expose a popup metadata panel. The bottom stripe 908 consists of a selection check box, time stamp hh:mm:ss, and audio overlay icon.
The snapshot display carousel 910 is a dedicated display area for six snapshots with a horizontal scroll capability with the following functions: six snapshots scrolling backwards/forwards; clicking the stripe jumps back/forward one frame; and default screen position is 0,0 but can be repositioned. Database entry for each snapshot includes: the procedure ID; the originator (surgeon, or assistant, or a permissioned collaborator); the SMPTE timestamp; the bit flags (selected, . . . , . . . , . . . , . . . ); the snapshot sequential identifier (a, b, c, . . . ); the top stripe; the compressed thumbnail; the bottom stripe; a link to first audio overlay database; a link to first annotation overlay database; and a link to first drag-n-drop to collaborator database. The post procedure options include options to record selected frames to USB or IP Port or to record selected frames to a USB or an IP Port.
In one embodiment, each transaction (or a block of transactions) is incorporated, confirmed, verified, included, or otherwise validated into the blockchain via a consensus protocol. Consensus is a dynamic method of reaching agreement regarding any transaction that occurs in a decentralized system. In one embodiment, a distributed hierarchical registry is provided for device discovery and communication. The distributed hierarchical registry comprises a plurality of registry groups at a first level of the hierarchical registry, each registry group comprising a plurality of registry servers. The plurality of registry servers in a registry group provide services comprising receiving client update information from client devices, and responding to client lookup requests from client devices. The plurality of registry servers in each of the plurality of registry groups provide the services using, at least in part, a quorum consensus protocol.
As another example, a method is provided for device discovery and communication using a distributed hierarchical registry. The method comprises Broadcasting a request to identify a registry server, receiving a response from a registry server, and sending client update information to the registry server. The registry server is part of a registry group of the distributed hierarchical registry, and the registry group comprises a plurality of registry servers. The registry server updates other registry servers of the registry group with the client update information using, at least in part, a quorum consensus protocol.
The VPD uses a combination of audio key words and the surgeon's hand movements to invoke various functionality. For example, the VPD can be used to overlay a synthetic visual dotted line starting from the end of the selected instrument and extending a specified distance in the direction the instrument is pointed. The VPD accomplishes this by analyzing the live endoscopic video and calculating the direction that the surgical instrument is pointing. Depending on the instrument employed, the system determines whether to use edge detection algorithms for resolving direction, or artificial intelligence, or a model derived from physical specifications and/or artificial intelligence neural networks trained with visual observations of the instrument.
While various embodiments of the disclosed technology have been described above, it should be understood that they have been presented by way of example only, and not of limitation. Likewise, the various diagrams may depict an example architectural or other configuration for the disclosed technology, which is done to aid in understanding the features and functionality that may be included in the disclosed technology. The disclosed technology is not restricted to the illustrated example architectures or configurations, but the desired features may be implemented using a variety of alternative architectures and configurations. Indeed, it will be apparent to one of skill in the art how alternative functional, logical or physical partitioning and configurations may be implemented to implement the desired features of the technology disclosed herein. Also, a multitude of different constituent module names other than those depicted herein may be applied to the various partitions. Additionally, with regard to flow diagrams, operational descriptions and method claims, the order in which the steps are presented herein shall not mandate that various embodiments be implemented to perform the recited functionality in the same order unless the context dictates otherwise.
Although the disclosed technology is described above in terms of various exemplary embodiments and implementations, it should be understood that the various features, aspects and functionality described in one or more of the individual embodiments are not limited in their applicability to the particular embodiment with which they are described, but instead may be applied, alone or in various combinations, to one or more of the other embodiments of the disclosed technology, whether or not such embodiments are described and whether or not such features are presented as being a part of a described embodiment. Thus, the breadth and scope of the technology disclosed herein should not be limited by any of the above-described exemplary embodiments.
Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open ended as opposed to limiting. As examples of the foregoing: the term “including” should be read as meaning “including, without limitation” or the like; the term “example” is used to provide exemplary instances of the item in discussion, not an exhaustive or limiting list thereof; the terms “a” or “an” should be read as meaning “at least one,” “one or more” or the like; and adjectives such as “conventional,” “traditional,” “normal,” “standard,” “known” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. Likewise, where this document refers to technologies that would be apparent or known to one of ordinary skill in the art, such technologies encompass those apparent or known to the skilled artisan now or at any time in the future.
While various embodiments of the disclosed technology have been described above, it should be understood that they have been presented by way of example only, and not of limitation. Likewise, the various diagrams may depict an example architectural or other configuration for the disclosed technology, which is done to aid in understanding the features and functionality that may be included in the disclosed technology. The disclosed technology is not restricted to the illustrated example architectures or configurations, but the desired features may be implemented using a variety of alternative architectures and configurations. Indeed, it will be apparent to one of skill in the art how alternative functional, logical or physical partitioning and configurations may be implemented to implement the desired features of the technology disclosed herein. Also, a multitude of different constituent module names other than those depicted herein may be applied to the various partitions. Additionally, with regard to flow diagrams, operational descriptions and method claims, the order in which the steps are presented herein shall not mandate that various embodiments be implemented to perform the recited functionality in the same order unless the context dictates otherwise.
Although the disclosed technology is described above in terms of various exemplary embodiments and implementations, it should be understood that the various features, aspects and functionality described in one or more of the individual embodiments are not limited in their applicability to the particular embodiment with which they are described, but instead may be applied, alone or in various combinations, to one or more of the other embodiments of the disclosed technology, whether or not such embodiments are described and whether or not such features are presented as being a part of a described embodiment. Thus, the breadth and scope of the technology disclosed herein should not be limited by any of the above-described exemplary embodiments.
Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open ended as opposed to limiting. As examples of the foregoing: the term “including” should be read as meaning “including, without limitation” or the like; the term “example” is used to provide exemplary instances of the item in discussion, not an exhaustive or limiting list thereof; the terms “a” or “an” should be read as meaning “at least one,” “one or more” or the like; and adjectives such as “conventional,” “traditional,” “normal,” “standard,” “known” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. Likewise, where this document refers to technologies that would be apparent or known to one of ordinary skill in the art, such technologies encompass those apparent or known to the skilled artisan now or at any time in the future.
The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent. The use of the term “module” does not imply that the components or functionality described or claimed as part of the module are all configured in a common package. Indeed, any or all of the various components of a module, whether control logic or other components, may be combined in a single package or separately maintained and can further be distributed in multiple groupings or packages or across multiple locations.
Additionally, the various embodiments set forth herein are described in terms of exemplary block diagrams, flow charts and other illustrations. As will become apparent to one of ordinary skill in the art after reading this document, the illustrated embodiments and their various alternatives may be implemented without confinement to the illustrated examples. For example, block diagrams and their accompanying description should not be construed as mandating a particular architecture or configuration.
Embodiments presented are particular ways to realize the invention and are not inclusive of all ways possible. Therefore, there may exist embodiments that do not deviate from the spirit and scope of this disclosure as set forth by appended claims, but do not appear here as specific examples. It will be appreciated that a great plurality of alternative versions are possible.
This application claims the benefit of U.S. Provisional Application Ser. No. 63/029,324, filed on May 22, 2020, and U.S. Provisional Application Ser. No. 63/146,530, filed on Feb. 5, 2021, the contents of which are incorporated herein.
| Number | Date | Country | |
|---|---|---|---|
| 63029324 | May 2020 | US | |
| 63146530 | Feb 2021 | US |
