Patent Application
20250217982
Managing surgical sponges is an area of importance in the modern surgical suite, most importantly, ensuring that no surgical sponges (or other objects) are inadvertently retained inside a patient or otherwise misplaced. In the past, healthcare professionals have relied upon manual sorting and counting of surgical sponges. More recently, surgical sponge management systems have utilized electronics to assist with counting the surgical sponges. One such system is sold under the tradenames SurgiCount and SurgiCount+ by Stryker Corporation (Kalamazoo, Mich.) and disclosed in commonly-owned United States Patent Publication No. 2013/0088354, published Apr. 11, 2013, hereby incorporated by reference in its entirety, in which a radiofrequency identification (RFID) reader detects RFID tags on the surgical sponges.
Determining blood loss during surgery may be used to monitor intraoperative patient health. It is known to estimate blood loss during surgery by visual evaluation of the surgical sponges and other fluid-absorbing articles (e.g., surgical gowns, bedding, or drapes), which is inherently subjective and therefore prone to human error. It is also known to estimate blood loss during surgery by weighing the surgical sponges on a scale in bulk, which requires the user transport the surgical sponges to the scale to be weighed. Advances in imaging and computing have provided for quantifying blood loss by capturing an image of the surgical sponge. One such system is sold under the tradename Triton by Gauss Surgical, Inc. (Menlo Park, Calif.) and disclosed in commonly-owned U.S. Pat. No. 8,897,523, issued Nov. 25, 2014, and U.S. Pat. No. 10,424,060, issued Sep. 24, 2019, the entire contents of each being hereby incorporated by reference.
The systems using image-based blood quantification may not adequately compensate for differing types of surgical sponges, and may not realize the advanced functionality of the surgical sponge management system. Therefore, there is a need in the art to provide for an improved system and methods that overcome the aforementioned disadvantages and does so in an intuitive and seamless workflow to the user who may be accustomed to existing surgical protocols.
The present disclosure is generally directed to analysis of surgical articles through image-based processing that is based on an article type of the surgical article being known. The primary embodiment is described in the context of a surgical sponge management system and methods for quantifying blood loss on surgical sponges bloodied during a surgical procedure. However, it is contemplated that the objects of the present disclosure may be applied for surgical articles other than surgical sponges, including sharps, tools (powered or manual), and other instruments. Among other advantages, the functionality of the system and methods obviate the need for the user to determine, recall, and/or manually enter an article type, and/or engage any screen or device to trigger capture of an image for analysis. For surgical sponges, the system is able to perform QBL on surgical sponges of differing sizes and plys, and may do so automatically and in real-time without requiring the user to meaningfully alter the workflow of counting out the surgical sponges. The need to transport the surgical sponges to a separate scale for weighing is likewise obviated. Further, the surgical sponge management system provides for QBL without any increase in footprint within the surgical suite, and accommodates the sponge sizes being imaged or scanned at a generally common distance from the camera and at a convenient height for the user. For other surgical articles, the system is able to confirm the article being presented to the system is of a same article type as previously identified. The image processing may further facilitate identification of damaged articles, used versus unused article status, article cleaning status, automated article counting, and the like. Therefore, the terminology associated with the sponge-based methods described herein may be modified to apply to non-sponge-based surgical articles as well.
The surgical sponge management system may include a user interface, a data reader, one or more processors, memory, communications device, and/or other hardware. The user interface may be a tablet with a touchscreen display. A camera including an optical sensor and optionally a depth sensor may be integrated on the tablet. Neural networks may be implemented or executed on the processor of the tablet, on one or more processors of hardware coupled to or remote from the tablet, via cloud computing, combinations thereof, or the like.
The data reader may be used either as a handheld device, or when supported by a cradle. The data reader is configured to detect tags associated with the surgical sponges. The data reader may a radiofrequency (RFID) reader configured to detect RFID tags. The surgical sponges may be counted in by causing the tag to be detected by the data reader. The detected tag transmits identifying data to the processor. The tag includes at least one unique identifier indicative of characteristics of the surgical sponge. The unique identifier may include sponge size or type. The processor is configured to determine a type of the surgical sponge based on the unique identifier. The processor may index a counter to reflect the one or more surgical sponges being counted in to surgical procedure. The counter may be displayed on the user interface.
The method includes counting out the surgical sponge. The tag of the surgical sponge is detected by the data reader, and the unique identifier stored on the tag are received by the processor. The processor identifies the sponge type of the surgical sponge from a database of predetermined sponge types based on the unique identifier. The user interface may index and display the counter including an updated count for each sponge type that has been counted in to and counted out of the surgical procedure.
The method may include triggering the system for sponge imaging. The camera may be automatically activated in response to the detection of the tag with the data reader. An image feed, as detected by the camera may be displayed on the user interface. The image feed may be based on data being received from the optical sensor and/or the depth sensor. The image feed may include image frames forming a video of the user holding the sponge in a field of view of the optical sensor. The image feed may include graphical augmentations or indicia based on data from the optical sensor and/or the depth sensor. The displaying of the image feed may be automatically performed in response to the detection of the tag with the data reader.
The method may include displaying, on the user interface, the presentation window overlying the image feed being captured by the camera. The dimensions of the presentation window are based on the identified sponge type of the surgical sponge corresponding to the detected tag of the surgical sponge being counted out. The processor may determine acceptance criteria based on the type of surgical sponge—and the presentation window may be sized and scaled so that at least some of the acceptance criteria are met if the user presents the surgical sponge in a manner to at least substantially match or conform to the presentation window. Alternatively, the presentation window may not be displayed, but rather the processor may determine a dynamic presentation window based on an actual location of the sponge within the optical and/or depth sensors.
The image feed is processed or analyzed for satisfaction of the acceptance criteria or guardrails. The acceptance criteria are generally designed to ensure the QBL image of the surgical sponge captured by the optical sensor is high quality. The step includes analyzing each image frame (or every few image frames) of the image feed to ensure a stretched, correct surgical sponge is presented at the appropriate distance, unfolded, and not moving, among other acceptance criteria. If any one or more of the acceptance criteria remains unsatisfied, the processor prevents the camera from capturing the QBL image of the surgical sponge for QBL analysis. Once the acceptance criteria are determined to be satisfied, the method includes the step of acquiring, with the camera, the QBL image(s) of the surgical sponge. The QBL image may be the subsequent frame(s) of the image feed immediately following satisfaction of the acceptance criteria and output from a sponge recognition algorithm executing a sponge recognition state machine (SRSM). The QBL image is analyzed with a hemoglobin estimation engine for determination of a blood component (e.g., hemoglobin) of the liquid on the surgical sponge. The processor may extract a color component value (e.g., a redness value) from the QBL image, and execute a trained hemoglobin estimation algorithm implementing a hemoglobin estimation engine to determine the mass of the blood component on a pixel-by-pixel or other suitable basis. The hemoglobin estimation algorithm may be unique to the sponge type having been counted out. The hemoglobin estimation algorithm may be trained on extensive datasets for each sponge type to be compatible with the system. The blood loss may be determined based on the blood component. By using the sponge type to determine characteristics of the presentation window and the acceptance criteria, the image-based QBL may be applied to more or most types of surgical textiles, including those of differing sizes, plys, and/or colors.
For certain sponge types of lower plys, the sponge may need to be folded before imaging. The presentation window being displayed on the user interface may be used to guide the user to fold the surgical sponge. The method may also include the optional step of displaying, on the user interface, a folding protocol to the user. The presentation window may be dimensioned to approximate the dimensions of the surgical sponge when folded according to the folding protocol. The folding protocol may include guidance to facilitate the user folding the surgical sponge in the desired manner. The guidance may be textual instructions or graphical indicia, and/or audible content.
A sponge recognition algorithm is executed by the one or more processors and configured to process the images of the surgical sponges to detect the sponge and apply the acceptance criteria or guardrails. The sponge recognition algorithm may include image pre-processing, neural networks, labeling logic, guardrail algorithms, the SRSM, and any post-image processing. There may be at least two neural networks: a localization neural network and a segmentation neural network. The localization neural network is configured to, among other actions, detect whether or not there is a sponge in the image, and provide a bounding box around the detected sponge. The bounding box may include information about the detected sponge, for example, a width, height, x-shift and y-shift. Additionally, the localization neural network may be configured to determine if a non-stretched, or a partial sponge or no sponge is presented within the presentation window. The segmentation neural network provides per-pixel prediction of a sponge segmentation mask of whether a pixel is a sponge pixel or background pixel. The segmentation neural network may utilize data from both the optical sensor and the depth sensor, or alternative from only the optical sensor. A localization label receives output from the localization neural network, and a segmentation labeler receives output from the segmentation neural network. The labels are fed to the SRSM to identify video-based states based on dominant class predictions over a preset number of previous frames. The output of the sponge recognition algorithm is provided to the hemoglobin estimation algorithm.
First, the sponge recognition algorithm determines whether a surgical sponge is present. Next, the method includes determining whether the surgical sponge is folded by determining whether the surgical sponge is at least substantially square or rectangular. The localization neural network generates the bounding box of the sponge being detected. Based on outer edges of the bounding box, the localization labeler determines whether the surgical sponge is sufficiently square or rectangular. The determination may be associated with the sponge type. The step may include determining whether an aspect ratio of the presented surgical sponge is within an acceptable range based on the sponge type being counted out. The localization neural network provides the bounding box and compares a determined aspect ratio of the bounding box against the acceptable range of the aspect ratio for that sponge type. If the determined aspect ratio of the bounding box is outside of the acceptable range, the localization labeler determines the sponge is a folded sponge, and generates a localization label accordingly.
The method may include determining whether the sponge being presented is a partial sponge. If the bounding box of the sponge extends beyond the field of view of the camera, the localization labeler determines a partial sponge condition. Further, the localization labeler may determine whether the bounding box is too close to the presentation window. A margin, i.e., a pixel-based distance between the bounding box and the presentation window, may be determined, and compared against margin threshold values. If the determined margin is less than the margin threshold value, the localization labeler generates the localization label the presented surgical sponge is a partial sponge.
The sponge recognition algorithm may include determining whether the sponge is positioned too close or too far from the camera by determining whether the sponge is too large or too small, respectively. The localization labeler determines a pixel-based area of the bounding box, and compares the bounding box area against an acceptable range of bounding box area that is unique to each sponge type. The acceptable range of the bounding box area may approximate the presentation window. If the processor determines the bounding box area is greater than or less than the acceptable range of the bounding box area, the localization labeler generates a localization label that the sponge is too close or too far from the camera, respectively.
The segmentation neutral network outputs the segmentation mask in which sponge pixels and the background pixels are separated. The segmentation neutral network may run on the image frames of the image feed only for which the localization neural network has determined a sponge is present. The segmentation neural network may run in parallel or sequentially after the localization network to optimize processing speed and resource tradeoff. The method includes validating the sponge segmentation mask by determining a number of pixels in the sponge segmentation mask, and comparing the determined number of pixels against an acceptable range of pixels. The acceptable range of pixels is based on the sponge type, which again is identified by the processor with the counting out of with the data reader. If the segmentation mask, as determined by the segmentation neutral network, does not have a sufficient number of pixels, the segmentation mask is determined to be invalid. The segmentation labeler may generate a segmentation label accordingly.
A segmentation mask refinement algorithm may reject pixels from the segmentation mask that have different depth values that exceed a threshold when compared to most of the mask pixels. Differences in the depth values may be estimated as the distance between respective points and plane fitted to the depth map. The threshold may be the same or different for different sponge types. The depth data may be used to establish thresholds on acceptable presentation distance. The method may include determining whether the presented surgical sponge is too close or too far. The segmentation labeler may output a corresponding segmentation label that the presented surgical sponge too close or sponge too far if the average depth of all pixels belonging to the sponge segmentation mask is less than or greater than the minimum threshold or maximum threshold, respectively.
The segmentation labeler may utilize a depth map of depth data obtained from the depth sensor to determine real-world dimensions of the presented surgical sponge for assessing further acceptance criteria. The method may further include determining whether the presented surgical sponge is the correct sponge. By leveraging the depth map to determine the sponge area of the segmentation mask, the segmentation labeler compares the determined real-world sponge area against an acceptable range of sponge area, which is based on the sponge type being counted out. If the determined real-world sponge area of the segmentation mask is outside the acceptable range of the sponge area, the segmentation labeler outputs a wrong sponge label. The step may further include comparing a sponge compactness against a predetermined compactness threshold. The predetermined compactness threshold may be a fraction of the area of the bounding box that must be exceeded by the determined sponge area of the segmentation mask to be determined as correct.
The method may further include determining whether the presented surgical sponge is moving. A centroid of the sponge segmentation mask may be determined, and an average motion magnitude of the centroid may be determined within a predetermined number of image frames of the image feed. The step may include comparing the average motion magnitude against a centroid change threshold, which may be based on the sponge type being counted out. If the average motion magnitude is greater than the centroid change threshold, the segmentation labeler determines the presented surgical sponge to be moving too quickly, and the acceptance criteria to be not satisfied.
The segmentation and localization neural networks can be combined into a single neural network that outputs both the classification labels (e.g., no sponge, partial sponge, folded sponge, and full sponge) as well as segmentation masks and the associated bounding box around the detected sponge. Further, the segmentation and localization labeler can be combined to check for each of the guardrail as a sequence of condition check based on detected sponge feature and expected sponge type. The segmentation and localization neural networks may be executed concurrently or sequentially.
The method may include performing sponge supervision in which the user's hand, torso, neck, and other associated body parts is detected using a pose estimation neural network, as well as the location of the presented surgical sponge using the localizer neural network or the segmentation neural network. Once the person and the presented surgical sponge are detected and located, the pose estimation neural network is configured to determine whether the presented surgical sponge is being held properly in front of the torso. The sponge supervision algorithm may identify a body centerline, and a sponge vertical centerline, and further determine a horizontal shift in location of the presented surgical sponge relative to body centerline. One or more of the additional centerlines may be compared against a horizontal sponge centerline of the presented surgical sponge to determine a vertical shift of the presented surgical sponge. The horizontal shift and/or a vertical shift may be compared against a respective shift threshold relative and/or a predetermined, preferred position. Guidance may be provided on the user interface to instruct the user to present the surgical sponge in front of their torso if the shift(s) exceed the shift threshold. The depth map from the depth sensor may be used to distinguish between people in background and the foreground, and the pose estimation algorithm determines which of multiple people in the field of view of the camera is presenting the sponge.
The sponge recognition algorithm includes the sponge recognition state machine (SRSM). The SRSM is configured to receiving the labels from each of the localization labeler and the segmentation labeler and performs further processing prior to the camera capturing the QBL image. The SRSM algorithm determines whether the acceptance criteria or guardrails, executed potentially separately by the localization labeler and the sponge segmentation labeler, have been satisfied, by combining the aggregate state from both the localization and segmentation labeler. The SRSM algorithm may execute a smoothing procedure to aggregate the localization and segmentation labels using a sliding window with an image frame size. A most frequent aggregated sponge label is called the dominant label. The SRSM may indicate the QBL image is ready for capture and analysis by the hemoglobin estimation engine if the dominant label in the required consecutive frames satisfy the acceptance criteria. If no dominant label is identified within the last consecutive image frames, then the SRSM may generate an uncertain condition state.
Once the QBL image is captured, the SRSM algorithm may not permit further capturing of QBL images regardless of the immediate and dominant labels, at least until the dominant label identifies the sponge as no longer being or not present for a required number of consecutive image frames of the image feed. Once the sponge as no longer present for the required number of consecutive frames, the SRSM algorithm is configured to return to a state in which the localization neural network and segmentation neural network is readied to perform the methods disclosed herein.
The present disclosure relates to a system for managing surgical sponges during a surgical procedure, and quantifying blood loss (QBL) associated with blood absorbed on the surgical sponges. Referring to
The data reader 38 may be used either as a handheld device, or when supported by a cradle as shown in
In a typical surgical procedure in which surgical sponges are utilized, the surgical sponges are counted in to be used during the surgical procedure. The surgical sponges may be counted in by causing the tag to be detected by the data reader 38. For example, the user may position the surgical sponge (or a bundle of the surgical sponges) near the data reader 38. The detected tag transmits identifying data to the processor 42, which includes at least one unique identifier indicative of characteristics of the surgical sponge. The unique identifier may include sponge size or type (e.g., 2×16 gauze, 4×4 gauze, 4×8 gauze, 4×12 sponge, 12×12 sponge, and 18×18 laparotomy sponge, etc.), material construction (e.g., four ply, eight ply, sixteen ply, etc.), absorbency, stretchability, dry weight, and the like. The processor 42 is configured to determine a type of the surgical sponge based on the unique identifier(s). Further, based on the data reader 38 detecting the tag, the processor 42 may index a counter 64 to reflect the one or more surgical sponges being counted in to surgical procedure. The counter 64 may be displayed on the user interface 36 (see
The system 20 and methods 100, 200 of the present disclosure facilitate quick and accurate assessment of fluids contained on the surgical sponges, and in particular blood. The relevant steps may be performed in just moments between the step of counting out the surgical sponge and placing the surgical sponge in the sponge sorter 32, and therefore with minimal disruption to the surgical workflow. Alternatively, the steps may be performed at other points in surgical procedure as desired by the user. Referring now to
The method may include triggering the system 20 for sponge imaging (step 104). The step may include automatically activating the camera 48 in response to the detection of the tag with the data reader 38. The step of automatic activation is optional, and it is alternatively contemplated that the user may provide an input to the user interface 36 to activate the camera 48 to perform the subsequent steps of the method 100. Alternatively, a mode may be selected on the user interface 36 prior to commencement of the surgical procedure for procedure types in which QBL protocols (e.g., the methods 100, 200) are to be utilized, after which the camera 48 is automatically activated in response to the detection of the tag with the data reader 38.
The step may further include displaying, on the user interface 36, an image feed 50 as detected by the camera 48. The image feed 50 may also be considered a sponge capture screen. The image feed 50 may be based on data being received from the optical sensor 44 and/or the depth sensor 46.
The displaying of the image feed 50 may be automatically performed in response to the detection of the tag with the data reader 38. Therefore, the camera 48 may be automatically activated, and the display of the user interface 36 automatically updated to include the image feed 50. As generally appreciated from
The method 100 includes displaying, on the user interface 36, the presentation window 54 overlying the image feed 50 being captured by the camera 48 (step 106). The presentation window 54 is visual indicia to guide the user to present the surgical sponge in the appropriate manner and at the appropriate distance from the camera 48. For some sponge types, the distance is between eighteen and thirty-six inches, for example. With continued reference to
The method 100 includes a sponge recognition algorithm analyzing the image feed 50 for satisfaction of the acceptance criteria 74 (step 110). The step includes analyzing each image frame (or every few image frames) of the image feed 50 to ensure a full or stretched, correct surgical sponge is presented at the appropriate distance, unfolded, and not moving, among other acceptance criteria 74, also referred to herein as guardrails (see
Once the acceptance criteria 74 is determined to be satisfied, the method 100 includes the step of acquiring, with the camera 48, the QBL image(s) of the surgical sponge (step 112), also referred to herein as “sponge scans.” In one example, the QBL image(s) may be the subsequent frame(s) of the image feed 50 immediately following satisfaction of the acceptance criteria 74 analyzed by the sponge recognition algorithm, including output from a sponge recognition state machine (SRSM) 73. The QBL images may be considered distinct from the image frames from the image feed 50 that are being analyzed for satisfaction of the acceptance criteria. Rather, the QBL image is analyzed with a hemoglobin estimation algorithm for determination of a blood component (e.g., hemoglobin) of the liquid on the surgical sponge (step 114), for example, in the manner disclosed in the aformentioned U.S. Pat. Nos. 8,897,523 and 10,424,060. In particular, the processor 42 may be configured to extract a color component value (e.g., a redness value) from the QBL image, and execute the hemoglobin estimation algorithm to determine the hemoglobin concentration on a pixel-by-pixel or other suitable basis. The hemoglobin estimation algorithm may be unique to the sponge type having been counted out. In other words, hemoglobin estimation algorithm may be trained on extensive datasets for each sponge type to be compatible with the system 20. Blood loss may be determined based on the blood component, and a pixel-based or real-world area of the surgical sponge.
The user interface 36 may provide a prompt—e.g., visual and/or audible indicia—to notify the user of a successful QBL image capture, and immediately present an item-specific blood loss metric 58 for that particular sponge. For example,
By using the sponge type to determine characteristics of the presentation window 54 and the acceptance criteria 74, the image-based QBL may be applied to more or most types of surgical textiles, including those of differing sizes, plys, and/or colors. By contrast, for example, should the system not know the sponge type (and thus its size), a smaller sponge type imaged at a close distance may be mistaken for a larger sponge type presented at a greater distance, or a larger sponge type imaged at a closer distance may have its outer portions truncated. Likewise, differing number of plys cause images of a different redness values, especially in instances in which it is imaged in a folded state. The system 20 and methods of the present disclosure overcomes such shortcomings by determining blood loss on surgical sponges of differing types in a seamless manner within the surgical procedure without needing to predefine the type of surgical sponge being used or counted out.
For certain surgical sponges, the presentation window 54 being displayed on the user interface 36 may require the user to fold the surgical sponge. One instance may include larger or elongate surgical sponges that, if presented unfolded, would require the user to be too far away from the camera 48 for accurate analysis. Moreover, requiring the folding of certain surgical sponges results in the dimensions of respective presentation windows 54 associated with different sponge types to be sufficiently different. Consequently, the acceptance criteria 74 are sufficiently different for each type of surgical sponge such that the user is generally prevented from satisfying the acceptance criteria 74 with an incorrect sponge type or an incorrect folding of the sponge.
In certain implementations, the method 100 may include the step of displaying, on the user interface 36, a folding protocol to the user (step 108). The step is optional, and alternatively the presentation window 54 may require the surgical sponge be presented in an unfolded configuration. The folding protocol may include at least two aspects. First, the presentation window 54 is dimensioned to approximate the dimensions of the surgical sponge when folded according to the folding protocol. In other words, the presentation window 54 may be too small for the surgical sponge if presented unfolded. Second, the folding protocol may include guidance 62 to facilitate the user folding the surgical sponge in the desired manner. The guidance 62 may be textual instructions or graphical indicia, as shown in
An exemplary workflow is described with reference to
As mentioned, the one or more processors 42 implement or execute the sponge recognition algorithm. The sponge recognition includes neural networks are configured to process the images of the surgical sponges to detect the sponge and apply the acceptance criteria 74 or guardrails. In an exemplary implementation, there are two neural networks: a localization neural network 66 and a segmentation neural network 68. The localization neural network 66 may be a deep neural network such as a convolutional neural network (e.g., ResNet-50, MobileNet-v1,2 etc.), a transformer-based neural network (e.g., ViT), or a combination thereof. The output of the localization neural network 66 is modified to have both a classification head as well as a regression head. The localization neural network 66 is configured to, among other actions, determine an image class: whether or not a sponge is being presented (sponge precent), is folded or crumbled (folded sponge), is only partially visible (partial sponge). The localization neural network 66 is further configured to provide a bounding box 70 around the detected sponge (see
The segmentation neural network 68 may be an encoder-decoder style convolutional neural network architecture such as a U-Net, Mask-R CNN, DeepLab, or transformer-based neural network architectures that support image segmentation. The output of the segmentation neural network 68 provides per-pixel prediction of a segmentation mask 72 (schematically represented in
As reflected in
The localization neural network 66 may run in real time on slower devices and facilitate real-time user interactions, whereas the segmentation neural network 68 may be considered a larger and more accurate convolutional neural network. The dual-mode design ensures real-time processing of the image frames of the image feed 50 without compromising on the accuracy of the sponge segmentation mask 72. Alternatively, both the neural networks 66, 68 can be combined into a single hybrid network that outputs both the classification labels, the segmentation mask 72, and the bounding box 70. Such a hybrid neural network may run in real-time on faster devices having sufficient graphics processing capabilities. The neural networks may be trained in manners disclosed herein in order to perform the methods of the present disclosure with the necessary accuracy. The output of the sponge recognition algorithm is provided to the hemoglobin estimation algorithms that are unique to the sponge type being counted out.
Referring now to
If the surgical sponge is detected in step 202, the method 200 includes determining whether the surgical sponge is folded (step 204).
The step 204 of determining whether the sponge is folded may further include determining whether an aspect ratio of the presented surgical sponge is within an acceptable range based on the sponge type being counted out. As mentioned, the bounding box 70 includes a width and a height-a ratio of the two being the determined aspect ratio. If the sponge is determined to be sufficiently square or rectangular and yet its determined aspect ratio is outside the acceptable range of the aspect ratio for that sponge type, the localization neural network 66 determines the sponge is a folded sponge (step 204). For example, for the 18×18 laparotomy sponge, its manufactured aspect ratio may be approximately 1.0. Owing to the training of the localization neural network 66 on the datasets of 18×18 laparotomy sponges being presented by users in varying manners, the algorithm implementing the guardrail may have established the acceptable range of the aspect ratio to be between, for example, 0.7 and 1.5. These variances account for instances where the user may stretch the sponge, stretchability of the sponge material, minor warping while being held from the upper corners, and the like. If the user counts out an 18×18 laparotomy sponge and presents the sponge in a manner where its determined aspect ratio is outside the acceptable range, the localization neural network 66 may determine that the 18×18 laparotomy sponge is folded or in a folded condition. Additionally or alternatively, the determined aspect ratio falling outside of the acceptable range may be indicative that a wrong sponge is being presented. If a folded sponge (or a wrong sponge) is detected, the acceptance criteria 74 is identified to be not satisfied. The localization neural network 66 generates a localization label the presented surgical sponge is a folded sponge.
As mentioned, requiring the folding of certain surgical sponges also results in the dimensions of the respective presentation windows 54 for different sponge types to be sufficiently different. By extension, this applies to the acceptable ranges of the aspect ratios. In other words, the acceptable ranges of the aspect ratios are sufficiently different for each type of surgical sponge such that the user is generally prevented from satisfying the step 204 with an incorrect sponge type. For example, the acceptable range for a 4×8 gauze, if folded according to the folding protocol, may be between 1.7 and 2.2. A compatible sponge type immediately smaller—e.g., a 2×16 gauze, if folded according to the folding protocol, if necessary—may be associated with an acceptable range between 4.9 and 7.8. Likewise, a compatible sponge type immediately larger—e.g., a 4×12 gauze, if folded according to the folding protocol, if necessary—may be associated with the acceptable range between 2.5 and 3.4. While these values are merely exemplary, it should be appreciated that none of the acceptable ranges overlap between the 2×16 gauze, the 4×8 gauze, and the 4×12 gauze. For square-shaped sponges—e.g., 12×12 sponge and 18×18 laparotomy sponge—the acceptable ranges may overlap; however, a correct sponge guardrail (step 216) to be described addresses such considerations.
The method 200 may include determining whether the sponge being presented is a partial sponge by being too close to a boundary (step 206). If the segmented representation of the presented surgical sponge extends beyond the field of view 52 of the camera 48, the localization neural network 66 determines a partial sponge condition. For example, if an outermost pixel of the field of view 52 of the camera 48 is determined to be a sponge pixel, and/or if the bounding box 70 is determined to extend through outermost pixel, the localization neural network 66 determines the partial sponge condition. With concurrent reference to
The localization neural network 66 and the localization labeler 67 may be further configured to determine whether the sponge is positioned too close or too far from the camera 48 by determining whether the sponge is too large or too small, respectively (step 208). In particular, the localization labeler 67 is configured to determine a pixel-based area of the bounding box 70, and compare the bounding box area against an acceptable range of bounding box area that is unique to each sponge type. The localization labeler 67 may be trained on the datasets for the algorithm to establish the acceptable ranges of bounding box areas, or the acceptable ranges may be inputted or otherwise determined. In one example, the acceptable range of the bounding box area approximates the presentation window 54 being displayed on the user interface 36, which again is unique to the sponge type being counted out. The acceptable ranges of bounding box areas are pixel-based such that the steps 208 may be based on only the optical image from the optical sensor 44. If the processor 42 determines the bounding box area is greater than or less than the acceptable range of the bounding box area, the localization labeler 67 generates the localization label that the sponge is too close or too far (from the camera 48), respectively.
If the localization neural network 66 determines the sponge to be presented as unfolded and otherwise appropriately positioned within the presentation window 54, the method 200 includes identifying the presented surgical sponge as a full sponge (step 210), and the localization labeler 67 generates the localization label accordingly for further processing by the SRSM 73.
With the processing by the localization neural network 66, the segmentation neural network 68 also processes the image frames of the image feed 50. The dual-mode processing may occur concurrently or sequentially. The segmentation neutral network 68 outputs the segmentation mask 72 in which sponge pixels and the background pixels are separated.
Referring to
The depth data may be used to establish thresholds on acceptable presentation distance; i.e., lower and upper limits on real-world distance by which the user must present the surgical sponge from the camera 48. The method 200 may include determining whether the presented surgical sponge is too close or too far (step 214). The segmentation labeler 69 is configured to output that the sponge is either too close or too far if the average depth of all pixels belonging to the sponge segmentation mask 72 is less than or greater than the minimum threshold or maximum threshold, respectively. As previously described, the localization neural network 66 does a similar process by which the size of the bounding box 70 is compared to the size of the presentation window 54 (see step 208), which is not reliant on the depth data or real-world dimensions.
Further, the real-world area of sponge being imaged may be determined if depth map 51 is available. The segmentation labeler 69 does so by determining a sponge area (of the segmentation mask 72) based on the depth map of the image frames of the image feed 50. The determined sponge area is compared against an acceptable range of sponge area. The determined sponge area may be computed as a squared root L2-squared norm of all values from the depth map corresponding to the pixels of the image frame. The training of the segmentation labeler 69 on the dataset may establish the minimum and the maximum area for each type of the surgical sponges, or the acceptance ranges may be inputted or otherwise determined.
The method 200 may further include determining whether the presented surgical sponge is the correct sponge (step 216). As mentioned, the extent by which the localization neural network 66 may determine a correct sponge is based on a comparison of the determined aspect ratio against the acceptance range of the aspect ratio. The segmentation neural network 68 may confirm the determination, and further account for scenarios in which certain sponge types may have acceptable ranges that overlap. By leveraging the depth map to determine the real-world area of the segmentation mask 72, the segmentation labeler 69 is capable of differentiating between square-shaped sponges of differing dimensions. The segmentation labeler 69 may be trained on the datasets to establish the acceptable range of sponge areas, or the acceptable ranges may be inputted or otherwise determined. For example, at the appropriate presentation distance, an upper sponge area threshold for a 12×12 sponge may be 14000 AU, whereas a lower sponge area threshold for an 18×18 laparotomy sponge may be 16000 AU. Consequently, if a user were to count out an 18×18 laparotomy sponge and present a 12×12 sponge in a manner to otherwise satisfy the acceptance criteria (e.g., conform to the presentation window 54), the segmentation labeler 69 would determine the determined sponge area of the presented surgical sponge to be too low for the distance at which it is being presented. The segmentation neural network 68 determines the acceptance criteria 74 to be not satisfied, and the segmentation label may be generated accordingly. Corresponding prompts may be provided on the user interface 36 to indicate to the user that a wrong sponge may be presented.
The step 216 of whether the presented surgical sponge is the correct sponge may also include comparing a sponge compactness against a predetermined compactness threshold. In some respects, the step of comparing the sponge compactness may again be considered a crosscheck between the segmentation neutral network 68 and the localization neural network 66, which also determines an area and the aspect ratio of the bounding box 70, as mentioned. The predetermined compactness threshold may be a fraction of the area of the bounding box 70 that must be exceeded by the determined sponge area of the segmentation mask 72 to be determined as correct. For example, the predetermined compactness threshold may be 50% such that, if the determined area of the segmentation mask 72 is less than 50% of the determined area of the bounding box 70, the acceptance criteria 74 is determined to be not satisfied. This may be indicative of the wrong sponge, or of an invalid mask (see step 218). A corresponding prompt may be provided on the user interface 36 to request the user to represent the surgical sponge to the camera 48.
The method 200 may further include determining whether the presented surgical sponge is moving (step 218). If the presented surgical sponge is moving too quickly, motion blur may result compromise accuracy of the QBL analysis. The step 218 may include determining a centroid of the sponge segmentation mask 72, and further determining an average motion magnitude of the centroid within a predetermined number of image frames of the image feed 50 (e.g., five consecutive image frames). The average motion magnitude may be computed as an L2-distance between consecutively estimated centroids. The step 218 may include comparing the average motion magnitude against a centroid change threshold, which is unique for each sponge type. The centroid change threshold may be based on training the segmentation labeler 69 with the extensive datasets, or inputted into the software. If the average motion magnitude is greater than the centroid change threshold, the segmentation labeler 69 determines the presented surgical sponge to be moving too quickly, and the acceptance criteria 74 to be not satisfied. An alternative implementation includes tracking motion of the bounding box 70 of the detected sponge across consecutive image frames of the image feed 50 against a threshold. A corresponding prompt may be provided on the user interface 36 to request the user to hold the sponge steady.
Another acceptance criteria 74 or guardrail of the method 200 may include performing sponge supervision (step 220). As mentioned, the processor 42 may be configured to extract a redness value from the bloodied sponge for determining the concentration of the blood component on a pixel-by-pixel or other suitable basis. The redness of the pixels of the image are affected by environmental lighting, such as the light variations in the medical facility. It is known to use a calibration placard for light normalization to account for linear variations in ambient light intensity and color temperature. While generally effective, light “bleed-through” from a directional light source through the back of the presented surgical sponge may cause underestimation in predictions of the blood component. The light bleed-through may be exacerbated by users who hold the sponge away from the body, thereby exposing a greater portion of the surgical sponge to light sources positioned behind the user within the medical facility.
The step 220 of performing sponge supervision may include detecting the user's hand, torso, neck, and other associated body parts using a pose estimation neural network and a pose estimation labeler (not identified), as well as the location of the presented surgical sponge in manners previously described. Once the person and the presented surgical sponge are detected, the pose estimation labeler is configured to determine whether the presented surgical sponge is being held properly in front of the torso, which tends to limit light bleed-through. In particular and with reference to
If the segmentation neural network 68 determines the sponge to be presented at the appropriate distance and position relative to the user's body, the correct sponge, not moving, and otherwise that the segmentation mask 72 to be valid, the segmentation labeler 69 generates the segmentation label for further processing by the SRSM 73. This is in addition to the localization label generated by the localization labeler 67.
The SRSM is configured to receive the labels from each of the localization labeler 66 and the sponge segmentation labeler 68, and perform further processing prior to the camera 48 capturing the QBL image. The SRSM is configured to determine whether the localization and segmentation labels are equal. In other words, the SRSM determines whether the acceptance criteria 74 or guardrails, executed potentially separate by the localization neural network 66 and the sponge segmentation neural network 68, have been satisfied. With the image feed 50 being a video and the image frames effectively occurring in imperceptible rapid succession, image states of one or more of the acceptance criteria 74 may change outcome rapidly. The SRSM may execute a smoothing procedure to aggregate the localization and segmentation labels using a sliding window with an image frame size (L). The sliding window incorporates previous states between time “t−L” to time “t−1,” and may not include the most immediate state (at time t). The most frequent aggregated sponge label is called the dominant label. The SRSM may indicate the QBL image is ready for capture and analysis by the hemoglobin estimation engine if the required consecutive frames (K), including immediate and dominant labels, satisfy the acceptance criteria 74. As the number of required consecutive frames is increased, the probability of false positive scans decreases. The parameters L, K may be tuned based on empirical evaluations. If no dominant label is identified within the last K consecutive image frames, then the SRSM may generate an uncertain condition video state. For example, the uncertain condition video state may be generated if the most frequent aggregated sponge label in the sliding window occurs less frequently than half of the sliding window size, or other suitable value.
Once the QBL image is captured, the sponge recognition algorithm is configured to momentarily put the system 20 in a “cool down” state for the user to remove the presented surgical sponge from the field of view 52 of the camera 48. The SRSM may not permit further capturing of QBL images regardless of the immediate and dominant labels, at least until the dominant label identifies the sponge as no longer being or not present (see step 202) for a required number of consecutive image frames (M) of the image feed. The parameter M may be tuned based on empirical evaluations or otherwise determined. Once the sponge as no longer present for the required number of consecutive frames, the SRSM is configured to return to a video state in which the localization neural network 66 and segmentation neural network 68 is readied to perform the methods 100, 200 disclosed herein. In effect, this allows that only one sponge scan is captured per presented surgical sponge. The user is appropriately instructed to remove the sponge from the field of view 52 of the camera 48 before presented the next surgical sponge.
As mentioned, the neural networks 66, 68 may be trained using supervised, semi-supervised, and/or unsupervised learning methodology in order to perform the methods of the present disclosure, and in particular with the necessary accuracy. A diverse and representative set of training dataset is collected to train each of the neural networks on diverse scenarios both within and outside of expected uses of the system 20. Training methods may include data augmentation, pre-trained neural networks, and/or use semi-supervised learning methodology to reduce the requirement of labeled training dataset. The data augmentation may increase sample size such as lighting augmentation (brightness, contrast, hue-saturation shift, PlanckianJitter, etc.), geometric transformation (rotation, homography, flips etc.), noise and blur addition, custom augmentation (artificially enlarging or reducing the size of the sponge, cropping the sponge), or the like.
The datasets vary parameters including sponge size, sponge saturation, ground truth (blood color), lighting, imaging distance, sponge orientation, and position. For examples, the sponges may be saturated by varying levels of total value up to maximum blood-carrying capacity; imaging distance may vary from 0.5 to 1.5 meters; lighting may be brighter or lighter; sponge orientation may be rotated in plane or out of plane; position may be varied along with images of folded, unfolded sponges, and slightly folded. Still further, an example set of sponge-like non-sponge objects may be included in the datasets to train the neural networks to recognize white or red non-sponge objects as part of background.
Additional approaches include training the neural networks with the color image and the depth data together to jointly learn the segmentation mask using both the color image as well as depth image. Specific architectures such as ESANet can be used to efficiently combine the color image and depth data. The segmentation neural network 68 may also be trained for images where depth data is not available (e.g., only the color image). If the depth data is not available, a dataset may be generated that includes the pixel area within the segmentation mask being correlated at different distance of imaging. From a depth of a sponge scan from the color image, the segmentation mask, and sponge type being known, the segmentation neural network 68 is configured to determine whether the presented surgical sponge is scanned too close or too far from the system. Therefore, the segmentation neural network 68 is configured to provide the segmentation mask 72 based only the color image from the optical sensor 44. Lastly, for the pose estimation neural network of sponge supervision, exemplary models may include AlphaPose, MobileHumanPose, MoveNet, OpenPose, etc., which are further trained on an application-specific dataset, e.g., nurses imaging the sponge.
The foregoing disclosure is not intended to be exhaustive or limit the disclosure to any particular form. The terminology which has been used is intended to be in the nature of words of description rather than of limitation. Many modifications and variations are possible in light of the above teachings and the invention may be practiced otherwise than as specifically described.
Further inventive aspects of the present disclosure are disclosed with reference to the following exemplary clauses:
Clause 1—A method of assessing surgical articles with a system including a data reader, one or more processors, a user interface, an optical sensor, the method comprising: identifying, with the one or more processors, an article type of the surgical article from a database of predetermined types of surgical articles; detecting, with a field of view of the optical sensor, a presented surgical article; capturing at least one image of the presented surgical article; determining, with the one or more processors, whether the presented surgical sponge satisfies at least one acceptance criteria, wherein the acceptance criteria are based on the identified article type that was scanned with the data reader; and displaying, on the user interface, at least one of confirmation that the presented surgical article is of the same type that was scanned with the data reader, and information related to a state or condition of the surgical article.
Clause 2—The method of clause 1, wherein each of the surgical articles includes a tag storing a unique identifier, the method further comprising: detecting, with the data reader, the tag of the surgical article, wherein the unique identifier stored on the tag is received by the processor; and identifying, with the one or more processors, the article type based on the unique identifier.
Clause 3—The method of clause 1 or 2, wherein the step of determining whether characteristics of the surgical article satisfy the acceptance criteria comprises: generating, with the one or more processors, a segmentation mask of the presented surgical article; determining, with the one or more processors, a number of pixels in the segmentation mask; comparing, with the one or more processors, the determined number of pixels of the segmentation mask against a database images in which the surgical article is in an unused or undamaged condition; and displaying, on the user interface, the information indicative that the condition of the surgical article is damaged if the determined number of pixels is outside of an acceptable range of pixels.
Clause 4—The method of clause 3, wherein the system includes a depth sensor, and wherein step of determining whether characteristics of the surgical article satisfy the acceptance criteria comprises: determining, with the one or more processors, a real-world article area of the segmentation mask based on depth data from the depth sensor; comparing, with the one or more processors, the determined real-world sponge area of the segmentation mask against an acceptable range of article area, wherein the acceptable range of article area is based on the identified article type; identifying, with the one or more processors, the presented surgical article as a wrong article if the determined real-world sponge area of the segmentation mask is outside the acceptance range of the article area.
Clause 5—The method of any one of clauses 1-4, wherein the step of determining whether characteristics of the surgical article satisfy the acceptance criteria comprises: generating, with the one or more processors, a bounding box about or around of the presented surgical article; determining, with the one or more processors, an aspect ratio of the bounding box; and comparing, with the one or more processors, the determined aspect ratio of the bounding box against an acceptable range of the aspect ratio, wherein the acceptable range of the aspect ratio is based on the identified article type; identifying, with the one or more processors, the presented surgical article to be a wrong type of surgical article if the determined aspect ratio of the bounding box is not within the acceptable range of the aspect ratio.
Clause 6—The method of clause 5, wherein the step of determining whether characteristics of the surgical article satisfy the acceptance criteria comprises: determining, with the one or more processors, an area within the bounding box; comparing, with the one or more processors, the determined area of the bounding box against an acceptable range of bounding box area, wherein the acceptable range of the bounding box area is based on the identified article type; identifying, with the one or more processors, the presented surgical article to be too close or too far from the optical sensor if the determined area of the bounding box is outside the acceptable range of the bounding box area.
This application claims priority to and all the benefits of U.S. Provisional Patent Application No. 63/321,400, filed Mar. 18, 2022, the entire contents of which are hereby incorporated by reference.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2023/015509 | 3/17/2023 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63321400 | Mar 2022 | US |
