Patent Application
20240129515
The present disclosure relates generally to using deep learning models to compress medical videos and detect target features in the medical videos.
In recent years, data-driven algorithms have been developed that aim to detect diseases from medical videos (for example, detecting polyps in colonoscopy videos). In order to train and evaluate these algorithms, hospitals may need to store these videos and transfer them to a company that will use them to train the algorithms or, eventually, to diagnose diseases. A large number of videos is useful to train the algorithms and each individual video can be quite large. This makes transmitting and storing these medical videos difficult and very expensive. Therefore, being able to efficiently and effectively compress a large number of medical videos is important to bring down the cost of training and deploying diagnostic algorithms, and storing and transmitting medical videos in general.
Methods of compressing and detecting target features of a medical video are presented herein. Some methods may include the steps of receiving an uncompressed medical video comprising at least one target feature; compressing the uncompressed medical video to generate a compressed medical video based on a predicted location of the at least one target feature using a first pretrained machine learning model; and detecting the location of the at least one target feature of the compressed medical video using a second pretrained machine learning model. In some contexts, target feature detection may also be referred to as feature extraction.
In some embodiments, the medical video may be obtained during a colonoscopy. Moreover, in some embodiments, the one or more target features of the at least one target feature may be a polyp.
In some embodiments, the first pretrained machine learning module may be a video compression transformer. In some embodiments, the second pretrained machine learning module may be a RetinaNet detector. In some embodiments, the first pre-trained machine learning model may be further configured to allocate a larger proportion of bits to the predicted location of the at least one target feature than to other parts of the medical video.
In some embodiments, the first pretrained machine learning model and the second pretrained machine learning model may be trained in tandem. In some embodiments, the second pretrained model may be designed to optimize detection of the at least one target feature. In some embodiments, the first pretrained model may be designed to optimize detection of the at least one target feature and to minimize the loss of the compressed medical video.
Systems for compressing and detecting target features of a medical video are described herein. In some embodiments, the systems may include an input interface configured to receive an uncompressed medical video, which may comprise at least one target feature, and a memory configured to store a plurality of processor-executable instructions. The memory may include a video compressor based on a first pretrained machine learning model, a target feature detector based on a second pretrained machine learning model, and a processor. The processor may be configured to execute the plurality of processor-executable instruction to perform operations including: receiving an uncompressed medical video comprising at least one target feature; compressing the uncompressed medical video to generate a compressed medical video based on a predicted location of the at least one target feature; and detecting the location of the at least one target feature of the compressed medical video.
In some embodiments, the first pretrained machine learning module may be a video compression transformer. the second pretrained machine learning module may be a RetinaNet detector. In some embodiments, the first pretrained machine learning model and the second pretrained machine learning model may be trained in tandem. In some embodiments, the second pretrained model may be designed to optimize detection of the at least one target feature. In some embodiments, the first pretrained model is designed to optimize detection of the at least one target feature and to minimize the loss of the compressed medical video.
In some embodiments a non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for compressing and detecting target features of a medical video is described. The storage medium storing a plurality of processor-executable instructions for compressing and detecting target features of a medical video, the instructions being executed by a processor to perform operations comprising: In some embodiments, the instructions stored on the storage medium may be executed on a processor to perform operations including: receiving an uncompressed medical video comprising at least one target feature; compressing the uncompressed medical video to generate a compressed medical video based on a predicted location of the at least one target feature using a first pretrained machine learning model; and, detecting the location of the at least one target feature of the compressed medical video using a second pretrained machine learning model.
In some embodiments, the first pretrained machine learning module may be a video compression transformer. In some embodiments, the second pretrained machine learning module may be a RetinaNet detector. In some embodiments, the first pretrained machine learning model and the second pretrained machine learning model may be trained in tandem. In some embodiments, the first pretrained model may be designed to optimize detection of the at least one target feature and to minimize the loss of the compressed medical video and the second pretrained model may be designed to optimize detection of the at least one target feature.
Illustrative embodiments of the present disclosure will be described with reference to the accompanying drawings, of which:
For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of the present disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and/or steps described with respect to one embodiment may be combined with the features, components, and/or steps described with respect to other embodiments of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately.
As used herein, the term “network” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system and/or any training or learning models implemented thereon or therewith.
As used herein, the term “module” may comprise hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.
Many scientists, physicians, programmers and others have been working on harnessing the power of artificial intelligence (AI) to quickly and accurately diagnose diseases. AI has been used in a variety of different diagnostic applications including, for example, identifying polyps in colonoscopy videos. Some of the most promising ways of detecting diseases from medical videos include using a machine learning (ML) and, in particular, neural networks (NN). By feeding an ML program hundreds or thousands of videos marked with the target feature, ML programs can develop methods, equations, and/or patterns for determining how to detect the target feature in future videos. For example, if a ML program is fed thousands of colonoscopy videos with the polyps marked by physicians, the ML program can learn what the polyp looks like and how to identify polyps in future colonoscopy videos.
However, storing these videos for training or for diagnosis may require a large amount of storage space. Moreover, if these videos need to be sent to the company developing the ML program, the large volume of videos may be difficult to transmit.
Compressing videos is a challenging task that requires balancing the size of the video with the quality of the video. In other words, there is a tradeoff between minimizing the bit rate and minimizing the distortion. When shrinking the size of the video file, a compressor often will often have to cut or combine certain parts of the file, resulting in a loss of quality. For diagnosing medical videos, the compression needs to result in a high enough quality for an algorithm to detect the target features in the video.
Data-driven image and video compression have surpassed standard codecs in the last several years. Learned compression algorithms improve upon traditional compression techniques in two ways that are both applicable to medical videos. First, neural network models learn complex nonlinear functions to maximize performance on the rate-distortion curve. These learned mappings often find better mappings than hand-crafted algorithms, in the general case. Second, learned compression algorithms are able to be finetuned for specific domains. For example, the machine learning (ML) training process can be used to tune an algorithm to maximize the rate-distortion curve specifically on medical videos.
Product applications of this technology are numerous. First, they decrease the cost of storing colonoscopy videos for algorithm training, medical records, and physician training. Second, they ensure that downstream models are not negatively affected by compression artifacts. Third, rapid compression can allow videos to be transmitted quickly, such as for report generation immediately after a procedure. Thus, improved ML algorithms are useful for intelligently compressing medical videos and diagnosing diseases from the videos.
As described herein, the present disclosure relates to an improved compressor that compresses medical videos by focusing on the target feature. Thus, the compressor can prioritize the quality of the most important parts of the video, improving subsequent detection of the target features. The compressor may be paired with the detector in a single program so that the compressor can be optimized for the accuracy of the subsequent detection.
These descriptions are provided for example purposes only and should not be considered to limit the scope of the invention described herein. Certain features may be added, removed, or modified without departing from the spirit of the claimed subject matter.
1. Overview
In the present embodiments, the video 110 is received as input data into the video compressor 120. The video compressor 120 is configured to compress the video 110. As described below, the video compressor 120 may include an ML model and in some cases may include a NN model. After the video 110 is compressed by the video compressor 120, the compressed video 130 is input into a target feature detector 140. The target feature detector 140 is configured to analyze the compressed video 130 and detect and target features that are shown in the video. For example, the video 110 may be a colonoscopy video and the target feature may be a polyp. The target feature detector 140 is configured to output the analyzed compressed video 150. In some embodiments, the target feature detector 140 may include an ML model and in some cases may include a NN model. In some embodiments, the analyzed compressed video 150 may identify the location of the target feature. The target feature detector 140 may output the location of the target features in any appropriate way. For example, the analyzed compressed video 150 may include a box, circle or other object surrounding the target features in the video. Moreover, the analyzed compressed video 150 may highlight the location of the target features in the video. In some embodiments, the output may be a list of the frames that include the target features and the area of those frames including the polyp. In some embodiments the output may include the number of target features detected.
In some embodiments, the video compressor 120 and the target feature detector 140 may be combined in a single module, described herein as a “combined compressor and detector module” or a “combined module”. By combining the video compressor 120 and the target feature detector 140 into a single combined module, the combined module can be optimized for the size (e.g., bit-rate) and quality (e.g., distortion) of the compressed video 130 as well as the accuracy of detecting a target feature in the compressed video 130, as described in more detail below. This improves the module by allowing the video compressor 120 to prioritize maintaining the quality of the parts having target features over other parts of the video. Therefore, the video can be meaningfully compressed without sacrificing the diagnostic accuracy of the combined module.
Furthermore, as would have been understood based on the principles of this disclosure, the video compressor 120 and target feature detector 140 may be placed in series in either order, such that target feature detection may occur followed by video compression.
2. Computing Device
The memory 230 may be used to store software executed by computing device 200 and/or one or more data structures used during operation of computing device 300. The memory 230 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor (e.g., the processor 220) or computer is adapted to read. In the present embodiments, for example, the memory 230 includes instructions suitable for training and/or using the combined module 210 described herein.
The processor 220 and/or the memory 230 may be arranged in any suitable physical arrangement. In some embodiments, the processor 220 and/or the memory 230 are implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), and/or the like. In some embodiments, the processor 220 and/or the memory 230 include distributed, virtualized, and/or containerized computing resources. Consistent with such embodiments, the processor 220 and/or the memory 230 may be located in one or more data centers and/or cloud computing facilities.
In some examples, the memory 230 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., the processor 220) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, the memory 230 includes instructions for an online adaptation module that may be used to implement and/or emulate the systems and models, and/or to implement any of the methods described further herein. In some examples, the combined module 210 may receive an input that includes the medical video 110 via a data input interface 240. The data input interface 240 can then send the medical video 110 to the processor 220. The combined module 210 may then generate one or more output data, such as the analyzed compressed medical video 150, in response to the input data.
In some embodiments, the combined module 210 includes at least the video compressor 120 and the target feature detector 140, as discussed in more detail below. In some examples, the combined module 210 and its components may be implemented using hardware, software, and/or a combination of hardware and software.
Some examples of computing devices, such as the computing device 200, may include non-transitory, tangible, machine-readable media that include executable code that when run by one or more processors (e.g., the processor 220) may cause the one or more processors to perform the processes of methods 400 described below. Some common forms of machine-readable media are discussed in detail above.
3. The Combined Compressor and Detector Module
The combined compressor and detector module 210 includes the video compressor 120 and the target feature detector 140, which are described individually below.
3.1 The Video Compressor
The video compressor 120 is configured to compress the medical video 110. The video compressor 120 may include any available video compression techniques. For example, the video compressor 120 may be an ML or NN based video compressor. In some embodiments, the NN based video compressor may be a video compressor transformer. In particular embodiments, the video compressor transformer may be a transformer-based temporal entropy model, such as the transformer-based temporal entropy model described in Mentzer et al., VCT: A Video Compression Transformer, arXiv:2206.07307 (Jun. 15, 2022), the entirety of which is incorporated herein by reference.
The medical video 110 may include some number of frames, each frame being denoted by “x”. Each frame x may be independently and lossily mapped into quantized representations y. From y a reconstruction z may be recovered. To store yi 304 with few bits, transformers may be used to model temporal dependencies and to predict a distribution for yi given previously transmitted representations. P may losslessly compress the quantized yi using entropy coding (“EC”). The better the transformer predicts P, the fewer bits are required to store yi.
The video compressor 120 may use transformers to compress videos in two steps. First, using lossy transform coding to map frames xi 302 of the medical video 110 from image space to quantized representations yi 304, independently for each frame. From yi 304 a reconstruction {circumflex over (x)}i 306 may be recovered. Second, a transformer may leverage temporal redundancies to model the distributions of the representations. The predicted distributions may losslessly compress the quantized yi 304 using EC. In this way, the transformer-based temporal entropy model may avoid complex state transitions or warping operations by letting the transformer learn to leverage arbitrary relationships between frames. Moreover, in some embodiments, there may be no temporal error propagation by construction since the reconstruction {circumflex over (x)}i 306 does not depend on previous reconstructions.
Each frame xi 302 may be encoded into a quantized representation yi=E(xi) using a convolutional neural network (CNN) based image encoder (“E”) 308 followed by quantization 310 to an integer grid. The encoder 308 downscales spatially and increases the channel dimension, resulting in yi being a (H, W, dC)-dimensional feature map, where H and W are 16× smaller than the input image resolution (H represents height, W represents width, and dC represents the number of convolutional filters in the convolutional layer). Thus, all y frames combined will result in the compressed medical video 130.
In some embodiments, the compressed medical video 130 may be decompressed using a decoder (“D”) 312. From yi 304 in the compressed medical video 130, a reconstruction {circumflex over (x)}i 306 can be recovered using the decoder 312. The encoder 308 and decoder 312 may be trained using standard neural image compression techniques to be lossy transforms reaching nearly any desired distortion d(xi, {circumflex over (x)}i 306) by varying how large the range of each element in yi is.
Given a probability mass function (PMF) P estimating the true distribution Q of symbols in yi 304, entropy coding (“EC”) can be used to transmit yi 304 with H·W·dC·Ey˜Q(yi)[−log2 P(y)] bits. By using EC, more frequently occurring values can be encoded with fewer bits, thus improving the efficiency. The expectation term representing the average bit count may correspond to the cross-entropy of Q with respect to P. P may be used for lossless EC, without sampling from the model to transmit data. In some embodiments, the model may be able to exploit the temporal redundancy across frames, and the spatial consistency within frames.
To compress a medical video 110 of F number of frames, x1, . . . , xF, each frame x may be encoded via encoder 308, then quantized via quantizer 310 to generate quantized representations y1, . . . , yF. Assuming y1, . . . , yi-1 have already been compressed or transmitted, to transmit yi 304, we use the transformer to generate a prediction distribution P(yi|yi-2,yi-1) 314. Using this distribution, yi 304 may be entropy encoded to create a compressed, binary representation that can be stored or transmitted.
To compress the full video, this procedure may be applied iteratively for each frame in the medical video 110, letting the transformer predict P(yj|yj-2,yj-1) for j∈{1, . . . ,F}, padding with zeros when predicting distributions for y1 and y2.
When the compressed medical video 130 is decompressed, the same procedure may be used to recover all yj, i.e., it iteratively calculates P(yj|yj-2,yj-1) to entropy decode each yj. After obtaining each representation, y1,y2, . . . , yF, the reconstructions may be generated.
Compressing the current representation yi 304, the representation may be split spatially into non-overlapping blocks 402 with size wc×wc 403.
Previous representations yi-2 316 can become overlapping blocks 317 and yi-1 318 can become overlapping block 319. The overlapping blocks 317, 319 may be sized wp×wp 321 (where wp>wc) to provide both temporal and spatial context for predicting (yi|yi−2,yi−1) 314. Intuitively, the larger spatial extent provides useful context to predict the distribution of the current block. All blocks may span a relatively large spatial region in image space due to the downscaling convolutional encoder 208. Each block 402, 404, 406 can then be flattened spatially to obtain tokens 410, 412, 414 for the transformers. The transformers then run independently on corresponding blocks 402, 404, 406 and tokens 410, 412, 414. In other words, each block and its corresponding token get processed together, trading reduced spatial context for parallel execution. In some embodiments, the processing may focus on a single block. For example block 422 and corresponding token 423 of yi 304, block 424 and corresponding token 425 of yi-2 and block 426 and corresponding token 427 of yi-1.
The second part is the masked transformer Tcur 510, which predicts PMFs for each token 423, 425, 427 using auto-regression within the block, which may create a powerful model by conditioning Tcur 510 on zjoint 508 as well as already transmitted tokens 425, 427 within the block 424, 426.
A start token tS may be chosen. After zjoint 508 is obtained, [tS] may be input to Tcur 510 to obtain P(t1|tS;zjoint) 512, and use entropy coding to store the dC symbols in token t1 into a bitstream using P(t1|tS;zjoint) 512. For t2, [tS,t1] may be fed into Tcur 510 to obtain P(t1|tS;zjoint) 512 and t2 may be stored in the bitstream, and so on. The resulting bitstream may be used during decompression through the same process. In
In some embodiments, this procedure is used in parallel over all blocks 402, 404, 406. In some embodiments, to ensure causality of Tcur 510 during training, the self-attention blocks may be masked. The process may assume that the blocks 402 in yi 304 are independent. Moreover, in some embodiments, it may be assumed that the symbols within each token 410 are conditionally independent given previous tokens 412, 414. In other words, Tcur 510 predicts the dC distributions required for a token at once. In other embodiments, a joint distribution may be predicted over all possible |S|dC realizations, channel-autoregression may be used, or vector quantization on tokens may be used.
The transformers 502, 504, 506, 510 may be based on standard architectures. In some embodiments, the dC-dimensional tokens may be projected to a dT-dimensional space using a single fully connected layer 514 and adding a learned positional embedding. In some embodiments, both Tsep 502, 504 and Tjoint 506 are stacks of multi-head self-attention (MHSA) layers. In some embodiments, Tcur 510 uses masked “conditional” transformer layers. These may alternate between masked MHSA layers and MHSA layers that use z joint 508 as keys and values, as shown in
The encoder 308 and decoder 312 may be any standard encoder or decoder used for video compression. In some embodiments, for the encoder 308, 4 strided convolutional layers may be used, which may downscale by a factor 16× in total. In some embodiments, for the decoder 312, transposed convolutions may be used and additionally add residual blocks at the low resolutions may be used. In some embodiments, dED=192 filters may be used for all layers.
In some embodiments, the powerful representation that the transformer learns may be further leveraged. In some embodiments, a latent residual predictor (“LRP”) may be used. In some embodiments, the final features zcur from Tcur have the same spatial dimensions as yi 304 and contain everything the transformer knows about the current and previous representations. These may be helpful to reconstruct {circumflex over (x)}i. We thus use zcur by feeding yi=yi+fLRP(zcur) to the decoder 312, where fLRP may consist of a 1×1 convolution mapping from dT to dED followed by a residual block. This may prevent temporal error propagation.
3.2 The Target Feature Detector
Once the medical video 110 has been compressed, the compressed medical video 130 may be input into the target feature detector 140. The target feature detector 140 may use any appropriate detection techniques. For example, the target feature detector 140 may be an ML or NN based object detector. In some embodiments, the NN based target feature detector may be a two stage, proposal-driven mechanism such as a region-based convolutional neural network (R-CNN) framework. In some embodiments, the target feature detector 140 may use a RetinaNet architecture, as described in, for example, Lin et al., Focal Loss for Dense Object Detection, arXiv:1708.02002 (Feb. 7, 2018) or in U.S. Patent Publication No. 2021/0225511, the entirety of which are incorporated herein by reference.
In some embodiments, a Feature Pyramid Network (FPN) may be used as the backbone network 602 for RetinaNet. FPN may augment a standard convolutional network with a top-down pathway and lateral connections so the network efficiently constructs a rich, multi-scale feature pyramid 608 from a single resolution input image 610. Each level of the pyramid 608 can be used for detecting objects at a different scale. FPN may be built on top of the ResNet architecture. A pyramid with levels P3 through P7 may be used, where l indicates pyramid level (Pl has resolution 2l lower than the input). In some embodiments, all pyramid levels have C=256 channels.
In some embodiments, translation-invariant anchor boxes may be used. In some embodiments, the anchors have areas of 322 to 5122 on pyramid levels P3 to P7, respectively. At each pyramid level anchors at three aspect ratios {1:2, 1:1, 2:1} may be used. For denser scale coverage, at each level anchors of sizes {20, 21/3, 22/3} of the original set of 3 aspect ratio anchors may be used. In total there may be A=9 anchors per level and across levels they may cover the scale range 32813 pixels with respect to the network's input image. Each anchor may be assigned a length K one-hot vector of classification targets, where K is the number of object classes, and a 4-vector of box regression targets. Anchors may be assigned to ground-truth object boxes using an intersection-over-union (IoU) threshold of 0.5 and to background if their IoU is in [0, 0.4). As each anchor is assigned to at most one object box, in some in some embodiments a corresponding entry in its length K label vector may be assigned to 1 and all other entries may be assigned to 0. If an anchor is unassigned, which may happen with overlap in [0.4, 0.5), the anchor may be ignored during training. Box regression targets may be computed as the offset between each anchor and its assigned object box. If there is no assignment, the anchor may be omitted.
In some embodiments, the classification subnet 604 predicts the probability of object presence at each spatial position for each of the A anchors and K object classes. The classification subnet 604 may be a small FCN attached to each FPN level. Parameters of this subnet can be shared across all pyramid levels. Taking an input feature map with C channels from a given pyramid level, the subnet 604 applies four 3×3 cony layers, each with C filters and each followed by ReLU activations, followed by a 3×3 cony layer with KA filters. Then, sigmoid activations can be attached to output the KA binary predictions 612 per spatial location. In some embodiments, C=256 and A=9 may be used. In some embodiments, the subnet 604 may be deep, may use only 3×3 convs, and may not share parameters with the box regression subnet 606.
In parallel with the object classification subnet 604, another small FCN may be attached to each pyramid level for the purpose of regressing the offset from each anchor box to a nearby ground-truth object, if one exists. In some embodiments, the design of the box regression subnet 606 is identical to the classification subnet 604 except that it terminates in 4A linear outputs 614 per spatial location rather than KA outputs 612 per spatial location. For each of the A anchors per spatial location, these 4 outputs 314 may predict the relative offset between the anchor and the ground truth box. In some embodiments, a class-agnostic bounding box regressor may be used. The class-agnostic bounding box regressor may use fewer parameters. In some embodiments, the object classification subnet 604 and the box regression subnet 606 may share a common structure, but may use different parameters.
4. Optimization and Training
During training, the video compressor 120 and the target feature detector 140 are optimized in tandem. When the video compressor 120 and the target feature detector 140 are optimized together, the video compressor 120 can be optimized for both improved compression metrics and the accuracy of detecting the target feature. In this way, the video compressor 120 may learn to compress the medical video 110 in a way that will allow the target feature detector 140 to accurately detect the target features. A novel feature of this disclosure is that training is performed using domain-specific videos, such as videos of a specific medical procedure (e.g., colonoscopy videos).
4.1 Optimizing Compression Metrics
The video compressor 120 may be optimized according to any appropriate compression metrics.
In embodiments where a video compression transformer, such as the transformer-based temporal entropy model described above, is used, the optimization and training may be split into three stages.
In Stage I, the per-frame encoder E 308 and decoder D 312 may be trained by minimizing the rate-distortion trade-off. Where U denote a uniform distribution in [−0.5,0.5], the following may be minimized:
using {tilde over (y)} to refer to the unquantized representation, and x˜pX are frames drawn from the training set. In some aspects, it may be desirable to minimize the reconstruction error under while still effectively quantizing the encoder output. Thus, λ may control the tradeoff between minimizing the reconstruction error and effectively quantizing the encoder output. For Stage I, the meanscale hyperprior approach may be used to estimate p, the de facto standard in neural image compression, which may be discarded for later stages. To enable end-to-end training, i.i.d. uniform noise u may be added to {tilde over (y)} when calculating r. Moreover, straight-through estimation (STE) for gradients may be used when rounding {tilde over (y)} to feed it to D.
For Stage II of the training stages, the video compression transformer may be trained to obtain p, and only minimize rate:
II=(x
[−log p({tilde over (y)}3+u|y1,y2)] {tilde over (y)}i=E(x), yi=round(yi), (2)
where (x1, x2, x3)˜pX1:3 are triplets of adjacent video frames. It can be assumed that each of the dC unquantized elements in each token follow a Gaussian distribution, p˜N, and the transformer may predict dC means and dC scales per token.
In Stage III of the training stages, everything may be finetuned jointly by adding the distortion loss to the Stage II result. To obtain a discrete PMF P for the quantized symbols (for entropy coding), standard practice may be followed, convolving p with a unit-width box and evaluating it at discrete points:
P(y)=p(y+u), y∈Z (3)
To train, random spatio-temporal crops of (B,NF,256,256,3) pixels may be used, where B is the batch size, and NF is the number of frames. The linearly decaying learning rate (LR) schedule with warmup may be used. In some embodiments, Stage I may be trained using λ=0.01. To navigate the rate-distortion trade-off and obtain results for multiple rates, 9 models may be finetuned in Stage III, using λ=0.01·2i, i∈{−3, . . . ,5}.
4.2 Optimizing Accuracy of Detecting Target Features
In addition to using the above methods to optimize compression, both the video compressor 120 and the target feature detector 140 may be trained to optimize detecting target features. This joint optimization may simultaneously take into account the quality of the video after compression and decompression, as well as reducing both the probability of mislabeling a frame as having a feature when it does not (also referred to as false positive) and reducing the probability of mislabeling a frame as not having a feature when it does (also referred to has false negative).
For the RetinaNet architecture described above, focal loss may be used as the loss on the output of the classification subnet 604. In some embodiments, a focal loss of γ=2 may be used or any appropriate focal loss may be used, including γ∈[0.5,5]. This focal loss may be applied to all anchors. However, in other embodiments heuristic sampling (RPN) or hard example mining (OHEM, SSD) may be used to select a small set of anchors (e.g., 256) for each minibatch for applying the focal loss. The total focal loss of an image may be computed as the sum of the focal loss over all anchors, normalized by the number of anchors assigned to a ground-truth box. Normalization may be performed by the number of assigned anchors or by total anchors. In some embodiments, α and the focal loss may be chosen together. In some embodiments, α can be decreased slightly as γ is increased such as, for example, γ=2, α=0.25.
In some embodiments, ResNet-50-FPN or ResNet-101-FPN backbones 602 may be used. The base ResNet-50 and ResNet-101 models may be pre-trained on ImageNet1k. All new cony layers except the final one in the RetinaNet subnets 604, 606 may be initialized with bias b=0 and a Gaussian weight fill with σ=0.01. For the final cony layer of the classification subnet 604, the bias initialization may be set to b=−log((1−π)/π), where π specifies that at the start of training every anchor should be labeled as foreground with confidence of ˜π. In some embodiments, π=0.01. This initialization may prevent the large number of background anchors from generating a large, destabilizing loss value in the first iteration of training.
In some embodiments, RetinaNet may be trained with stochastic gradient descent (SGD). In some embodiments, synchronized SGD may be used over 8 GPUs with a total of 16 images per minibatch (2 images per GPU). In some embodiments, all models may be trained for 90k iterations with an initial learning rate of 0.01, which is then divided by 10 at 60k and again at 80k iterations. In some embodiments, horizontal image flipping may be used for data augmentation. In some embodiments, a weight decay of 0.0001 and a momentum of 0.9 can be used. The training loss may be the sum of the focal loss and the standard smooth Li loss used for box regression.
4.3 Optimizing and Training the Combined Compressor and Detector Module
The training and optimization described above for each individual component may be used in training the overall program.
The method 700 of training the combined compressor and detector module 210 according to some aspects of the present disclosure is illustrated in the flow diagram of
The method 700 of training the combined module 210 may also include the step 720 of inputting a medical video into the video compressor 120. Any appropriate number of medical videos 110 may be used for training. For example, dozens, thousands, or millions of videos may be used during training. Thus, any appropriate number of medical videos 110 may be input into the video compressor 120.
The method 700 may further include the step 730 of generating a compressed medical video 130. In this step, the video compressor 120 may compress the medical video 110. As described in more detail above, any appropriate type of video compressor may be used. For example, a video compression transformer or, in particular, a transformer-based temporal entropy model may be used to compress the medical video 110.
The method 700 may further include the step 740 of inputting the compressed medical video into a target feature detector 140. Once the medical video 110 has been compressed, the compressed medical video 130 output from the video compressor 120 may be input into the target feature detector 140. As described in more detail above, any appropriate type of target feature detector may be used. For example, the target feature detector 140 may be an object detector and, in particular, may be a RetinaNet detector.
The method 700 may also include the step 750 of predicting the presence and location of the target features. The target feature detector 140 may detect the presence and location of the target features in the compressed medical videos 130. As described in more detail above, the analyzed compressed medical video 150 output of the target feature detector 140 may be in any appropriate form. For example, in some embodiments, the target feature detector 140 may mark a box around the location it predicted as showing a target feature, similar to the box the healthcare provider marked around the target feature as shown in
The method 700 may further include the step 760 of computing a loss associated with the analyzed compressed medical video 150. As described above, the analyzed compressed medical video 150 may be compared to the medical video 110. For example, the loss may be calculated according to Eqns. (1), (2), and (3) as described above.
The method 700 may also include the step 770 of computing an accuracy of the predicted location of the target feature based on the location of the marked by the healthcare provider. The location marked by the healthcare professional may be compared to the location predicted by the target feature detector 140. The accuracy of the prediction may be calculated using SGD, as described in more detail above. The accuracy of the prediction may also take into account false positives/false alarms, which reflect indication of a target feature when none exists, and also false negatives, which reflect absence of a target feature when one exists.
The method 700 may also include the step 780 of updating the video compressor 120 and the target feature detector 140 based on the loss and the accuracy. For example, the trade-off between the bit-rate and distortion may be optimized according to Eqns. (1), (2), and (3) as described above. Moreover, the accuracy of predicting the location of the target feature may be optimized. In this way, the video compressor 120 learns how to compress medical videos 110 to minimize the bit rate and distortion, but also to maximize the accuracy of target feature detection. Thus, in some embodiments, the video compressor 120 may learn to distribute a larger proportion of bits to the location of the target features so that the target feature detector will be able to accurately detect the target feature. By preserving the quality of the most important parts of the medical video, the video compressor 120 may perform a better compression by allowing more distortion in unimportant parts.
5. Inference
After the combined compressor and detector module 210 has been trained, the combined module 210 enters an inference phase in which a subset of the steps in
Step 910 may include receiving medical videos 110. In some embodiments, the medical videos 110 may have one or more target features. In other embodiments, some of the medical videos 110 may have no target features. The medical videos are domain-specific in the sense that they are the same category of video used to train the model.
Step 920 may include inputting a medical video 110 into a video compressor 120.
Step 930 may include generating compressed medical videos 130. As described in more detail above, any appropriate type of video compressor may be used. For example, a video compression transformer or, in particular, a transformer-based temporal entropy model may be used to compress the marked medical video 110.
Step 940 may include inputting the compressed medical videos 130 into a target feature detector 140.
Step 950 may include detecting the target features, if any, in the compressed medical video 130. The target feature detector 140 may output a result, including an analyzed compressed medical video 150. As described in more detail above, the analyzed compressed marked medical video 150 output of the target feature detector 140 may be in any appropriate form. For example, in some embodiments, the target feature detector 140 may mark a box around the location it predicted as showing a target feature. Any appropriate output indicating the presence and/or location of the target features is contemplated.
A number of variations are possible on the examples and embodiments described above. Accordingly, the logical operations making up the embodiments of the technology described herein are referred to variously as operations, steps, objects, elements, components, layers, modules, or otherwise. Furthermore, it should be understood that these may occur in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.
Generally, any creation, storage, processing, and/or exchange of user data associated with the method, apparatus, and/or system disclosed herein is configured to comply with a variety of privacy settings and security protocols and prevailing data regulations, consistent with treating confidentiality and integrity of user data as an important matter. For example, the apparatus and/or the system may include a module that implements information security controls to comply with a number of standards and/or other agreements. In some embodiments, the module receives a privacy setting selection from the user and implements controls to comply with the selected privacy setting. In some embodiments, the module identifies data that is considered sensitive, encrypts data according to any appropriate and well-known method in the art, replaces sensitive data with codes to pseudonymize the data, and otherwise ensures compliance with selected privacy settings and data security requirements and regulations.
In several example embodiments, the elements and teachings of the various illustrative example embodiments may be combined in whole or in part in some or all of the illustrative example embodiments. In addition, one or more of the elements and teachings of the various illustrative example embodiments may be omitted, at least in part, and/or combined, at least in part, with one or more of the other elements and teachings of the various illustrative embodiments.
Any spatial references such as, for example, “upper,” “lower,” “above,” “below,” “between,” “bottom,” “vertical,” “horizontal,” “angular,” “upwards,” “downwards,” “side-to-side,” “left-to-right,” “right-to-left,” “top-to-bottom,” “bottom-to-top,” “top,” “bottom,” “bottom-up,” “top-down,” etc., are for the purpose of illustration only and do not limit the specific orientation or location of the structure described above. Connection references, such as “attached,” “coupled,” “connected,” and “joined” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and/or” rather than “exclusive or.” Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.
Additionally, the phrase “at least one of A and B” should be understood to mean “A, B, or both A and B.” The phrase “one or more of the following: A, B, and C” should be understood to mean “A, B, C, A and B, B and C, A and C, or all three of A, B, and C.” The phrase “one or more of A, B, and C” should be understood to mean “A, B, C, A and B, B and C, A and C, or all three of A, B, and C.”
Although several example embodiments have been described in detail above, the embodiments described are examples only and are not limiting, and those skilled in the art will readily appreciate that many other modifications, changes, and/or substitutions are possible in the example embodiments without materially departing from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications, changes, and/or substitutions are intended to be included within the scope of this disclosure as defined in the following claims.
The present application claims the benefit of U.S. Provisional Application No. 63/410,522, filed on Sep. 27, 2022, which is incorporated by reference herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63410522 | Sep 2022 | US |
