Patent Application
20250029410
The present invention relates generally to machine learning systems for video processing and, more specifically, to an active sparse labeling system and method for enhancing video understanding. Frames are selected from a multimedia input using an adaptive scoring mechanism and a dynamic loss formulation, aimed at reducing the costs and resources associated with annotating video data. The system utilizes both spatial and temporal data from video frames to optimize the frame selection process, thereby improving the efficiency of action detection models through targeted sparse annotations.
Video action detection is a challenging problem with many applications, such as security settings, autonomous driving, robotics, and similar industries. Video action detection requires spatio-temporal localization of an action within a video segment or frame. Capturing and selecting segments or frames from a larger multimedia file has led to innovative methods over the past few years, with most methods relying on annotation on every frame of a video sample, including bounding box sampling or pixel-wise sampling. Such annotations are different from action classifications, in which a class label for each video is sufficient for training. Accordingly, it is challenging and costly to annotate an action detection dataset at large scale, and existing datasets are much smaller in size compared to classification datasets.
Attempts have been made to improve label efficient learning techniques for action detection, which generally focus on semi-supervised or weakly-supervised approaches. These methods rely on either video-level annotations, point annotations, pseudo-labels, or reduced bounding-box annotations to reduce labeling effort. In addition, these methods rely on separate (and often external) actor detectors and tube linking methods, coupled with weakly-supervised multiple instance learning or pseudo-annotations, which limit the practical simplicity of the systems for general use on datasets. Moreover, the video-level and pseudo-label approaches include trade-offs of lower performance for saving annotations; in addition, the point and reduced bounding-box approaches require annotations for each instance to improve performance. These methods are also limited based on a lack of selection criteria for annotating only informative data.
Specifically, recent attempts at action detection in videos use a conventional neural network-based approach to perform spatio-temporal localization of actors in videos, commonly using a two-stage approach. The two-stage approach includes the use of object detection methods to detect actors per frame based on action classification models and to combine the actors using temporal aggregation for classification. In addition, attempts have been made to use active learning to iteratively select unlabeled data for assigning labels based on certain utility factors. Labeling a large set of data often proves to be expensive and unnecessary; as such, active learning can be vital in selecting related unlabeled data for further annotation in an iterative fashion. Some active learning models use uncertainty, entropy, heuristics and mutual information, or core-set selections to select samples that are most likely to provide maximum utility to the learning algorithm. Active learning-based classifications are effective for different modalities, such as images, videos, text, and speech. Classification only requires class labels for an entire sample, making the scoring easier for the model. However, extending the classification to a complex task, such as object detection, is challenging, since it requires dense annotations in each sample. Further extension to video segments adds additional complexity, due to the requirement of spatio-temporal annotations and selections of portions of the video for extra annotation. While some attempts have been made to perform frame selection using active learning for object segmentation, these attempts do not leverage temporal aspects of videos to avoid sequential annotation, thereby increasing the overall annotation cost.
Accordingly, what is needed is an active sparse labeling system to annotate the most informative frames within a video segment, thereby reducing annotation costs associated with dense video understanding tasks (such as action detection). However, in view of the art considered as a whole at the time the present invention was made, it was not obvious to those of ordinary skill in the field of this invention how the shortcomings of the prior art could be overcome.
While certain aspects of conventional technologies have been discussed to facilitate disclosure of the invention, Applicant in no way disclaims these technical aspects, and it is contemplated that the claimed invention may encompass one or more of the conventional technical aspects discussed herein.
The present invention may address one or more of the problems and deficiencies of the prior art discussed above. However, it is contemplated that the invention may prove useful in addressing other problems and deficiencies in a number of technical areas. Therefore, the claimed invention should not necessarily be construed as limited to addressing any of the particular problems or deficiencies discussed herein.
In this specification, where a document, act or item of knowledge is referred to or discussed, this reference or discussion is not an admission that the document, act or item of knowledge or any combination thereof was at the priority date, publicly available, known to the public, part of common general knowledge, or otherwise constitutes prior art under the applicable statutory provisions; or is known to be relevant to an attempt to solve any problem with which this specification is concerned.
The long-standing but heretofore unfulfilled need for an active sparse labeling system is now met by a new, useful, and nonobvious invention.
The novel method of sparsely labeling a multimedia input for dense video understanding includes a step of selecting, via an adaptive proximity-aware uncertainty selection model, a portion of frames from the multimedia input. The adaptive proximity-aware uncertainty selection model calculates an estimated frame-level uncertainty based on a pixel-wise confidence score of localization for each frame, and calculates an adaptive distance metric based on a proximity of a new frame to existing selected frames. Based on the estimated frame-level uncertainty and the adaptive distance metric, the adaptive proximity-aware uncertainty selection model calculates a selection score and selects the portion of frames having the selection score above a threshold, such that the selected portion of frames have diversity in a temporal domain.
The method includes a step of annotating labels for each frame of the selected portion of frames. The method also includes a step of determining a loss formulation for the multimedia input via a max-Gaussian weighted loss model that calculates a localization loss for each frame using the annotated labels and pseudo-labels generated for non-selected frames. The max-Gaussian weighted loss model also calculates, for each frame, a frame-wise weight based on an actual ground-truth frame location, and adjusts, for each frame, the localization loss based on the frame-wise weight. An active sparse labeling system for a multimedia input for dense video understanding includes a processor and a non-transitory computer-readable medium operably coupled to the processor. The computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the active sparse labeling system to automatically sparsely select and label frames of a multimedia input for action detection by executing certain instructions. In an embodiment, the instructions include the steps involved in the method of sparsely labeling a multimedia input for action detection.
An embodiment of the invention includes a system for actively labeling sparse segments of multimedia input to enhance the understanding of densely packed video data. This system includes a computer processor designed to manage several functions. The processor receives a multimedia input that consists of multiple frames. It then employs an adaptive proximity-aware uncertainty selection model, which to iteratively select a subset of frames. This model operates by calculating the uncertainty at the frame level for each frame, based on pixel-wise confidence scores regarding their localization. It also determines an adaptive distance metric, which assesses the proximity of a new frame to those previously selected and computes a selection score for each frame by considering both the frame-level uncertainty and the distance metric. Frames are then chosen for annotation when their selection scores surpass a predefined threshold, ensuring a selection of frames that is both diverse and informative across the multimedia input's temporal domain.
Once a subset of frames has been selected, they are annotated to create a labeled dataset. This dataset is used to train an action detection model, which the system aims to enhance to meet or exceed a predetermined precision benchmark or threshold. Such benchmarks may include, but are not limited to, video-metric average precision (v-mAP), frame-metric average precision (f-mAP) and mean average precision (mAP). The action detection model is updated iteratively by reapplying the selection model, refining frame selection based on insights gained from updated model performance and new annotations. This iterative process continues until the precision benchmark threshold is achieved or the annotation cost budget is exhausted. Additionally, the system may integrate a max-Gaussian weighted loss model that assigns weights to each frame based on how closely their localization matches ground-truth annotations. This model adjusts the localization loss for each frame according to the assigned weights, applying these adjustments iteratively to continuously enhance the model's accuracy. The weights follow a Gaussian distribution centered on the frame's distance to the nearest ground-truth annotation, with a variance that adapts based on the model's performance.
The system also may include an intra-sample approach within the uncertainty model to ensure that the selected subset of frames represents various temporal segments of the input. An uncertainty-based scoring mechanism prioritizes frames with higher uncertainty scores for annotation. Furthermore, the system features a pseudo-label generation function that creates pseudo-labels for frames not selected for annotation using methods like interpolation and spatio-temporal superpixel techniques, refining these labels as new annotations are acquired. Finally, the system is equipped to handle multiple types of annotations, such as bounding boxes, pixel-wise masks, and scribbles, and applies these annotations across the multimedia input in a mix that best suits the data's needs. The processor is configured to apply the uncertainty selection model iteratively, enhancing the action detection model's training and refining its accuracy over time.
An object of the invention is to reduce annotation costs associated with annotating, classifying, and detecting actions within a multimedia input by sparsely labeling only a portion of frames of the multimedia input while obtaining action detection accuracies that are comparable to more computationally demanding methods.
These and other important objects, advantages, and features of the invention will become clear as this disclosure proceeds.
The invention accordingly comprises the features of construction, combination of elements, and arrangement of parts that will be exemplified in the disclosure set forth hereinafter and the scope of the invention will be indicated in the claims.
For a fuller understanding of the invention, reference should be made to the following detailed description, taken in connection with the accompanying drawings, in which:
In the following detailed description of the preferred embodiments, reference is made to the accompanying drawings, which form a part thereof, and within which are shown by way of illustration specific embodiments by which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the invention.
As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term “or” is generally employed in its sense including “and/or” unless the context clearly dictates otherwise.
The present invention includes an active sparse labeling system that provides high performance and low annotation costs by performing partial instance annotation (i.e., sparse labeling) by frame level selection to annotate the most informative frames, thereby improving action detection task efficiencies. The active sparse labeling system utilizes a frame level cost estimation to determine the utility of each frame in a video based on the frame's impact on action detection. As such, the system includes an adaptive proximity-aware uncertainty model, which is an uncertainty-based frame scoring mechanism. The adaptive proximity-aware uncertainty model estimates a frame's utility using the uncertainty of detections of the frame's proximity to existing annotations, thereby determining a diverse set of frames in a video which are effective for learning the task of dense video understanding (such as action detection). In addition, the active sparse labeling system includes a loss formulation training model (max-Gaussian weighted loss) that uses weighted pseudo-labeling. The active sparse labeling system will be described in greater detail in the sections herein below.
As shown in
Since some frames have more utility than others for learning action detection (such as due to lack of motion, variation in action dynamics, redundancy in appearance, or redundancy in action), the active sparse labeling system includes annotations of only l frames fv,l in a video v instead of labeling each frame, leaving a set of u unannotated frames fu,l. As such, the active sparse labeling system avoids annotation of frames with lower utility and reduces the overall labeling cost. Each video v has a class label vcls for the action category and a set of l annotated frames fv,l, which indicates the localization of actions.
During each active learning cycle, the active sparse labeling system selects video frames for labeling that have the highest utility for learning action detections. Uncertainty (42) provides a measure to estimate a model's confidence on decisions and has been used for selecting informative samples. The active sparse labeling system requires informativeness of each frame in a video to generate a partial sample. The action detection model M (18) provides spatio-temporal localization for the entire video, and the system uses a pixel-wise confidence score of localization on each frame to estimate frame-level uncertainty. An embodiment of the system uses MC-dropout to estimate the model's uncertainty for each pixel in the video, which is a more efficient form of uncertainty estimation as compared to a Bayesian neural network. The uncertainty is estimated over T different trials, and the score is averaged over all pixels in a frame. For a given video v with I frames, the uncertainty Ui for the ith frame over T trials is computed as:
Unlike images, motion in videos results in some continuity between frames, and it is highly likely that nearby frames will include similar uncertainty scores (42). Therefore, selecting frames based on uncertainty favors adjacent frames, which may have similar utility for learning action detection. As such, the active sparse labeling system instead uses an adaptive proximity-aware uncertainty (APU) selection mechanism (50) to ensure that the selected frames (46) have diversity in the temporal domain. The APU scoring (50) incorporates a distance measure into cost estimation and uses the proximity (44) to existing annotated frames (36). As more frames are selected, the distance measure adapts to the additional selected frames. The system uses a normal distribution (μ, σ2) for distance measure D, where each annotated frame has its own distribution centered around its temporal location in the video. Given a video with K annotated frames, the distance measure Di for the ith frame of the video is computed as:
After obtaining the UAPU score for all frames in V videos, the frame with the highest global score is selected (46), and the remaining frames are scored again using the adapted distance measure. By rescoring the remaining frames, the probability of picking frames from the same region is reduced, since a deficient region is likely to have more frames that score higher in the selection process. The rescoring model only requires a minimal computational expense to recompute the distance measure. After Fannot frames are selected (52) in accordance with B budget, the frames are annotated by an oracle (30) and the training set is updated (32) with the new annotations (16), thereby completing one active learning cycle. The model M (18) is then trained using the updated annotations.
Non-activity regions withing a frame can negatively influence the score, as the model easily determines background pixels compared to the actual action region within a frame. A low uncertainty score from background pixels lowers the overall frame uncertainty even if the activity region has high uncertainty, especially in videos with a relatively larger background area compared to the actual action region. Therefore, the active sparse labeling system ignores pixels that are predicted as background (true negatives and false negatives) high a confidence (using threshold τ) when computing the frame-level uncertainty. While this can exclude some foreground pixels (i.e., false negatives) from the uncertainty estimation, such pixels are not useful in the system since they have low uncertainty.
Given a video clip V={f1, f2, . . . fN} with N frames where K frames (48) are annotated such that K<N, the action must be detected through the entire clip. A traditional action detection network is trained with the help of two different objectives: a classification loss Lc for action category and a localization loss La for spatio-temporal detection. The classification loss Lc is computed for the entire video clip, and the localization loss Ll is computed for every frame of the video.
Since the system utilizes sparse labeling, it is not possible to compute the localization loss Ll on every frame, since there are missing annotations. As such, the localization loss Ll with sparse labeling is computed as:
The active sparse labeling system uses a loss formulation to leverage both masking and pseudo-labels. Since the pseudo-labels that are close to ground-truth labels are more reliable, the system uses a Max-Gaussian Weighted Loss (MGW-Loss) model which discounts the approximated pseudo-labels, since they are not as reliable as the actual ground-truth. The localization loss is computed for each frame using both available and pseudo-labels, where the pseudo-labels have a varying weight in the overall loss component. The approximated annotations do not have a similar weight, since their distances from the annotated frames varies. A mixture of Gaussian distribution is used to assign the weight of each frame w∈{1 . . . . W}˜(μgt, σ2), given gt∈{1 . . . . K} actual ground-truth frame location as the mean of the distribution and σ is the variance of the distribution. The weighted localization loss LlMGW is defined as:
Video action detection is a challenging problem, and existing methods typically follow a complex pipeline. For example, a region proposal-based approach has been found to be effective. However, training such two-step methods is inefficient, especially relating to an iterative active learning framework. As such, the active sparse labeling system includes an end-to-end training approach for both classification and detection, further achieving memory and training speed efficiencies by replacing 3D routing with 2D routing. The system also includes added dropout layers used for uncertainty, and MGW-Loss (Eq. 5) is used to handle sparse labels by obtaining the frame-wise weight from the max-Gaussian weighted method and adjusting the loss using the obtained weight. The network is then trained using margin-loss for classification and binary cross-entropy loss for spatio-temporal localization.
In an embodiment of the active sparse labeling system, given a set of N videos V={v1, v2, . . . vn} with F total frames, a subset of videos V8T⊂V are selected with F8T frames and AT % of frames are annotated from subset V8T based on a total budget B after T active learning cycles. The resulting subset of videos V8T have F8T=(FLT, FUT) frames, where FLT frames are annotated frames and FUT are unannotated frames. The active sparse labeling system enables the use of partial spatio-temporal annotation, utilizing both FLT and FUT frames for model training. AN embodiment of the system begins with an initial set of V80⊂V videos with F80=(FL0, FU0) frames, where A % (FL0) of these frames are annotated. The action detection model M0 is trained using (Vs0, Fs0), and the trained model is used to select additional videos and frames using the active sparse labeling system to obtain new annotations. The system selects a diverse set of informative videos for annotation from (V−Vs0), which is added to Vs0 to obtain Vs1 videos. Subsequently, A % informative frames are selected from the selected videos Vs1 for annotation. The iterative process is repeated until the desired performance is met or the total budget B is exhausted.
An overview (11) of the embodiment of active sparse labeling system is shown in
Embodiments of video action detection require spatial localization of the activity in each frame with the temporal consistency of the predicted action location throughout the video. While existing methods include complex multi-stage training with dense frame-level annotations, such iterative training is challenging due to large resource requirements and dependencies on good region proposals. As such, embodiments of the active sparse labeling system include a one-stage approach with state-of-the-art performance on the action detection task, which can be efficiently trained end-to-end using a single GPU and including reduced complexity, with the model being trained using margin-loss for classification and binary-cross entropy loss for action localization.
As such, an embodiment of the active sparse labeling system includes a hybrid active learning approach that enables selection across unlabeled videos to identify diverse and important samples, while also selecting limited frames within those samples for annotation, thereby significantly reducing overall annotation costs. As shown in
In some embodiments, model uncertainty can predict the utility of a video sample. In the case of a classification task, video-level classification uncertainty can be a sufficient predictor; however, video action detection also requires localization of actions on every frame of a video. Therefore, spatio-temporal localization also plays a important role in estimating a sample's utility. To take this into account, the hybrid model relies on spatio-temporal uncertainty. Specifically, uncertainty in the model's prediction at pixel-levels is used to compute spatio-temporal uncertainty. The activity and non-activity region in a video will vary across action classes as well as across video samples. Therefore, uncertainty scores based on all pixels in a video for sample utility will not be comparable across all unlabeled videos VU for learning action detection, since such an approach provides low uncertainty score of videos with short uncertain actions and long easy non-action regions, which is not favorable for such videos. To overcome this issue, the system selects limited frames in each video, in which video frames are ranked based on uncertainty and the top At frames with high uncertainty are selected. Given a pixel-level uncertainty U, the spatio-temporal uncertainty at video-level is calculated as Eq. 6:
The informative videos selected in the inter-sample selection Vs-primet are added to the existing set Vst-1 to obtain Vst. Within the intra-sample selection, frames with high utility are selected from the videos Vs-prime for frame-level annotation. The system relies on frame-level model uncertainty Uf=ΣiII(Ui) for all I pixels in a frame to estimate frame utility for action detection, where U is pixel-level uncertainty as described in Eq. 7. Since pixel-level uncertainty U is already computed for spatio-temporal uncertainty, there is no computation overhead for intra-sample selection.
Model uncertainty can be used for sample selection focusing on their informativeness. However, model uncertainty does not ensure diversity among selected videos and there can be redundancy in such a selection strategy. Moreover, using class labels to address diversity incurs additional annotations costs. As such, an embodiment of the active sparse labeling system utilizes an implicit clustering approach including latent video features, which does not require additional annotations. Specifically, the system uses deep clustering which learns the cluster representation for each category from the known labeled subset Vs0 and adapts the clusters as the latent features of each video changes during training.
To enable diverse sample selection, the system models the relationships between diversity of each unlabeled sample VU with already labeled samples VL. The clustering approach allows the model M to learn latent features which represent each sample in a cluster. The objective of the model M is to improve the latent features such that it is close to the corresponding cluster center for that sample. The clustering objective is defined as Eq. 8:
Information scores are first computed for each video in VU using Eq. 6, and each cluster in C=[c1, c2, . . . . ck] is found, where K total clusters corresponds to each unlabeled video. The total number of videos selected in a cycle is constrained by budget Bv. The samples selected per cluster are limited, such that the selection is proportional to the cluster size. For any cluster having no video, a budget of nc×Bv/NU is assigned, where NU represents the total number of unlabeled videos. Since nearby frames in a video have similar model uncertainty and redundant utility, the system avoids selecting nearby frames in intra-sample selections to ensure diversity during frame selection.
By using partial annotations, it is not possible to train localization without annotations, thereby limiting the use of actor annotations for each frame to train models for action localization and classification. As such, an embodiment of the system utilizes a loss formulation (spatio-temporal weighted loss, or STeW loss) that effectively uses partial annotations for localization. The partial spatio-temporal annotations are converted into dense pseudo-labels using interpolations. However, since these pseudo-labels can have errors due to motion of actor/camera in a video and temporal gap between the partial labels, the model uses temporal continuity of actions to enable effective utilization of partial annotations. Since actions have some temporal continuity across time which may vary with different actions temporal continuity within a video is leveraged to compute spatio-temporal weight for each pixel independently, thereby capturing the confidence of a pseudo-label.
As such, in an embodiment, the system first computes the pseudo-labels using interpolation between the annotated frames, and subsequently applies a spatio-temporal weight to suppress incorrect pseudo-labels. The overlap of annotation for nearby frames is computed and each pixel is assigned a weight based on the overall consistency in accordance with Eq. 9:
In an embodiment of the system, the overall training objective is given as Eq. 11:
For each stage of annotation, a fixed budget B is assumed, which is separated to annotate video labels (Bv) and to annotate frames within the video (Bf). Annotating each video label requires a cost Cv since the annotator must identify the class of each label; similarly annotating each frame with bounding-box or pixel-wise labels requires a cost Cf. For each stage, embodiments of the system only annotate videos and frames such that Cvtotal≤Bv and Cftotal≤Bf.
An embodiment of the active sparse labeling system was evaluated on three different datasets: UCF-101, which contains 3207 videos from 24 different classes with spatio-temporal bounding box annotations; J-HMDB, which contains 928 videos from 21 classes with pixel-level spatio-temporal annotations; and YouTube-VOS, which contains 3471 training videos from 65 categories with pixel-level annotation for multi-object segmentation. Following prior action detection works on the UCF-101 and J-HMDB datasets, the spatial intersection-over-union (IoU) was calculated for each frame per class to obtain the frame average precision score, and the spatio-temporal IoU was calculated per video per class to obtain the video average precision score. These scores were then averaged to obtain the frame-metric average precision (f-mAP) and video-metric average precision (v-mAP) scores over various thresholds. For video segmentation, the average IoU (Jscore) and average boundary similarity (Fscore) were evaluated.
The active sparse labeling system was implemented using an I3D encoder head with pre-trained weights from the Charades dataset. An Adam optimizer with a batch size of 8 was used to train for 22k iterations in each active learning cycle. Dropouts were used to generate uncertainty by enabling uncertainty during inference. For the YouTube-VOS task, the value τ=0.9 was used for non-active suppression, and σ=1.3 was used for Eq. 2 and Eq. 5.
In the initialization stage, the availability of annotations was assumed for 1% of frames in each video in V to make the sparse annotation set S). These frames are randomly selected for the first stage. Amounts of 1%, 3%, and 5% of the initial frames were used for UCF-101, J-HMDB, and YouTube-VOS, respectively. Annotation costs for each frame were assigned as Cframe=Actor×Clicks based on clicks per actor (bounding box/pixels).
Several baselines were explored to understand their limitations on video action detection. First, random and equidistant frame selections were used, in which random selection includes the selection of frames at random in each stage, and in which equidistant frame selection includes the use of equal distances between the frames during selection. Next, existing active learning methods were extended to video action detection, in which each frame was scored using the active learning algorithm for frame selection. Uncertainty was calculated at pixel-level in each baseline. Each baseline was trained using the same action detection backbone to provide a comparison basis, and comparisons were made against random, equidistant, uncertainty-based, and entropy-based approaches. Results are shown in Table 1 (shown in
As shown in
The active sparse labeling system was evaluated on the UCF-101 dataset and the J-HMDB dataset for action detection, and the system was compared with fully-supervised training, as shown in Table 2 (shown in
Turning to Table 3, shown in
The effectiveness of the MGW-Loss function was evaluated for video action detections with sparse labels and the MGW-Loss function was compared to baseline masking and interpolations-based loss, with results shown in
The APU technique was evaluated in comparison to entropy and uncertainty-based selection methods when using the same loss function from Eq. 5. As shown in
In addition, the effects of adding additional frames until the score saturates were evaluated, with results shown in
Lower budget steps enable the selection of fewer frames with a high utility in each step, instead of selecting more frames with low utility in higher budget steps. Since the annotation set is more curated in each step in lower steps, better frames are obtained for the same annotation budget as higher steps. The effects of using step sizes of 1% and 5% were evaluated (with results shown in
The active sparse labeling system is focused on sparse labeling, in which frames with high utility within a video are selected for annotations. However, it is important to note that videos as a whole have varying utility. To exploit this aspect, two different frame selection strategies were evaluated: local selection and global selection. In local selection, each video has a fixed budget b/Nv, where b is the budget per cycle and Nv is the total number of videos in the training set. However, frames in global selection are taken from a global pool which includes frames from all videos, ranking based on overall dataset utility. As shown in
An approach was followed, such that the entire sample (video) was annotated instead of finding the most useful frames within each sample. Pixel level uncertainty was calculated and averaged over all the pixels in a frame using Eq. 1, and subsequently averaged over all frames in a video to obtain the video level score. While this approach is simpler, it has higher cost during annotation with lower data variation. For example, assuming a fixed cost of c per frame with f frames to annotate, a budget of B=c×f can be assumed. The frames can be distributed across the set by picking only few important frames from each video, which would increase variation in the training set. However, if the entire sample is annotated, there will be many redundant annotations with little gain; as such, frame selection performs better for an action detection task, as shown in
The generalization of the proposed cost and loss function was tested for a video object segmentation task on the YouTube-VOS 2019. Table 4 (shown in
The active sparse labeling system uses an uncertainty-based scoring mechanism for selecting informative and diverse set of frames for action detection. In addition, the system includes a simple yet effective loss formulation which is used to train a model with sparse labels. The system results in annotation cost savings and achieves performance comparable to fully supervised methods while using only 10% of labels. Moreover, the system can be generalized for video object segmentation.
An embodiment of the active sparse labeling system was evaluated on two different datasets: UCF-101, which contains 3207 videos from 24 different classes with spatio-temporal bounding box annotations; and J-HMDB, which contains 928 videos from 21 classes with pixel-level spatio-temporal annotations. The standard frame-mAP and video-mAP scores were measured for different threshold to evaluate the system's action detection results, with the frame-mAP reflecting the average precision of detection at the frame level for each class (which is then averaged to obtain the f-mAP), and the video-mAP reflecting the average precision of detection at the video level (which is then averaged to obtain the v-mAP score).
The training was initialized with a set of videos VL0 with class labels and A % of annotated frames within the videos that are randomly selected for the first stage. For clustering, the system uses K=5 centers; for each stage, the system selects v % of videos for annotation based on the budget Bv, Bf, where videos are given class labels and A % of the frames are annotated and added to the set of videos VL0. The system repeats this process until the total budget is exhausted or the desired performance is achieved.
An embodiment of the active sparse labeling system was implemented using 2D capsules and an I3D encoder head with pre-trained weights from the Charades dataset. An Adam optimizer with a batch size of 8 and a learning rate 5e-4 was used to train. The values for Plow=0.1 and Phigh=0.9 are set empirically. Random crops and horizontal flips are used for video augmentation during training. Interpolation is performed using linear point interpolation for bounding-box (UCF-101-24) and using CyclicGen for pixel-wise (J-HMDB) dataset experimentations, with uncertainty being computed based on dropout during inference.
Several baselines were explored to understand their limitations on video action detection. First, random and equidistant frame selections were used, in which random selection includes the selection of frames at random in each stage, and in which equidistant frame selection includes the use of equal distances between the frames during selection. Next, existing active learning methods were extended to video action detection, in which each frame was scored using the active learning algorithm for frame selection. Uncertainty was calculated at pixel-level in each baseline. Each baseline was trained using the same action detection backbone to provide a comparison basis, and comparisons were made against random, equidistant, uncertainty-based, and entropy-based approaches.
As shown in
The cluster-based video and frame selection approach of the active sparse labeling system selects limited samples and can be compared to prior weakly supervised methods for video action detection which rely on multiple instance learning or instance learning paired with off-the-shelf actor detector or user-generated points to create ground truth annotations for training. These results are shown in particular in
The effect of clustering for video selection in the embodiment of the active sparse labeling system was evaluated, with results shown in
The STeW loss of the embodiment of the active sparse labeling system was evaluated by training the action detection network using a frame loss (computing loss for the annotated frame while ignoring pseudo-labels) and an interpolation loss (equally computing loss for all real and pseudo-labels) for the UCF-101-24 dataset. During evaluation, the same active learning algorithm was used for all approaches, with results for the UCF-101-24 dataset for different steps being shown in
Different scoring functions were also evaluated in comparison to the CLAUS method of the active sparse labeling system, with each scoring function being evaluated using the STeW loss (results shown in
Turning to
The effect of increasing only samples for a constant frame annotation rate of 5% and increasing both samples and frames annotations was evaluated to better understand the importance of each variation. Increasing only samples with a constant frame annotation rate has a lower associated annotation cost than the cost associated with increasing both samples and frames for the same step. As such, an evaluation was performed to determine whether the added annotation is worth the cost, since the goal is to obtain maximum performance gain with the lowest cost. Turning again to
While samples from different classes add diversity, too many samples for easy classes also add redundancy and can increase costs as a result. As shown in
Selection using the hybrid active sparse labeling system was compared to traditional approaches for the same annotation budget. Inter selection assumes that each video is fully annotated and randomly selects videos for a given budget, thereby selecting fewer videos as more budget is used to annotate all frames. Intra selection assumes that each video of the dataset is annotated for at least one frame, thereby spreading the budget over all videos. The comparison is shown in
In this embodiment of the active sparse labeling system, a hybrid active learning strategy was implemented and evaluated for reducing annotation costs for video action detection. The hybrid approach uses a cluster-aware strategy to select informative and diverse samples to reduce sample redundancy while also performing intra-sample selection to reduce frame annotation redundancy. The active sparse labeling system also includes a STeW loss to help the model train with limited annotations, thereby removing the need for dense annotations for video action detection. In contrast to traditional active learning approaches, the hybrid approach adds more annotation diversity at the same resource cost. The active sparse labeling system was evaluated on two different action detection datasets, demonstrating the effectiveness of the system in learning from limited labels with minimal trade-offs on performance.
Turning now to
The labeled dataset 92 is used to train an action detection model 94 to achieve or exceed a predetermined precision benchmark 96. If not meeting benchmark 96, the action detective model is updated iteratively by reapplying the adaptive proximity-aware uncertainty selection model to further refine frame selection based on updated model insights and annotations until the precision benchmark 96 is met or an annotation cost budget 98 is exhausted.
All referenced publications are incorporated herein by reference in their entirety. Furthermore, where a definition or use of a term in a reference, which is incorporated by reference herein, is inconsistent or contrary to the definition of that term provided herein, the definition of that term provided herein applies and the definition of that term in the reference does not apply.
Active Learning (AL) means a machine learning approach that selectively chooses the most informative data points from a large dataset for labeling and training. This method is iterative, involving cycles of training a model, using the model to assess the informativeness of unlabeled data, and then selecting the most valuable data points for annotation. In video action detection, Active Learning strategies such as uncertainty sampling, entropy-based sampling, heuristic-based selection, and coreset selection are utilized. These strategies evaluate frames based on predicted uncertainties, entropy levels, or pre-defined heuristics to select those that are likely to have the highest impact on model performance. By focusing on such frames, Active Learning aims to significantly improve the efficiency and effectiveness of the training process by reducing the number of labels needed while still maintaining or enhancing the model's accuracy and robustness.
Adaptive Distance Metric means a quantitative measure used within machine learning frameworks to evaluate the similarity or proximity of new data points, such as frames in a video, to previously labeled data points. This metric is helpful in scenarios where maintaining diversity in the training dataset is necessary to enhance model generalization. By assessing the proximity of a new frame to existing labeled frames, the Adaptive Distance Metric ensures that the selected frames are varied not just in their visual content but also in their spatial and temporal characteristics. This diversity supports the development of more robust models capable of generalizing well across different, unseen datasets. The metric often adapts dynamically as new data are labeled and added to the dataset, optimizing ongoing learning and selection processes.
Adaptive Proximity-Aware Uncertainty Selection Model means a framework in machine learning that integrates both spatial proximity and predictive uncertainty to guide the annotation of data points. This model assesses each frame of a video for its level of uncertainty—typically how much the model predictions vary for that frame—and its proximity to previously selected frames. By considering these factors, the model strategically selects a set of frames that not only encompasses a broad range of scenarios but also targets those where the model's current performance is weakest, thereby providing a balanced and information-rich dataset for training. This method is especially effective in large-scale video datasets where exhaustive labeling is impractical, helping to refine the model with a focus on improving accuracy in uncertain and diverse conditions.
Annotation means the process of labeling video frames with data that describes the content, such as the location of objects or the type of actions occurring. This information is used to train the model to recognize similar patterns in unlabeled data.
Annotation Costs means the cumulative resources required to manually label data within machine learning projects. These costs encompass both the direct labor hours spent in the manual identification and labeling of objects within frames and the computational costs associated with supporting technologies like software tools and infrastructure. In dense video understanding, where each frame of potentially thousands in a video may require annotation, these costs can be substantial. Strategies to reduce annotation costs include the use of automated tools that generate pseudo-labels and techniques that prioritize the labeling of particularly informative frames, thereby reducing the need for widespread manual annotation while maintaining or improving the quality of the data used for model training.
Bounding Box Annotation means a method in image processing and computer vision where a rectangular box is drawn around a specific object or region of interest within an image or video frame. This annotation type is fundamental in training machine learning models for tasks such as object detection and localization. The bounding box clearly defines the spatial extent of the object, allowing the model to learn not only the appearance of the object but also its size and position within different contexts. This form of annotation is important for applications where the precise dimensions and locations of objects affect the outcome, such as in surveillance, autonomous driving, and robotic vision.
Clustering-Assisted Active Learning Strategy means a method in which active learning is supported by clustering techniques to enhance sample selection for annotation in machine learning tasks, particularly in video action detection. This strategy leverages both model uncertainty and the diversity of data points to optimize the training process with limited labels. A key component is the use of Clustering-Aware Uncertainty Scoring (CLAUS), which assesses both the informativeness of samples based on model uncertainty and the diversity based on clustering outcomes. The dual focus on informativeness and diversity helps in selecting not just the most uncertain samples, but also those that will contribute to a well-rounded training dataset. This is particularly useful in tasks requiring detailed spatio-temporal annotations, such as video action detection, where labeling can be both costly and time-consuming. By integrating clustering, the approach aims to reduce redundancy among selected samples, ensuring a more efficient annotation process and effective learning, even with significantly fewer annotated examples.
Confidence Scores means numerical values assigned by the model to its predictions, indicating the probability or likelihood that a particular classification or localization is correct. These scores are used to assess the certainty of the model's outputs on individual frames.
Coreset Selection means a technique in machine learning that involves selecting a subset of a dataset that best represents the full dataset. This method is particularly useful in scenarios where processing or labeling the entire dataset is computationally expensive or otherwise impractical. By focusing on a carefully chosen coreset, models can be trained more efficiently and effectively, as the selected samples capture the underlying distribution and diversity of the full dataset. Coreset selection reduces redundancy, minimizes annotation costs, and can significantly speed up the training process without sacrificing the quality of the model output.
Dense Video Understanding means the detailed analysis and interpretation of video content to identify and classify actions, objects, and events across every frame of the video. This high level of analysis requires extensive annotations that cover a wide range of spatial and temporal details within the video, making it possible to achieve a deep understanding of complex scenes. Dense video understanding is essential for applications such as automated surveillance, where understanding nuanced activities over time can be important for decision-making.
Diversity means the variety in the types of frames selected for annotation across the temporal domain of the video. Ensuring diversity helps the model learn from a broad range of scenarios and actions, enhancing its ability to generalize across different videos.
Entropy-Based Sampling means a strategy in active learning where data points (such as video frames) are selected based on the entropy of their predicted class distributions. Entropy, a measure of uncertainty or unpredictability, helps identify frames that contain the most ambiguous or diverse information according to the current state of the model. High-entropy frames are deemed more informative because they are likely to contain new, varied, or complex information that can challenge and thus improve the model. Selecting such frames for annotation helps in building robust models that are better at generalizing from complex or variable data inputs.
Frames means individual images or pictures that make up a video. Each frame represents a static snapshot of the visual content at a specific point in time within the video sequence.
Frame-Level Uncertainty means the measure of uncertainty associated with the prediction of each individual frame in a video as processed by a machine learning model. High frame-level uncertainty indicates that the model has low confidence in its prediction for that frame, which typically occurs in scenarios with ambiguous or complex visual content. Identifying frames with high uncertainty is useful for directing annotation efforts towards the most challenging parts of the data, which can significantly improve the model's accuracy and robustness by focusing training on these difficult cases.
Gaussian Weighting means a method of assigning weights to frames using a Gaussian distribution, where frames closer to the ground truth have higher weights. This method helps focus the model's learning on more accurate and relevant data.
Ground-Truth Frame Location means precisely defined and verified coordinates within video frames that delineate the presence and boundaries of objects or actions of interest, as determined through manual annotation. This term is important in the context of training and evaluating machine learning models involved in video analysis, such as action detection and object tracking systems. Ground-truth annotations are typically established by human experts who meticulously label video frames to mark the locations of specific activities or entities. These annotations might appear as bounding boxes, outlines, or other marking systems that clearly indicate the extent and position of an object or action within the frame's spatial context. Ground-truth frame locations serve multiple functions: they provide a definitive dataset against which the predictions of automated systems can be compared, offering a benchmark for measuring the accuracy, precision, and recall of such systems. By supplying a reliable standard, ground-truth frame locations enable developers to quantitatively assess and refine the performance of detection algorithms, ensuring that these technologies meet the stringent accuracy requirements needed in applications ranging from automated surveillance to interactive media. The creation of these annotations, while resource-intensive, is indispensable for developing robust vision-based models that perform reliably across diverse scenarios.
Heuristic-Based Selection means a method in data annotation and machine learning where frames are selected based on specific rules or heuristics that assess their relevance or importance. These rules may consider factors like the presence of motion, changes in lighting, or the occurrence of particular events that are known to be challenging or informative for the model. By using heuristic-based selection, the annotation process can be made more efficient by focusing efforts on frames that are most likely to enhance the model's performance.
Hybrid Selection Approach means a method used in machine learning to optimize the selection of data for annotation by integrating multiple selection strategies to balance informativeness, diversity, and annotation costs. This approach selects important samples and high-utility frames for annotation, effectively reducing overall costs and enhancing the efficiency of the training process. It leverages a clustering-aware uncertainty scoring system, known as CLAUS (Clustering-Assisted Active Learning Strategy), which assesses both the informativeness and diversity of samples. The hybrid selection approach involves using a combination of intra-sample and inter-sample selection strategies. Intra-sample selection focuses on selecting the most informative frames within individual video samples based on model scores, specifically choosing the top frames that significantly contribute to the understanding of the action depicted. Inter-sample selection, on the other hand, evaluates entire video samples based on video scores and cluster assignments to identify and annotate samples that provide diverse insights into the action detection task. This hybrid methodology incorporates deep clustering techniques to ensure diversity in the selected samples, supporting a comprehensive learning process. It addresses redundancy both within and across samples by selecting frames and samples that add unique value to the dataset, avoiding wasteful annotation of similar or less informative content. The approach is designed to work efficiently even with limited computational resources, supporting end-to-end training on a single graphics processing unit (GPU) and using advanced loss calculations like spatio-temporal weighted (STeW) loss and binary-cross entropy loss for precise action localization and classification.
Interpolation means a mathematical technique used to estimate missing data points within a sequence by utilizing known data points. In the context of this invention, interpolation is used to generate pseudo-labels for frames that are not explicitly annotated.
Intra-Sample Approach means a frame selection method where informative frames are chosen from each video sample, considering the temporal relationships within the sample to ensure diverse representation. This approach aims to select frames that capture significant variations or key events within a single video sequence, maximizing the range of actions and interactions depicted. By focusing on temporal diversity within samples, the method helps to create a training dataset that comprehensively represents the dynamic nature of the video content. It effectively supports models in learning to recognize patterns and predict outcomes over sequences, rather than just from isolated frames, thus enhancing the model's ability to generalize across similar yet temporally distinct events.
Intra-Sample Approach means a method used in data analysis and machine learning for selecting informative frames or data points within individual samples of a dataset, particularly when dealing with sequential data like video or time-series. This approach focuses on identifying and utilizing the internal structure and temporal relationships within each sample to enhance the diversity and representativeness of the selected data points for training or analysis purposes. The intra-sample approach examines the sequence of frames within each video clip to identify those that best capture key actions or variations in the scene. By analyzing the temporal flow within a single sample, this method ensures that the selected frames span the dynamic range of activities and interactions, providing a comprehensive view of the events in the video. This selective process helps in creating a balanced dataset where each sample contributes maximally to the model's understanding of the actions being analyzed.
Localization means the process of identifying the precise location of objects or actions within video frames. This involves determining the spatial coordinates or boundaries of objects within the frame.
Localization Loss means a type of loss function used in machine learning models to quantify the error in determining the position and extent of objects within images or video frames. This metric is key for tasks involving object detection and tracking, where the model must not only identify the presence of objects but also accurately predict their spatial boundaries. Localization loss is calculated by comparing the predicted locations of objects-typically represented as bounding boxes or segmentation masks-against their corresponding ground-truth annotations provided during training. The purpose of localization loss is to fine-tune the model's ability to precisely outline objects, ensuring that the predicted boundaries closely match the actual locations as indicated by the ground-truth data. Common methods for calculating localization loss include Mean Squared Error (MSE), Intersection over Union (IoU), and Smooth L1 loss, each offering different advantages depending on the specific requirements of the training scenario. For instance, IoU is particularly favored in object detection tasks because it directly measures the overlap between predicted and true bounding boxes, providing a clear and intuitive metric for spatial accuracy. By effectively minimizing localization loss during the training process, machine learning models can achieve higher accuracy in tasks requiring precise spatial understanding, such as autonomous vehicle navigation, robotic vision, and advanced video surveillance systems.
Loss Formulation means the mathematical calculation used to measure and minimize the difference between predicted and actual annotations. In video action detection, this often involves a max-Gaussian weighted loss model, which adjusts localization loss based on a combination of pseudo-labels and ground truth data. The loss formulation serves as a feedback mechanism for training machine learning models, guiding them towards more accurate predictions by penalizing errors in a way that reflects the relative importance or confidence of different data points. Such formulations are integral to refining model parameters during the learning process, especially in complex tasks like video understanding where precision in spatial and temporal localization is important.
Max-Gaussian Weighted Loss Model means a loss model that incorporates Gaussian weighting to emphasize frames that have more accurate annotations. This model calculates localization loss for each frame and assigns frame-wise weights based on their proximity to ground truth locations. By weighting the frames differently according to their annotation accuracy, the model prioritizes learning from the most reliable data. This approach helps to fine-tune the model's performance, particularly in applications such as video action detection where the spatial accuracy of predictions is essential. The Gaussian distribution applied in the weighting considers the variance around each annotated frame, thereby smoothing the impact of outlier frames and reinforcing the influence of frames with high confidence annotations.
Multimedia Input means any form of content that combines different types of media such as video, audio, and text. In the context of this invention, multimedia input specifically refers to video content that comprises a sequence of frames, each containing visual data that the system processes for action detection and understanding.
Non-Transitory Computer-Readable Medium means a storage device capable of retaining computer-readable instructions, which, when executed by a processor, perform specific functions like frame selection and loss calculation. These mediums include various forms of digital memory such as SSDs, HDDs, USB drives, and memory cards. They are essential for storing software that instructs machine learning models how to process and analyze data, enabling complex tasks such as video action detection and frame annotation. The term “non-transitory” distinguishes these mediums from transient signals and emphasizes their role in permanently storing data that can be repeatedly accessed and executed by computing devices.
Omni-Supervised Learning means an approach where models are trained using annotations from diverse sources, including bounding boxes, scribbles, points, and tags. This method allows a machine learning model to benefit from a rich mixture of annotated data types, enhancing its ability to understand complex visual content from limited or varied labeling inputs. By aggregating different types of annotations, the model can leverage comprehensive contextual clues, reducing the reliance on densely labeled datasets and enabling more scalable training processes.
Pixel-Level Mask Annotation means the detailed labeling of objects or regions within a frame at the pixel level. This type of annotation provides highly accurate spatial information by specifying the exact boundaries of objects within images or video frames. While pixel-level mask annotation offers precision that is important for tasks such as semantic segmentation, it is also resource-intensive, requiring significant manual effort and computational resources to produce detailed masks for large datasets.
Precision Benchmark means a metric used to evaluate the accuracy of a detection model in identifying and localizing objects within multimedia content. Common precision benchmark algorithms include video-metric average precision (v-mAP), which assesses accuracy across entire video sequences; frame-metric average precision (f-mAP), which measures accuracy at the individual frame level; and mean average precision (mAP), which calculates the average precision at various recall levels across multiple object categories.
Proximity refers to the closeness or distance between frames within a video or between a new frame and previously selected frames. Measuring proximity helps ensure that selected frames are not clustered too closely together, thereby maintaining diversity in the samples chosen for annotation.
Pseudo-Label Generation means the creation of estimated labels for frames that lack ground truth annotations. This process utilizes techniques such as interpolation, superpixels, and predictive modeling to assign provisional labels to unlabeled data. Pseudo-labels serve as temporary training data, allowing models to extend their learning beyond the limits of manually annotated data. They play a role in semi-supervised learning frameworks by providing additional information that can be refined through successive training iterations, thereby enhancing the model's overall accuracy and robustness.
Pseudo-Labeling means a semi-supervised learning technique where a model's predictions on unlabeled data are used as pseudo-labels to enable further training. This approach helps in extending the training dataset without additional manual annotation efforts. By treating the model's predictions as provisional labels, training can continue to progress, utilizing the generated data to refine the model's understanding. Pseudo-labeling is particularly useful in scenarios with limited annotated data, allowing continuous model improvement while mitigating the costs associated with extensive manual labeling.
Pixel-Wise means the analysis or processing of digital images and videos at the level of individual pixels, which are the smallest addressable elements in a digital image. This approach is important for tasks that require high granularity and precision, such as object detection, image segmentation, and action recognition in videos. Pixel-wise processing involves examining or manipulating each pixel independently based on its characteristics, such as color, intensity, or spatial relationships with neighboring pixels. In machine learning models, particularly those involved in computer vision, pixel-wise operations are fundamental for accurately understanding and interpreting the content of images and videos. For instance, in pixel-wise classification, each pixel in an image is classified into a category based on its features. This is commonly used in semantic segmentation, where the goal is to label each pixel of the image with a class corresponding to what the pixel represents (e.g., car, road, tree). This results in a detailed, pixel-level map of the image which is useful for autonomous vehicles, medical imaging, and other applications requiring precise visual details. In video action detection, pixel-wise analysis might involve determining the motion or change of each pixel over time to identify actions or events within the video. This could include detecting changes in pixel values that signify movement or applying models that predict the future state of a pixel based on its past appearances, contributing to more nuanced and detailed understanding of dynamic scenes. Pixel-wise methods are typically more computationally intensive than methods that process images or videos at higher levels of abstraction (e.g., patch-wise or image-wise), as they require operations to be carried out on potentially millions of pixels.
Sample Selection Approach means a method used in data analysis and machine learning to choose subsets of data (samples) from a larger dataset for processing, training, or analysis. This approach aims to optimize the training of models by carefully selecting the most representative or informative samples from a dataset, thus reducing computational overhead and improving learning efficiency. The selection criteria can vary depending on the specific goals and the nature of the data but typically involve measures of diversity, informativeness, or potential to improve model performance. In the context of machine learning, particularly in scenarios dealing with large and complex datasets such as images or videos, the sample selection approach involves analyzing the characteristics of each sample and determining its value to the model's ability to learn generalized features. For instance, in image recognition, samples that contain rare or underrepresented features might be prioritized to ensure that the model can accurately recognize these features when encountered in real-world scenarios. The sample selection approach can employ various strategies, such as random sampling, stratified sampling where the dataset is divided into homogeneous subgroups, or more complex methods like adaptive sampling which adjusts the selection criteria based on the model's performance as training progresses.
Scoring Mechanism means a method of evaluating frames or samples to assign a numerical score that indicates their priority for annotation within a dataset. This mechanism plays a important role in machine learning, particularly in tasks involving large volumes of data such as video analysis, where manually annotating every frame is impractical. By assigning scores that reflect the importance or informativeness of each frame, the scoring mechanism helps to streamline the annotation process, focusing resources on the most valuable data points. The scoring mechanism often integrates various selection metrics to form a comprehensive evaluation criteria. These metrics typically include proximity, which assesses how close a frame is to other important or already annotated frames, and uncertainty, which measures the confidence level of the model's predictions for a particular frame. High uncertainty indicates a low confidence in the prediction and suggests that annotating the frame could significantly improve the model's accuracy. Other metrics might include entropy, which quantifies the randomness or unpredictability of the prediction outcomes, and heuristic scores based on specific rules or conditions relevant to the task at hand. By combining these metrics, the scoring mechanism can prioritize frames that will add the most value to the training dataset, either by filling gaps in the model's understanding, enhancing diversity within the dataset, or correcting areas where the model currently performs poorly. This targeted approach not only optimizes the use of annotation resources but also accelerates the overall training process, leading to more robust and effective models.
Scribble Annotation means a sparse labeling technique where thin lines or regions are drawn over the object of interest within a frame. This method is part of semantic segmentation in computer vision, aimed at categorizing each pixel according to the object or region it represents. Scribble annotations are used to train machine learning models for object detection, segmentation, and tracking, providing a way to delineate objects with minimal input, thus reducing the resources needed for full pixel-wise labeling. In practice, users might draw lines on objects in an image or video frame using a mouse or stylus. These scribbles guide the model to learn object boundaries in complex scenes. The annotations do not need to cover the entire object but highlight key features or outlines to help the model segment the entire object in other, unannotated frames.
Selection Score means a value calculated by a selection model to prioritize frames or samples for annotation. The score is derived from various metrics, such as frame complexity, presence of key features, or predictive uncertainty. Higher scores indicate frames that will likely yield more training value, focusing annotation efforts on samples that improve the model's accuracy and robustness. The selection score helps in efficiently allocating resources by focusing on data that maximizes learning impact rather than uniformly distributing effort across potentially redundant information.
Sparse Annotation means a form of labeling where only a subset of the data is annotated, reducing costs while still providing sufficient information to train models. This approach targets the annotation of key frames or regions within a dataset, which are determined to be most informative for the learning process. Sparse annotation is particularly useful in large datasets or continuous video streams where full annotation is impractical. It allows for the effective training of models on less data by focusing on the most significant or representative samples.
Spatio-Temporal Deep Superpixel Method means a technique that identifies regions within video frames by grouping pixels based on visual similarity and temporal continuity. These regions, or superpixels, are analyzed over consecutive frames to ensure consistency and continuity in the features. The method facilitates efficient handling of video data by reducing the granularity of pixel-level processing while maintaining significant structural details. These regions are used to create pseudo-labels that improve training by providing intermediate structural information that bridges fully labeled and unlabeled data.
Spatio-Temporal Localization means the identification of the location and timing of an action or object within a video. This process involves detecting the action's spatial boundaries across consecutive frames and aligning this information with temporal data to understand when and where actions occur. Spatio-temporal localization is important for applications like video surveillance, sports analysis, and interaction recognition, where the timing and positioning of events are useful for accurate interpretation.
Spatio-Temporal Weighted Loss (STeW) Model means a model that applies different weights to frames based on their spatial-temporal proximity to annotations, ensuring the loss function prioritizes more accurate data. This weighting approach adjusts the impact of each data point based on its relevance and relationship to key annotated frames, enhancing the model's ability to focus on significant events and reducing the influence of less relevant data. It is useful in scenarios where annotations are sparse or unevenly distributed across the data.
Superpixel Segmentation means grouping visually similar pixels into larger regions based on color, texture, and other visual features. This technique reduces the complexity of image data by aggregating pixels into meaningful clusters, which simplifies further processing and analysis. In video analysis, 3D superpixels consider not only the spatial but also the temporal consistency of pixel clusters, thereby extending sparse annotations across frames and improving the generation of pseudo-labels for unannotated regions.
Temporal Aggregation means the process of combining multiple video frames to enhance the detection of actions or objects over time. This technique leverages the continuity and progression of visual information in videos to build a more comprehensive understanding of actions. Temporal aggregation captures the temporal context of actions, aiding in the recognition of patterns and behaviors that unfold over time, which is advantageous in dynamic scenes.
Temporal Domain means the aspect of the video that pertains to time, specifically the sequence and duration of frames. Managing diversity in the temporal domain involves selecting frames from different times or events within the video to capture a wide array of actions and interactions.
Temporal Segments refers to divisions within the video timeline that help identify and differentiate between various scenes or actions occurring at different times.
Threshold means to a predefined cutoff value that selection scores must exceed for frames to be selected for annotation. Frames with scores above this threshold are considered important for adding informational value to the model.
Tube Linking Methods means a set of techniques used in video analysis to connect detected actions or objects across successive frames, forming coherent trajectories or “tubes.” These methods are integral to action detection and tracking applications where understanding the continuity and progression of an action within a video is important. Tube linking involves analyzing the spatial and temporal properties of detected objects or actions in individual frames and then linking these detections over time to construct a continuous path that represents the movement and behavior of the object or action across the video. The primary challenge addressed by tube linking methods is maintaining the identity and accuracy of the tracked objects or actions in conditions where the appearance, speed, or trajectory may change due to various factors like occlusions, rapid movements, or shifts in perspective.
Two-Stage Approach means a detection method involving an initial step of object detection followed by action classification. This approach first localizes objects of interest within frames and then classifies the detected objects based on their actions. Providing a more accurate spatio-temporal understanding of actions in videos, this method is beneficial for applications requiring detailed recognition and categorization of complex activities.
Uncertainty Sampling means a technique in active learning where frames with high prediction uncertainty are selected for annotation. By focusing on data points where the model's confidence is low, this approach enables the model to learn from the most challenging data points. Uncertainty sampling is useful in refining model performance, especially in cases where the dataset is large and diverse, allowing for targeted improvement of the model's accuracy in ambiguous or difficult scenarios.
Weight means a factor assigned to each frame based on its importance or reliability, which influences how much that frame affects the overall loss calculation during model training.
The advantages set forth above, and those made apparent from the foregoing description, are efficiently attained. Since certain changes may be made in the above construction without departing from the scope of the invention, it is intended that all matters contained in the foregoing description or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
It is also to be understood that the following claims are intended to cover all of the generic and specific features of the invention herein described, and all statements of the scope of the invention that, as a matter of language, might be said to fall therebetween.
The present application is a continuation of U.S. Provisional Patent Application Ser. No. 63/514,482 entitled “Active Sparse Labeling System and Method,” filed Jul. 19, 2023.
This invention was made with Government support under Grant No. 2022-21102100001 awarded by the Office of the Director of National Intelligence (Intelligence Advanced Research Projects Activity). The government has certain rights in the invention.
| Number | Date | Country | |
|---|---|---|---|
| 63514482 | Jul 2023 | US |
