Low-latency Captioning System

Abstract

An artificial intelligence (AI) low-latency processing system is provided. The low-latency processing system includes a processor; and a memory having instructions stored thereon. The low-latency processing system is configured to collect a sequence of frames jointly including information dispersed among at least some frames in the sequence of frames, execute a timing neural network trained to identify an early subsequence of frames in the sequence of frames including at least a portion of the information indicative of the information, and execute a decoding neural network trained to decode the information from the portion of the information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify the smallest number of subframes from the beginning of a training sequence of frames containing a portion of training information sufficient to decode the training information.

Description

TECHNICAL FIELD

This invention relates generally to scene captioning and scene captioning based interaction more particularly to end-to-end scene captioning that utilizes a latency optimization function.

BACKGROUND

At any time instant, countless events that happen in the real world are captured by sensors such as cameras, microphones, LiDAR, mmWave and stored as massive sensor data resources. To effectively retrieve such recordings, whether in offline or online settings, scene captioning is an essential technology thanks to its ability to understand scenes and describe events in natural language. Furthermore, robots can interact with humans or other robots to decide next actions based on scene understanding through online scene captioning.

Scene captioning technology has been actively researched in the field of computer vision for video captioning. A deep recurrent neural network (RNN) has been applied to video captioning, where an RNN is trained to convert a sequence of image features extracted from a video clip to a sequence of words that describes the video content. Its goal is to generate a video description (caption) about objects and events in any video clip. Recently, a Transformer model is getting more popular than RNNs as a model for video captioning, since it improves the captioning performance.

Moreover, not only image features but also audio features have been extracted from video clips and utilized to improve the caption quality, where an attention-based multimodal fusion technique has been introduced to effectively fuse the image and audio features according to the video content [U.S. Pat. No. 10,417,498].

Prior methods for video captioning are basically assumed to work in an offline manner, where each video clip is given before captioning, and therefore the system can access all frames of the video clip to generate the caption. However, such prior methods are not practical in real-time monitoring, surveillance systems, scene-understanding based interaction systems for car navigation and robots, in which it is essential not only to describe events accurately but also to produce captions as soon as possible to find and report the events quickly to take next actions. Thus, low-latency captioning for online systems is required to realize such functionality, where the system needs to decide an appropriate timing to generate a correct caption using only the limited number of frames the system has received so far. Additionally, such a low-latency captioning function allow robots to decide next actions as soon as possible to interact with humans and other robots.

SUMMARY

Some embodiments of the present disclosure are based on recognition that scene captioning is an essential technology to understand scenes and describe events in natural language. To apply it to real-time monitoring systems, the systems require not only to describe events accurately but also to produce the captions as soon as possible. Low-latency captioning is needed to realize such functionality, but this research area for online scene captioning has not been pursued yet. This paper proposes a novel approach to optimize each caption's output timing based on a trade-off between latency and caption quality. An audio-visual Transformer is trained to generate ground-truth captions using only a small portion of all scenes, and to mimic outputs of a pre-trained Transformer to which all the scenes are given. A CNN-based timing detector is also trained to detect a proper output timing, where the captions generated by the two Transformers become sufficiently close to each other. With the jointly trained Transformer and timing detector, a caption can be generated in the early stages of an event, as soon as an event happens or when it can be forecasted.

An object of some embodiments of the invention is to provide a system and a method for end-to-end scene captioning capable for online/offline surveillance system and scene-aware interaction to understand the events using natural language as soon the system recognizes the events as possible.

This disclosure includes a low-latency scene captioning system, which is trained to optimize the output timing for each caption based on a trade-off between latency and caption quality. The present invention may train a low-latency caption generator according to the following strategy: (1) generate groundtruth captions using a low-latency caption generator that only sees a small portion of all scenes acquired by sensors or a small portion of all signals; (2) mimic outputs of a pre-trained caption generator, where the outputs are generated using the entire scene; and (3) train a timing detector that finds the best timing to output a caption, such that the caption ultimately generated by the low-latency caption generator becomes sufficiently close to its groundtruth caption or the caption generated by the pre-trained caption generator using the entire scene. The low-latency caption generator based on (1) and (2) and the timing detector of (3) are jointly trained.

The jointly trained low-latency caption generator and timing detector can generate captions in an early stage of scene acquired by sensors, as soon as an event happens. Additionally, this framework can be applied to forecast future events in low-latency captioning. Furthermore, by combining multimodal sensing information, an event can be recognized at an earlier timing triggered by the earliest cue in one of modalities without waiting for other cues in other modalities. In particular, an audio-visual Transformer built as a low-latency caption generator according to embodiments of the present invention can be used to generate captions earlier than a visual cue's timing based on the timing of an audio cue. Such a low-latency video captioning system using multimodal sensing information can contribute not only to retrieving events quickly but also to respond scenes earlier.

Some embodiments are based on recognition that experiments with the ActivityNet Captions dataset show that a system based on the present invention achieves 94% of the caption quality of the upper bound given by the pre-trained Transformer using the entire video clips, using only 28% of frames from the beginning.

According to some embodiments of the present invention, a scene captioning system can be provided. In this case, the scene captioning system may include an interface configured to acquire a stream of signals captured by multimodal sensors to caption scenes; a memory to store a computer-executable scene captioning model including a multimodal sensor feature extractor, a multimodal sensor feature encoder, a timing detector, and a scene caption decoder, wherein the multimodal sensor feature encoder is shared by the timing detector and the scene caption decoder; a processor, in connection with the memory, configured to perform steps of: extracting multimodal sensor features from the multimodal sensor signals by use of the multimodal sensor feature extractor; encoding multimodal sensor features by use of the multimodal sensor feature encoder; determining a timing of generating a scene caption by use of the timing detector, wherein the timing is arranged an early stage of the stream of multimodal sensor signals; and generating the scene caption describing events based on the multimodal sensor features by using the scene caption decoder according to the timing.

Further, some embodiments of the present invention are based on recognition that a computer-executable training method is provided for training a multimodal sensor feature encoder, a timing detector, and a scene caption decoder. The method may include providing a training dataset that includes a set of multimodal sensor signals and a set of groundtruth scene captions; converting the multimodal sensor signals to a feature vector sequence using a feature extractor; eliminating future frames from the feature vector sequence using a future frame eliminator, wherein the future frame eliminator takes first feature vectors from the feature vector sequence and removes other feature vectors to generates a future-eliminated feature vector sequence; encoding the future-eliminated feature vector sequence to a hidden activation vector sequence; training the low-latency scene caption generator by computing a loss value; computing a loss value based on a posterior probability distribution and the groundtruth captions; training the timing detector by computing a supervision signal based on a scene caption similarity between a scene caption for the future-eliminated feature vector sequence and the groundtruth caption.

Yet further, when the scene captioning system is configured as a video captioning system, the video captioning system may include an interface configured to acquire a stream of audio-visual signals including image and sound data; a memory to store a computer-executable video captioning model including an audio-visual feature extractor, an audio-visual feature encoder, a timing detector, and a video caption decoder, wherein the audio-visual encoder is shared by the timing decoder and the timing detector and the video caption decoder; a processor, in connection with the memory, configured to perform steps of: extracting audio and visual features from the audio-visual signals by use of the audio-visual extractor; encoding audio and visual features from the audio-visual signals by use of the audio-visual encoder; determining a timing of generating a video caption by use of the timing detector, wherein the timing is arranged an early stage of the stream audio-visual signals; and generating the video caption describing audio-visual scenes based on the audio and visual features by using the video caption decoder according to the timing.

According to some embodiments of the present invention, an artificial intelligence (AI) low-latency processing system is provided. The low-latency processing system may include a processor and a memory having instructions stored thereon that, when executed by the processor, cause the low-latency processing system to collect a sequence of frames jointly including information dispersed among at least some frames in the sequence of frames; execute a timing neural network trained to identify an early subsequence of frames in the sequence of frames including at least a portion of the information indicative of the information; execute a decoding neural network trained to decode the information from the portion of the information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify the smallest number of subframes from the beginning of a training sequence of frames containing a portion of training information sufficient to decode the training information.

Further, an embodiment of the present invention provides a computer-implemented method for an artificial intelligence (AI) low-latency processing system including a processor and a memory storing instructions of the computer-implemented method performing steps using the processor. The computer-implemented method may include the steps of collecting a sequence of frames jointly including information dispersed among at least some frames in the sequence of frames; executing a timing neural network trained to identify an early subsequence of frames in the sequence of frames including at least a portion of the information indicative of the information; executing a decoding neural network trained to decode the information from the portion of the information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify the smallest number of subframes from the beginning of a training sequence of frames containing a portion of training information sufficient to decode the training information.

BRIEF DESCRIPTION OF THE DRAWINGS

The presently disclosed embodiments will be further explained with reference to the attached drawings. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments.

FIG. 1 is a schematic diagram illustrating a process of low-latency video captioning, according to embodiments of the present invention;

FIG. 2 is a block diagram illustrating a low-latency video captioning system including a low-latency caption generator, a pre-trained caption generator, and a timing detector, according to embodiments of the present invention;

FIG. 3 is a block diagram illustrating a training procedure for a low-latency caption generator and a timing detector, which exploits a pre-trained caption generator to jointly optimize the low-latency caption generator and the timing detector, according to embodiments of the present invention;

FIG. 4 is a schematic diagram illustrating an audio-visual Transformer built as a low-latency caption generator combined with a timing detector for low-latency video caption, according to embodiments of the present invention; and

FIG. 5 is an evaluation result obtained by performing a low-latency video captioning test using an audio-visual Transformer built as a low-latency caption generator combined with a timing detector, according to embodiments of the present invention;

FIG. 6 illustrates the model architecture, which consists of a question encoder, an AV encoder, a timing detector, and a response decoder, according to embodiments of the present invention;

FIG. 7 illustrates a video stream, where the video has started at time T_s and ends at T_e, which are associated with ground-truth answer Y_e′, according to embodiments of the present invention;

FIG. 8 shows process steps performed by the low-latency processing system using a processor and a memory or memory storages storing instructions, according to an embodiment of the present invention;

FIG. 9 shows the relationship between latency ratio and answer accuracy on MSRVTT-QA, according to embodiments of the present invention;

FIG. 10 shows a comparison of the quality of answer sentences on the AVSD task, according to embodiments of the present invention; and

FIG. 11A and FIG. 11B show the distribution of the QAs over the latency with detection thresholds F=0.3 and F=0.4 on the AVSD-DSTC7 task, according to embodiments of the present invention.

While the above-identified drawings set forth presently disclosed embodiments, other embodiments are also contemplated, as noted in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation. Numerous other modifications and embodiments can be devised by those skilled in the art which fall within the scope and spirit of the principles of the presently disclosed embodiments.

DETAILED DESCRIPTION

The following description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

Specific details are given in the following description to provide a thorough understanding of the embodiments. However, understood by one of ordinary skill in the art can be that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the subject matter disclosed may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Further, like reference numbers and designations in the various drawings indicated like elements.

Also, individual embodiments may be described as a process, which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in a figure. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the function's termination can correspond to a return of the function to the calling function or the main function.

Furthermore, embodiments of the subject matter disclosed may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be executed, or at least assisted, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium. A processor(s) may perform the necessary tasks.

Modules and networks exemplified in the present disclosure may be computer programs, software or instruction codes, which can execute instructions using one or more processors. Modules and networks may be stored in one or more storage devices or otherwise stored into computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape, in which the computer readable media are accessible from the one or more processors to execute the instructions.

Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media may be RAM, ROM, EEPROM or flash memory, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both using one or more processors. Any such computer storage media may be part of the device or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media.

Low-Latency Captioning

According to some embodiments of the present disclosure, a scene captioning system can provide low-latency scene captioning to understand scenes and describe events in natural language from real-time monitoring of scenes or online/offline scene streaming.

In some cases, the scene captioning system can be configured as a video captioning system, and the video captioning system may be referred to as a low-latency video captioning method/system.

In the following, a low-latency video captioning method/system is provided as an example descriptions of the low-latency scene captioning method/system.

For instance, a stream of signals obtained by unimodal or multimodal sensors can be translated into captions using natural language. Unimodal is included by multimodal when a single sensor is applied. The multimodal sensors capture information such as visual image, audio signals, 3D-localization, thermography, Wi-Fi and so on. In some cases, the sensors may be image sensors, microphone, audio-visual signal sensors, LiDAR, mmWave, thermo-sensor, smell-sensor, tactile-sensor or any combinations thereof.

When the low-latency scene captioning method/system is applied to realtime or offline video streams, multimodal sensor signals may be a stream of audio-video signal, a scene encoder may be an audio-visual encoder, unimodal/multimodal sensor features may be audio and visual features. Further, multimodal sensor signals may be signals/information acquired by multimodal sensors. For instance, the signals can be any of sensing information such as video frames, sound data, 3-D localization data, thermo sensor data, smell sensor data, tactile sensor data or any combinations thereof.

FIG. 1 is a schematic diagram illustrating a process of low-latency video captioning, according to embodiments of the present invention. Suppose a video stream 101 is given as X, in which an event has occurred between time s and time T, where a groundtruth caption 102 as Y may be “One of workers is hit by a bulldozer” for the event. The caption is typically a sentence that explains the event in natural language. Video captioning systems based on prior art generate such a caption for a video clip using the entire frames x_s:T of the video stream X, where x_s:T denotes a sequence of video frames x_s, x_s+1, . . . , x_T-1, x_T, each of which corresponds to a video frame for each time index. Therefore, such prior video captioning systems are assumed to work only in an offline manner.

However, such prior methods are not practical in real-time monitoring or surveillance systems, in which it is essential not only to describe events accurately but also to produce captions as soon as possible. To this end, a low-latency video captioning module 110 needs to have a timing detector 103 that decides if the current time t is an appropriate timing to output a caption, using partial video frames 104, i.e., x_s:t the video frames from the beginning time s to the current time t. Only if the timing detector 103 detects the appropriate timing, a low-latency caption generator 105 generates a caption 106 based on the partial video frames 104. The beginning time s may be decided based on a timing the system generated the previous caption or a timing a certain change on pixel intensities in the video is detected (not shown in FIG. 1).

One remaining problem is that there is no sufficient dataset to train the timing detector 103 and the low-latency caption generator 105, where the dataset should be annotated with appropriately early timings and their captions for various videos. There exist datasets only for offline video captioning, i.e., the datasets were annotated with timings where the events have already finished in the video. In general, correcting a large amount of new dataset with such early timing annotations is very expensive.

To solve this problem, embodiments of the present invention train the timing detector 103 and the low-latency caption generator 105 using only datasets for offline video captioning, where the detector and the generator are optimized not only to describe events accurately but also to produce captions as soon as possible. Thus, low-latency video captioning is realized by detecting an appropriate timing to generate a correct caption using only the partial video frames 104 until the current time t. The system may generate a caption 106, which is equal to the groundtruth 102 “One of workers is hit by a bulldozer” even though it uses only partial video frames 104. On the other hand, prior methods that train the generator for offline video captioning may generate an incorrect caption 107 as “Workers are walking” for the partial video frames 104, because the generator is trained to generate a caption about an event occurred in each video clip. In the example of low-latency video captioning 100, the event “One of workers is hit by a bulldozer” has not yet occurred at time t. Therefore, it is difficult for systems based on prior art to generate the correct caption, while systems based on the present invention potentially generate the correct caption since the generator is trained to utilize some signs on future events in the partial video frames to generate correct captions.

Low-Latency Captioning Based Interaction System

FIG. 2 is a block diagram illustrating a low-latency video captioning system 200 including a low-latency caption generation training module 300 consisting of a traing data set 301, a feature extractor 302, a future frame eliminator 303, a pre-trained caption generator 320, a caption generation loss calculator 309, a caption similarity checker 310 and a timing detection loss calculator 311, a low-latency captioning module 110 consisting of a timing detector module 103 and a low latency caption generator 105, interaction module 201 consisting of a caption understanding module 202 and a response generator 203 and, according to embodiments of the present invention;

The low-latency video captioning system 200 includes a human machine interface (HMI) 210 connectable with a keyboard 211 and a pointing device/medium 212, one or more processor 220, a memory 240, a network interface controller 250 (NIC) connectable with a network 290 including local area networks and internet network, a display and/or speaker interface 260 connectable with a display and/or speaker device 261, a machine interface 262 connectable with a machine actuator 263, an multimodal sensor interface 271 such as an audio interface 273 and a visual interface 275 connectable with input devices including multimodal sensors 272 such as a microphone device 274 and a camera device 276 respectively, a printer interface 280 connectable with a printing device 285. The memory 240 may be one or more memory units. The low-latency video captioning-based interaction system 200 can receive multimodal sensing data 295 via the network 290 connected to the NIC 250. The storage device 230 includes a feature extractor 302, a low-latency caption generation training module 300, a low latency captioning module 110 and an interaction module 201. In some cases, the feature extractor 302 may be configured as a multimodal sensor feature extractor 302 when the system is configured as a scene captioning system. Further the feature extractor 302 may be configured as an audio-visual feature extractor 302 when the system is configured as a video captioning system.

For performing the low-latency captioning, instructions may be transmitted to the low-latency video captioning system 200 using the keyboard 211, the pointing device/medium 212 or via the network 290 connected to other computers (not shown in the figure). The system 200 receives instructions via the HMI 210 and executes the instructions for performing low-latency video captioning using the processor 220 in connection with the memory 240 by loading the low-latency captioning module 110.

The low-latency video captioning module 110 outputs a token sequence as captioning result for a given multimodal sensing feature sequence obtained by feature extractor 302, and sends the token sequence to the display/speaker device 265 via the display/speaker interface 260, the printer device 285 via the printer interface 280, or other computers (not shown in the figure) via the network 290. Each token in the token sequence may be a single word, a single letter, a single character, or a word piece in a text form.

Training Procedure for Low-Latency Video Captioning System

FIG. 3 is a block diagram illustrating a training module 300 included in a low-latency video captioning system 200 including a low latency captioning module 110, which exploits a pre-trained caption generator 320 to jointly optimize the low-latency caption generator 103 and the timing detector 105, according to embodiments of the present invention.

Given a training dataset 301 including a set of video clips X and a set of groundtruth captions Y, wherein each video clip X in X is associated with its groundtruth caption Y in Y, first a feature extractor 302 converts the video clip X to a feature vector sequence Z for the video clip X. Then, a future frame eliminator 303 eliminates future frames from Z to simulate a situation of low-latency video captioning, where the future frames are not available. The future frame eliminator 303 takes the first K feature vectors of Z, removes the rest, and generates a future-eliminated feature vector sequence Z′, where K may be determined randomly from 1≤K≤|Z| (|Z| denotes the length of Z).

The low-latency caption generator 105 may include an encoder 304, a decoder 305, and a search module 306, wherein the encoder 304 encodes the future-eliminated feature vector sequence Z′ to a hidden activation vector sequence H, the decoder 305 estimates a posterior probability distribution P (|H) over a random variable of captions for H, and the search module 306 searches for the best caption Ŷ such that

$\hat{Y}=\underset{y\in {𝒱}^{*}}{\arg \max }P\left(𝒴=y❘H\right),$

• • where * indicates a set of all possible word sequences based on a predefined vocabulary .

The timing detector 103 may include an encoder 307 and a timing detection module 308, wherein the encoder 307 encodes the future-eliminated feature vector sequence Z′ to a hidden activation vector sequence H′. The timing detector 103 may use the encoder 304 of the low-latency caption generator 105 instead of the encoder 307. In this case, the timing detector 103 may not have the encoder 307, and may receive the output of the encoder 304, i.e., H′ may be obtained as H′=H.

The timing detection module 308 estimates a detection probability distribution P (|H′) over a random variable indicating whether the current timing is appropriate or not based on H′. The random variable may take 0 or 1, where 0 indicates that the current timing is inappropriate while 1 indicates that the current timing is appropriate. In low-latency video captioning 100, the timing detector 103 may detect the appropriate timing when P(=1|H′)>F, where F is a pre-determined threshold such that 0≤F≤1.

To train the low-latency caption generator 105, a caption generation loss calculator 309 may compute a loss value based on the posterior probability distribution P(|H) and the groundtruth caption Y. The loss value may be computed as a cross-entropy loss;

_CE=−log P(=Y|H)

If the encoder 304 and the decoder 305 are designed as differentiable functions such as neural networks, the parameters of the neural networks can be trained to minimize the cross-entropy loss using the back-propagation algorithm. Minimizing the cross-entropy loss means letting the low-latency caption generator 105 generate appropriate captions as close as possible to the groundtruth captions. Further, when the system is configured as a scene captioning system, the encoder 304 may be configured as a multimodal sensor feature encoder 304. When the system is configured as video captioning system, the encoder 304 may be configured as an audio-visual feature encoder 304.

To train the timing detector 103, it needs a supervision signal on whether the current timing is appropriate or not. However, such kinds of signals are not included in the training dataset 301. Some embodiments of the present invention may have a caption similarity checker 310 to determine the appropriate timing based on the caption similarity between the generated caption Ÿ and the groundtruth caption Y. This is based on an idea that the appropriate timing is a timing when the low-latency caption generator can generate a caption Ŷ for the future-eliminated feature vector sequence Z′, which is sufficiently close to the groundtruth caption Y. The caption similarity may be computed based on any sequence similarity measures such as Word accuracy, BLEU score, and METEOR score defined between the word sequences of the two captions.

• • The supervision signal d may be computed as

$d=\{\begin{array}{cc}1& \mathrm{if}\mathrm{Sim}\left(\hat{Y},Y\right)>S,\\ 0& \mathrm{otherwise},\end{array}$

• • where Sim(Ŷ, Y) is a similarity measure between Ŷ and Y, and S is a predefined threshold such that 0≤S≤1.

Based on the supervision d and the detection probability distribution P(|H′) estimated by the timing detection module 308, a binary cross-entropy loss can be computed as

_BCE=−log P(=d|H′)

As well as the low-latency caption generator 105, the timing detector 103 can be trained using the back-propagation algorithm if the encoder 307 and the timing detection module 308 are designed as differentiable functions such as neural networks.

In some embodiments of the present invention, the training procedure 300 for a low-latency video captioning system 200 may also include a pre-trained caption generator 203 to improve the performance of the low-latency video captioning system 200. The pre-trained caption generator 203 may include an encoder 312, a decoder 313, and a search module 314, which have already been trained using the training dataset 301 to minimize the loss function

_CE=−log P(=Y|H″)

where H″ may be generated by the encoder 312 from the feature vector sequence Z, which are not eliminated. The generated caption Y′ obtained as

$Y^{\prime }=\underset{y\in {𝒱}^{*}}{\arg \max }P\left(𝒴=y❘H^{″}\right),$

is expected to be closer to the groundtruth caption Y than the low-latency caption Ŷ because the pre-trained caption generator 203 can use sufficient video frames (without future frame elimination) to generate captions unlike the low-latency caption generator 105.

Similarly, the estimated posterior probability distribution P(|H″) is expected to be better than P(|H) to generate correct captions.

With the pre-trained caption generator 203, the caption generation loss calculator 310 may use a Kullback-Leibler (KL) divergence loss as

${ℒ}_{KL}=-\sum _{y\in {𝒱}^{*}}P\left(𝒴=y❘H^{″}\right)\log P\left(𝒴=y❘H\right)$

where the loss value _KL is computed as a KL divergence that indicates the similarity between two probability distributions. Minimizing the KL divergence means training the low-latency caption generator 105 to mimic the pre-trained caption generator 203, where the low-latency caption generator 105 can use only the future-eliminated feature vector sequence Z′ while the pre-trained caption generator 203 can use the entire feature vector sequence Z. More specifically, the training is performed to let the low-latency caption generator 105 estimate the posterior probability distribution P(|H) as close as possible to the posterior probability distribution P(|H″) estimated by the pre-trained caption generator 203 using the entire video frames. This framework potentially avoids overfitting the parameters of the low-latency caption generator 105 to the groundtruth caption Y.

A linear combination of the cross-entropy loss _CE and the KL divergence loss _KL may also be used to train the low-latency caption generator 105 as

_conv=λ_CE+(1−λ)_KL

where λ is a pre-defined scaling factor to balance the two losses.

With the pre-trained caption generator 203, the caption similarity checker 310 may compute the supervision signal d as

$d=\{\begin{array}{cc}1& \mathrm{if}\max \left(\mathrm{Sim}\left(\hat{Y},Y\right),\mathrm{Sim}\left(Y^{\prime },Y\right)\right)>S,\\ 0& \mathrm{otherwise},\end{array}$

This mechanism encourages the timing detector 103 to detect the timing when the low-latency caption generator 105 can generate a caption not only close to the groundtruth caption Y but also to the caption Y′ generated by the pre-trained caption generator 203 for the entire feature vector sequence Z. This also avoids the supervision for the timing detector 103 relies only on the similarity to the groundtruth caption Y, which hopefully improves the robustness of the timing detector 103.

Some Embodiments Based on Audio-Visual Transformer

FIG. 4 is a schematic diagram illustrating an audio-visual Transformer 400 built as a low-latency caption generator combined with a timing detector for low-latency video captioning, according to embodiments of the present invention. The audio-visual Transformer 400 for low-latency video captioning includes a feature extractor 401, an encoder 402, a decoder 403, and a timing detection module 404.

Given a video stream 405, the feature extractor 401 extracts VGGish features 406 and I3D features 407 from the audio signal 408 and the visual signal 409 from the audio and visual tracks of the video stream 405, respectively, where the audio signal 408 may be an audio waveform and the visual signal 409 may be a sequence of images. The frame rate for feature extraction may be different on each track. The encoder 402 encodes the VGGish 406 and I3D 407 feature vectors, where the sequences of audio and visual features from a starting point to the current time are fed to the encoder 402 and converted to hidden activation vector sequences through self-attention layers 410, bi-modal attention layers 411, and feed-forward layers 412. Typically, this encoder block 413 is repeated N times, e.g., N=6 or greater. The final hidden activation vector sequences are obtained via the N-th encoder block.

Let X^A and X^V be audio signal 408 and visual signal 409. First, the feature extractor 401 is applied to the input signals as

A ⁰=VGGish(X^A),V⁰=I3D(X^V),

to obtain feature vector sequences corresponding to the VGGish 406 and I3D 407 features, respectively. Each encoder block 413 computes hidden vector sequences as

Ā ⁿ =A ^n-1 +MHA(A^n-1,A^n-1,A^n-1),

V ⁿ =V ^n-1 +MHA(V^n-1,V^n-1,V^n-1),

à ⁿ =Ā ⁿ +MHA(Āⁿ,Vⁿ,Vⁿ),

{tilde over (V)} ⁿ =V ⁿ +MHA(Vⁿ,Āⁿ,Āⁿ),

A ⁿ =Ã ⁿ +FFN(Ãⁿ),

V ⁿ ={tilde over (V)} ⁿ +FFN({tilde over (V)}ⁿ),

where MHA( ) and FFN( ) denote a multi-head attention and a feed-forward network, respectively.

MHA( ) takes three matrices query Q, key K, and value V, each of which has the length times feature dimensions as Q∈^Tq×dk, K∈^Tk×dk, V∈^Tv×dv, and returns a matrix as

$MHA\left(Q,K,V\right)=\mathrm{Concat}\left({\mathrm{head}}_1,\dots ,{\mathrm{head}}_h\right)W^O$ ${\mathrm{head}}_i=\mathrm{Atte}\mathrm{ntion}\left(Q{W}_i^Q,{\mathrm{KW}}_i^K,{\mathrm{VW}}_i^V\right)$ $\mathrm{Attention}\left(Q,K,V\right)=\mathrm{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

where w_i^Q ∈^dmodel×dk, W_i^K∈^dmodel×dk, W_i^V∈R^dmodel×dv and W^O ∈R^hdv×dmodel are parameter matrices of each multi-head attention layer, h is the number of heads, and d_model is the model size. Concat( ) concatenates matrices in the feature dimension.

The feed-forward network FFN( ) is computed as

FFN(x)=max(0,xW₁+b1)W₂+b₂,

where W₁∈^dk×dff and W₂∈^dff×dk are parameter matrices and b₁∈^dff and b₂∈^dk are parameter vectors of each feed-forward layer.

The self-attention layer 410 extracts temporal dependency within each modality, where the arguments for MHA( ) are all same, i.e., A^n-1 or V^n-1. The bi-modal attention layers 411 further extract cross-modal dependency between audio and visual features, where they take the key and the value from the other modality. After that, the feed-forward layers 412 are applied in a point-wise manner. The encoded representations for audio and visual features are obtained as A N and V N.

The timing-detection module 404 receives the encoded hidden vector sequences based on the audio-visual information available at the moment. The role of the timing-detection module 404 is to estimate whether the system should generate a caption or not for the given encoded features. The timing-detection module 404 first processes the encoded vector sequence from each modality with stacked 1D-convolution layers 414 as

A _c=Conv1D(A^N),V_c=Conv1D(V^N).

Each time-convoluted sequences are then summarized into a single vector through pooling 415 and concatenation 416 operations.

H=Concat(MeanPool(A_c),MeanPool(V_c)

Feed-forward layer FFN( ) 417 and sigmoid function σ( ) 418 convert the summary vector to the probability of , where ∈{0, 1} is a random variable indicating whether a relevant caption can be generated or not.

P(=1|X^A,X^V)=σ(FFN(H))

P(=O|X^A,X^V)=1−P(=1|X^A,X^V)

Once the timing-detection module 404 provides a higher probability than a threshold, e.g., P(d=1|X^A,X^V)>0.5, the decoder generates a caption based on the encoded hidden vector sequences (A^N, V^N).

The decoder 403 iteratively predicts the next word from a starting word (<sos>). At each iteration step, the decoder receives the previously generated words 419, and estimate the posterior probability distribution of the next word 420 by applying word embedding 421, M decoder blocks 422, a linear layer 423, and a softmax operation 424.

Let Y_i⁰ be partial caption <sos>, y₁, . . . , y_i after i iterations. Each decoder block 422 has self-attention 425, bi-modal source attention 426, concatenation 427, and feed-forward layer 428 as

Y _i ^m =Y _i ^m-1 +MHA(Y_i^m-1,Y_i^m-1,Y_i^m-1)

Y _i ^Am =Y _i ^m +MHA(Y_i^m,A^N,A^N)

Y _i ^Vm =Y _i ^m +MHA(Y_i^m,V^N,V^N)

{tilde over (Y)} _i ^m=Concat(Y_i^Am,Y_i^Vm)

Y _i ^m ={tilde over (Y)} _i ^m +FFN({tilde over (Y)}_i^m).

The self-attention layer 425 converts the word embedding vectors to high-level representations considering their temporal dependency. The bi-modal source attention layers 426 update the word representations based on the relevance to the encoded multi-modal hidden vector sequences. The feed-forward layer 428 is then applied to the outputs of the bi-modal attention layers after concatenation 427. Finally, the linear transform 423 and the softmax operation 424 are applied to the output of the M-th decoder block to obtain the posterior probability distribution of the next word 420 as

$P\left(y_{i+1}❘Y_i,A^N,V^N\right)=\mathrm{Softmax}\left(\mathrm{Linear}\left(Y_i^M\right)\right)$ ${\hat{y}}_{i+1}=\underset{y\in 𝒱}{\arg \max }P\left(y_{i+1}=y❘Y_i,A^N,V^N\right),$

where denotes the vocabulary.

After picking the one-best word ŷ_i+1, the partial caption is extended by adding the selected word to the previous partial caption as Y_i+1=Y_i, ŷ_i+1. This is a greedy search process that ends if ŷ_i+1=<eos>, which represents an end word. The posterior probability distribution of caption can be computed as

$P\left(𝒴❘H\right)=\prod _{i=0}^{L:{\hat{y}}_{L+1}=<eos>}P\left(y_{i+1}❘Y_i,A^N,V^N\right)$

where H=(A^N,V^N).

It is also possible to pick multiple words with highest probabilities and consider multiple candidates of captions according to the beam search technique in the search module 306.

Training

The multi-modal encoder, the timing detector, and the caption decoder are jointly trained, so that the model achieves a caption quality comparable to that for a complete video, even if the given video is shorter than the original one by truncating the later part.

• • The training process for model θ=(θ_C, θ_D) repeats the following steps:
  • Sample T_o˜Uniform(T_s, T_e),
  • Compute loss =α_CE+β_KL+γ_D,
  • Update θ using ∇_θ
  • Inference
  • The inference is performed in two steps:
  • Find t that first satisfies P(=1|X_s:t; θ_D)>F,
  • Generate a caption based on

$\hat{Y}=\underset{Y\in {𝒱}^{*}}{\arg \max }P\left(𝒴=Y❘X_{s:t};{\theta }_C\right),$

Evaluation for Low-Latency Video Captioning Quality

FIG. 5 is an evaluation result obtained by performing a video captioning bench marks, according to embodiments of the present invention;

The proposed low-latency caption generation was tested using the ActivityNet Captions dataset Krishna et al. (2017), which consists of 100 k caption sentences associated with temporal localization information based on 20 k YouTube videos. The dataset is split into 50%, 25%, and 25% for training, validation, and testing. The validation set was split into two subsets on which the performance was reported. The average duration of a video clip is 35.5, 37.7, and 40.2 seconds for the training set and the validation subsets 1 and 2, respectively. VGGish and I3D features were used. The VGGish features were configured to form a 128-dimensional vector sequence for the audio track of each video, where each audio frame corresponds to a 0.96 s segment without overlap. The I3D features were configured to form a 1024-dimensional vector sequence for the video track, where each visual frame corresponds to a 2.56 s segment without overlap.

A multi-modal Transformer was first trained with entire video clips and their ground-truth captions. This model was used as a baseline and teacher model. N=2 was used for encoder blocks and M=2 for decoder blocks, and the number of attention heads was 4. The vocabulary size was 10,172, and the dimension of word embedding vectors was 300.

The proposed model for online captioning was trained with incomplete video clips according to the steps in the invention. In the training process, α=β=γ=⅓ was used for the loss function. The dimensions of hidden activations in audio and visual attention layers were 128 and 1024, respectively. The timing detector had 2 stacked 1D-convolution layers, with a ReLU non-linearity in between. The performance was measured by BLEU3, BLEU4, and METEOR scores.

The latency ratio indicates the ratio of the video duration used for captioning to the duration of the original video clip. With the baseline model, the latency ratio is always 1, which means all frames are used to generate captions.

FIG. 5 compares captioning methods in METEOR scores on validation subset 1. The model selected for evaluation was trained with S=0.6 (empirically decided) and had the best METEOR score on validation subset 2. The latency with the detection threshold F was controlled. As shown in FIG. 5, the proposed method at a 55% latency achieves 10.45 METEOR score with only a small degradation, which corresponds to 98% of the baseline score 10.67. It also achieves 10.00 METEOR score at a 28% latency, which corresponds to 94% of the baseline. A naive method was tested which takes video frames from the beginning with a fixed ratio to the original video length and runs the baseline captioning on the truncated video clip. The results show that the proposed approach clearly outperforms the naive method at an equivalent latency.

The table also includes the results using a unimodal Transformer that receives only the visual feature. The results show that the proposed method works for the visual feature only, but the performance is degraded due to the lack of the audio feature. This result indicates that the audio feature is essential even in the proposed low-latency method.

A low-latency caption is input into an interaction system trained using a pair of captions and action commands to understand scenes and decide next actions. The interaction system may use a sequence-to-sequence neural network model that converts a received caption to a sequence of words for responding to a human or a sequence of action commands to control robots.

Further, some embodiments are based on recognition that the scene-aware interaction technologies allow machines to interact with humans based on shared knowledge obtained through recognizing and understanding their surroundings using various kinds of sensors. Audio-visual scene-aware dialog (AVSD) is one of the scene-aware interaction technologies. According to some recognition based on AVSD, the end-to-end approaches can be better handle flexible conversations between a user and a system by training models on large conversational data sets. Such approaches have been extended to carry out conversations about objects and events taking place around the machines or the users, based on understanding of the dynamic scenes captured by multimodal sensors such as a video camera and a microphone. This extension allows users to ask questions about what is happening in their surroundings. For example, this framework is applicable to visual question answering (VQA) for one-shot QA about a static image, Visual dialog in which an AI agent holds a meaningful dialog with humans about a static image using natural conversational language, Video QA for one-shot QA about a video clip, and AVSD that performs a QA-based dialog with humans about a video clip. Video QA is a QA (question answering) task which consists in answering a single question about video clips such as those found on YouTube, and was formalized within the MSVD (Microsoft Research Video Description Corpus) and MSRVTT (Microsoft Research Video to Text) datasets. AVSD is a multi-turn dialog task which consists in generating responses to a user's questions about daily-life video clips, and has been addressed in the Dialog Systems Technology Challenges (DSTC). AVSD systems are trained to generate sentences to answer user's questions about video clips, where the systems need to understand what, when, how, and by whom events took place based on a time series of audio-visual features to provide correct answers.

At any time instant, countless events that happen in the real world are captured by cameras and stored as massive video data resources. To effectively retrieve such recordings, whether in offline or online settings, since the S2VT (Sequence to Sequence—Video to Text) system was first proposed, video captioning has been actively researched in the field of computer vision using sequence-to-sequence models in an end-to-end manner. Its goal is to generate a video description (caption) about objects and events in a video clip. To further leverage audio features to identify events, the multimodal attention approach allows to fuse audio and visual features such as VGGish (Visual Geometry Group (VGG)-like audio classification model) and I3D (Interactive Three Dimensional) to generate video captions. Such video clip captioning technologies have been expanded to offline video stream captioning technologies such as dense video captioning and progressive video description generator, where all salient events in a video stream are temporally localized, and event-triggered captions are generated in a multi-thread manner. While all video captioning technologies had so far been based on LSTM, the Transformer can be successfully applied together with the audio-visual attention framework.

Current dialog systems for video QA typically process offline video data to generate answers to questions after the end of a clip. In order for an interaction system to enable real-time conversations about scenes for surveillance and robot applications, it needs to understand scenes and events and respond to user queries about the scenes promptly. In that work, the audio-visual Transformer was tested using the ActivityNet Captions dataset within an offline video captioning system and achieved the best performance for the dense video captioning task.

However, such scene-aware interaction tasks assume that the system can access all frames of the video clip when predicting an answer to a question. This assumption is not practical for real-time dialog systems that are monitoring an ongoing audio-visual scene, where it is essential not only to predict an answer accurately but also to respond to the user as soon as possible by finding the question-related events quickly in the online video stream and generating an appropriate answer. Such functionality requires the development of new low-latency QA techniques.

In previous work, we proposed a low-latency audio-visual captioning method, which can describe events accurately and quickly without waiting for the end of video clips, optimizing the timing for captioning. In parallel, we recently introduced a new AVSD task for the third AVSD challenge at DSTC10, which notably asks systems to demonstrate temporal reasoning by finding evidence from the video to support their answers. This was based on a new extension of the AVSD dataset for DSTC10, for which we collected human-generated temporal reasoning data. This work proposes to extend our low-latency captioning approach to the scene-aware interaction task, combining it with our reasoning AVSD system to develop a low-latency online scene-aware interaction system.

Some extensions to the work on low-latency video captioning can build a novel approach that can optimize the timing to generate each answer under a trade-off between latency of generation and quality of answer. For video QA, the timing detector is now in charge of finding a timing for the question-relevant event, instead of determining when the system has seen enough to generate a general caption as in the video captioning case. The audio visual scene-aware dialog system built for the 10th Dialog System Technology Challenge can be extended to exploit a low-latency function. For instance, the experiments with the MSRVTT-QA and AVSD datasets show that our approach achieves between 97% and 99% of the answer quality of the upper bound given by a pre-trained Transformer using the entire video clips, using less than 40% of frames from the beginning.

In the same spirit as the low-latency captioning approach, another system according to another embodiment optimizes the output timing for each answer based on a trade-off between latency and answer quality. We train a low-latency audio-visual Transformer composed of (1) a Transformer-based response generator which tries to generate the ground-truth answer after only seeing a small portion of all video frames, and also to mimic the outputs of a similar pre-trained response generator that is allowed to see the entire video, and (2) a CNN-based timing detector that can find the best timing to output an answer for the input question, such that the responses ultimately generated by the above two Transformers become sufficiently close to each other. The proposed jointly-trained response generator and timing detector can generate answer responses in an early stage of a video clip, as soon as a relevant event happens, and may even forecast future frames. Thanks to the combination of information from multiple modalities, the system has more opportunities to recognize an event at an earlier timing by relying on the earliest cue in one of the modalities. Experiments with the MSR-VTT QA and AVSD datasets show that our approach achieves low-latency video QA with competitive answer quality to an offline video QA baseline that utilizes the entire video frames.

According to an embodiment of the present invention, video QA, which is different from video captioning is provided to generate appropriate answers for a user's questions as soon as possible. To this end, we introduce a question encoder to provide question embeddings to the timing detector, and extend the sentence generator to accept a question as contextual information. Thus, while the proposed method utilizes the same underlying mechanism for low-latency processing, the model is extended for video QA tasks.

Low-Latency Video QA Model

We build our proposed model for low-latency video QA upon the DSTC10-AVSD system, which employs an AV-transformer architecture. For the DSTC10-AVSD challenge, we extended the AV-transformer with joint student-teacher learning and attentional multimodal fusion to achieve state-of-the-art performance. As in our low-latency video captioning system, the low-latency QA model receives video and audio features in a streaming manner, and a timing detector decides when to generate an answer response for the feature sequence the model has received until that moment. FIG. 6 illustrates a model architecture (low-latency video QA architecture) 600 for low-latency video QA according to some embodiments of the present invention. The model architecture 600 includes function parts similar to that used in the audio-visual Transformer 400. Identical numbers are used in the figure for the parts (or layers) corresponding to that used in the audio-visual Transformer 400. The low-latency video QA architecture 600 further includes a word embedding 621, a question encoder (text encoder) 613 having a self-attention layer 610 and a feed-forward layer 612, a pooling 615 connected to the contact 416, an AV encoder 402, a timing detector 404, and a response decoder 403, where the AV encoder 402 is shared by the timing detector 404 and the response decoder 403.

Further, the mathematical expressions or equations used to explain the audio-visual Transformer 400 are used in the following descriptions to explain the low-latency video QA architecture 600.

Given a video stream and a question text as inputs, the AV encoder encodes VGGish and I3D features extracted from the audio and video tracks, while a Transformer-based text encoder encodes the question. The sequences of audio and visual features from a starting point to the current time are fed to the encoder, and converted to hidden vector sequences through self-attention, bi-modal attention, and feed-forward layers. This encoder block is repeated N times, and the final encoded representation is obtained via the N-th encoder block. The question word sequence is also encoded via a word embedding layer followed by a Transformer with N′ blocks.

We denote by X^A and X^V the audio and visual signals, and by Q the question sentence. First, the feature extraction module is applied to the input signals as

A ⁰=VGGish(X^A),V⁰=I3D(X^V),  (1)

to obtain feature vector sequences corresponding to the VGGish and I3D features, respectively. Each encoder block computes hidden vector sequences as

Ā ⁿ =A ^n-1 +MHA(A^n-1,A^n-1,A^n-1),  (2)

V ⁿ =V ^n-1 +MHA(V^n-1,V^n-1,V^n-1),  (3)

à ⁿ =Ā ⁿ +MHA(Āⁿ,Vⁿ,Vⁿ),  (4)

{tilde over (V)} ⁿ =V ⁿ +MHA(Vⁿ,Āⁿ,Āⁿ),  (5)

A ⁿ =Ã ⁿ +FFN(Ãⁿ),  (6)

V ⁿ ={tilde over (V)} ⁿ +FFN({tilde over (V)}ⁿ),  (7)

where MHA and FFN denote multi-head attention and feed-forward network, respectively. Layer normalization is applied before every MHA and FFN layers, but it is omitted from the equations for simplicity. MHA takes three arguments, query, key, and value vector sequences. The self-attention layer extracts temporal dependency within each modality, where the arguments for MHA are all the same, i.e., A^n-1 or V^n-1, as in (2) and (3). The bi-modal attention layers further extract cross-modal dependencies between audio and visual features, taking the keys and values from the other modality as in (4) and (5). The feed-forward layers are then applied in a point-wise manner. The encoded representations for audio and visual features are obtained as A^N and V^N. Question Q is also encoded using a

Transformer with N′ blocks as

Q ⁰=WordEmbedding(Q),  (8)

Q ^n′ =Q ^n′−1 +MHA(Q^n′−1,Q^n′−1,Q^n′−1),  (9)

Q ^n′ =Q ^n′ +FFN(Q^n′).  (10)

With the AV encoder and the question encoder, we obtain encoded representations A and V of the audio-visual features, and Q of the question word sequence Q. The timing detector receives the encoded representations A and V available up to the current time and the encoding Q of the input question. The role of the timing detector is to decide whether the system should generate an answer or not for the given encoded features and the question. The detector first processes the encoded vector sequence from each modality with stacked 1D-convolution layers (Conv1d) as

A′=Conv1d(A),V′=Conv1d(V).  (11)

These time-convoluted sequences and the question encoding are then summarized into a single vector through mean pooling and concatenation operations:

=Concat(Mean(A′),Mean(V′),Mean(Q))  (12)

A feed-forward layer FFN and sigmoid function a convert the summary vector to the probability of d, where d indicates whether a relevant answer can be generated or not:

P(d=1|A,V,Q)=σ(FFN()).  (13)

Once the timing detector outputs a probability higher than a threshold, e.g., P(d=1|A, V, Q)>0.5, the decoder generates an answer based on the encoded representations.

The decoder iteratively predicts the next word, starting from question word sequence Q plus a starting token <sos>. At each iteration step, it receives the partial answer sentence that has already been generated, and predicts the next word by applying M decoder blocks and a prediction network. Let Y_i be a partial answer sentence <sos>, y₁, . . . , y_i after i iterations and y_i⁰ be its question-conditioned sequence Q^0Y_i⁰, where each word in the question and the partial answer sentence is converted to a word embedding vector. Each decoder block has self-attention, bi-modal source attention, and feed-forward layers:

_i ^m=_i^m-1+MHA(_i^m-1,_i^m-1,_i^m-1),  (14)

_i ^Am=_i^m+MHA(_i^m,A,A),  (15)

_i ^Vm=_i^m+MHA(_i^m,V,V),  (16)

_i ^m=Concat(_i^Am,_i^Vm),  (17)

_i ^m=_i^m+FFN(_i^m),  (18)

where MHA denotes a multi-head attention network. The self-attention layer maps the word embedding vectors to high-level representations that contain their temporal dependency in (14). The bi-modal attention layers update the word representations based on the relevance to the encoded bi-modal representations in (15) and (16). The feed-forward layer converts the concatenated outputs of the bi-modal attention layers in (17) and (18). These operations are repeated to the M-th block. The linear transform and the softmax operation are applied to the M-th output to obtain the probability distribution of the next word as

$\begin{array}{cc}P\left(y_{i+1}|\overline{Q}{Y}_i,A,V\right)=\mathrm{Softmax}(\mathrm{Linear}\left(y_i^M\right)),& \left(19\right)\end{array}$ $\begin{array}{cc}{\hat{y}}_{i+1}=\underset{y\in 𝒱}{\arg \max }P\left(y_{i+1}=y❘\overline{Q}{Y}_i,A,V\right),& \left(20\right)\end{array}$

where denotes the vocabulary. The partial answer is extended by appending the best word ŷ_i+1 to the previous partial sentence as Y_i+1=Y_i, _i+1. This is a greedy search process repeated until it receives an end-of-sentence token <eos> as ŷ_i+1. We can also use a beam search technique, which selects top-K words with highest probabilities and keeps multiple word sequences. Finally, the best word sequence is selected as the final answer for the input question.

Training and Inference

We jointly train the AV encoder, the question encoder, the response decoder, and the timing detector, so that the system achieves an answer quality comparable to that for a complete video, even if the given video is shorter than the original one by truncating the later part.

We utilize two loss functions, a response loss to improve the answer quality and a timing detection loss to detect a proper timing to emit an answer sentence. FIG. 7 illustrates a video stream, where the video has started at time T_s and ends at T_e, which are associated with ground-truth answer Y_e′. If time T_o is picked as the emission timing, the response decoder generates an answer based on the audio-visual signals X_Ts:To=(A_Ts:To, V_Ts:To) from time T_s to time T_o, and the question Q.

The response loss is based on a standard cross-entropy loss for the ground-truth answer Y_e′.

_CE=−log P(Y_e′|X_Ts:To,Q;θ_R),  (21)

and a Kullbackâ “Leibler (KL) divergence loss between predictions from a pre-trained model allowed to process the complete video and the target model that can only process incomplete videos, i.e.,

$\begin{array}{cc}{ℒ}_{\mathrm{KL}}=-\sum _{i=1}^{❘Y_{e^{\prime }}❘}\sum _{y\in 𝒱}P\left(y❘Y_{e,i}^{\prime },X_{T_s:T_e},\stackrel{\_}{Q};{\stackrel{\_}{\theta }}_R\right)\log P\left(y❘Y_{e,i}^{\prime },X_{T_s:T_e},\stackrel{\_}{Q};{\stackrel{\_}{\theta }}_R\right).& \left(22\right)\end{array}$

This student-teacher learning approach can exploit the superior description power of the teacher model θ_R, which predicts an answer using entire video clip X_Ts:Te, by pushing the student model θ_R to mimic the teacher's predictions using only the truncated video clip X_Ts:To. This improves the training stability and leads to better performance.

The timing detection loss is based on a binary cross-entropy for appropriate timings. In general, however, such timing information does not exist in the training data set. As in [?], we decide the right timing based on whether or not the response decoder can generate an appropriate answer, that is, an answer sufficiently close to the ground-truth Y_e′ or the answer Ŷ_e generated for the entire video clip X_Ts:Te using the pre-trained model θ_R. The detection loss is computed as

_D=−log P(d|X_Ts:To,Q;θ_D),  (23)

where d is determined based on

$\begin{array}{cc}d=(\begin{array}{cc}1& \mathrm{if}\max \left(\mathrm{Sim}\left(Y_{e^{\prime }},{\hat{Y}}_o\right),\mathrm{Sim}\left({\hat{Y}}_e,{\hat{Y}}_o\right)\right)\ge S,\\ 0& \mathrm{otherwise},\end{array}& \left(24\right)\end{array}$

(23) where Sim(⋅, ⋅) denotes a similarity measure between two word sequences. In this work, we use the matched word percentage computed in a teacher-forcing manner, where we obtain Ŷ_e and Ŷ_o, as sequences of highest-probability words given the ground-truth word sequence as the left context, and count the matched words between them. S∈(0,1] is a pre-determined threshold which judges whether or not the online answer Ŷ_o is sufficiently close to the references Y_e′ and Ŷ_e.

The model θ=(θ_R, θ_D) is trained by repeating the steps of sampling the emission timing T_o˜Uniform(T_s, T_e), computing the loss =α_CE+β_KL+γ_D, and updating the parameters θ using ∇_θ.

At inference time, the emission time {circumflex over (T)}_o is determined as the first time that meets P(d=1|X_{Ts:{circumflex over (T)}o}, Q; θ_D)>F, where F is a pre-determined threshold controlling the sensitivity of timing detection. An answer is then generated based on

$\begin{array}{cc}{\hat{Y}}_o=\arg \underset{Y\in {𝒱}^{*}}{\max }P\left(Y❘X_{T_S:{\hat{T}}_o},\overline{Q};{\theta }_R\right).& \left(25\right)\end{array}$

Note that we assume that T_s is already determined.

According to some embodiments of the present invention, an artificial intelligence (AI) low-latency processing system is provided. FIG. 8 shows a computer-implemented method 800 including process steps performed by the low-latency processing system using a processor and a memory or memory storages, according to an embodiment of the present invention.

The low-latency processing system may include a processor and a memory having instructions as a computer-implemented method 800 stored thereon that, when executed by the processor, cause the low-latency processing system to collect a sequence of frames jointly including information dispersed among at least some frames in the sequence of frames; execute a timing neural network trained to identify an early subsequence of frames in the sequence of frames including at least a portion of the information indicative of the information; execute a decoding neural network trained to decode the information from the portion of the information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify the smallest number of subframes from the beginning of a training sequence of frames containing a portion of training information sufficient to decode the training information.

In some cases, the timing detector neural network is jointly trained with the decoder neural network on features of different subsequences of the sequence of frames to minimize a multi-task loss function including a time detection loss and an information generation loss. The multi-task loss function may include three losses defining (1) an accuracy of decoded information, (2) a difference between the information decoded from the subsequence of frames and the information decoded from the full sequence of frames, and (3) an accuracy of prediction of the timing detector neural network. Further, the processor may be configured to execute a feature extractor neural network to extract features from each frame in the sequence of frames; execute a feature encoder neural network to encode the extracted features of each frame to produce a sequence of encoded features; submit the sequence of encoded features to the timing detector neural network and to identify a subsequence of encoded features representing the subsequence of frames; and submit the subsequence of encoded features to the decoder neural network to decode the information. For the other cases, the processor may trigger the execution of modules of the AI low-latency processing system upon receiving a new input frame appended to the sequence of frames. The information can be a caption for an audio scene, a video scene, or an audio-video scene. Also, the information is an answer to a question about the sequence of frames. In some cases, the frames may include multi-model information coining from different sensors of different modalities. Further, the information may be an answer to a question about the sequence of frames, wherein the processor executes a text encoder neural network to encode the question, submits the encoded question to the timing neural network, and submits the question or the encoded question to the decoding neural network.

Experiments

We evaluate our low-latency video QA method using the MSRVTT-QA and AVSD datasets. MSRVTT-QA is based on the MSR-VTT dataset, which contains 10 k video clips and 243 k question-answer (QA) pairs. The QA pairs are generated automatically from the manually-annotated captions for each video clip, where the question is a sentence and the answer is a single word. We follow the data split in the MSR-VTT dataset which is 65% for training, 5% for validation, and 30% for test. AVSD is a set of text-based dialogs on short videos from the Charades dataset, which consists of untrimmed multi-action videos, which each include an audio track. In AVSD, two parties, dubbed questioner and answerer, have a dialog about events in the provided video. The job of the answerer, who has already watched the video, is to answer questions asked by the questioner. We follow the AVSD challenge setting, where the train, validation, and test sets consist of 7.7 k, 1.8 k, and 1.8 k dialogs, respectively, and each dialog includes 10-turn QA pairs, where both questions and answers are sentences. The duration of video clips ranges from 10 to 40 seconds.

The VGGish features were configured to form a 128-dimensional vector sequence for the audio track of each video, where each audio frame corresponds to a 0.96 s segment without overlap. The I3D features were configured to form a 2048-dimensional vector sequence for the video track, where each visual frame corresponds to a 2.56 s segment without overlap.

We first trained a multi-modal Transformer with entire video clips and QA pairs. This model was used as a baseline and teacher model. We used N=2 audio-visual encoder blocks, N′=4 question encoder blocks, M=4 decoder blocks, and set the number of attention heads to 4. The vocabulary size was 7,599 for MSRVTT-QA and 3,669 for AVSD. The dimension of the word embedding vectors was 300.

The proposed model for low-latency video QA was trained with incomplete video clips according to the steps in Section 3.2. The architecture was the same as the baseline/teacher model except for the addition of the timing detector. In the training process, we consistently used α=β=γ=⅓ for the loss function and threshold S=0.9 in Eq. (24). We set the dimensions of the hidden activations in the audio and visual attention layers to 256 and 1024, respectively, the dropout rate to 0.1, and we applied a label smoothing technique. The timing detector consisted of 2 stacked 1D-convolution layers with a ReLU non-linearity in between. The performance was measured by answer accuracy for MSRVTT-QA, and BLEU4 and METEOR scores for AVSD.

FIG. 9 shows the relationship between latency ratio and answer accuracy on MSRVTT-QA. The latency ratio denotes the ratio of actually used frames (from the beginning) to the entire video frames. The baseline results were obtained with the baseline (teacher) model by simply omitting the future frames at various ratios. Results for the proposed models were obtained by changing the detection threshold F. The accuracy for MSRVTT-QA shows a percentage of one-word answers matching with the ground-truths. The result demonstrates that our proposed method achieves low-latency video QA with much smaller accuracy degradation compared to the baseline. Our approach achieves 97% of the answer quality of the upper bound given by the pre-trained Transformer using the entire video clips, using only 40% of frames from the beginning.

FIG. 10 shows a comparison of the quality of answer sentences on the AVSD task. We controlled the latency ratio by setting the detection threshold F to obtain average ratios of 1.0, 0.5, and 0.2 for the proposed method. We omitted the future frames with the above fixed ratios for the baseline system. As shown in the table, our proposed method slightly outperforms the baseline even with 1.0 latency. This could be due to the increased robustness of the model by training with randomly truncated video. Moreover, the proposed method keeps the same level of BLEU4 and METEOR scores at the 0.5 latency and achieves competitive scores even at the 0.2 latency with little degradation, reaching 98% to 99% of the scores at the 1.0 latency condition.

FIG. 11A and FIG. 11B show the distribution of the QAs over the latency with detection thresholds F=0.3 and F=0.4 on the AVSD-DSTC7 task, which correspond to the results for latency 0.2 (left) and 0.5 (right) in FIG. 10. These results illustrate that most of the QAs of AVSD require either only the early frames or all frames to generate accurate answers. We investigated the reason for the polarized distribution. The most frequent pattern for questions leading to an early decision is “How does the video starts?”. Furthermore, there are some consistent answers such as “one person” in response to “How many people are in the videos?” in the training data. Such frequent linguistic patterns could also be a cause for the early decision. The late decision case contains such patterns as “How does the video ends?”. Such a question is natural for questioners who need to generate video captions through 10 QAs without watching the full videos.

According to some embodiments described above, a low-latency video QA method can answer a user's questions accurately and quickly without waiting for the end of video clips. The proposed method optimizes each answer's output timing based on a trade-off between latency and answer quality. The system described above can generate answers in early stages of video clips using the MSRVTT-QA and AVSD datasets, achieving between 97% and 99% of the answer quality of the upper bound given by a pre-trained Transformer using the entire video clips, using less than 40% of frames from the beginning.

Claims

1. An artificial intelligence (AI) low-latency processing system, the low-latency processing system comprising: a processor; and a memory having instructions stored thereon that, when executed by the processor, cause the low-latency processing system to: collect a sequence of frames jointly including information dispersed among at least some frames in the sequence of frames;execute a timing neural network trained to identify an early subsequence of frames in the sequence of frames including at least a portion of the information indicative of the information;execute a decoding neural network trained to decode the information from the portion of the information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify the smallest number of subframes from the beginning of a training sequence of frames containing a portion of training information sufficient to decode the training information.

2. The AI low-latency processing system of claim 1, wherein the timing detector neural network is jointly trained with the decoder neural network on features of different subsequences of the sequence of frames to minimize a multi-task loss function including a time detection loss and an information generation loss.

3. The AI low-latency processing system of claim 2, wherein the multi-task loss function includes three losses defining (1) an accuracy of decoded information, (2) a difference between the information decoded from the subsequence of frames and the information decoded from the full sequence of frames, and (3) an accuracy of prediction of the timing detector neural network.

4. The AI low-latency processing system of claim 1, wherein the processor is configured to execute a feature extractor neural network to extract features from each frame in the sequence of frames; execute a feature encoder neural network to encode the extracted features of each frame to produce a sequence of encoded features;submit the sequence of encoded features to the timing detector neural network and to identify a subsequence of encoded features representing the subsequence of frames; andsubmit the subsequence of encoded features to the decoder neural network to decode the information.

5. The AI low-latency processing system of claim 1, wherein the processor triggers the execution of modules of the AI low-latency processing system upon receiving a new input frame appended to the sequence of frames.

6. The AI low-latency processing system of claim 1, wherein the information is a caption for an audio scene, a video scene, or an audio-video scene.

7. The AI low-latency processing system of claim 1, wherein the information is an answer to a question about the sequence of frames.

8. The AI low-latency processing system of claim 4, wherein the information is an answer to a question about the sequence of frames, wherein the processor is configured to: execute a text encoder neural network to encode the question;submit the encoded question to the timing neural network; andsubmit the question or the encoded question to the decoding neural network.

9. The AI low-latency processing system of claim 1, wherein the frames include multi-model information coining from different sensors of different modalities.

10. A computer-implemented method for an artificial intelligence (AI) low-latency processing system including a processor and a memory storing instructions of the computer-implemented method performing steps using the processor, the steps comprising: collecting a sequence of frames jointly including information dispersed among at least some frames in the sequence of frames;executing a timing neural network trained to identify an early subsequence of frames in the sequence of frames including at least a portion of the information indicative of the information; andexecuting a decoding neural network trained to decode the information from the portion of the information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify the smallest number of subframes from the beginning of a training sequence of frames containing a portion of training information sufficient to decode the training information.

11. The computer-implemented method of claim 10, wherein the timing detector neural network is jointly trained with the decoder neural network on features of different subsequences of the sequence of frames to minimize a multi-task loss function including a time detection loss and an information generation loss.

12. The computer-implemented method of claim 11, wherein the multi-task loss function includes three losses defining (1) an accuracy of decoded information, (2) a difference between the information decoded from the subsequence of frames and the information decoded from the full sequence of frames, and (3) an accuracy of prediction of the timing detector neural network.

13. The computer-implemented method claim 10, wherein the processor is configured to execute a feature extractor neural network to extract features from each frame in the sequence of frames; execute a feature encoder neural network to encode the extracted features of each frame to produce a sequence of encoded features;submit the sequence of encoded features to the timing detector neural network and to identify a subsequence of encoded features representing the subsequence of frames; andsubmit the subsequence of encoded features to the decoder neural network to decode the information.

14. The computer-implemented method of claim 10, wherein the processor triggers the execution of modules of the AI low-latency processing system upon receiving a new input frame appended to the sequence of frames.

15. The computer-implemented method of claim 10, wherein the information is a caption for an audio scene, a video scene, or an audio-video scene.

16. The computer-implemented method of claim 10, wherein the information is an answer to a question about the sequence of frames.

17. The computer-implemented method of claim 10, wherein the frames include multi-model information coining from different sensors of different modalities.

18. The computer-implemented method of claim 13, wherein the information is an answer to a question about the sequence of frames, wherein the processor is configured to: execute a text encoder neural network to encode the question;submit the encoded question to the timing neural network; andsubmit the question or the encoded question to the decoding neural network.