The invention relates to the field of extraction of key frames in a sequence of frames constituting a shot for representing the shot in video summarization, browsing, searching and understanding.
With the rapid growth of popularity in storing and viewing digital video in Internet, mobile devices and a wide range of video applications, an effective management of the video data becomes much more important than ever before.
For automatic video retrieval, it is almost impossible to use keywords to describe video sequences. The reasons are that manual annotation requires tremendous manpower, and the keywords used tend to be inaccurate and subjective. Therefore, content-based techniques which can provide efficient indexing, retrieval and browsing to video sequences will be a solution.
A generic approach for managing video data is to segment a video into groups of related frames called “shots” by means of shot cut detection or scene break detection. After indentifying the shot boundaries, one or more key frames or representative frames can be extracted from each group of frames (GoF) or video shot. The visual contents on these key frames are then used to represent the video shots for indexing and retrieval.
Key frame extraction is an essential part in video analysis and management, providing a suitable video summarization for video indexing, browsing and retrieval. The use of key frames reduces the amount of data required in video indexing and provides the framework for dealing with the video content.
Key frame extraction can be done either in scene or shot level. Usually the analysis in shot level is preferred as it preserves the time sequence of the selected key frame in the video frame set.
Current key frame extraction techniques can be categorized into the following six classes:
Shot boundary based approach, visual content based approach, motion analysis based approach, shot activity based approach, unsupervised clustering based approach, and macro block based approach. These methods have their merit respectively.
For instance, document US2005/0002452 discloses a key frame extraction based on an entropy measure which is defined by a luminance distribution and a comparison with adjacent frames so that the frame with the least motion activity is selected.
It appears that known extraction methods do not perform well to select frames containing complex and fast-changing motions which may be used for action recognition.
It would advantageous to achieve a method of extracting key frames representative of the movement(s) captured by the shot.
To better address one or more concerns, in a first aspect of the invention a method of extracting a key frame from a sequence of frames constituting a shot, each frame being constituted by a matrix of pixels, comprises:
The method has the particular advantage to select frame(s) with complex and fast-changing motions.
In a particular embodiment,
In a second aspect of the invention a computer software product stored on a recording media and comprising a set of instructions to enable a computer to practice the method disclosed hereabove when the computer executes the set of instructions.
In a third aspect of the invention an apparatus for extracting a key frame from a sequence of frames constituting a shot, each frame being constituted by a matrix of pixels, comprises:
a frame optical flow calculator for computing the optical flow of each frame of said sequence of frames compared to the following frame as a matrix of displacement of each pixel from the frame to the following frame;
a motion entropy measure calculator based on the output of the frame optical flow calculator;
a key frame selector for selecting the frame of the sequence of frames having the maximum motion entropy measure.
Depending on the type of image, a particular embodiment may be preferred as easier to adapt or as giving a better result. Aspects of these particular embodiments may be combined or modified as appropriate or desired, however.
These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment described hereafter where:
In reference to
Each step is now discussed in details with specific embodiments.
Considering the computation of optical flow, it should be noted that each human activity gives rise to characteristic motion patterns, which can be easily recognized by an observer. The optical flow is a motion descriptor suitable for recognizing human actions.
In a first step, the displacement of each pixel of the frame is computed by comparison with the following frame as an optical flow field. For instance, the sequence of optical flow fields is computed using standard approaches such as the Lucas-Kanade algorithm.
So, for the frame k, the optical flow Fk between frame i and frame i+1 is a matrix of velocity vectors Fi(x, y) having each a modulus Mi(x, y) and an angle Θi(x, y). The velocity vector Fi(x, y) measures the displacement of the pixel (x, y) from the frame i to the frame i+1.
Entropy is a good way of representing the impurity or unpredictability of a set of data since it is dependent on the context in which the measurement is taken.
Based on the optical flow defined here above, a motion entropy measure is computed.
Each velocity vector based on the optical flow output is quantized by its magnitude Mi(x, y) and orientation Θi(x, y). A motion histogram is defined as a predetermined number of bins, each bin being a combination of magnitude and orientation so that the entire spectrum of magnitude and orientation value is covered. For instance, 40 histogram bins which represent 5 magnitude levels and 8 orientation angles are used.
The probability of appearance of the kth bin in a frame is given as:
where M, N is the size of the frame and h denotes the count of the kth bin. Pf(k) is thus the ratio of the pixel count contained in bin k on the total number of pixels.
where Kmax is the total bin number in the histogram, in the example Kmax=40, and the sum of all the bin entropies ef(k) is the global entropy of the motion in this frame. the bin entropy measure ef(k) is thus the probability of appearance of the bin weighted by the absolute value of the logarithmic probability of appearance of the bin. As the logarithmic probability is always negative, the absolute value is taken to obtain a positive value as entropy.
Intuitively, a peaked motion histogram contains less motion information thus produces a low entropy value; a flat and distributed histogram includes more motion information and, therefore, yields a high entropy value.
The entropy maximum method disclosed here above provides the information about which frames contain the most complex motions. In some situations frames in which the motion histograms change fast relatively to the surrounding frames also contain important information. Therefore, a second embodiment is disclosed which will be called the inter-frame method, or the histogram intersection method, and which measures the differences between the motions of consecutive frames.
The measure calculates the similarity between two histograms.
The motion histograms of a frame i and its neighborhood frame (x frames leading or lagging) are Hf(i) and Hf(i±x) respectively, and each contains Kmax bins Hf(i, k) and Hf(i±x, k) respectively. The intersection HI of two histograms are defined as
The denominator normalizes the histogram intersection and makes the value of the histogram intersection between 0 and 1. This value is actually proportional to the number of pixels from the current frames that have corresponding pixels of the same motion vectors in the neighborhood frame. A higher HI value indicates higher similarity between two frames.
In this method, HI is used as the motion entropy measure and key frame is selected as the frame having the highest HI.
This method may be used as a supplemental method for the first disclosed method since it provides extra information about the motion vector distribution between two frames.
In a variant of these two methods, it is noted that a video frame usually has both foreground (objects) and background (camera) motions, and the background motion is usually consistent and dominant in the motion histogram.
As shown in
For certain applications such as action recognition, one key frame may not be sufficient and multiple key frames are needed to summarize a shot. Therefore, instead of finding the global maximum of the entropy function for the complete shot, local maxima are searched for. For instance, the local maximum in a sliding window with the length of n frames is considered. Of course, more advanced techniques for finding local maxima can be also employed.
The key frames selected by using the local maxima approach may be used for applications, such as video summarization. For low-activity shots, one single key frame may be sufficient, but most of the time, multiple key frames are needed to represent the contents of the shot. By observing a set of key frames instead of a single key frame, a better understanding of the layout of the shots, e.g. the direction of the movements, changes in the background, etc. may be obtained.
Key frames may be obtained by combining the entropy maxima and the inter-frame algorithms. The combined algorithm extracts frames which not only contain the most complex motions but also have salient motion variations relative to its neighborhoods.
The disclosed methods may be implemented by an apparatus,
The apparatus may comprises input means for receiving shots to be analyzed and output means to send the key frame(s) to a video database index for instance.
While the invention has been illustrated and described in details in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiment.
The apparatus may be implemented by using a programmable computer and a computer software product stored on a recording media and comprising a set of instructions to enable a computer to practice the disclosed methods when the computer executes the set of instructions. However, due to the highly parallelism of each operations, and the high throughput required specifically by video processing, the man skilled in the art may implement advantageously the system into a specific hardware component such as a FPGA (Field Programmable Gate Arrays) or by using some specific digital signal processor.
Other variations to the disclosed embodiments can be understood and effected by those skilled on the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements and the indefinite article “a” or “an” does not exclude a plurality.
Number | Date | Country | Kind |
---|---|---|---|
09305316.3 | Apr 2009 | EP | regional |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IB2010/051620 | 4/14/2010 | WO | 00 | 10/7/2011 |