This application claims the benefit of priority to GB Application No. 1615374.4, filed Sep. 9, 2016, the contents of which are incorporated herein by reference in its entirety.
This invention relates to apparatus and methods for analysing a set of images in order to arrange them based on the content of the images.
Linear playback of a sequence is well known. With traditional physical recording media such as tape or DVD, playback is performed by a dedicated device such as a tape player, controlled by buttons which perform well-known functions such as Play, Pause, Stop, Rewind and Fast Forward. Some playback devices have more sophisticated control functions such as Jog and Shuttle, controlled by a knob which allows fast access to, and detailed frame-by-frame viewing of, different parts of the recorded content.
In order to graphically show a set of images that comprise a video sequence the set of images is normally arranged in chronological order so that the first image shown is the first image from the linear sequence, the next image shown from the set is the second image in the sequence, until the final image in the sequence is shown. Variants of this arrangement include identifying the key images in a set of images and showing these chronologically.
In one embodiment, a method for performing analysis on a set of images is provided. This method may comprise measuring, for each image, a feature value for each of a plurality of image features and determining over the set of images a correlation measure representing for at least some combinations of the image features the correlation in the respective feature values. This may be followed by selecting, in accordance with said correlation measure, at least one closely correlated combination of image features and ordering the set of images in accordance with those closely correlated combinations of image features.
The inventor has recognized that the prior art visualization methods described above have the limitation that the organization of the content is related only to the temporal position of frames within the sequence. In other words, the condition for frames to be close together in the visualization is that they be close together in time in the sequence itself. For some sets of images this known arrangement works well, however for others it is not an optimal solution. There is further described below techniques which allow the images from a set to be arranged not merely in chronological order but in an order based on features of the content of each image.
The invention will now be described by way of example with reference to the accompanying drawings, in which:
The filmstrip visualisation is in chronological order, and this allows the user to scroll through the entire sequence of images from the image set in order to find a desired image, or section of images.
An alternative to this embodiment is shown in
However, both
These repetitions of content are used by the viewer to build a semantic model of what is seen: to make sense of the things that are seen, to concentrate on the important aspects and to filter out superfluous information. A human observer will establish links and will group scenes according to their visual appearance. Search engines rely on establishing and retrieving connections and relationships between data. Non-linear visual representations of textual information, such as “mind maps” or “word clouds” are often used successfully in many schemes for visualization of a variety of information.
The present invention extends the above principles of non-linear grouping of types of information to video data.
These steps combine to create one embodiment of performing analysis on a set of images to identify features which are most closely correlated, and then arranging or ordering the set of images in accordance with those closely correlated features.
The step of measuring a feature value for each image 302 comprises calculating a series of features values for each image feature of each image. These feature values form a feature vector. A feature vector is a multidimensional quantity consisting of measurable features of an image. For example, such image features may include the average luminance level, average red, blue or green level, the proportion of the picture occupied by defined colours such as flesh tones, the standard deviation of pixel level such as luminance, the average level of detail such as horizontal or vertical detail, the average speed of motion between the current image and previous or next image, the estimated quantity of text present in the image, the volume or loudness of associated audio, the time stamp or frame number of the image. Alternatively any other measurable feature of the image may be included. By calculating a feature vector including at least two features the values of the features for each image can be analysed, as can the relationship between the features across all of the images from the set.
Determining a correlation measure 304 comprises analysing the feature values, or feature vectors, for each image and the relationship between the features. This allows correlations between the features (and between combinations of features) to be found. For example, in an action sequence from an action movie it would be expected that the sound associated with an image in the action sequence would be loud, and the average speed of motion between the current image and the previous image would be high. It would therefore be expected that these features would correlate well in this section of the movie. In a video of a sunrise the set of images would be expected be brighter as the sun rises. Therefore in this example average luminescence of each image will likely increase as the time, or image number, of each image increases. By detecting the features, or combinations of features, that are most correlated an image sequence can be characterised.
Selecting at least one closely correlated combination of image features 306 comprises using the correlation measure to find combinations of image features that are closely correlated. One or more of these may then be selected. For example, it may be advantageous to use two combinations of features (especially for a two dimensional map of images). The highest two combination values (or the highest two combinations that fulfil a user specified criteria) may then be selected.
Ordering the images in accordance with the closely correlated features 308 comprises using the determined most correlated features (or combinations of features) to place each image on a two or three dimensional map.
In one example the sunrise and the action sequence described above may be spliced together to form a single set of images. The sunrise has a high correlation between time and average luminescence, whilst the action sequence has a high correlation between speed and sound level. Therefore these may be the combinations of features that are determined to be most closely correlated across the set of images 304. They may then be selected as the closely correlated combinations of features 306. A combination of both of these pairs of features may be determined for each image of the set. The combination values may then be used to determine where each image should be placed on a two or three dimensional map 308. In this example it is likely the sunrise images will be grouped together because the value of the time and average luminescence combination will be high, and the action sequence images will be grouped together because the speed and sound level combination value will be high. Therefore the map will separate out the unrelated sections of the set of images from one another. There may be a non-linear mapping between the values of the combinations and the placement on the map. For example, clusters of images with similar values may be spread out slightly, whilst large gaps between groupings may be narrowed so that the images can be scaled to an appropriate size, and so that the map is easy to use. There may be an overlap between certain images that are close to one another on a map. In another example, the overlapping could be restricted to images whose features were close to one another in the original image set, and a degree of positional adjustment could be applied to groups of overlapping images so that the different groups could be viewed separately. Alternatively, overlapping could be reduced or avoided altogether by choosing to display only key images from each scene in the sequence are displayed.
Step 502 of calculating a feature vector for each image is one embodiment of measuring a feature value for each feature for each image 302, which is described above.
Steps 504 and 506 together may comprise the steps to perform step 304 of
Step 506 describes calculating a covariance matrix for the entire set of images. This may comprise averaging the value of each combination of features across all of the images and associated feature vector matrices. Alternatively a covariance matrix may be calculated straight from the feature vectors associated with each image. A covariance matrix is shown below where C is the covariance matrix, n is the number of features, xij, from feature-vector matrix, is the value of feature j in picture i, and is an averaging operation across the sequence. Each element of the covariance matrix (305) indicates the correlation between a different pair of features.
Steps 508, 510 and 512 together may comprise the step 306 of
C=U′W′V′
T
The symmetry of the covariance matrix means that the matrices U′ and V′ are identical. This use of singular value decomposition on covariance matrix C is shown below:
This produces a first matrix U′, a diagonalised matrix W′, and a second matrix V′T 510. This diagonalised matrix is formed of singular values. It can be determined which of these have the highest or largest value 512. This allows the closely correlated feature combinations to be selected 306.
Steps 514, 516 and 518 together may comprise step 308 of
The original feature-vector matrix for each image may then be reduced by applying the following matrix multiplication formula 516:
This result can then be used determine the arrangement of the images based on the Y′matrices associated with each image 518.
It will be appreciated from the discussion above that the embodiments shown in the Figures are merely exemplary, and include features which may be generalised, removed or replaced as described herein and as set out in the claims. With reference to the drawings in general, it will be appreciated that schematic functional block diagrams are used to indicate functionality of systems and apparatus described herein. For example the steps shown in
The above embodiments are to be understood as illustrative examples. Further embodiments are envisaged. It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.
In some examples, one or more memory elements can store data and/or program instructions used to implement the operations described herein. Embodiments of the disclosure provide tangible, non-transitory storage media comprising program instructions operable to program a processor to perform any one or more of the methods described and/or claimed herein and/or to provide data processing apparatus as described and/or claimed herein.
The processor of any apparatus used to perform the method steps (and any of the activities and apparatus outlined herein) may be implemented with fixed logic such as assemblies of logic gates or programmable logic such as software and/or computer program instructions executed by a processor. Other kinds of programmable logic include programmable processors, programmable digital logic (e.g., a field programmable gate array (FPGA), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM)), an application specific integrated circuit, ASIC, or any other kind of digital logic, software, code, electronic instructions, flash memory, optical disks, CD-ROMs, DVD ROMs, magnetic or optical cards, other types of machine-readable mediums suitable for storing electronic instructions, or any suitable combination thereof.
Number | Date | Country | Kind |
---|---|---|---|
1615374.4 | Sep 2016 | GB | national |