DETECTION OF SIMILAR VIDEO SEGMENTS

Abstract

A method and apparatus for processing a first sequence of images and a second sequence of images to compare the first and second sequences is disclosed. Each image of the first sequence and each image of the second sequence is processed by: (i) processing the image data for each of a plurality of pixel neighbourhoods in the image to generate at least one respective descriptor element for each of the pixel neighbourhoods; and (ii) forming an overall image descriptor from the descriptor elements. Each image in the first sequence is compared with each image in the second sequence by calculating a distance between the respective overall image descriptors of the images being compared. The distances are arranged in a matrix, and the matrix is processed to identify similar images.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the right of priority based on British patent application number 09 012 63.4 filed on 26 Jan. 2009, which is hereby incorporated by reference herein in its entirety as if fully set forth herein.

FIELD OF THE INVENTION

The invention relates to a method, apparatus and computer program product for the detection of similar video segments.

BACKGROUND TO THE INVENTION

In recent years there has been a sharp increase in the amount of digital video data that consumers have access to and keep in their video libraries. These videos may take the form of commercial DVDs and VCDs, personal camcorder recordings, off-air recordings onto HDD and DVR systems, video downloads on a personal computer or mobile phone or PDA or portable player, and so on. This growth of digital video libraries is expected to continue and accelerate with the increasing availability of new high capacity technologies such as Blu-Ray. However, this abundance of video material is also a problem for users, who find it increasingly difficult to manage their video collections. To address this, new automatic video management technologies are being developed that allow users efficient access to their video content and functionalities such as video categorisation, summarisation, searching and so on.

One problem that arises is the need to identify similar video segments. The potential applications include the identification of recurrent video-segments (e.g. TV-station jingles), and video database retrieval, based for instance on the identification of a short fragment provided by the user within a large database of video. Another potential application is the identification of repeated video segments before and after commercials.

In GB 2 444 094 A “Identifying repeating video sections by comparing video fingerprints from detected candidate video sequences” a method is devised to identify repeated sequences as a mean of identifying commercial breaks. Initially, the detection of hard cuts, fades, and audio level changes identifies candidate segments. Whenever a certain number of hard cuts/fades is identified, a candidate segment is considered and stored. This will be compared against the subsequent identified candidate segments. Comparison is performed using features from a set of possible embodiments: audio level, colour histogram, colour coherence vector, edge change ratio, and motion vector length.

The problem with this method is that it relies on clear boundaries between a segment and its neighbours in order for the segment to be identified in the first place, and then compared against other segments. Also, partial repetitions (i.e. only one section of a segment is repeated) cannot be detected. Furthermore colour coherence vectors provide very little spatial information and therefore are unsuitable for frame-to-frame matching. Finally, some of the features suggested are not available in uncompressed video and therefore must be calculated ad-hoc, noticeably increasing the computational and time requirements.

In WO 2007/053112 A1 “Repeat clip identification in video data” a method and system for identifying repeated clips in video data is presented. The method comprises partitioning the video data into ordered video units utilising content-based keyframe sampling, wherein each video unit comprises a sequence interval between two consecutive keyframes; creating a fingerprint for each video unit; grouping at least two consecutive video units into one time-indexed video segment; and identifying the repeated clip instance based on correlation of the video segments.

The video is firstly scanned and for each frame a colour histogram is calculated. When a change in histogram is detected between two frames, according to a given threshold, the second frame is marked as keyframe. The set of frames between one keyframe and the next constitutes a video unit. A unit-level colour signature is then extracted, as well as frame-level colour signatures. Furthermore, unit time length is also considered as feature. A minimum of two consecutive video units are then united to form a segment. This is compared against each other segment in the video. L1 distances are calculated for the unit-level signature and time lengths and if both are below fixed thresholds, a match is detected and the corresponding point in a correlation matrix is set to 1 (0 otherwise). Sequences of is then indicate sequences of matching segments. The frame-level features are used only as a post-processing verification step, and not in the proper detection process.

One drawback with the technique in WO 2007/053112 A1 is that it is based on video units, a video unit being the video between non-uniformly sampled content-based keyframes. Thus, a unit is a significant structural element, e.g. a shot or more. This is a significant problem since, in the presence of very static or very dynamic video content, the key-frame extraction process itself will become unstable and detect too few or too many units. Also, for video segments which are matching but also differ in small ways, e.g. by the addition of a text overlay, or a small picture-in-picture, and so on, the key-frame extraction may also become unstable and detect very different units. A segment is then defined as the grouping of two or more units, and the similarity metric is applied at the segment level, i.e. similarities are detected at the level of unit-pairs. So, the invention is quite limited in that it is targeted to the matching of longer segments, e.g. groups of shots, and cannot be applied to ad-hoc segments that last only a few frames. The authors acknowledge this and claim that this problem can be addressed by assuming for example, sampling at more than one keyframes per second. This, however, can only be achieved by uniform rather than content-based sampling. A major problem that emerges in that case is that video unit-level features will lose all robustness to frame rate changes. In all cases, a fundamental flaw of this method is that it makes decisions on the similarity of segments (i.e. unit-pairs) based on a fixed threshold, but without taking into consideration what similarity levels the neighbouring segments exhibit. The binarized correlation matrix may provide an excessively coarse description of the matching, and result in an excessive number of 1s, e.g. due to the presence of noise. Then, linear sequences of matching segments are searched for. With non-uniform key-frame sampling these lines of matching unit-pairs may be non-contiguous and made of breaking and non-collinear segments, and a complex line-tracking algorithm is employed to deal with all these cases. And although frame-level features are available, these are only used for verification of already detected matching segments, not for the actual detection of matching segments.

In general, the aforementioned prior art is mostly concerned with the identification of equal length segments with very high similarity, and distinctive boundaries with respects to neighbouring segments. This situation can reasonably suit the application of such methods to the identification of repeated commercials, which are usually characterized by sharp boundaries (e.g. few dark frames before/after commercial), distinctive audio levels, and equal length of the repetitions. However, the aforementioned prior art lacks the generality necessary to deal with more arbitrary applications.

One problem that is not addressed is the partial repetition of even a short segment, i.e. only a portion of a segment is repeated. In this case, it is not possible to use segment length as a feature/fingerprint for identification.

Another problem that is not addressed is the presence of text overlay in one of the two segments, or linear/non-linear distortion of one of the two segments (e.g. blurring, or luminance/contrast/saturation changes). Such distortion must be taken into account when considering more general applications.

In WO 2004/040479 A1 “method for mining content of video” a method for detecting similar segments in video signal is illustrated. A video of unknown and arbitrary content and length is subject to feature extraction. Features can be audio and video based, e.g. motion activity, colour, audio, texture, such as MPEG-7 descriptors. A feature progression in time constitutes a time series. A self-distance matrix is constructed from this time series using Euclidean distance between each point of the time series (or each vector of a multi-dimensional time series). In the claims, other measures are mentioned, specifically dot product (angle distance) and histogram intersection. Whether multiple features are considered (e.g. audio, colour, etc), for each feature the method of finding paths in the distance matrix is applied independently. The resulting identified segments are subsequently fused.

The method finds diagonal or quasi-diagonal line paths in the diagonal matrix using dynamic programming techniques i.e. finding paths of minimal cost, defined by an appropriate cost function. This cost function includes a fixed threshold that defines, in the distance matrix, where the match between two frames is to be considered “good” (low distance) or “bad” (high distance). Therefore points whose value is above the threshold are not considered, while all the points in the distance matrix whose value is below the threshold are considered. Subsequently, paths which are consecutive (close endpoints) are joined, and paths that partially or totally overlap are merged. After joining and merging, short paths (less than a certain distance between the end points) are removed.

One drawback with the technique in WO 2004/040479 A1 is that the application of dynamic programming to search linear pattern in the distance matrix may be computationally very intensive. Furthermore one should consider that dynamic programming is applied to all points in the distance matrix that fall below a certain fixed threshold. This fixed threshold may lead to a very large or very small number of candidate points. A large number of points is produced if segments in a video are strongly self-similar, i.e. the frames in the segment are very similar. In this case a fixed threshold that is too high may generate a impractically large number of points to be tracked.

In the eventuality that a repeated segment is composed of identical frames, the problem of finding a least cost path could be ill-posed since all diagonal paths connecting a point of the first segment with a point of the second segment would yield same cost. This would generate a very large number of parallel patterns. An example of these patterns is illustrated in FIG. 4. The invention does not provide a method to merge groups of parallel segments generated by a region of strong self-similarity.

On the other hand, in the presence of strong non linear editing (e.g. text overlay, blur, brightening/darkening) the distance between frames may rise above the fixed threshold, resulting in an insufficient number of candidate points.

Another problem may arise when a replicated segment is partially edited, e.g. some frames of the segment are replicated with blur, or text overlay. In this case a break is generated in the path of minimal cost, resulting in two split segments even if the two segments a semantically connected.

Another problem with both WO 2007/053112 A1 and WO 2004/040479 A1 is the complexity and cost of calculating the distance matrix and storing the underlying descriptors, which become prohibitive for very large sequences when a real-time or faster operation is required. What is required is a method which alleviates these problems so as to allow fast processing of large sequences, e.g. entire programmes.

SUMMARY OF THE INVENTION

Certain aspects of the present invention are set out in the accompanying claims. Other aspects are described in the embodiments below and will be appreciated by the skilled person from a reading of this description.

An embodiment of the present invention provides a new method and apparatus for detecting similar video segments, which:

• • Describes frames by inexpensive binary descriptors which may be compared by the Hamming distance, giving rise to a Hamming distance matrix, with great computational savings.
  • Finds line patterns in the distance matrix on a small subset of points in the distance matrix. These are points who are local minima for the distance matrix, or neighbouring points of the local minima, where a minimum is defined by a finite-difference approximation of the first and second derivatives of the distance matrix.
    • These points are further processed and only those whose values is below a certain threshold are kept. This threshold is determined adaptively according to the number of minima found per column of the distance matrix, i.e. it guarantees that no less than a minimum number and no more than a maximum number of minima (if found) are kept.
    • Furthermore, whenever a sequence of identical or quasi-identical local minima is found, i.e. a local valley betraying a zone of strong self-similarity, a method is provided that finds and retains only selected points in the valley, reducing the number of parallel patterns generated.
    • By doing so, the method has a great advantage with respects to WO 2004/040479 A1 as it minimizes the computational effort by minimizing the number of potentially valid matches in the Hamming distance matrix.
  • Provides a method to eliminate multiple parallel patterns generated by segments with high self-similarity (valleys in the distance matrix).
  • Is robust to luminance shifts, text overlay and non-linear editing (e.g. blurring) and detecting weak similarities via adaptive threshold on local minima.
  • Is robust to partial non-linear editing of a segment by providing a method to join split segments via an hysteresis threshold joining method.
  • Can operate on compressed MPEG video stream as well as uncompressed video. Can operate only on I-frames of a compressed MPEG stream, therefore not requiring the decoding of P and B frames in the video stream. Consequently the method can also operate on time-subsampled version of the video.
  • Can operate on DC or sub-DC frame resolutions, therefore minimizing the computational effort and the memory requirements and not requiring the decoding of the frame to its full-resolution.
  • Operates on a compact vector of features for each individual frame, based on a multi-level spatial transformation.
  • Exploits details and high-frequency spatial contents in the frame as a measure of similarity.
  • Is based on a frame-to-frame matching, and does not require the grouping of frames prior the analysis.
  • Does not rely on the audio track, transition/hard cut/scene change detections, dynamic content analysis.
  • Does not require the segments to have equal or similar length.
  • Is robust to frame rate changes.
  • Has a high recall rate with negligible false detections.

More particularly, given two video sequences, an embodiment of the invention performs processing for each frame of each sequence to:

• • Calculate a compact, computationally efficient descriptor based on a multi-level transform that captures multi-level luminance and chrominance content (average values/low pass) and interrelations (differences/high pass).
  • Binarize the elements of the descriptor.
  • Calculate a matching score between frames of one sequence with all the frames in the other sequence according to the corresponding descriptors' binary distance, and store the result in a Hamming distance matrix.
  • Find local minima along rows and/or columns in the distance matrix preserving continuity information to deal with uncertain/imperfect/multiple matching and coarse sampling.
  • Detect sequences of consecutive and neighbouring minima over diagonal paths, tacking misalignments and missing matches, and assess them according to their overall matching scores.

LIST OF FIGURES

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

FIGS. 1 and 2 comprise flowcharts showing the processing operations in an embodiment;

FIG. 3 illustrates the detection of local minima and valley points;

FIG. 4 illustrates the detection of local minima lying on straight lines;

FIG. 5 shows a flowchart of the processing operations to apply a hysteretic line segment joining algorithm;

FIG. 6 shows an example of the results of the processing;

FIG. 7 shows an embodiment of a processing apparatus for performing the processing operations.

EMBODIMENTS OF THE INVENTION

A method that is performed by a processing apparatus in an embodiment of the invention will now be described. The method comprises a number of processing operations. As explained at the end of the description, these processing operations can be performed by a processing apparatus using hardware, firmware, a processing unit operating in accordance with computer program instructions, or a combination thereof.

Given two video sequences, S_a and S_b, the processing performed in an embodiment finds similar segments between the two sequences.

According to the present embodiment, video frames

F(n,m)={F^c(n,m)}, n=1K N, m=1K M, c=1K C

may be described by their pixel values in any suitable colour space (e.g. C=3 in RGB or YUV colour space, or C=1 for greyscale images), or in any suitable descriptor derived thereof.

In one embodiment of the invention, each frame in S_a and S_b is described by its pixel values. In a preferred embodiment of the invention (FIG. 1), each frame in S_a and S_b is described by a descriptor which captures the high-pass and low-pass content of the frame in the YUV colour channel (step S1).

Such descriptors may be calculated using the techniques described in EP 1,640,913 and EP 1,640,914, the full contents of which are incorporated herein by cross-reference. For example, such descriptors may be calculated using a multi-resolution transform (MRT), such as the Haar or Daubechies wavelet transforms. In a preferred embodiment, a custom, faster transform is used that is calculated locally on a 2×2 pixel window and is defined as

$\hspace{1em}\{\begin{array}{c}{\mathrm{LP}}^c\left(n,m\right)=\left[\begin{array}{c}{F}^c\left(n,m\right)+F^c\left(n+1,m\right)+\\ F^c\left(n,m+1\right)+F^c\left(n+1,m+1\right)\end{array}\right]/4\\ {\mathrm{HP}}_1^c\left(n,m\right)=\left[F^c\left(n,m\right)-F^c\left(n+1,m\right)\right]/2\\ {\mathrm{HP}}_2^c\left(n,m\right)=\left[F^c\left(n+1,m\right)-F^c\left(n+1,m+1\right)\right]/2\\ {\mathrm{HP}}_3^c\left(n,m\right)=\left[F^c\left(n,m\right)-F^c\left(n,m+1\right)\right]/2\end{array}$

In a similar fashion to the Haar transform, this MRT is applied to every 2×2 non-overlapping window in a resampled frame of dimensions N=M=a power of 2. For a N×M frame F(n,m) it produces, for each colour channel c, (N×M)/4 LP^c elements and (3×N×M)/4 HP^c elements. Then, it may be applied to the LP^c elements that were previously calculated, and so on until eventually only 1 LP^C and (N×M−1) HP^c elements remain per colour channel.

For each frame F(n,m) LP and HP elements, or a suitable subset of them are arranged in a vector (hereinafter referred as descriptor) Φ=[φ_d] d=1k D (step S2), where each element φ_d belongs to a suitable subset of LP and HP components (e.g. D=C×N×M).

Each element of the vector φ_d is then binarized (quantized) according to the value of its most significant bit (MSB) (step S3)

Φ^bin=[φ_d^bin] d=1k D:φ_d^bin=MSB(φ_d), φ_dεΦ

In different embodiments of the invention, different frame descriptors, or different elements of each descriptor, are subject to individual binarization (quantisation) parameters, such as MSB selection, locality-sensitive hashing (for example as described in Samet H., “Foundations of Multidimensional and Metric Data Structures”, Morgan Kaufmann, 2006), etc.

Each frame F_i^(a) in S_a=[F_i^(a)], i=1K A is compared against each frame F_j^(b) in S_b where S_b=[F_j^(b)], j=1K B by means of Hamming distance δ_ij of their respective binarized descriptors.

The elements δ_ij are arranged in a distance matrix (step S4)

Δ=[δ_ij] i=1K A, j=1K B

In the preferred embodiment of the invention (FIG. 2), for each column of Δ (step S5), local minima μ are searched for (step S6). Minima are defined as zero-crossings in the first derivative of the column under exam, yielding positive second derivative. A general approach interpolates the column with a smooth differentiable curve (e.g. a high-order polynomial) that is subsequently analytically differentiated twice in order to calculate first and second derivatives. More practical approaches calculate first derivatives as combinations of smoothing and finite differences. In one embodiment, in order to minimize computational costs, an implicit combination of first and second order finite difference is implemented where minima are found when the previous and next values (column-wise) are higher (step S6)

${\delta }_{\mathrm{ij}}=\mathrm{minimum}\iff \{\begin{array}{c}{\delta }_{\mathrm{ij}}<{\delta }_{\left(i+1\right)j}\\ {\delta }_{\mathrm{ij}}<{\delta }_{\left(i-1\right)j}\end{array}$

A local minimum μ_ij at the i-th row of the j-th column of Δ indicates that the frame F_i^(a) is the most similar to F_j^(b) within its column-wise neighbourhood . In the simple minimum finding procedure described above, the neighbourhood is defined as ={F_j−1^(a), F_j^(a), F_j+1^(a)}. Consequently a local minimum μ_ij which is also global in the j-th column indicates that the frame F_i^(a) is the best match to F_j^(b). Local minima are evaluated against a threshold (step S7). The algorithm preserves only those minima whose value is sufficiently small i.e. that imply a sufficiently strong match between the corresponding frames in S_a and S_b.

The threshold in S7 is adaptively calculated so that a minimum amount mm and no more than a maximum amount Mm of minima are kept. However, if the number of minima found at step S6 is smaller than mm, then the threshold is consequently adapted in order to preserve all of them.

For each local minimum μ a set V of valley points is found (step S8). These are defined as the non-minima points immediately below and above (column-wise in Δ) the corresponding minimum, i.e.

∀μ_ij=δ_ijV=[δ_(i−v)jKδ_(i−1)jδ_(i+1)jKδ_(i+v)j]

where v is a default parameter (such as 3) or alternatively is defined heuristically. The goal of V is to provide continuity information in the neighbourhood of each μ and therefore harness discontinuity and non-colinearity that arises from any form of sampling, non-linear editing, and in general lack of “strong” matching between the two sequences S_a and S_b.

Valley points are evaluated against a threshold (step S9). The algorithm preserves only those valley points whose value is sufficiently small i.e. that imply a sufficiently strong match between the corresponding frames in S_a and S_b.

Local minima and valley points are denominated altogether as candidate matching segment points π (step S10). An example of π is illustrated in FIG. 3, where local minima are indicated with circles and valley points as crosses.

It should be noted that in a different embodiment of the invention, local minima and valley points may be searched in an analogous fashion along rows of the distance matrix instead of columns. In yet another embodiment of the invention, local minima and valley points may be searched in an analogous fashion in both dimensions of the distance matrix.

A line segment searching algorithm is applied to the set of π (step S11). The rationale is that if a video segment of S_a is repeated in S_b, this will raise a set of consecutive (adjacent) π in Δ arranged in a line segment σ orientated at θ=tan⁻¹(ρ_a/ρ_b) where ρ_a and ρ_b are respectively the frame rates of S_a and S_b. If frame rate does not change from S_a to S_b it follows that ρ_a=ρ_b and θ=45°.

Valley points V therefore help to fill any gap that may arise due to the presence of noise or imperfect matching due to any coarse time sampling. An example of the line segment searching algorithm is illustrated in FIG. 4.

In a preferred embodiment of the invention, further to the line segment searching, a hysteretic line segment joining algorithm follows (FIG. 5). This helps to further fill the gaps between line segments that may arise from local non-linear editing, noise, sampling or incorrect matching. If two collinear line segments are closer than a given distance in terms of number of points in Δ between the proximal ends of the two line segments (step S12), the corresponding intermediate δ values are averaged. If this average value

${\stackrel{\_}{\Delta }}_{\mathrm{interm}}\left({\delta }_{\mathrm{ij}}\right)=\frac{1}{L\left({\delta }_{\mathrm{ij}}|i,j\in \mathrm{interm}\right)}\sum _{i,j\in \mathrm{interm}}{\delta }_{\mathrm{ij}}$

is lower than a given threshold, therefore indicating sufficient matching between the intermediate frames in S_a and S_b, then the two line segments are connected (step S13).

In a preferred embodiment, line segments σ (step S14), and therefore matching video segments, are validated according to their average value in Δ calculated as

$\stackrel{\_}{\Delta }\left(\sigma \right)=\frac{1}{L\left(\sigma \right)}\sum _{i,j}{\delta }_{\mathrm{ij}}|{\delta }_{\mathrm{ij}}\in \sigma$

where L(σ) is the length (number of π) of the line segment σ (step S15). Line segments yielding Δ(σ) higher than a given threshold are discarded as erroneous matches, since a high Δ(σ) betrays insufficient matching of the frames (FIG. 5).

In a preferred embodiment, an ambiguity resolution procedure (AR) is employed to remove multiple matches and ambiguous results. An example of the final result is provided in FIG. 6.

The AR works in two stages as follows:

Stage 1: Shadow Removal

• 1. Line segments are sorted accorded to their length. Longer line segments are considered first. Each line segment σ projects a “square shadow” ζ(σ) i.e. defines a square area whose diagonal is σ. If σ is defined by its start and end coordinates x_xtart(σ), x_xtop (σ), y_xtart (σ), y_xtop(σ) then a point π=(x_π,y_π) is shadowed by σ if

$\pi \in \zeta \left(\sigma \right)\iff \{\begin{array}{c}{x}_{\pi }\in \left[x_{\mathrm{xtart}}\left(\sigma \right),x_{\mathrm{xtop}}\left(\sigma \right)\right]\\ y_{\pi }\in \left[y_{\mathrm{xtart}}\left(\sigma \right),y_{\mathrm{xtop}}\left(\sigma \right)\right]\end{array}$

• • Therefore, a line segment σ_a is shadowed by σ_b if

${\sigma }_a\in \zeta \left({\sigma }_b\right)\iff \{\begin{array}{c}{x}_{\pi }\in \left[x_{\mathrm{xtart}}\left({\sigma }_b\right),x_{\mathrm{xtop}}\left({\sigma }_b\right)\right]\\ y_{\pi }\in \left[y_{\mathrm{xtart}}\left({\sigma }_b\right),y_{\mathrm{xtop}}\left({\sigma }_b\right)\right]\end{array},\forall \pi \left(x_{\pi },y_{\pi }\right)\in {\sigma }_a$

• • It trivially follows that

σ_aεζ(σ_b)L(σ_b)≧L(σ_a)

• • Partial shadowing between two line segments implies that only a subset of points from one line segment is shadowed by the other line segment, and vice versa. In this case, no assumptions on the relative lengths can be drawn.
• 2. A line segment σ_shorter shadowed by a longer line segment σ_longer is removed. However if σ_shorter is only partially shadowed by σ_longer, only the points π_shorterπεσ_shorter:π_shorterεζ(σ_longer) are removed. However, if the length of σ_shorter (or alternatively the length of its shadowed part) is equal or larger than half the length of σ_longer i.e. L(σ_shorter)≧L(σ_longer)/2 and the average value of σ_shorter (or alternatively the average value of its shadowed part) is lower than the average value of σ_longer i.e. Δ(σ_shorter)< Δ(σ_longer), hence σ_shorter inferring a better average match for the respective video sequences, then those points of σ_longer that are shadowed by σ_shorter i.e. those points longer π_longer=πεσ_longer:π_longerεζ(σ_shorter) are removed and the procedure is repeated.

Stage 2: Multiple Matches

In one embodiment of the invention, it is considered the case where two or more video segments in S_a (in S_b) have the same match in S_b (in S_a). The corresponding line segments in Δ are said to be competing as they “compete” to associate the same frames in S_b (in S_a) with different frames in S_a (in S_b). Trivially, competing line segments do not shadow each other (this eventuality would be dealt by stage 2). Given two line segments σ₁, σ₂, σ₁ is said to compete with σ₂ if

• • Competing for the same segment in S_a:

[x_xtart(σ₁),x_xtop(σ₁)]∩[x_xtart(σ₂),x_xtop(σ₂)]≠0

[y_xtart(σ₁),y_xtop(σ₁)]∩[y_xtart(σ₂),y_xtop(σ₂)]=0

• • Competing for the same segment in S_b:

[x_xtart(σ₁),x_xtop(σ₁)]∩[x_xtart(σ₂),x_xtop(σ₂)]=0

[y_xtart(σ₁),y_xtop(σ₁)]∩[y_xtart(σ₂),y_xtop(σ₂)]≠0

Although competing frame segments may occur, the presence of competing line segments may in fact betray a false result by the algorithm, and therefore they are assessed as follows:

• 1. Consider the average value Δ of all the competing line segments σ. The one yielding the lowest Δ(σ) is initially considered as the true (winner) match σ_winner.
• 2. If any other competing segment σ yields Δ(σ) within an upper bound from the winner average Δ(σ_winner), Δ(σ_winner)≦ Δ(σ)≦ Δ(σ_winner)+κ with κ>0 a suitable threshold then σ is considered another instance of σ_winner. Should that not be the case, σ is considered a false detection and discarded.

In different embodiments of the invention, and according to the target application, either Stage 1 or Stage 2 or the entire AR procedure may be omitted.

In one embodiment of the invention, the two video sequences S_a and S_b are one and the same, i.e. S_a=S_b=S, and the method is aimed at finding repeated video segments within S. In that case only the upper-triangular part of Δ requires processing, since S_b=S_a trivially implies that Δ is symmetric, and the main diagonal is a locus of global minima (self-similarity). So we have to guarantee that given a line segment σ={x_xtart, x_xtop, y_xtart, y_xtop} then x_xtart<y_xtart, x_xtop<y_xtop. Furthermore, to avoid detection of self-similarity we have to ensure that any detected line segment infer two non-overlapping time-intervals in S_a and S_b. In other words y_xtop<x_xtart i.e. the repeated video segment in S_b must start after the end of its copy in S_a. Since however y_xtart<y_xtop, x_tart<x_xtop, the condition y_xtop<x_xtart is sufficient as it also implies that the segment lies in the upper triangular part. In an alternative embodiment of the invention, the lower-triangular part of the distance matrix may be processed instead of the upper-triangular part in an analogous fashion.

In different embodiments of the invention, S_a and S_b may be described by multiple descriptors, e.g. separately for different colour channels and/or for LP and HP coefficients, resulting in multiple distance matrices Δ. This is understood to better harness the similarity between frames by addressing separately the similarity in colour, luminosity, detail, average colour/luminosity, etc.

In a preferred embodiment, we consider the YUV colour space, and we separate HP and LP coefficients for the Y-channel, and retain only the LP coefficients of the U- and V-channels. This results in three distance matrices Δ_Y-HP, Δ_Y-LP, and Δ_UV-LP. In such an embodiment, each distance matrix may be processed individually. For example, the minima and valley points found on the Δ_Y-HP may be further validated according to their value in Δ_Y-LP and Δ_L-HP. In a similar fashion, line segments σ may be validated according to their average values in the three matrices, i.e. according to Δ_Y-HP(σ), Δ_Y-LP(σ) and Δ_UV-LP(σ).

In different embodiments of the invention, the descriptor elements are not binarised but quantised to a different number of bits, e.g. 2 or 3 bits, in which case the Hamming distance is replaced by a suitable distance measure, e.g. L1, which may be efficiently implemented using table lookup operations, in a fashion similar to the commonly employed for the Hamming distance.

In different embodiments of the invention, one or more of the aforementioned multiple descriptors may be calculated from only a portion, e.g. the central section, of the corresponding frames. This can reduce computational costs and may improve accuracy.

In different embodiments of the invention, the frame descriptors may be calculated from spatially and/or temporally subsampled video, e.g. from low-resolution video frame representations, and employing frame skipping. In one embodiment, S_a and/or S_b are MPEG coded and frame matching is performed based on the DC or subsampled DC representations of I-frames. This means that no video decoding is required, which results in a great increase in computational efficiency.

A data processing apparatus 1 for performing the processing operations described above is shown in FIG. 7. The apparatus can, for example, be a personal desktop computer or a portable computer.

The apparatus 1 comprises conventional elements of a data processing apparatus, which are well-known to the skilled person, such that a detailed description is not necessary. In brief, the apparatus 1 of FIG. 7 comprises an input data interface 3 for receiving computer program instructions from a computer program product such as a storage medium 5 or a signal 7, as well as video data to be processed. A processing system is provided, for example, by a CPU 9, a random access memory 11, and a read-only memory 13, which are connected by a bus 15. The CPU 9 controls the overall operation. The RAM 11 is a working memory used by the CPU 9 to execute programs and control the ROM 4, which stores the programs and other data. The processing apparatus of apparatus 1 is configured to perform a method of processing image data defining an image as described herein above. The results of this processing are output by output interface 17.

Although the processing apparatus 1 described above performs processing in accordance with computer program instructions, an alternative processing apparatus can be implemented in any suitable or desirable way, as hardware, software or any suitable combination of hardware and software. It is furthermore noted that the present invention can also be embodied as a computer program that executes one of the above-described methods of processing image data when loaded into and run on a programmable processing apparatus, and as a computer program product, e.g. a data carrier storing such a computer program.

The foregoing description of embodiments of the invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Alterations, modifications and variations can be made without departing from the spirit and scope of the present invention.

Claims

1. A method of processing a first sequence of images and a second sequence of images with a physical computing device to compare the first and second sequences, the method comprising the physical computing device: (a) for each image of the first sequence and each image of the second sequence: processing the image data for each of a plurality of pixel neighbourhoods in the image to generate at least one respective descriptor element for each of the pixel neighbourhoods; andforming an overall image descriptor from the descriptor elements;(b) comparing each image in the first sequence with each image in the second sequence by calculating a distance between the respective overall image descriptors of the images being compared;(c) arranging the calculated distances in a matrix; and(d) processing the matrix to identify similar images.

2. A method according to claim 1, wherein each distance comprises a Hamming distance

3. A method according to claim 1, wherein the physical computing device forms each overall image descriptor from binarised descriptor elements.

4. A method according to claim 1, wherein the matrix is processed by the physical computing device to identify similar images by: processing the matrix to identify local minima in the distances therein;comparing each identified local minima against a threshold, the threshold being determined adaptively according to the number of minima identified per row or column of the matrix, and retaining minima which are below the threshold; andidentifying similar images in accordance with the retained minima.

5. A method according to claim 1, wherein the matrix is processed by the physical computing device to identify similar images by: processing the matrix to identify local minima in the distances therein;detecting a local valley in the matrix values;retaining a subset of the points in the local valley; andidentifying similar images in accordance with the retained points.

6. A method according to claim 1, wherein the matrix is processed by the physical computing device to identify similar images by: processing the matrix to identify local minima in the distances therein;applying a line segment searching algorithm to identify local minima lying on a straight line;applying a hysteretic line segment joining algorithm to fill gaps between identified line segments; andusing the results of the processing to identify matching images.

7. Apparatus operable to process a first sequence of images and a second sequence of images to compare the first and second sequences, the apparatus comprising: an image descriptor generator operable to process each image of the first sequence and each image of the second sequence by:processing the image data for each of a plurality of pixel neighbourhoods in the image to generate at least one respective descriptor element for each of the pixel neighbourhoods; andforming an overall image descriptor from the descriptor elements;an image comparer operable to compare each image in the first sequence with each image in the second sequence by calculating a distance between the respective overall image descriptors of the images being compared;a matrix generator operable to arrange the calculated distances in a matrix; anda similar image identifier operable to process the matrix to identify similar images.

8. Apparatus according to claim 7, wherein the image comparer is operable to calculate a distance between the respective overall image descriptors of the images being compared comprising a Hamming distance.

9. Apparatus according to claim 7, wherein the image descriptor generator is operable to form each overall image descriptor from binarised descriptor elements.

10. Apparatus according to claim 7, wherein the similar image identifier comprises: a local minima identifier operable to process the matrix to identify local minima in the distances therein;a local minima comparer operable to compare each identified local minima against a threshold, the threshold being determined adaptively according to the number of minima identified per row or column of the matrix, and operable to retain minima which are below the threshold; anda similar image identifier operable to identify similar images in accordance with the retained minima.

11. Apparatus according to claim 7, wherein the similar image identifier comprises: a local minima identifier operable to process the matrix to identify local minima in the distances therein;a local valley detector operable to detect a local valley in the matrix values;a point retainer operable to retain a subset of the points in the local valley; anda similar image identifier operable to identify similar images in accordance with the retained points.

12. Apparatus according to claim 7, wherein the similar image identifier comprises: a local minima identifier operable to process the matrix to identify local minima in the distances therein;a line segment searcher operable to apply a line segment searching algorithm identify local minima lying on a straight line;a line gap filler operable to apply a hysteretic line segment joining algorithm to fill gaps between identified line segments; anda similar image identifier operable to use the results of the processing to identify matching images.

13. A computer-readable medium having computer-readable instructions stored thereon that, if executed by a computer, cause the computer to perform processing operations comprising: (a) for each image of a first sequence and each image of a second sequence: processing image data for each of a plurality of pixel neighbourhoods in the image to generate at least one respective descriptor element for each of the pixel neighbourhoods; andforming an overall image descriptor of binarised descriptor elements;(b) comparing each image in the first sequence with each image in the second sequence by calculating a Hamming distance between the respective overall image descriptors of the images being compared;(c) arranging the Hamming distances in a matrix; and(d) processing the matrix to identify similar images.

14. The computer-readable medium according to claim 13, wherein the computer-readable instructions, when executed, cause the computer to calculate the distance between the respective overall image descriptors of the images being compared as a Hamming distance.

15. The computer-readable medium according to claim 13, wherein the computer-readable instructions, when executed, cause the computer to form each overall image descriptor from binarised descriptor elements.

16. The computer-readable medium according to claim 13, wherein the computer-readable instructions, when executed, cause the computer to identify the similar images by: processing the matrix to identify local minima in the distances therein;comparing each identified local minima against a threshold, the threshold being determined adaptively according to the number of minima identified per row or column of the matrix, and retaining minima which are below the threshold; andidentifying similar images in accordance with the retained minima.

17. The computer-readable medium according to claim 13, wherein the computer-readable instructions, when executed, cause the computer to identify the similar images by: processing the matrix to identify local minima in the distances therein;detecting a local valley in the matrix values;retaining a subset of the points in the local valley; andidentifying similar images in accordance with the retained points.

18. The computer-readable medium according to claim 13, wherein the computer-readable instructions, when executed, cause the computer to identify the similar images by: processing the matrix to identify local minima in the distances therein;applying a line segment searching algorithm to identify local minima lying on a straight line;applying a hysteretic line segment joining algorithm to fill gaps between identified line segments; andusing the results of the processing to identify matching images.