The invention relates to a method of characterizing a digital image and to applications thereof.
In numerous applications, such as recognizing images, for example, it is desired to extract characteristics from digital images in order to discriminate between them.
Methods of detecting points of interest or “salient” points in an image are known for this purpose, for example from reference [6]. Such points of interest are points containing a large amount of information and they can be situated, for example, on the outlines of shapes present in the image. These points of interest can be defined as being points corresponding to high frequency zones in the image.
Several techniques are known for characterizing images.
Thus, a first category of techniques categorize color, e.g. using an autocorrelogram or a color histogram, as in references [7], [8], [9], mentioned below.
In a second category, according to reference [10] mentioned below, texture is characterized with the help of Gabor filters.
Finally, in a third category, according to reference [11] mentioned below, form is characterized with the help of a histogram of orientation distribution in the image. An approach of that type is used in the MPEG7 standard.
Those techniques nevertheless present the drawback of not characterizing localized singularities in the image.
The invention seeks to mitigate the drawbacks inherent to the prior art and to obtain a method of characterizing a digital image making use of its points of interest, in order to make the characterization that is performed more discriminating.
To this end, in a first aspect, the invention provides a characterization method for characterizing a digital image in which at least one point of interest in the digital image is detected in association with a region of the image presenting high frequencies, the method being characterized by the steps of:
The invention thus makes it possible to calculate a signature for the local environment of the point of interest, situated in the working region, and to characterize said point of interest.
According to other characteristics of the invention:
the foveal energy signal Eθ,s(k)(n) is equal to:
E
θ,s
(k)(n)=|Σmfθ(k)(m)·ψs(1)(n−m)|2+|Σmfθ(k)(m)·ψs(2)(n−m)|2,
where:
θ designates each of said plurality of I orientations;
{(fθ(k)(n)}1≦k≦K designates said family of said K one-dimensional lines fθ(k)(n) of points extracted for each orientation θ;
K is a natural integer prescribed for each orientation θ;
s designates each level of sub-sampling amongst said J prescribed levels of sub-sampling;
n designates the argument of the foveal energy signals Eθ,s(k)(n); and
{ψj(1), ψj(2)}1≦j≦s is a basis of said foveal wavelets, going as far as sub-sampling level s, of the foveal space Vs of discrete one-dimensional signals {ψj(1)}1≦j≦s, being a basis of odd foveal wavelets and {ψj(2)}1≦j≦s being a basis of even foveal wavelets.
the basis of odd foveal wavelets {ψj(1)}1≦j≦s is defined by:
ψj(1)(n)=Φj(n)·sign(n−½)−231 1/2·Φj−1(n)·sign(n−½),
the basis of even foveal wavelets {ψj(2)}1≦j≦s is defined by:
ψj(2)(n)=Φj(n)−21/2·Φj−1(n),
where:
sign(t) is the sign function of the real variable t, equal to +1 for t greater than 0 and equal to −1 for t less than 0; and
Φj(t) being the even function of the real variable t satisfying;
∫−∞+∞Φ0(t)·dt≠0
the foveal wavelet basis {ψj(1), ψj(2)}1≦j≦s are constructed from spline functions Φj(t) defined by:
the family of one-dimensional lines of points along straight lines from the first two-dimensional working region is extracted using the Bresenham algorithm enabling sloping straight lines to be plotted on a discrete grid;
said foveal signature is calculated by statistically processing foveal energy signals on the lines and on their arguments;
said foveal signature SI is equal to:
SI=(μisσis)1=i=I, 1=s=J
with
μis=(1/NM)·ΣkΣnEi,s(k)(n) corresponding to the mean of the foveal energy signal Ei,s(k)(n) for the orientation i and the sub-sampling level s over the working region;
σis=(1/NM)·[ΣkΣn(Ei,s(k)(n)−μis)2]1/2 corresponding to the standard deviation of the foveal energy signal Ei,s(k)(n) for the orientation i and the sub-sampling level s over the working region; and
NM corresponds to the size of the working region;
the first two-dimensional working region surrounds the detected point and is of a size that is less than one-tenth of the size of the digital image;
the first two-dimensional working region surrounds the detected point and is of a size less than or equal to 15 pixels×15 pixels;
said plurality of different prescribed orientations in the digital image subdivide the angular sector going from −90° to +90° relative to the horizontal in equal angles;
the number of said orientations is eight; and
the foveal signature is associated with the digital image and is stored in memory in association with said digital image.
In a second aspect, the invention provides the application of the characterization method to digital images for comparison purposes, in which the foveal signatures calculated for the digital images are compared with one another.
For example, a reference foveal signature is calculated for a reference image and, in order to compare foveal signatures, a distance relative to the reference signature is calculated for the foveal signatures calculated for the other images, the image closest to the reference image being the image having the shortest distance relative thereto.
In another example, a prescribed number of the images closest to the reference image are selected.
A third aspect of the invention is the application of the characterization method to recognizing shapes in the digital image, in which the foveal signature is used to characterize a shape situated in the working region.
For example, this application relates to recognizing faces, in which the foveal signature is used to characterize a feature of the face, situated in the working region.
A fourth aspect of the invention is the application of the characterization method to indexing digital images by partial request, in which digital images containing an item specified by the request are extracted from a digital image base, a digital image being determined as containing said item when the distance of the foveal signature calculated for said image relative to the foveal signature calculated for the item situated at least in part in the working region is less than a prescribed threshold.
A fifth aspect of the invention is the application of the characterization method to classifying digital images by comparing their foveal signatures.
The invention can be better understood on reading the following description given purely by way of non-limiting example and with reference to the accompanying drawings, in which:
a and 6b show experimental results of the method,
By way of example, the invention comprises the following steps:
detecting salient points;
for each salient point, extracting K lines (preferably 8 lines) from K orientations of a neighborhood of the salient point (e.g. a 5×5 neighborhood);
projecting each line on a foveal wavelet basis (e.g. obtained using splines);
obtaining a set of (K×number of levels) coefficients (foveal response amplitude): e.g. (24× number of points in each line) coefficients;
carrying out a statistical study on these coefficients by level and by orientation (mean, standard deviation);
storing this information for each salient point of the image, associated with the image; and
During a first step of the method of the invention, one or more points of interest PI are detected amongst the P points of the digital image I.
Below, it is assumed that the point of interest PI of
Naturally, points of interest can be detected using methods other than that described in Appendix 1 and
During a second step, a two-dimensional working region RPI around the point of interest PI is selected in the image I. By way of example, this working region RPI is rectangular or square parallel to the sides of the image I and is of a size considerably smaller than the image I. By way of example, the region RPI is centered on the point of interest PI, e.g. having odd numbers of pixels in its rows and in its columns. For example, in
During a second step, lines fθ(k)(n) are extracted from the working region RPI along straight lines of orientation θ relative to the horizontal. This extraction is performed for a plurality of prescribed orientations θ. By way of example, these orientations θ are spaced apart from one another by the same angle. For example, they are equal to:
For example, in
By way of example, the extraction of the lines fθ(k)(n) is shown for
in
The lines of points are, for example, traced using the Bresenham algorithm of reference [12] mentioned below, that enables lines to be traced on an image made up of discrete points. That algorithm is used in particular for displaying sloping straight line segments on a computer screen. For a line y=a·x+b that is to be traced between two points (x1, y1) and (x2, y2) on the discrete raster of screen points, the coordinates of the points of the segment are given by y=round(ax+b) for all x lying in the range x1 to x2, with the rounding function returning the closest natural integer.
Thus, the algorithm for tracing inclined straight line segments on a discrete raster can be implemented as follows:
According to the invention, K lines fθ(k)(n) are extracted from the working region RPI for each orientation θ, representing K parallel straight lines at said orientation θ, crossing the region RPI, and each offset by one pixel in the abscissa and/or the ordinate direction. K is a natural integer depending on each orientation θ, or constant for all I orientations θ. Extraction is performed in such a manner as to cover for each orientation θ all of the points of the region RPI. Consequently, for each orientation θ, each point of the working region RPI belongs to a single line fθ(k)(n) as extracted in this way. Naturally, line fθ(k)(n) could be extracted from the region RPI that do not cover all of the points thereof. According to the invention, provision is made for at least one line containing the point of interest PI in the orientation θ to be extracted from the working region RPI (fθ(5) in
The family {fθ(k)(n)}1≦k≦K of lines fθ(k)(n) is determined in the working region RPI for each of the selected I orientations θ. The lines fθ(k)(n) are thus one-dimensional discrete signals having as argument n giving the number of the point in the line and having as value the value of the point n of said line in the working region RPI, e.g. its gray level in the image I.
During a third step, for each extracted line fθ(k)(n), a foveal energy signal Eθ,s(k)(n) is calculated where s indicates one sub-sampling level taken from amongst J prescribed sub-sampling levels for which the signal is calculated, where −s gives a level of resolution.
The theory of foveal wavelets is summarized below.
For a rectangular image of size A×B, sub-sampling leads to a rectangular image of size A′×B′ smaller than the size A×B.
Reference can be made to Appendix 2 and to reference [5] mentioned below for foveal wavelet theory.
Foveal wavelets are constituted for all dyadic expansions of a symmetrical or even wavelet ψ(2) and of an antisymmetrical or odd wavelet ψ(1), these wavelets being constructed from a mother function Φ.
The foveal energy signal is equal to:
E
θ,s
(k)(n)=|<fθ(k)(m),ψs,m(1)(n)>|2+|<fθ(k)(m),ψs,m(2)(n)>|2
where:
ψs,m(1)(n)=ψs(1)(n−m)
and
ψs,m(2)(n)=ψs(2)(n−m),
by analogy with the continuous case in which gs,u(t)=2−s/2·g(2−s(t−u)) is a function of the real variable t, obtained by translation of u and expansion of the sub-sampling level 2−s.
Consequently,
E
θ,s
(k)(n)=|Σmfθ(k)(m)·ψs(1)(n−m)|2+|Σmfθ(k)(m)·ψs(2)(n−m)|2.
During a fourth step, the foveal signature SI is calculated from foveal energy signals Eθ,s(k)(n)) in order to characterize the working region RPI and the digital image I.
In this foveal signature SI, the variables k representing the various lines extracted for a given orientation θ in the working region RPI, and the argument n are eliminated from the foveal energy signal Eθ,s(k)(n) so that the foveal signature SI depends only on the selected I orientations θ and on the J prescribed sub-sampling levels. By way of example, the foveal signature SI might be equal to:
SI=(μisσis)1=i=I, 1=s=J,=(μ00, σ00, μ10, σ10 . . . μIJ, σIJ)
with
In the above, m, n, k, K, s, j, I, J, i are natural integers.
For a working region that is square, NM=N2.
The foveal signature SI can be used in various ways, as described below.
By way of example, the foveal signature SI associated with the digital image is stored in memory in association therewith.
In a first application, a plurality of digital images are compared with one another by means of their foveal signatures.
The foveal signatures are compared by means of a conventional L2 distance, for example.
For example, images are compared with a reference image, in order to determine which images are the closest or the most similar to the reference image, which is determined as being the image for which the foveal signature has the shortest distance relative to the foveal signature of the reference image.
Thus, by way of example, the reference image is the large image I1 on the left in
If it is desired to select images of the same person from an image base, the foveal signature of the reference image I1 is calculated and it is compared with the foveal signatures of other images stored in the base, in order to extract those that have the shortest distances relative to the foveal signature of the reference image I1. Thus, this extraction produces again the image I1 at distance 0, together with images I2, I3, I4, I5, and I6 at short and increasing distances from the reference image I1 these images corresponding to images Nos. 102, 139, 274, 303, 171, and 181 in a base of 700 images. It can thus be seen that the images I2, I3, I4, I5, and I6 are images of the same person as in the image I1 but with different facial expressions than in the image I1, these images being taken in succession while the person was speaking. In contrast, an image of some other person would have a signature with a much greater distance from the reference image I1 than the distances of images I2 to I6.
A second application is recognizing shapes in an image. In this application, the point of interest is detected as forming part of the outline of the shape to be recognized. The working region then contains in full or in part the shape to be recognized and the foveal signature of all or part of this shape to be recognized is obtained. This shape may be a feature of a face, such as an eye, the mouth, or the nose, which are zones having outlines that are well-marked. Since the features of the face of a person are characteristic of that person, this application makes it possible to search for people in images or to search for images containing a particular person.
A third application concerns indexing digital images by partial requests, thus making it possible to extract from an image base all of the images that contain an item similar to a request item specified by the user.
A fourth application is classifying images, enabling images in an image base to be grouped together in a small set of groups, each group presenting some particular theme (landscape, sport, etc. . . . ). This application can help in browsing through image bases, for example bases of images created by users with their digital cameras.
Naturally, the method of the invention operates with any mathematical tool having the same characteristics as the above-described foveal wavelet basis.
In addition, the method of the invention applies to characterizing any digital item, which may be a two-dimensional image as described above, but which could also be a three-dimensional image, in which case the lines should be considered as planes, and the steps of the method should then be adapted accordingly.
An object of the invention is thus to detect salient points of an image I. These points correspond to the pixels of I that belong to high frequency regions. To do this, use is made of wavelet theory [1], [2], [3], and [6].
The wavelet transformation is a multi-resolution representation of the image that enables the image to be expressed at different resolutions ½, ¼, etc. Thus, at each resolution level 2j (j≦−1), the wavelet transformation represents the image I of size n×m=2k×2l (k, lεZ) in the form:
Each of these images is of size 2k+j×2l+j.
Each of these three images is obtained from A2
The invention then consists firstly in selecting a wavelet basis and a minimum resolution level 2r (r≦−1). Once the wavelet transformation has been performed, we propose going through each of the three detail images D2
Thus, a coefficient having large salience corresponds to a region of I presenting high frequencies. A wavelet coefficient of large modulus at resolution 2r (r≦−1) corresponds to an outline of the image A2
On the basis of the salience map as constructed in this way, the invention proposes a method serving to select the pixel from the 2−r×2−r pixels of I that is the most representative of said zone.
The wavelet transformation is a powerful mathematical tool serving to perform multi-resolution analysis of a function [1], [2], [3].
In the invention, the functions under consideration are digital images, i.e. discrete two-dimensional functions. Without loss of generality, we assume that the images being processed are samples on a discrete grid of n rows and m columns with a value in brightness space that is sampled over 256 values. Furthermore, it is assumed that n=2k (kεZ) and that m=2l (lεZ).
If the original image is written I, then:
As mentioned, the wavelet transformation of I obtains a multi-resolution representation of I. At each resolution level 2j (j≦−1), the representation of I is given by an approximate image, A2
The wavelet transformation requires a scale function Φ(x) to be selected and also a wavelet function Ψ(x). From these two functions, a scale filter H and a wavelet filter G are derived having respective impulse responses h and g defined by:
h(n)=φ2
∀nεZ,
Φij(x)=Φ(2jx−i)
g(n)=ψ2
∀nεZ,
Ψij(x)=Ψ(2jx−i)
The filters constituting the mirrors of H and G are written respectively {tilde over (H)} and {tilde over (G)} (i.e. {tilde over (h)}(n)=h(−n) and {tilde over (g)}(n)=g(−n)).
It can then be shown [1] (cf.
A2
D2
D2
D2
Once the wavelet transformation has been performed as far the resolution 2r (r≦−1), the following are available:
an approximate image A2
three detail images D2
A tree of wavelet coefficients is then constructed using the Zerotree technique [4]. The trees are constructed as follows (cf.
each pixel p(x,y) of the image A2
each root p(x,y) is allocated to three child nodes designated by the wavelet coefficients of the three detail images D2
because of the sub-sampling by a factor of two that is performed by the wavelet transformation on each change of resolution, each wavelet coefficient α2
A tree is constructed recursively in which each wavelet coefficient α2
Once the tree has been constructed, each wavelet coefficient α2
From the tree obtained by the preceding step, we propose constructing a set of −r salience maps (i.e. one salience map per level of resolution). Each salience map S2
It should be observed that each wavelet coefficient privileges one particular direction (horizontal, vertical, or oblique) depending on the detail image to which it belongs. However we have decided not to privilege any particular direction so we have therefore merged the information contained in the three wavelet coefficients α2
Furthermore, the salience of each coefficient of resolution 2j must take account of the salience of its descendents in the coefficient tree.
In order to take all of the properties into account, the salience of a coefficient located at (x,y) at resolution 2j is given by the following recursive relationship:
Equation 1: expression for the salience of a coefficient where:
max(D2
αk (0≦αk≦1) serves to adjust the importance of the salience coefficients depending on the level of resolution. It should be observed that:
it is important to observe that the salience values are normalized (i.e. 0≦S2
As can be seen in equation 1, the salience of a coefficient is a linear relationship of wavelet coefficients. As mentioned, we consider salient points as image pixel belonging to high frequency regions. A wavelet coefficient α2
Thus, the formulation of the salience of a coefficient given in the equation . . .
Once the construction of salience maps has been completed, we propose a method of selecting the most salient points in the original image.
To do this, we construct a tree of salient values from the −r salience maps that have been constructed. In a manner analogous to constructing the tree of wavelet coefficients, we construct 2k+l+2r trees of salient coefficients, each having at its root a coefficient of S2
In order to locate the most salient points in I, we proceed as follows:
1. sorting the 2k+l+2r salient values present in S2
2. selecting the branch having the greatest salience in each of the 2k+l+2r trees as sorted in this way.
In order to select this branch, we propose working through the tree from the root in which we select at each level of the tree the child node having the most important salience value (cf.
Salient branch={s2
with
(xk,yk)=ArgMax{s2
From the most salient branches of each tree, the pixel of I that is selected as being most representative of the branch is located at 2x−r, 2y−r). In practice, only a subset of the 2k+l+2r trees is passed through. In numerous applications, a search is made for a fixed number n of salient points. Under such circumstances, it suffices to pass through only the n trees having the n most salient roots.
As mentioned, it is necessary to begin by selecting a wavelet basis and a minimum resolution level 2r (r≦−1). For this particular implementation, we propose using the Haar basis and r=−4.
The Haar basis is defined by:
for the scale function and by:
for the wavelet function.
In this step, no parameter is required. The process is thus as described.
In this step, it is necessary to select the parameters αk (−1≧k≧r) in order to adjust the importance given to the salience coefficients depending on the resolution level to which they belong.
In this particular implementation, we propose using:
αk=(−1/r)∀kε[r,−1]
The results obtained on natural images using the parameters proposed in this particular implementation are shown in
Definition 1: a space V0⊂L2(R) is a foveal space centered on 0 if there exists a finite generating family {ψm} 1≦m≦M belonging to L2(R) such that {ψjm} 1≦m≦MjεZ is a Riesz basis of V0 and such that:
fεV
0
f·1[0,+∞[εV0
(where R designates the set of real numbers and Z the set of positive and negative natural integers).
Definition 2: let f′:R→R defined from f by ∀tεR, f′(t)=f(t+u), be a foveal space centered on u if it satisfies fεVuf′εV0.
The projection of a function f on Vu is then a function of resolution that decreases on going away from the foveal u, whence the analogy with the foveal nature of human vision.
Let φ be an even function with ∫φ(t)·dt≠0. V0 is resolved into two sub-spaces V0e and V0o made up respectively of even functions and odd functions. By defining:
ψ1(t)=(φ(t)−φ(2t))·sign(t)=
ψ2(t)=φ(t)−2·φ(2t)
where sign(t) is the sign function of the real variable t, equal to +1 for t greater than 0 and equal to −1 for t less than 0, it is shown in reference [5] that {ψj1} jεZ is a Riesz basis of V00 and that {ψj2} jεZ is a basis of V0e. Since V0o and V0e are orthogonal supplementaries, it can be deduced therefrom that {ψj1,ψj2,} jεZ is a basis of V0 (foveal wavelet basis).
Since the bases {ψj1} jεZ and {ψj2} jεZ are not always orthogonal on construction, it is possible to make use of the Bram-Schmidt orthogonalization technique.
In this section, we describe the procedure of constructing a basis of foveal wavelets that are discrete in n, as developed in reference [5]. The signals under consideration are sampled at t=n−½. The origin t=0 is then to be found between n=0 and n=1. Furthermore, the following notation is used below:
Definition 3: if a family of discrete foveal windows {φj[n]} 0≦j≦s satisfies:
φj[n]=φj[1−n]
there exists K>0 such that the support of φj[n] is [−K2j+I,K2j]
there exists C>0 such that
then the family {φj−,φj+} 0≦j≦s is a basis of the foveal space V0,s, said foveal space V0,s corresponding to the discrete functions sub-sampled up to sub-sampling level j=s.
By analogy with the continuous case of the paragraphs above, if the following are defined:
ψj1[n]=
ψj2[n]=φj[n]−21/2φj−1[n]
it is shown in reference [5] that the family {
On construction, these bases are not necessarily orthogonal. Nevertheless, it is also possible to apply a Gram-Schmidt orthogonalization procedure.
In the above, the mother functions φ are selected by way of example from spline functions.
Spline functions are in very widespread use in the field of interpolation. They make it possible to interpolate a sequence of points by pieces of polynomial functions. The order of the interpolating polynomials corresponds to the order of the spline function used for interpolation.
Definition 4: a spline function of order 0 is φ0=1[−1,1]. Spline functions of higher order are defined recursively by φP(t)=∫−∞t(φp−1(2x)−φp−1(x))sign(x)dx.
It is then possible to use these functions to construct foveal wavelet basis. An example of a foveal wavelet basis obtained using 0 order splines (staircase functions) is shown in
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/FR2004/001280 | 5/24/2004 | WO | 00 | 6/8/2007 |