This application is based upon and claims priority of Japanese Patent Application No. 2004-045135, filed on Feb. 20, 2004, the contents being incorporated herein by reference.
1) Field of the Invention
The present invention relates to a similarity determination program, a multimedia-data search program, a similarity determination method, and a similarity determination apparatus. In particular, the present invention relates to a similarity determination program, a multimedia-data search program, a similarity determination method, and a similarity determination apparatus for determining a degree of similarity between multimedia-data items.
2) Description of the Related Art
In the field of computers, conventionally searches have been conducted based on character strings and numerical values which represent, for example, keywords. However, with the recent widespread use of the Internet, digital cameras, mobile telephones, and the like, interest in searches for multimedia data such as images, sounds, and documents is growing.
The search based on an annotation or a keyword is a method for searching for multimedia-data items. In this method, a group of keywords called an annotation is attached to each image for searching. For example, the keywords are a text phrase such as “deep blue sea shot in Okinawa,” or words such as “Okinawa” and “sea.” Conventionally, keyword searches for images have been conducted based on keywords attached to the images. However, the above method for searching based on annotations has two problems.
The first problem is that the human cost for attachment of annotations is great. Further, the attachment of annotations is becoming more difficult with the rapid increase in the numbers of images. The second problem is that the features of the images cannot be completely described by the annotations. Actually, the images have various features such as colors, shapes, and patterns, which cannot be completely characterized by characters.
Therefore, a method for searching for a multimedia-data item by automatically extracting a feature of the multimedia-data item, and using a feature space, a color histogram, and a feature quantity is known. The image data is multimedia data to which this method can be applied. In the similarity search of image data, features such as colors and shapes are automatically extracted as numerical values without human assistance. A typical method frequently used in the case of colors is a method called the color histogram, where the histogram means a bar graph.
In the color histogram, pixels are classified into n colors, and the number of pixels having each color is extracted, where n is a natural number. Then, the feature concerning each color is represented by the proportion of the number of pixels having the color to the total number of the pixels in the entire image. A quantity which represents a feature, as the above proportion, is called a feature quantity. The above number n of color classification should be a rather large number such as 64.
Consider a simple case where n=3, and the pixels are classified into the three primary colors, red, green, and blue. In this case, a feature quantity of an image can be represented by coordinates in a three-dimensional feature space.
In the above case, the point B represents an image which does not contain red, and the point C represents an image which does not contain green and blue. When attention is focused on the distances between three points, images are regarded to be similar when the distance between the points representing the images is small. As understood from
As described above, the basic concept of the similarity search is that one-to-one correspondences are set between images and points in a feature space, and images corresponding to points nearer to each other are regarded as more similar to each other.
The similarity search as above are used in various fields. For example, the similarity search using a feature space is widely used in the fields of sounds and documents as well as the fields of images including movies. For example, in the case of similarity search in the fields of sounds, when an introduction to a piece of music is inputted, the piece of music is searched for. In the case of similarity search in the fields of documents, a frequently used feature quantity of a document is a product of a frequency of occurrence of a word contained in the document and a logarithm of the total number of documents divided by the number of documents containing the word. In this case, the dimension of the feature space is the number of words considered as bases. Therefore, the dimension of the feature space is very great. Thus, the similarity search using a feature quantity are widely used for a variety of multimedia data.
As described above, in the similarity search, a feature of an object such as a document or an image as a multimedia data item is associated with a vector (point) in a multidimensional space called a feature space, where the coordinates of the point indicate the feature quantities of the object. In most cases, the feature quantities are represented in floating-point format. That is, in most cases, the feature space is an n-dimensional space with coordinates represented by real numbers.
Hereinbelow, the meanings of the terms “base” and “feature vector,” which will be frequently used in this specification, are explained.
[Base and Normalized Orthogonal Bases]
As is well known, an arbitrary vector in a so-called vector space such as a Euclidean space can be represented by using n vectors called base vectors when the dimension of the vector space is n. In the case of a three-dimensional Euclidean space, the following three vectors e1, e2, and e3 are base vectors.
An arbitrary vector v can be expressed as a so-called linear combination of the above base vectors as follows.
The set of the above n base vectors is called a basis. That is, the basis corresponds to the coordinate system, and conversely the coordinate system can be conceived based on the basis.
The set of the base vectors e1, e2, and e3 in the above example are called a (normalized) orthogonal basis. The expression “orthogonal” means that the base vectors ei and ej are perpendicular to each other, where i and j are natural numbers, and i≠j. In addition, the expression “normalized” means that the length of each of the base vectors is “1.”
[Feature Vector]
Hereinafter, the vector expressed by the following linear combination (9) is called an entire-feature vector corresponding to an object,
f=c1e1+c2e2+ . . . +cnen (9)
where the dimension of the feature space is n, the vectors e1, e2, e3, . . . , en are the base vectors of the feature space, and the feature quantities of the object are c1, c2, C3, . . . Cn. The entire-feature vector represents an overall feature of the object. The distance between objects is measured as the distance between the entire-feature vectors corresponding to the objects, whereas the feature quantities are quantities indicating respective features of the object.
[Orthogonal Basis+Euclidean Distance]
In the most basic method, n feature quantities are represented as a point x in an n-dimensional Euclidean space as expressed below.
The distance between two points is represented as an Euclidean distance. That is, when another point is expressed as
the distance d between the above points is expressed as follows.
d={square root}{square root over ((x1−y1)2+(x2−y2)2+ . . . +(xn−yn)2)}. (12)
However, the above method has the problems explained below.
For example, consider a case where the number of colors is twelve. At this time, the twelve colors can be expressed by a hue circle.
Distance between Red and Green=21/2
Distance between Red and Red-orange=21/2
Distance between Red-orange and Green=21/2
Since the distance between any two of the images is identical, numerically the degree of similarity between each pair of the images is regarded as identical. However, when human beings see the above three colors, blue-green and red do not look similar, whereas red and red-orange look similar. That is, the similarity perceived by human beings is not reflected in the manner in which the points are arranged in the feature space. This problem occurs not only in the cases of images, and can generally occur in every type of multimedia data. An example of text data is indicated below.
[Example of Document]
Although normally each document contains a number of words, three simple documents each of which is composed of only one word are considered below for simple explanation of representation of documents in a feature space.
Assume that the set of the words {premier, chancellor, tennis} are considered as bases, and the feature quantity corresponding to the ith dimension is the number of occurrences of the ith word in the bases. In this case, the above documents can be represented by the following vectors, respectively.
As in the case of the images, the distance between each pair of the documents is calculated as follows.
[Orthogonal Basis+Quadratic-form Distance]
Various methods have been proposed for solving the aforementioned problem in the use of the orthogonal basis and the Euclidean distance. Basically, the orthogonal basis is also used in the proposed methods. However, in the proposed methods, distance functions d(x, y) in which similarity between features is reflected are used instead of the aforementioned Euclidean distance, where d(x, y) indicates a distance between two points x and y.
Although it is easy to calculate the Euclidean distance, which is used in the aforementioned method based on the orthogonal basis and the Euclidean distance, the distance functions used in the proposed methods are generally complex, and in most cases it takes much time to perform calculation based on the distance functions. Therefore, it is necessary to solve this problem.
Hereinbelow, the quadratic-form distance, which is obtained by the most typical one of the above distance functions, is explained.
When a vector x is expressed as
the length ∥x∥ of the vector x is defined by using a matrix S as
where tx is a transposed vector of the vector x. For example, when x is a column vector, the transposed vector tx is a row vector. Therefore, the distance d (x, y) between the vectors x and y is obtained as the length of a difference vector between the vectors x and y,
d(x, y)2=t(x−y)S(x−y). (21)
The matrix S is a matrix indicating similarity between features, and is hereinafter referred to as a similarity matrix. Each element Sij of the matrix is called a degree of similarity, and indicates a degree of similarity between the ith and jth features. When the matrix is a unit matrix, the quadratic-form distance is the normal Euclidean distance. In this sense, the quadratic-form distance is a generalization of the Euclidean distance. The method based on the quadratic-form distance is used in the QBIC (Query By Image Contact) system of IBM Corporation, where QBIC is a trademark of IBM Corporation. For example, see James Hafner et al., “Efficient Color Histogram Indexing for Quadratic Form Distance Functions,” IEEE Trans. Pattern Anl. Machine Intell. 17(7), pp. 729-736 (1995).
[Oblique Basis+Euclidean Distance]
A similarity search method using an oblique basis corresponding to an oblique coordinate system has also been proposed. As well known in mathematics, the angles between oblique base vectors are not required to be 90 degrees. Coordinates based on oblique base vectors, which are not necessarily perpendicular to each other, are called oblique coordinates, and are widely used in a number of technical fields as well as in mathematics and physics. In this specification, a basis constituted by oblique base vectors is referred to as an oblique basis.
In order to represent the point P by oblique coordinates, points A and B are obtained, where the point A is an intersection of a line passing through the point P and being parallel to the oblique base vector e2 and a line containing the oblique base vector e1, and the point B is an intersection of a line passing through the point P and being parallel to the oblique base vector e1 and a line containing the oblique base vector e2. Hereinafter, generally a vector from a point X to a point Y is referred to as a vector XY. In the example of
The oblique coordinates and the orthogonal coordinates of the point P have the following relationship.
The rightmost side of the above equations expresses a product of a matrix and a vector. In this specification, the above matrix is referred to as a feature-vector conversion matrix (from oblique coordinates to orthogonal coordinates). The above matrix, which is denoted T, can be produced by arranging the elements of the oblique base vectors e1 and e2 in order, i.e., T=(e1 e2).
The basic concept of the method using the oblique basis is to reflect similarity in the distances between oblique base vectors. In this method, the distance function of the Euclidean distance is uses as it is. In this case, calculation of the distance is easy. In addition, the distance between two objects obtained by this method is basically identical to the quadratic-form distance. That is, from the viewpoint of precision, the distance between two objects according to the method using the oblique basis is basically equivalent to the quadratic-form distance. However, the amount of data required to be stored in the method using the oblique basis is almost half of the amount of data required to be stored in the method using the quadratic-form distance. Further, the amount of data required to be stored affects the processing speed. Therefore, the small amount of data required to be stored is an advantage of the method using the oblique basis.
In a method used in an idea of a prototype of the oblique basis, a new feature vector is produced by converting a feature vector represented by orthogonal coordinates by use of the aforementioned matrix T, instead of a linear combination based on the oblique basis. For example, see Jack. S. N. Jean, “A New Distance Measure for Binary Images,” Proceedings of the 1990 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '90), Apr. 3-6, 1990, Vol. 4, pp. 2061-2064 (paper#: 2061).
Concerning the method using the oblique basis, the present assignee has filed a Japanese patent application No. 2003-172217, “Apparatus for Similarity Search of Image Data and Method for Determining Similarity in the Apparatus,” on Jun. 17, 2003.
Further, a technique for searching for a similar image, which is called the Earth Mover's Distance (EMD) technique, is known. Hereinbelow, this technique is briefly explained.
According to the EMD technique, similarity between images is determined based on distances between a plurality of points. The definition of this distance is briefly explained below by comparison to a transportation problem between holes and masses of earth. This distance is based on a solution of the transportation problem.
First, sets x and y each called a signature are defined in correspondence with respective images, as follows.
x={(p1, x1), (p2, x2), . . . (pm, xm)}
y={(q1, y1), (q2, y2) , (qn, yn)} (26)
In the above definitions, the set x corresponds to a set of masses of earth, the set y corresponds to a set of holes, and m and n are natural numbers. Generally, it is unnecessary that m and n are identical. This flexibility is one of the features of EMD. In addition, pi and qj each indicate a point in a space in which an arbitrary distance is defined. It is assumed that xi indicates an amount of earth placed at the point pi, yj indicates the capacity of a hole excavated at the point qj, and the total amount of the earth is sufficient to fill all of the holes. When the distance between the points pi and qj is denoted dij, and the amount of earth transported from the point pi to the point qj, is denoted fij, the cost for filling all of the holes is expressed as
Then, the amounts fij which minimize the above cost are obtained, and the EMD between the sets x and y is defined as
In the formula (28), the denominator is provided for normalization, which prevents the tendency toward selection of a signature corresponding to a small total amount. In the technique disclosed by Yosshi Rubner et al. (“A Metric for Distribution with Applications to Image Databases,” Proceedings of the 1998 IEEE International Conference on Computer Vision, January 1998, pp. 59-66), the EMD is applied to the colors and textures (patterns) of images. In addition, the above definition of the EMD can cope with partial matching in the case where the total numbers of feature quantities in the two signatures are different. Unlike the aforementioned methods using the color histogram, the EMD technique is so flexible that the numbers m and n of elements can be arbitrarily designated. In addition, according to the EMD technique, it is easy to calculate the lower limit of the distance.
However, the conventional techniques lack discriminability.
On the other hand, the subheading “SELECT DISSIMILAR OBJECT” under the column heading “TENDENCY TO . . . ” means a tendency to determine a dissimilar object to be similar. When one of the methods has this tendency, a blank circle (O) is indicated in the row corresponding to the method under the subheading “SELECT DISSIMILAR OBJECT”. The method using the orthogonal basis and the quadratic-form distance and the method using the oblique basis and the Euclidean distance, in both of which similarity between features is taken into consideration, have the tendency to determine a dissimilar object to be similar. According to these methods, in extreme cases, completely different objects are determined to be similar. In this specification, this drawback is called “lack of discriminability,” and the object of the present invention is to overcome the lack of discriminability as indicated later.
Referring back to
As illustrated in
In the case of the EMD technique, partial matching is performed when the total numbers of feature quantities in the two signatures are different. When comparison to the transportation problem between a set of masses of earth and a set of holes is used again for explanation, comparison processing for similarity search is completed at the time all of the masses of earth are exhausted for filling the holes, even if a portion of the holes is left unfilled. Therefore, when all of features of a first object are similar to a portion of features of a second object, the first and second objects are determined to be similar, even if the first object is dissimilar to the second object as a whole. That is, in the case where the total numbers of feature quantities of two objects are different, it is advantageous that partial matching is possible. However, when similarity between entire objects is considered, an object which is dissimilar as a whole can be selected, i.e., discriminability is impaired.
The present invention is made in view of the above problems, and the object of the present invention is to provide a similarity determination program, a multimedia-data search program, a similarity determination method, and a similarity determination apparatus, in which discriminability in determination of similarity by comparison of entire multimedia-data items is improved.
In order to accomplish the above object, a similarity determination program is provided for determining similarity between multimedia-data items by using a computer. The similarity determination program makes the computer comprise the functions of: an oblique-base-vector storage unit which stores oblique base vectors being respectively provided in correspondence with representative features of the multimedia-data items, and respectively indicating the representative features by directions of the oblique base vectors; an input unit which inputs first and second multimedia-data items to be compared; a vector-set generation unit which analyzes the first and second multimedia-data items inputted by the input unit, determines feature quantities, respectively corresponding to the representative features, of each of the first and second multimedia-data items so that each of the feature quantities indicates a degree of inclusion of information corresponding to one of the representative features in each of the first and second multimedia-data items, generates first and second feature vectors for the first and second multimedia-data items, respectively, by multiplying each of the oblique base vectors by one of the feature quantities corresponding to the oblique base vector for each of the first and second multimedia-data items, and forms first and second vector sets respectively corresponding to the first and second multimedia-data items so that the first feature vectors constitute the first vector set, and the second feature vectors constitute the second vector set; a vector-pair generation unit which makes the first and second vector sets have an identical number of feature vectors, and generates a plurality of vector pairs by establishing one-to-one correspondences between the feature vectors in the first vector set and the feature vectors in the second vector set; a vector-to-vector distance calculation unit which calculates distances in the plurality of vector pairs, respectively, where each of the distances indicates a first degree of similarity between two feature vectors forming one of the plurality of vector pairs; a degree-of-similarity calculation unit which calculates a second degree of similarity between the first and second multimedia-data items by summing the distances calculated by the vector-to-vector distance calculation unit; and an output unit which outputs the second degree of similarity calculated by the degree-of-similarity calculation unit.
Further, in order to accomplish the aforementioned object, a multimedia-data search program for searching multimedia-data items by using a computer is provided. The multimedia-data search program makes the computer comprise the functions of: an oblique-base-vector storage unit which stores oblique base vectors being respectively provided in correspondence with representative features of multimedia-data items, and respectively indicating the representative features by directions of the oblique base vectors; a vector-set storage unit which stores first vector sets each including first feature vectors representing features of each of first multimedia-data items which are to be searched; an input unit which inputs a second multimedia-data item as a search condition; a vector-set generation unit which analyzes the second multimedia-data item inputted by the input unit, determines feature quantities, respectively corresponding to the representative features, of the second multimedia-data items so that each of the feature quantities indicates a degree of inclusion of information corresponding to one of the representative features in the second multimedia-data item, generates second feature vectors for the second multimedia-data item by multiplying each of the oblique base vectors by one of the feature quantities corresponding to the oblique base vector, and forms a second vector sets constituted by the second feature vectors; a vector-pair generation unit which makes the first vector sets and the second vector sets have an identical number of feature vectors, and generates a plurality of vector pairs by establishing one-to-one correspondences between the feature vectors in each of the first vector sets and the feature vectors in the second vector set; a vector-to-vector distance calculation unit which calculates distances in the plurality of vector pairs, respectively, where each of the distances indicates a first degree of similarity between two feature vectors forming one of the plurality of vector pairs; a degree-of-similarity calculation unit which calculates a second degree of similarity between the second multimedia-data item and each of the first multimedia-data items by summing the distances calculated by the vector-to-vector distance calculation unit; and an output unit which outputs information identifying one of the first multimedia-data items corresponding to a highest value of the second degree of similarity calculated by the degree-of-similarity calculation unit.
The above and other objects, features and advantages of the present invention will become apparent from the following description when taken in conjunction with the accompanying drawings which illustrate preferred embodiment of the present invention by way of example.
In the drawings:
An embodiment of the present invention is explained below with reference to drawings.
First, an outline of the present invention which is realized in the embodiment is explained, and thereafter details of the embodiment are explained.
The oblique-base-vector storage unit 1 stores oblique base vectors 1a, each of which is arranged in correspondence with one of a plurality of representative features of multimedia-data items, and represents the representative feature by the direction of the oblique base vector. For example, when the multimedia-data items are image data items, a plurality of representative colors are defined as the representative features. The colors constituting a hue circle can be used as the representative colors. In this case, for example, each vector having a unit length and pointing to the position of one of the representative colors can be defined as one of the oblique base vectors 1a.
The input unit 2 inputs two multimedia-data items 2a and 2b which are to be compared. For example, the multimedia-data items 2a and 2b are designated by a user through an input device such as a keyboard, and the input unit 2 inputs the multimedia-data items 2a and 2b into the vector-set generation unit 3.
The vector-set generation unit 3 analyzes each of the multimedia-data items 2a and 2b, and determines a feature quantity indicating a degree of inclusion of information corresponding to each representative feature. Then, the vector-set generation unit 3 generates a feature vector by multiplying one of the oblique base vectors corresponding to each representative feature by the feature quantity corresponding to the representative feature, and sorts the generated feature vectors into groups corresponding to the multimedia-data items 2a and 2b, which are denoted vector sets 3a and 3b.
For example, in the case where the multimedia-data items are image data items, correspondences between the representative colors and colors which the image data items represent are predefined by the vector-set generation unit 3. The vector-set generation unit 3 determines the proportion of pixels corresponding to each of the representative colors in each image as the feature quantity corresponding to the representative feature (representative color).
The vector-pair generation unit 4 generates vector pairs each of which is formed of a first feature vector extracted from one of the vector sets 3a and 3b and a second feature vector extracted from the other of the vector sets 3a and 3b. For example, in order to generate the vector pairs, the vector-pair generation unit 4 extracts the first feature vector from the one of the vector sets 3a and 3b, and then extracts the second feature vector from the other of the vector sets 3a and 3b so that the second feature vector is directed in a direction nearest to the direction of the first feature vector among the feature vectors in the vector set 3b. It is possible to estimate the proximity of the directions of the feature vectors (being respectively extracted from the vector sets 3a and 3b and forming a vector pair) to each other by normalizing each of the feature vectors and calculating an inner product of the normalized feature vectors.
In addition, when the number of feature vectors included in the vector set 3a and the number of feature vectors included in the vector set 3b are not identical, the vector-pair generation unit 4 equalizes the numbers of the feature vectors included in the vector sets 3a and 3b. For example, the numbers can be equalized by dividing a portion of the feature vectors included in one of the vector sets 3a and 3b which includes a smaller number of feature vectors. When the numbers of feature vectors included in the vector sets 3a and 3b are identical, it is possible to calculate the degree of similarity 7 by using all of the feature vectors. That is, comparison can be performed on all of the feature vectors instead of the comparison performed in only a portion of the feature vectors. Further, when a feature vector is divided, for example, the division is performed so that the length of each of feature vectors produced by the division becomes equal to the length of a feature vector which is to be paired with the feature vector produced by the division.
The vector-to-vector distance calculation unit 5 calculates a distance indicating a degree of similarity between the feature vectors forming each vector pair generated by the vector-pair generation unit 4.
The degree-of-similarity calculation unit 6 calculates a sum of the distances calculated by the vector-to-vector distance calculation unit 5 so as to obtain the degree of similarity 7 between the multimedia-data items to be compared.
The output unit 8 outputs the degree of similarity 7 obtained by the degree-of-similarity calculation unit 6. For example, the output unit 8 causes display of the degree of similarity 7 on a screen, and stores the degree of similarity 7 in a hard disk device or the like.
In the above construction, the following processing is performed.
First, two multimedia-data items 2a and 2b are inputted by the input unit 2. Then, the vector-set generation unit 3 analyzes each of the multimedia-data items 2a and 2b inputted by the input unit 2, determines a feature quantity indicating a degree of inclusion of information corresponding to each representative feature, generates a feature vector by multiplying one of the oblique base vectors corresponding to each representative feature by the feature quantity corresponding to the representative feature, and generates vector sets 3a and 3b. Next, the vector-pair generation unit 4 generates vector pairs each of which is formed of a first feature vector extracted from one of the vector sets 3a and 3b and a second feature vector extracted from the other of the vector sets 3a and 3b. Subsequently, the vector-to-vector distance calculation unit 5 calculates a distance indicating a degree of similarity between feature vectors included in each vector pair generated by the vector-pair generation unit 4. Thereafter, the degree-of-similarity calculation unit 6 calculates a sum of the distances calculated by the vector-to-vector distance calculation unit 5 so as to obtain the degree of similarity 7 between the multimedia-data items to be compared, and then the output unit 8 outputs the degree of similarity 7 obtained by the degree-of-similarity calculation unit 6.
Consider the case where two image data items 9b and 9c which are to be compared are inputted. In the vector-set generation unit 3, correspondence relationships which indicate which color in the hue circle 9a is near to each color which the pixels constituting the image data items 9b and 9c can have are defined. Then, the vector-set generation unit 3 calculates the ratio of pixels having colors corresponding to each color in the hue circle 9a to the total number of pixels of each image represented by the image data items 9b and 9c. In the example of
The vector-set generation unit 3 generates vector sets 9d and 9e respectively corresponding to the image data items 9b and 9c. In the example of
Next, the vector-pair generation unit 4 generates a vector pair. For example, the vector-pair generation unit 4 acquires the feature vector 0.5e1 from the vector set 9d, and then acquires the feature vector 0.5e2 from the vector set 9e since the direction of the feature vector 0.5e2 is nearest to the direction of the feature vector 0.5e1. Then, the vector-pair generation unit 4 generates a vector pair of the two feature vectors acquired as above. Similarly, the vector-pair generation unit 4 generates another vector pair of the two feature vectors 0.5e7 and 0.5e8.
The vector-to-vector distance calculation unit 5 calculates a distance (d1 and d2) between feature vectors forming each vector pair generated by the vector-pair generation unit 4. The degree-of-similarity calculation unit 6 calculates a degree of similarity 9f by summing the distances d1 and d2.
As explained above, according to the present invention, a degree of similarity between multimedia-data items is determined by summing distances between respective vector pairs formed of feature vectors of multimedia-data items to be compared. Therefore, it is possible to efficiently calculate the degree of similarity without impairing discriminability between multimedia-data items.
In addition, since the features of multimedia-data items are represented by vectors, similar but different vectors can be easily discriminated based on the directions of the vectors, and processing load caused by generation of vector pairs is small.
The RAM 102 temporarily stores at least portions of an OS (operating system) program and application programs which are executed by the CPU 101, as well as various types of data necessary for processing by the CPU 101. The HDD 103 stores the OS and application programs.
A monitor 11 is connected to the graphic processing device 104, which makes the monitor 11 display an image on a screen in accordance with an instruction from the CPU 101. A keyboard 12 and a mouse 13 are connected to the input interface 105, which transmits signals sent from the keyboard 12 and the mouse 13, to the CPU 101 through the bus 107.
The communication interface 106 is connected to a network 10, and exchanges data with other computers through the network 10.
By using the above hardware construction, it is possible to realize processing functions in the embodiment of the present invention.
The storage device 110 stores an image file group 111 and vector sets 112. The image file group 111 includes a plurality of image data items to be compared, and is stored in the storage device 110 in advance of search operations. Each of the vector sets 112 is a set of feature vectors representing features of an image, and generated for each image data item which is included in the image file group 111.
The similarity search apparatus 120 comprises a generation unit 121, a search unit 123, and a distance calculation unit 124.
When an unprocessed image file is added to the image file group in the storage device 110, or when an image file as an object to be compared is passed from the search unit 123 to the generation unit 121, the generation unit 121 generates a vector set which represents the features of the image file. In the generation unit 121, a feature-quantity extraction unit 121a and a vector-set generation unit 121b perform operations for generation of the vector set.
The feature-quantity extraction unit 121a acquires an image file to be processed, and extracts a feature quantity of an image represented by the image file for each of prescribed representative features, which are, for example, predetermined colors. In the case where a feature quantity for each representative color is extracted, information indicating a designated color (representative color) similar to each of colors which can be represented in image files is predefined in the feature-quantity extraction unit 121a. Next, the feature-quantity extraction unit 121a classifies colors which are represented in each image file, into groups corresponding to the representative colors. Then, the feature-quantity extraction unit 121a determines the proportion of regions corresponding to each representative color in each image to be a feature quantity corresponding to the representative color (i.e., extracts the feature quantity), and temporarily stores the extracted feature quantity in the RAM 102.
The oblique base vectors are defined in advance in the vector-set generation unit 121b. The oblique base vectors are respectively defined in correspondence with the representative features, which correspond to feature quantities of image files. The vector-set generation unit 121b acquires from the RAM 102 the feature quantity extracted by the feature-quantity extraction unit 121a, multiplies the oblique base vector corresponding to the feature quantity by the feature quantity so as to produce a feature vector. When the production of the feature vectors for the respective feature quantities extracted by the feature-quantity extraction unit 121a is completed, the vector-set generation unit 121b generates a set of the feature vectors (i.e., a vector set).
When an image file to be processed is acquired from the storage device 110, the vector-set generation unit 121b stores the generated vector set 112 in association with the image file to be processed. When an image file to be processed is passed from the search unit 123, the vector-set generation unit 121b passes the generated vector set 112 to the search unit 123.
The search unit 123 receives input of an image file as an object to be compared, and searches the image file group 111 in the storage device 110 for an image file similar to the received image file. Specifically, the search unit 123 passes the received image file to the generation unit 121, and receives a vector set from the generation unit 121. Next, the search unit 123 sequentially receives from the generation unit 121 vector sets 112 respectively corresponding to image files in the storage device 110, and passes to the distance calculation unit 124 the vector set corresponding to the image file as the object to be compared and the vector sets 112 corresponding to image files in the storage device 110. Then, the distance calculation unit 124 calculates distances between the above vector sets, and passes the calculated distances to the search unit 123.
The search unit 123 recognizes the distances between the image files based on the vector set corresponding to the image file as the object to be compared and the vector sets corresponding to the respective image files in the storage device 110. The search unit 123 recognizes that image files less distant from (nearer to) the image file as the object to be compared are more similar to the image file as the object to be compared. Thus, the distance calculation unit 124 outputs as a search result a predetermined number of image files relatively less distant from the image file as the object to be compared (or identification information indicating the predetermined number of image files).
When the distance calculation unit 124 receives from the search unit 123 vector sets corresponding to two image files to be compared, the distance calculation unit 124 calculates the distance between the received image files. Specifically, the distance calculation unit 124 establishes one-to-one correspondences between feature vectors in the two inputted vector sets so as to form a plurality of vector pairs. The distance calculation unit 124 calculates distances between the respective vector pairs. Then, the distance calculation unit 124 calculates the sum of the distances between the respective vector pairs as a distance between the two vector sets, which is information indicating a degree of similarity between the two image files. The value indicating this distance decreases with increase in the degree of similarity. Finally, the distance calculation unit 124 passes the calculated distance to the search unit 123.
Thus, when an image file as an object to be compared is inputted into the above similarity search apparatus 120 by a user, the image file is passed from the search unit 123 to the generation unit 121. Then, the generation unit 121 generates a vector set of the image file passed from the search unit 123, and passes the generated vector set to the search unit 123. Subsequently, the search unit 123 extracts the vector sets 112 from the storage device 110, and the distance calculation unit 124 calculates distances between the generated vector set of the image file as the object to be compared and the extracted vector sets. Finally, the search unit 123 outputs as a similar image file an image file corresponding to one of the vector sets 112 which is nearest to the generated vector set of the image file as the object to be compared.
Hereinbelow, details of processing performed in the multimedia-data search apparatus 100 illustrated in
1. Method for Obtaining Oblique Basis from Similarity Matrix
It is necessary to define an oblique basis in advance in the multimedia-data search apparatus 100. The oblique basis can be calculated based on a similarity matrix as indicated below.
In the following explanations, the term “square matrix” means a matrix in which the number of rows is equal to the number of columns, the term “regular matrix” means a matrix which has an inverse matrix, and the term “positive definite matrix” means a square matrix of which all eigenvalues are positive.
1.1 When Similarity Matrix is Positive Definite Matrix
Without losing generality, oblique base vectors e1, e2, e3, . . . , en constituting a requested oblique basis can be expressed as
Therefore, a transformation matrix T from a vector having feature quantities as its elements to feature vectors is expressed as follows.
When a similarity matrix is expressed as
the oblique basis should satisfy the following four conditions.
The left side of the equation of the condition C2 indicates an inner product of the oblique base vectors ei and ej. However, the condition C3 is not a necessary condition for comparison of vector sets. That is, in the case where vector sets are compared, it is possible to achieve discriminability even when oblique base vectors which are not linearly independent of each other are used. However, when the oblique base vectors are linearly independent of each other, the discriminability is improved. Therefore, in this embodiment, oblique base vectors satisfying the condition C3 are used.
Since the condition C4 depends on human judgement, it is difficult to evaluate the condition C4. However, the ultimate goal of the similarity search is to satisfy the condition C4. On the other hand, the conditions C1 to C3 are mathematical, and it is possible to definitely determine whether or not the conditions C1 to C3 are satisfied. According to the method explained below, a solution which satisfies the conditions C1 to C3 is obtained while taking the condition C4 into consideration. First, a way of obtaining a solution which satisfies the conditions C1 to C3 is explained.
According to the condition C1, ∥e1∥=1, i.e., e11=1. In addition, according to the condition C2, the inner product of the oblique base vectors e1 and ej is determined as (e1, ej)=s1j. Thus, the first row of the transformation matrix is determined.
Next, the second row is determined. First, since
∥e2∥={square root}{square root over (e122+e222)}=1 , (32)
e22={square root}{square root over (1−e222)}. (33)
Although more precisely
e22=±{square root}{square root over (1−e122)} (34)
is derived from the equation (32), the positive sign is chosen in this specification since the condition C2 allows either of the positive and negative signs in the equation (32). Hereinafter, in every similar case, a positive sign will be chosen. Since the value of the element e12 is already determined, it is possible to determine the value of the element e22.
Subsequently, the values of the elements e2j are obtained as follows.
Although
(e2, ej)=e12e1j+e22e2j=s2j (35)
the values of the elements e2j can be obtained by
e2j=(s2j−e12e1j)/e22 (36)
since the value of the elements e12, e1j, and e22 are already determined. It should be noted that e22 must be nonzero as explained later. Hereinafter, It is assumed that e22±0. Thus, the values of the elements eij for the other values of j can be sequentially obtained. Specifically, the following formulas are used.
Incidentally, the amount of data required to be stored is considered. When the oblique base vectors are obtained by calculation of only real numbers, and each real number is represented by w bytes, the amount of data required for representing a vector is wn bytes.
1.2 Introduction of Imaginary Number (When Similarity Matrix is Regular and Not Positive Definite)
In the above explanations, the case where the quantity in the square root in the right side of the formula (38) is equal to zero or negative is not mentioned.
(a) When the quantity in the square root in the right side of the formula (38) is zero, it is impossible to proceed calculation. This problem will be explained later in the next section, “1.3 When Similarity Matrix Is Not Regular.”
(b) When the quantity in the square root in the right side of the formula (38) is negative, the values of the elements eii become imaginary numbers. In this embodiment, the values of the elements eii are allowed to be imaginary numbers. Hereinbelow, a case where a value of an element eii becomes an imaginary number is indicated, and then a calculation method in the case where a value of an element eii is an imaginary number is explained.
First, it should be noted that the imaginary number is a pure imaginary number. In addition, when a value of an element eii is a pure imaginary number, all of the other elements eij satisfying i<j≦n in the same column become pure imaginary numbers. Therefore, all of the values of elements in each row in the matrix T are real numbers or pure imaginary numbers. For the sake of convenience, in this specification, each zero element in the matrix T is regarded as a real number or a pure imaginary number.
In addition, manners of definitions of the inner product, the length (norm) of each vector, and the distance between vectors are also important. Normally, the inner product and the length (norm) of vectors having components represented by complex numbers can be defined by using their complex conjugates. That is, when two vectors x and y are expressed as
yi is a complex number expressed as a+bi, and {overscore (yi)} is the complex conjugate a-bi of yi, the inner product of the vectors can be defined as
where a and b are real numbers, and i is the imaginary unit. In addition, the length of the vector x can be defined as
the distance between the vectors x and y is expressed as
According to the above definition, the length of every vector is guaranteed to be zero or positive. However, the above definition is not used in the present embodiment, and instead the inner product of the vectors x and y is defined as
That is, in the present embodiment, the inner product of vectors having elements represented by complex numbers is defined as the inner product of vectors having elements represented by real numbers. Therefore, the length of each vector and the distance between two vectors are defined as
The reason why the above definitions are used is that in some of the cases where the distances are defined by using the complex conjugates, it is impossible to concurrently satisfy both of the conditions C1 and C2. On the other hand, when the distances are defined as in the present embodiment, it is possible to concurrently satisfy both of the conditions C1 and C2. The definitions of the distances as in the present embodiment are used for defining the space-time intervals in the special theory of relativity.
In the present embodiment, according to the above definitions, it is possible to obtain a solution which concurrently satisfies all of the conditions C1, C2, and C3 when the values of the elements e11 are nonzero.
<Example in which Imaginary Element Appears (First Example)>
Hereinbelow, an example where an imaginary element occurs is indicated below as a first example. In the following explanation, the Munsell color solid and in particular, black, white, and grey in the Munsell color solid are considered.
the oblique base vectors corresponding to the similarity matrix are expressed as
That is, a pure imaginary number appears in the oblique base vector e3, i.e., the third component of the oblique base vector e3 is a pure imaginary number.
The amount of data required to be stored in the case where the similarity matrixes can have an element represented by a pure imaginary number is indicated below.
As mentioned before, in the case where all of the elements of the similarity matrixes are represented by real numbers, and each real number is represented by w bytes, the amount of data required for representing a vector is wn bytes. Normally, the amount of data required for representing a vector represented by complex numbers is twice the amount of data required for representing a vector represented by real numbers, i.e., 2wn bytes. However, in the method used in this embodiment, the imaginary numbers are pure imaginary numbers, and the row or rows in which imaginary numbers appear are fixed. Therefore, in this embodiment, only the information indicating the row or rows in which imaginary numbers appear is stored separately from vectors. Thus, the substantial amount of data required for representing a vector is still wn bytes, as in the case where the similarity matrixes are positive definite.
The J.S.N. Jean reference discloses that when the feature vectors obtained by the transformation are represented by real numbers, the amount of data required for representing a each feature vector is wn bytes. In addition, as explained above, even in the more generalized case where the components of feature vectors obtained by a similarity matrix may be pure imaginary numbers, the substantial amount of data required for representing each feature vector is still wn bytes.
1.3 When Similarity Matrix is not Regular
Hereinbelow, a method which enables acquisition of a solution even in the case where an element eii is zero is explained. In this method, it is assumed that the dimension of the oblique base vectors is 2n, and the oblique base vectors are expressed as
In addition, each of the components eii (1≦i≦n) is initially assumed to be one, i.e., eii=1 (1≦i≦n). Therefore, each of the components eii (1≦i≦n) is not zero. In this case, the length of each oblique base vector seems to exceed one, and not to satisfy the condition C1. However, the components en+i, i in the (n+1)st to 2nth rows adjust for the above excess. The values of these components eij can be obtained in a similar manner to those explained in the previous sections 1.1 and 1.2.
The amount of data required to be stored in the case of 2n dimensions is indicated below.
In the case of n dimensions, as explained before, even in the case where imaginary numbers are introduced, the substantial amount of data required for representing each feature vector is wn bytes when each real number is represented by w bytes. On the other hand, in the case of 2n dimensions, the vector components corresponding to the (n+1)st to 2nth dimensions are pure imaginary numbers. Therefore, it is unnecessary to use complex numbers as in the case of n dimensions, and vectors can be substantially represented by 2n real numbers. Therefore, the amount of data required to be stored in the case of 2n dimensions is 2nw bytes, i.e., doubled compared with the case of n dimensions.
When the above method using 2n dimensions is used, although the amount of data required to be stored increases, it is possible to obtain oblique base vectors in every case regardless of regularity of the similarity matrix.
Further, it is possible to reduce the amount of data required to be stored, by combining the above method explained in this section and the method explained in the previous section 1.2, as indicated in the following section 1.4.
1.4 Reduction of Dimension
In the method explained below, the methods explained in the previous sections 1.2 and 1.3 are combined for reduction of the dimension. The methods explained in this section are classified into a first method in which importance is placed on the method of sections 1.2 and a second method in which importance is placed on the method of sections 1.3. Hereinafter, the first method is referred to as the minimum dimension method since the dimension can be minimized according to the first method, and the second method is referred to as the separation method since, according to the second method, components represented by imaginary numbers are separately arranged in the (n+1)st and subsequent rows.
Sequences of the first and second methods are indicated below. Since the first and second methods are different in only portions of their sequences, the common portions of the sequences of the first and second methods are commonly explained below. In addition, in the following explanations, an array defined for memorizing integers is indicated by a.
The dimension of the oblique base vectors is determined to be n+m, and the values of the matrix elements for k=1, 2, . . . , m are calculated based on the following formulas, where i=a[k].
The vector components corresponding to 1st to nth dimensions are real numbers, and the vector components corresponding to (n+1)st to (n+m)th dimensions are pure imaginary numbers. Thus, the oblique base vectors can be expressed as follows. First, each oblique base vector which has no imaginary component is expressed as follows.
In addition, each oblique base vector which has at least one imaginary component is expressed as follows.
As explained above, when the number of oblique base vectors is n (where n is an integer), and the oblique base vectors are not linearly independent within n dimensions, it is possible to realize linear independency by defining oblique base vectors with a dimension in the range from n+1 to 2n. When the above method for dimension reduction is used, the amount of data required to be stored is as small as (n+m)w bytes.
2. Measure for Overcoming Lack of Discriminability
A measure for overcoming the aforementioned lack of discriminability is explained below. This problem does not occur in the method using the orthogonal basis and the Euclidean distance, since, according to the method using the orthogonal basis and the Euclidean distance, the distance between two different objects is necessarily positive, i.e., nonzero. However, in the method using the orthogonal basis and the quadratic-form distance and in the method using the oblique basis and the Euclidean distance, in some cases, vectors corresponding to two different objects become identical, or distances between different vectors become zero. The former problem occurs since the oblique base vectors are not linearly independent of each other, and the latter problem can occur when an imaginary number appears in the aforementioned solution.
Hereinbelow, attempts are made to solve the above problem by the following two approaches:
The basic concept of the approach (a) is to bring the similarity matrix obtained as above close to a unit matrix. In addition, in the approach (b), a solution is sought without deforming the similarity matrix.
2.1 Loss of Discriminability
Hereinbelow, two simple examples in which discriminability is lost are indicated.
Consider four colors, red, yellow, green, and blue out of the colors in the hue circle, and assume that the four colors are located at the quartering points of the hue circle. At this time, a similarity matrix in which the distances between the four colors in the hue circle are directly reflected is expressed as
In addition, the following vectors are oblique base vectors corresponding to the above similarity matrix.
However, actually the above oblique base vectors are not linearly independent. Therefore, mathematically, the vectors e1, e2, e3, and e4 (55) should not be called base vectors. For example, when the set of vectors (55) are used, the feature vector f1 of an image composed of 50% of red pixels and 50% of green pixels (i.e., pixels of complementary colors) and the feature vector f2 of an image composed of 50% of yellow pixels and 50% of blue pixels are each a zero vector as indicated below, since the pair of red and green and the pair of yellow and blue are each a pair of complementary colors.
This is because the set of vectors e1, e2, e3, and e4 (55) are not linearly independent.
<Example of Linearly Independent Vectors which Lack Discriminability (Third Example)>
Consider again the aforementioned example of white, black, and grey, which is used for explaining occurrence of an imaginary number. In this case, by using the aforementioned vectors
the feature vector f1 of an image composed of 50% of white pixels and 50% of black pixels and the feature vector f2 of an image composed of only grey pixels are respectively expressed as
The distance between the above two feature vectors is calculated to be zero, i.e., d(f1, f2)2=0. That is, although the vectors e1, e2, and e3 are linearly independent, the vectors e1, e2, and e3 lack discriminability.
The lack of discriminability is a serious problem. This problem does not occur in the method using the orthogonal basis and the Euclidean distance since the base vectors in the orthogonal basis are linearly independent, and the Euclidean distance satisfies the distance axiom. Actually, when the base vectors are linearly independent, different objects correspond to different vectors. In addition, the distance between different vectors is nonzero.
One of the following causes is considered to lead to the lack of discriminability in the aforementioned third example:
Next, an attempt to overcome the lack of discriminability is made from the viewpoint of each of the above causes (R1) and (R2).
First, an attempt to overcome the above problem within the framework of the quadratic-form distance (i.e., by modification of the similarity matrix) is made from the viewpoint of the cause (R1).
It is considered that there is a trade-off relationship between discriminability and similarity between features. Actually, the method using the orthogonal basis and the Euclidean distance is included in the method using the oblique basis and Euclidean distance and the method using the quadratic-form distance, and the similarity matrix in the method using the orthogonal basis and the Euclidean distance is a unit matrix. Therefore, when the similarity matrix is brought closer to a unit matrix, the method using the quadratic-form distance can be brought closer to the method using the orthogonal basis and the Euclidean distance, which does not have the problem of lack of discriminability. At this time, the value “one” of the elements sii in the similarity matrix is unchanged. In addition, the other elements sij are reduced by multiplying sij by a real number a satisfying 0≦a≦1. Thus, the similarity matrix is brought closer to a unit matrix. That is, the real number a is a parameter for controlling the degree of proximity to the unit matrix. When a=1, the matrix is the unchanged similarity matrix. When a=0, the method using the quadratic-form distance becomes the method using the orthogonal basis and the Euclidean distance. Hereinafter, the above method for bringing the similarity matrix closer to a unit matrix is referred to as the similarity-matrix deformation method.
2.2 Loss of Similarity
The similarity between features are maintained in the method using the quadratic-form distance and the method using the oblique basis and the Euclidean distance. In other words, when objects each have only a single feature, the similarity between the objects is maintained. However, in some cases, similarity between nonzero feature vectors each of which represents a plurality of feature quantities is lost. This problem is considered in this section. First, a simple example in which the above problem occurs is indicated below.
<Example in which Similarity is Lost (Fourth Example)>
A hue circle comprised of twelve colors is considered. In the following explanations, when an image composed of pixels of a pair of complementary colors Color1 and Color2, and the amounts of the pixels of each color is identical, the image is denoted Color1+Color2. For example, according to the human color perception, the red+green images look more similar to the (red-orange)+green images than to the yellow+blue images. However, according to the method using the oblique basis and the Euclidean distance or the method using the quadratic-form distance, the degrees of similarity between the above images cannot be determined as the human beings perceive, by simply deforming the similarity matrix based on the aforementioned single parameter a, and instead all of the above images are determined to be similar.
The above fact is explained in detail below. Assume that the colors correspond to twelve points which equally divide the hue circle into twelve portions, as in the aforementioned first example, in which an imaginary element appears. When entire-feature vectors each corresponding to an image composed of identical amounts of pixels of a pair of complementary colors are expressed as
fi=0.5ei+0.5ei+6(1≦i≦6), (61)
relationships between the distances d(f1, fj) (1≦i, j<6) and the parameter a are considered below.
The applicants had expected the similarity between features to be reflected in the distance between the feature vectors fi and fj, i.e., the following relationships exist.
d(f1, f2)<d(f1, f3)<d(f1, f4)<d(f1, f5) (62)
In addition, the applicants had also expected the differences between the above four distances to be greater when the parameter a is closer to one within the range of 0<a≦1, while the above distances in the relationships (62) are identical when a=0 (i.e., in the method using the orthogonal basis and the Euclidean distance).
However, the applicants actually calculated the above distances for various values of the parameter a, and found that the following relationships exist regardless of the value of the parameter a.
d(f1, f2)=d(f1, f3)=d(f1, f4)=d(f1, f5) (63)
That is, the same relationships as the relationships in the method using the orthogonal basis and the Euclidean distance exist even in the methods using the quadratic-form distance or the method using the orthogonal basis and the Euclidean distance. The above relationships indicate that the similarity between features is lost, and remains lost even when the parameter a is brought closer to zero.
In the case where the similarity is lost as in the above fourth example, the matrix is deformed by using a plurality of parameters. Thereby, the problem of the loss of similarity between features as in the fourth example is overcome.
2.3 Feature Space Based on Distances Between Vector Sets (Multivector Feature Space)
In this section, a method for solving a problem of the loss of discriminability and similarity based on an assumption that the aforementioned cause (R2) leads to the problem is explained.
The discriminability and similarity are lost in the case where feature vectors are synthesized from oblique base vectors as indicated in the fourth example. However, attention is focused on unsynthesized nonzero vectors (i.e., vectors ciei (1≦i≦n), where ci≠0), these vectors hold information on feature quantities and information on similarity between features. In addition, discriminability is not lost.
Consider a set of vectors
F={ciei|1≦i≦n, ci≠0}. (64)
In this specification, a set of vectors as indicated in (64) are referred to as a vector set. Each vector set may redundantly include identical vectors. In this sense, to be more precise, the vector set is a multiset (multivector).
Hereinbelow, in order to facilitate conceptual understanding of effectiveness of use of vector sets, values of vectors are replaced with material points in the following explanations. In multivector spaces, each object is generally represented by a plurality of vectors. It is assumed that a material point having an identical mass is placed at a point corresponding to each of a plurality of vectors. At this time, it is possible to consider a kind of solid formed of these material points. The δ-distance defined in the following explanations is a definition of a distance between solids each of which is formed as above. In this case, an approximation to a feature set is produced by dividing material points forming each solid into a plurality of groups, and replacing each group with (an essential equivalent to) a center of gravity.
a12=a1+a2, and
a34=a3+a4,
where aij is the center of gravity of the material points ai and aj. In addition, the points in the second solid are respectively determined as
b14=b1+b4, and
b23=b2+b3,
where bijis the center of gravity of the material points bi and bj. Although, to be more precise, the right side of each of the above four definitions of the centers of gravity should be divided by two, aij and bij are each referred to as a center of gravity in this specification since the essence is unchanged except for the factor of ½.
Further, the centers of gravity of the above centers of gravity aij and bij are respectively determined as
a1234=a12+a34=a1+a2+a3+a4, and
b1234=b12+b34=b1+b2+b3+b4.
Thus, the original, first and second solids are respectively approximated by their centers of gravity. Since the centers of gravity of the first and second solids are assumed to coincide, the points a1234 and b1234 coincide. This means that discriminability is lost.
Therefore, according to the present embodiment, vectors in each vector set in a multivector feature space are not synthesized, and the concept of the distance between solids is defined by one-to-one comparison of individual vectors. The basic concept is to define a distance between vector sets in order to measure a degree of similarity between the vector sets. Although the distance can be defined in various manners, two basic examples are indicated below.
<Example of Calculation of Distance between Multivectors (Fifth Example)>
First, the situation of the aforementioned fourth example, in which the similarity is lost according to the conventional method, is considered below.
A first multivector set for the image 20 includes a vector 21 being oriented in the direction of red and having a length of 0.5 and a vector 22 being oriented in the direction of green and having a length of 0.5. A second multivector set for the image 30 includes a vector 31 being oriented in the direction of red-orange and having a length of 0.5 and a vector 32 being oriented in the direction of blue-green and having a length of 0.5. A third multivector set for the image 40 includes a vector 41 being oriented in the direction of yellow and having a length of 0.5 and a vector 42 being oriented in the direction of blue and having a length of 0.5.
Then, multivector sets are defined as follows.
Fi={0.5ei, 0.5ei+6}(1≦i<6). (65)
In addition, the δ-distance between the multivector sets Fi and Fj (i≦j) are defined as
where M is a set containing all of one-to-one mappings from vectors in the multivector set Fi to vectors in the multivector set Fj, and each of the multivector sets Fi and Fj contains m vectors. In this example, the δ-distance can be expressed as
δ(Fi, Fi+j)=2{square root}{square root over (1−|cos(πj/6)|)}, (67)
where 1≦j<6, and |cos(πj/6)| is an absolute value of cos(πj/6). Therefore, δ (F1, F2)=0.732, δ (F1, F3)=1.414, δ (F1, F4)=2, δ (F1, F5)=1.414, and δ (F1, F6)=0.732. That is, the discriminability is maintained, and the problem of the loss of similarity is also solved. In addition, the similarity between features is natural.
In order to calculate the δ-distance between the images 20 and 30, first, a distance d1 between the vector 21 included in the multivector set for the image 20 and the vector 31 included in the multivector set for the image 30 is calculated. Similarly, a distance d2 between the vector 22 included in the multivector set for the image 20 and the vector 32 included in the multivector set for the image 30 is calculated. Thus, the δ-distance between the images 20 and 30 is obtained by calculating a sum of the distances d1 and d2.
In order to calculate the δ-distance between the images 20 and 40, first, a distance d3 between the vector 21 included in the multivector set for the image 20 and the vector 41 included in the multivector set for the image 40 is calculated. Similarly, a distance d4 between the vector 22 included in the multivector set for the image 20 and the vector 42 included in the multivector set for the image 40 is calculated. Thus, the δ-distance between the images 20 and 40 is obtained by calculating a sum of the distances d3 and d4.
<Provision for Linearly Dependent Vectors (Sixth Example)>
Next, a provision for the situation of the aforementioned second example, in which the vectors are not linearly independent, is considered below.
In this embodiment, a distance between vector sets as above is referred to as a multivector distance.
2.4 Feature Set and Approximation
As an extreme example, it is possible to consider a multivector distance between vector sets which are defined as
F={ciei|1≦i≦n}. (68)
Unlike the vector set defined in (64), the vector set F defined in (68) includes the zero vector, and is referred to as a feature set. However, the cost of calculation of distances between feature sets as above is considered to be very great. Therefore, each feature set is approximated by a vector set containing m vectors, which is hereinafter referred to as an m-vector set, where 1≦m<n. An example of a 2-vector set, which is defined in the situation of the aforementioned fourth example, is indicated below. In the fourth example, the similarity is lost.
A feature set F is defined as
F={c1e1, c2e2, . . . , c12e12}, and (69)
the 2-vector set A is defined as
A={a1, a2}, (70)
where
a1=c1e1+c2e2+ . . . +c6e6, and (71)
a2=c7e7+c8e8+ . . . +c12e12. (72)
That is, the 2-vector set A approximates the feature set F. When 2-vector sets for feature sets corresponding to the feature vectors in the fourth example, as the above 2-vector set A for the above feature space F, are denoted Ai (1≦i≦6), the 2-vector sets Ai are respectively identical to the vector sets Fi in the aforementioned fifth example, in which a distance between multivectors is calculated. Therefore, the distance between the 2-vector sets can be defined by using the formula (67). That is, the distances between the above feature sets can be approximated by the distance defined by the formula (67).
Based on the above consideration, the conventional distance between feature sets can be regarded as an approximation using 1-vector sets. That is, the m-vector sets is an extension of the conventional feature vector space.
The approximation method explained in this section is based on division of oblique base vectors. In this example, the oblique base vectors are divided into the two groups, E1={e1, e2, . . . , e6} and E2={e7, e8, . . . , e12}. The vectors e1, e2, . . . , e6 correspond to the warm colors near yellow, and the vectors e7, e8, . . . , e12 correspond to the cold colors near blue. That is, the oblique base vectors should be divided into groups each of which includes oblique base vectors corresponding to colors similar to each other for the reason explained below.
The problem of the loss of discriminability does not occur when no approximation is used, i.e., when the distance between feature sets is used. However, discriminability can be lost when the approximation is used. Nevertheless, loss of discriminability does not occur over the entire feature space since the approximation explained in this section can localize the extent of occurrence of loss of discriminability. In the above example, loss of discriminability does not occur over the two groups E1 and E2. Next, general consideration of the distance between two vector sets is given below.
Although the definition by the formula (66) succeeds in the aforementioned fifth example, this definition does not necessarily succeed. An example in which the above definition fails is explained below.
<Approach to Very Similar Feature (Seventh Example)>
Consider two 2-vector sets, A1={0.7e1, 0.3e3} and A2={e2, 0 (zero vector)}. At this time, it is assumed that the three oblique base vectors e1, e2, and e3 are close to each other. For example, the three oblique base vectors e1, e2, and e3 correspond to whitish grey, grey, and blackish grey, respectively. Therefore, according to the human color perception, the above two 2-vector sets A1 and A2 look similar.
However, when the definition by the formula (66) is used, the distance between the two 2-vector sets A1 and A2 is d(0.7e1, e2)+d(0.3e3, 0 (zero vector))≈0.6. Therefore, the distance between the two 2-vector sets is redefined as explained below.
First, each element (the vector ai, 1≦i≦m) of an m-vector set A={a1, a2, . . . , am} is divided as
a1=a11+a12+ . . . +a1m,
a2=a21+a22+ . . . +a2m,
Similarly, each element (the vector bi, 1≦i≦m) of another m-vector set B={b1, b2, . . . , bm} can be divided.
Next, a D-distance between the two m-vector sets, A={a1, a2, . . . , am} and B={b1, b2, . . . , bm} is defined as
When the above definition of the D-distance is used, it is possible to define the distance between the two 2-vector sets A1 and A2 in the seventh example so that the distance between the two 2-vector sets A1 and A2 becomes close to zero.
Variations in the D-distance according to the way of division of vectors are explained below.
2.5 Approximate Calculation of D-Distance
If the aforementioned definition of the D-distance is directly adopted, the vectors can be divided in an infinite number of ways, and therefore the amount of calculation becomes extremely great. At this time, a method of approximately obtaining the D-distance is explained below. Specifically, an algorithm for obtaining an approximate value of the D-distance between two m-vector sets A={a1, a2, . . . , am} and B={b1, b2, . . . , bm} is indicated. In the following example, in order to enable application to the case where feature quantities are represented by absolute quantities, the feature vectors are divided in correspondence with the sums of the absolute values of the feature quantities of the m-vector sets A and B. The sums of the absolute values of the feature quantities of the m-vector sets A and B are respectively expressed as
When the feature quantities are relative quantities, the above sums α and β are equal to one. Hereinbelow, the D-distance is indicated by D, the zero vector is denoted vector0, and a vector set containing only zero vectors is indicated by O. For example, a vector set {vector0, vector0, vector0} is indicated by O.
Next, a sequence of processing for approximate calculation of the D-distance is explained below.
[Step S11] It is determined whether or not the condition that A=O or B=O is satisfied. When yes is determined, the operation goes to step S12. When no is determined, the operation goes to step S15.
[Step S12] It is determined whether or not the condition that A=O is satisfied. When yes is determined, the operation goes to step S13. When no is determined, the operation goes to step S14.
[Step S13] The D-distance D is set as D=β, and the sequence of
[Step S14] The D-distance D is set as D=α, and the sequence of
[Step S15] The D-distance D is set as D=α.
[Step S16] It is determined whether or not the condition that A≠O is satisfied. When yes is determined, the operation goes to step S17. When no is determined, the sequence of
[Step S17] One ai of nonzero vectors included in the vector set A and one bj of nonzero vectors included in the vector set B which minimize (ai, bj)/(|ai|·|bj|) are determined.
[Step S18] It is determined whether or not the condition that |ai|/|bj|≧α/β is satisfied. When yes is determined, the operation goes to step S19. When no is determined, the operation goes to step S20.
[Step S19] The D-distance D is set as D=D+d(αai/β, bj), the vector ai in the m-vector sets A is replaced with {1-(α|bj|/β|ai|)}ai, and the vector bj in the m-vector sets B is replaced with a zero vector. Thereafter, the operation goes to step S16.
[Step S20] The D-distance D is set as D=D+d(ai, βbj/α), the vector ai in the m-vector sets A is replaced with a zero vector, and the vector bj in the m-vector sets B is replaced with {1-(β|ai|/α|bj|)}bj. Thereafter, the operation goes to step S16.
The basic concept of the above algorithm is as follows.
The one ai of nonzero vectors included in the vector set A and the one bj of nonzero vectors included in the vector set B which minimize (ai, bj)/(|ai·|bj|) are chosen. That is, a pair of nonzero vectors ai and bj are chosen so that a unit vector in the same direction as the vector ai and a unit vector in the same direction as the vector bj are nearest. Then, the whole of one of the vectors ai and bj and the whole or a portion of the other of the vectors ai and bj are extracted (cut out) as vectors corresponding to each other so that the ratio between the length of the whole of the one and the length of the whole or the portion of the other is α:β. Subsequently, the distance between the extracted vectors is obtained and added to the current value of the D-distance D, which is zero or an accumulation of at least one distance between previously extracted vectors. Further, the above vectors ai and bj are respectively shortened by the lengths of the corresponding vectors extracted as above. Since at least one of the vectors ai and bj is fully extracted, the at least one of the vectors ai and bj is shortened to a zero vector. The ratio between the lengths of the corresponding vectors extracted as above is α:β, and this ratio is unchanged in every pair of corresponding vectors. Therefore, finally, the vector sets A and B concurrently become a vector set O containing only zero vectors.
The above operations of cutting out vectors determine the division of vector sets, and the correspondence relationships between the vectors generated by the division determines the one-to-one correspondences in the δ-distance between the divided vector sets.
According to the above method, every time, a pair of nonzero vectors ai and bj are chosen so that a distance between a unit vector in the same direction as the vector ai and a unit vector in the same direction as the vector bj is minimized. That is, processing for choosing a pair of feature vectors which have directions nearest to each other, and cutting out portions which realize a pair of corresponding vectors is repeated. Therefore, the distance calculated as above can be expected to be near to the D-distance.
The approximation of the distance between feature sets is based on the above definition. In addition, although the problems of the discriminability and the similarity is not completely solved as long as approximation is performed, the problems are more localized as the value m increases. Further, the conventionally used feature vectors are the same as approximation based on 1-vector sets. That is, the approximation of the distance defined in this section is a generation of the conventional distance between feature vectors.
3. Search Method
Hereinbelow, a search method in a multivector feature space is explained. The search is performed by the search unit 123 illustrated in
(1) Method in which Vector Sets are Generated at the Time of Searching
In a secondary storage such as the HDD 103, a plurality of sets of identifiers of objects such as images and feature quantities which are automatically extracted from the objects are stored in advance. In addition, information on an oblique basis is also stored in advance. Further, at the time of searching, m-vector sets are generated based on the feature quantities and the oblique basis, and a similarity search is performed by calculating D-distances. When this search method is used, the plurality of sets of the feature quantities and the identifiers of the objects are also stored in the storage device 110 illustrated in
According to the above search method, it is unnecessary to store vector sets in advance. Therefore, when m>2, it is sufficient for the secondary storage to have small capacity, although it is necessary to generate the vector sets at the time of searching.
(2) Method in which Vector Sets are Generated in Advance of Searching
In a secondary storage such as the HDD 103, m-vector sets generated from feature quantities and an oblique basis are stored in advance. In addition, a similarity search is performed by using the m-vector sets and D-distances. The explanations of the present embodiment are basically based on the search method (2).
According to the search method (2), it is necessary to store vector sets in advance. Therefore, when m>2, the load of storing the vector sets is heavy.
Although there is a trade-off relationship between the search methods (1) and (2) as explained above, generally, the search method (2) is considered to be appropriate in the case where m=1, and the search method (1) is considered to be appropriate in the case where m>2.
When the processing explained above is performed, the present embodiment has the following advantages.
(a) Improvement in Accuracy
Since a multivector feature space is used, it is possible to improve the accuracy compared with the conventional method using the quadratic-form distance.
(b) Improvement in Performance
When an approximation of the feature space is performed, it is possible to improve the performance. In addition, the performance can also be improved by approximately obtaining the D-distance.
(c) Improvement in Discriminability
When distances between each pair of vectors in a multivector feature space are obtained according to the present embodiment, it is possible to improve the discriminability without impairing the similarity between features.
4. Differences from EMD
Hereinbelow, the differences of the present embodiment from the EMD technique disclosed in the aforementioned Y. Rubner reference are explained below. A great difference is that entire feature vectors are fully compared in every operation of comparison in a multivector feature space defined in the present embodiment, while partial matching is performed in the EMD technique in the case where the total numbers of feature quantities in two signatures are different.
In the aforementioned image histogram, the feature quantities are relative quantities in some cases, or absolute quantities in other cases. In the former cases, each feature quantity is based on a proportion of a predetermined color in an entire area. In the latter cases, each feature quantity is based on the number of pixels having a predetermined color. According to EMD, when the feature quantities are relative quantities, and the total numbers of the feature quantities in two signatures are different, partial matching is performed, although full matching is performed when the feature quantities are absolute quantities.
On the other hand, according to the method in the present embodiment, the numbers of the feature vectors included in two vector sets to be compared are always equalized. Specifically, a portion or all of feature vectors in at least one of the two vector sets including a smaller number of feature vectors are divided. Thus, every feature vector in the two vector sets can be used as one of a pair of vectors in a one-to-one correspondence for calculation of a distance. Therefore, even when the total numbers of feature quantities in two vector sets are different, full matching can be performed.
The capability of full matching on every occasion is especially effective when the absolute quantities of feature quantities is significant. For example, in the case of documents, full matching of feature quantities based on absolute quantities is important. That is, in the similarity search of a document, a frequency of occurrences of each word or a weighted frequency of occurrences of each word may be used as a feature quantity. In this case, the dimension is equal to the number of words. However, the similarity search is not performed based on all of the words, and instead at least one word which describes the document well is chosen. Therefore, the commonplace words such as “this” and “do” are excluded. Even when such words are excluded, normally the dimension becomes about one thousand to ten thousand. The frequency of occurrences of each word in each document is an absolute quantity, while the pixels in images, which are mentioned later, are relative. For example, a fact that a specific word is frequently used per se is important.
For example, when a word appears only once in a document U and ten times in a document V, this fact means that this word is important in the document V, or this word more strongly characterizes the document V than other words which appear less frequently in the document V. Therefore, in the case of documents, feature quantities are absolute quantities (the numbers of occurrences of respective words).
In the method according to the present embodiment, even when the total numbers of feature quantities are different (i.e., even when the absolute quantities are significant) as in the similarity search of a document, the feature quantities are fully, not partially, compared. Therefore, similarity between entire objects can be accurately determined.
Further, in order to indicate the advantage of the method according to the present embodiment in the similarity search of images, for example, consider comparison of an image X composed of 1,000 black pixels and 1,000 white pixels and an image Y composed of 1,000 black pixels. According to the EMD, feature quantities are partially compared, and a portion of the feature quantities of the image X (corresponding to the 1,000 black pixels) and all of the feature quantities of the image Y (corresponding to the 1,000 black pixels) match.
On the other hand, according to the method in the present embodiment, as illustrated in
5. Additional Matters
The above processing functions can be realized by a computer. In this case, a program describing details of processing for realizing the functions which the multimedia-data search apparatus should have is provided. When the computer executes the program, the above processing functions can be realized on the computer.
The program describing the details of the processing can be stored in a recording medium which can be read by the computer. The recording medium may be a magnetic recording device, an optical disk, an optical magnetic recording medium, a semiconductor memory, or the like. The magnetic recording device may be a hard disk drive (HDD), a flexible disk (FD), a magnetic tape, or the like. The optical disk may be a DVD (Digital Versatile Disk), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disk Read Only Memory), a CD-R (Recordable)/RW (ReWritable), or the like. The optical magnetic recording medium may be an MO (Magneto-Optical Disk) or the like.
In order to put the program into the market, for example, it is possible to sell a portable recording medium such as a DVD or a CD-ROM in which the program is recorded. Alternatively, it is possible to store the program in a storage device belonging to a server computer, and transfer the program to another computer through a network.
The computer which executes the program stores the program in a storage device belonging to the computer, where the program is originally recorded in, for example, a portable recording medium. The computer reads the program from the storage device, and performs processing in accordance with the program. Alternatively, the computer may directly read the program from the portable recording medium for performing processing in accordance with the program. Further, the computer can sequentially execute processing in accordance with each portion of the program every time the portion of the program is transferred from the server computer.
As explained above, according to the present invention, features of multimedia-data items are represented by feature vectors, and a sum of distances between a pair of vectors in a one-to-one correspondence is obtained, where the vectors are feature vectors of the multimedia-data items to be compared. Then, the degree of similarity between the multimedia-data items is determined based on the sum of the distances. Thus, it is possible to accurately calculate the degree of similarity without impairing discriminability between multimedia-data items.
The foregoing is considered as illustrative only of the principle of the present invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and applications shown and described, and accordingly, all suitable modifications and equivalents may be regarded as falling within the scope of the invention in the appended claims and their equivalents.
Number | Date | Country | Kind |
---|---|---|---|
2004-045135 | Feb 2004 | JP | national |