Pyramid match kernel and related techniques

Abstract

A method for classifying or comparing objects includes detecting points of interest within two objects, computing feature descriptors at said points of interest, forming a multi-resolution histogram over feature descriptors for each object and computing a weighted intersection of multi-resolution histogram for each object. An alternative embodiment includes a method for matching objects by defining a plurality of bins for multi-resolution histograms having various levels and a plurality of cluster groups, each group having a center, for each point of interest, calculating a bin index, a bin count and a maximal distance to the bin center and providing a path vector indicative of the bins chosen at each level. Still another embodiment includes a method for matching objects comprising creating a set of feature vectors for each object of interest, mapping each set of feature vectors to a single high-dimensional vector to create an embedding vector and encoding each embedding vector with a binary hash string.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of this invention, as well as the invention itself, may be more fully understood from the following description of the drawings in which:

FIG. 1 is a block diagram of a preferred embodiment of an electronic computer system for implementing the current invention;

FIG. 1A is a pictorial diagram of a pyramid match kernel intersecting histogram pyramids formed over local features, approximating the optimal correspondences between the sets' features according to the invention;

FIG. 2 is a diagram showing a pyramid match determines a partial correspondence by matching points once they fall into the same histogram bin;

FIG. 3 is four plots showing various matching characteristics;

FIG. 4. shows plots of pyramid match and L₁ embedding comparison on bijective matchings with equally-sized sets and partial matchings with variably-sized sets;

FIG. 5 is a plot showing a comparison of object recognition matching techniques;

FIG. 6. shows example images where three images are shown for each of 28 objects;

FIG. 7. is a flow chart showing the process steps to implement the invention;

FIG. 8 shows graphs inferring the time of publication for papers from 13 volumes of NIPS proceedings;

FIG. 9 is an example generated with graphics software composed of a silhouette and its corresponding 3-D pose, as represented by the 3-D positions of 15 joint positions;

FIG. 10 are diagrams of various Pose inference results;

FIG. 11 are examples of Pose inference on real images;

FIG. 12 shows an example where partial matching may be difficult;

FIG. 12A shows additional examples where partial matching may be difficult;

FIG. 13 shows examples of explicit feature correspondences extracted from a pyramid matching;

FIG. 14 shows a schematic view of category feature mask inference;

FIG. 15 is a chart showing accuracy of categories learned without supervision, as measured by agreement with ground truth labels;

FIG. 16 is a chart showing recognition performance on unseen images using categories learned with varying amounts of weak semi-supervision;

FIG. 17 depicts an embodiment of the invention for organizing media files;

FIG. 18 depicts an embodiment of the invention integrating other information with the organized files;

FIG. 19 shows a plot of uniformly-shaped partitions in contrast with a plot where the feature space determines the partition;

FIG. 20 is a comparison of optimal and approximate matching rankings on image data;

FIG. 21 shows a number of graphs of new matches formed at each pyramid level for either uniform (dashed) or VG (solid) bins for increasing feature dimensions;

FIG. 22 shows a comparison of correspondence field errors and associated computation times for the VG and uniform pyramids;

FIG. 23 shows a table that shows our improvements over the uniform-bin pyramid match kernel;

FIG. 23A shows a flow diagram implementing a vocabulary guided pyramid matching technique;

FIG. 24 shows a schematic of a pyramid match hashing technique;

FIG. 25 shows pseudocode illustrating the steps to perform the pyramid match hashing algorithm;

FIG. 26. is a plot showing approximation robustness to outliers; and

FIG. 27 shows image retrieval results for Caltech-4 and Caltech-101 databases using PMK hashing.

Claims

1. A method for classifying or comparing data objects comprising: detecting points of interest within two data objects;computing feature descriptors at said points of interest;forming a multi-resolution histogram over feature descriptors for each data object; andcomputing a weighted intersection of multi-resolution histogram for each data object.

2. The method as recited in claim 1 where the weight of each weighted intersection is determined by a bin size.

3. The method as recited in claim 1 wherein the forming a multi-resolution histogram comprises: setting a histogram resolution level and detecting matching points within two data objects at said resolution level;changing the histogram resolution level and confirming existing matching points and providing a weight reflecting distance estimate between matched points; andchanging again the histogram resolution level and confirming existing matching points and providing a weight reflecting distance estimate between matched points accordingly to provide an identifying feature.

4. The method as recited in claim 3 wherein each histogram resolution level is changed by a factor of two.

5. A method for assessing data objects comprising: characterizing an data object by a set of feature vectors;partitioning feature space into a plurality of bins with multiple levels with the size of the bins changing at each level;computing a histogram over the partitioned feature space using the set of feature vectors for the data object: andapproximating the similarity to another data object according to partial matching in similar feature space.

6. The method as recited in claim 5 wherein characterizing the data object by a set of feature vectors comprises: detecting points of interest within a data object that relate with the plurality of bins at each level of the feature space;recognizing corresponding points along the different levels of the feature space which provide feature descriptors; andfrom the feature descriptors and points of interest provide the set of feature vectors.

7. The method as recited in claim 5 wherein the partitioning of feature space is in the form of a pyramid with variable bin sizes.

8. The method as recited in claim 5 wherein the plurality of bins with multiple levels vary in bin size from the smallest bin at the finest resolution varying in size getting larger at each level until at the last level the bin size encompasses the entire feature space.

9. The method as recited in claim 5 wherein approximating the similarity to another object comprises comparing the histograms with a weighted histogram intersection computation to approximate the similarity of the best partial matching between feature sets.

10. A method for matching data objects comprising: characterizing each data object by a set of feature vectors;partitioning feature space into a plurality irregularly shaped and sized bins with multiple levels and forming a pyramid shape;encoding point sets of interest from the feature vectors into multi-resolution histograms;providing a matching value indicative of the probability of a match among data objects from any two histogram pyramids.

11. The method as recited in claim 10 wherein partitioning feature space into a plurality of irregularly shaped and sized bins with multiple levels and forming a pyramid shape comprises: randomly selecting some example feature vectors from a feature type of interest;form a representative feature corpus from the randomly selected examples; andperform a hierarchical clustering operation to build a pyramid tree.

12. The method as recited in claim 11 wherein the hierarchical clustering operation comprises performing a hierarchical k-means clustering using the Euclidean distance to build the pyramid tree.

13. The method as recited in claim 10 wherein each irregularly shaped and sized bin has a center and for each point of interest, calculating a bin index, a bin count and a maximal distance to the bin center; choosing a bin at each level of the pyramid; andproviding a path vector indicative of the bins chosen at each level.

14. The method as recited in claim 10 wherein partitioning feature space into a plurality of irregularly shaped and sized bins with multiple levels and forming a pyramid shape comprises determining the number of levels and determining the number of bins in each level.

15. The method as recited in claim 11 wherein boundaries of the plurality of irregularly shaped and sized bins are determined by Voronoi cells surrounding cluster centers.

16. The method as recited in claim 10 wherein encoding point sets of interest from the feature vectors into multi-resolution histograms is determined by nearest bin centers of the pyramid.

17. The method as recited in claim 10 wherein encoding point sets of interest from the feature vectors into multi-resolution histograms includes providing a bin index, a bin count and a distance to a bin center value for each point.

18. A method for matching objects comprising: creating a set of feature vectors for each object of interest;mapping each set of feature vectors to a single high-dimensional vector to create an embedding vector; andencoding each embedding vector with a binary hash string.

19. The method as recited in claim 18 further comprising providing, from an encoded embedding vector, a matching value indicative of the normalized partial matching score among any two objects that generated the histograms pyramids.

20. The method as recited in claim 19 wherein providing a matching value comprises taking the dot product of two encoded embedding vectors.