This application claims the priority of Korean Patent Application No. 10-2021-0096529 filed on Jul. 22, 2021, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.
The present disclosure relates to a method for performing image convolution by considering a hierarchical relationship between hyperbolic feature vectors in a hyperbolic space.
Recently, in a deep learning field using a convolutional neural network (CNN), a data learning method in a hyperbolic space has been highlighted because of effective formation of a hierarchical relationship.
In particular, as a need for deep learning on graph-structured data that may represent a relationship between data is on the rise, a data learning method considering a hierarchical relationship is emerging as a core research field to keep pace with the explosive growth of an AI field.
However, the data learning method in the hyperbolic space studied so far has a limit to be applied only to fields designed so that data itself has a hierarchical structure, such as a knowledge graph or a synonym hierarchy, and thus there was a problem of low usability in a computer vision field.
The above-described technical configuration is the background art for helping in the understanding of the present invention, and does not mean a conventional technology widely known in the art to which the present invention pertains.
An object of the present disclosure is to provide an image convolution method capable of allocating importance to each pixel constituting a hyperbolic feature vector in consideration of a hierarchical relationship between hyperbolic feature vectors embedded on a hyperbolic space.
The objects of the present disclosure are not limited to the above-mentioned objects, and other objects and advantages of the present disclosure, which are not mentioned, will be understood through the following description, and will become apparent from the embodiments of the present disclosure. In addition, it will be appreciated that the objects and advantages of the present disclosure will be easily realized by those skilled in the art based on the appended claims and a combination thereof.
According to an aspect of the present disclosure, there is provided an image convolution method in a hyperbolic space including steps of embedding an image feature vector on a Euclidean space into a hyperbolic feature vector on a hyperbolic space, allocating a hierarchical weight on the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector, and convolutioning the hyperbolic feature vector by applying the hierarchical weight.
In one embodiment, the embedding step may include embedding each pixel constituting the image feature vector on the hyperbolic space through a mapping function.
In one embodiment, the embedding step may include embedding the image feature vector into the hyperbolic feature vector on a Poincaré ball through a one-to-one correspondence function connecting the Euclidean space and the Poincaré ball.
In one embodiment, the allocating of the hierarchical weight may include allocating the hierarchical weight based on a geodesic distance of the hyperbolic feature vector.
In one embodiment, the allocating of the hierarchical weight may include allocating a hierarchical weight inversely proportional to the geodesic distance of the hyperbolic feature vector.
In one embodiment, the allocating of the hierarchical weight may include allocating the hierarchical weight based on a distance between a reference pixel and remaining pixels constituting the hyperbolic feature vector.
In one embodiment, the allocating of the hierarchical weight may include generating a hierarchical weight vector consisting of the hierarchical weight.
In one embodiment, the allocating of the hierarchical weight may include rearranging the hyperbolic feature vector according to the hierarchical weight, and arranging the hierarchical weight in the rearrangement order to generate a hierarchical weight vector.
In one embodiment, the allocating of the hierarchical weight may include rearranging the hyperbolic feature vector in the size order of the hierarchical weight, and generating a hierarchical weight vector arranged in the size order of the hierarchical weight.
In one embodiment, the convolutioning step may include multiplying the hyperbolic feature vector by the hierarchical weight vector consisting of the hierarchical weight, and convolutioning the hyperbolic feature vector multiplied by the hierarchical weight vector.
According to the present disclosure, by allocating a hierarchical weight to each pixel constituting a hyperbolic feature vector, it is possible to give importance to a pixel having high affinity in performing deep learning. Accordingly, it is possible not only to increase a learning effect and learning efficiency of a spatial propagation neural network, but also to significantly improve the learning accuracy for learnable parameters used in deep learning.
In addition, according to the present disclosure, it is possible to rearrange hyperbolic feature vectors according to importance in allocating a weight to a hyperbolic feature vector. Accordingly, the arrangement of pixels in the hyperbolic feature vector may have a tendency, and when the tendency is applied to a convolutional neural network, learning consistency and learning accuracy are improved to establish a robust model.
In addition to the above-described effects, specific effects of the present disclosure will be described together with explanation of specific matters for carrying out the present disclosure.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The above and other aspects, features and other advantages of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:
The above-described objects, features and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present disclosure pertains will be able to easily implement the technical idea of the present disclosure. In describing the present disclosure, a detailed description of related known technologies will be omitted if it is determined that they unnecessarily make the gist of the present disclosure unclear. Hereinafter, preferred exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, like reference numerals are used to indicate like or similar components.
Hereinafter, the fact that an arbitrary configuration is disposed at the “upper (or lower)” of the component or “on (or below)” the component may mean that not only that an arbitrary component is disposed in contact with an upper surface (or lower surface) of the component, but also that other components may be interposed between the component and any component disposed on (or below) the component.
Further, a singular form used in the present specification may include a plural form if there is no clearly opposite meaning in the context. In this specification, the term such as “comprising” or “including” should not be interpreted as necessarily including all various components or various steps disclosed in the specification, and it should be interpreted that some component or some steps among them may not be included or additional components or steps may be further included.
Throughout this specification, unless otherwise described to the contrary, “A and/or B” means A, B, or A and B, and unless otherwise described to the contrary, “C to D” means C or more and D or less.
The present disclosure relates to a method for performing image convolution by considering a hierarchical relationship between hyperbolic feature vectors in a hyperbolic space. More specifically, the present disclosure is a method applied to a neural network using spatial propagation, and may be used to spatially propagate a sparse input to generate a dense output.
Hereinafter, an image convolution method of the present disclosure will be described in detail with reference to
Referring to
In addition, the image convolution method may include allocating a hierarchical weight on the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector (S120). In addition, the image convolution method may include convolutioning the hyperbolic feature vector by applying the previously allocated hierarchical weight (S130).
Each of steps S110 to S130 described above may be programmed and implemented through software, or may also be implemented through hardware including a processing unit such as a graphic processing unit (GPU) and a central processing unit (CPU). Hereinafter, for convenience of description, it is assumed that each of steps S110 to S130 of the present disclosure is performed by a convolution module. Hereinafter, each step illustrated in
The convolution module may receive the image feature vector on the Euclidean space and embed the received image feature vector on the hyperbolic space (S110). More specifically, the convolution module may convert the image feature vector located on the Euclidean space into a hyperbolic feature vector located on the hyperbolic space.
The hyperbolic space is a non-Euclidean space, and may be a homogeneous space having a uniform negative curvature at all points on the space. The hyperbolic space may include a multidimensional vector that is difficult to be shaped.
In the following drawings, it is assumed that the hyperbolic space is expressed in a three dimension in order to shape the hyperbolic space. However, as described above, the hyperbolic space may include a multidimensional vector of four or more dimensions.
Referring to
The mapping function M may be a one-to-one correspondence function for mapping a Euclidean space to a hyperbolic space or the hyperbolic space to the Euclidean space. For example, the mapping function M may be defined as a linear function, and in this case, each coefficient constituting the linear function may be a learnable parameter.
Meanwhile, referring to
Meanwhile, since a multiple operation and a sum operation constituting the mapping function M are performed on the hyperbolic space, Möbius sum operation and multiple operation may be used. More specifically, the Möbius multiple operation may be expressed as in [Equation 1] below, and the Möbius sum operation may be expressed as in [Equation 2] below.
(M represents a matrix, u represents a vector, k represents a curvature of a Poincaré ball, and ||·|| represents a Euclidean norm)
(u and v represent vectors, k represents a curvature of a Poincaré ball, and ·, · represents a Euclidean inner product)
As described above, when the embedding is completed, the convolution module may allocate the hierarchical weight on the hyperbolic feature vector based on the hierarchical property of the hyperbolic feature vector (S120).
Here, the hierarchical property is a property of similarity and/or affinity between the hyperbolic feature vectors. More specifically, the hierarchical property may include any parameter indicating any value or degree regarding the similarity and/or affinity of the vectors on the hyperbolic space.
In the mapping function M described in step S110, the hyperbolic feature vectors are uniformly distributed on the hyperbolic space, such that the image feature vectors are arranged on a regular grid in the Euclidean space. However, since the hyperbolic feature vector is projected into a manifold space through dimensionality reduction for data visualization, the above-described premise may not be satisfied. Accordingly, when the hyperbolic feature vector is convolutioned without additional data processing, there is a problem in that non-Euclidean spatial characteristics are not faithfully reflected.
In order to alleviate this problem, the convolution module may allocate the hierarchical weight to each hyperbolic feature vector based on a geodesic distance of the hyperbolic feature vector.
Referring to
The convolution module may calculate a hyperbolic feature vector embedded on the hyperbolic space, specifically, a geodesic distance between respective pixels constituting the hyperbolic feature vector, and allocated a hierarchical weight according to the calculated distance.
Referring to
The convolution module may calculate a geodesic distance between the respective pixels in the hyperbolic feature vector
The geodesic distance may be calculated according to
[Equation 3] below.
(dk represents a geodesic distance, u and v represent two points on the Poincaré ball, k represents a curvature of the Poincaré ball, and ||·|| represents a Euclidean norm)
The closer the geodesic distance between hyperbolic feature vectors in the hyperbolic space, the higher the affinity between respective pixels. In the neural network using spatial propagation, neural network learning may be performed in consideration of such affinity.
The convolution module of the present disclosure may give higher importance to a pixel having high affinity in order to improve the learning efficiency of a spatial propagation neural network (SPN). To this end, the convolution module may allocate a hierarchical weight that is inversely proportional to the geodesic distance between the hyperbolic feature vectors
Hereinafter, step S120 of allocating the hierarchical weight to the hyperbolic feature vectors {acute over (h)} will be described in detail with reference to
The convolution module may allocate a hierarchical weight based on a distance between a reference pixel rf and the remaining pixels of constituting the hyperbolic feature vector
Referring to the positions of pixels embedded in the Poincaré ball, a geodesic distance between a dimgray reference pixel rf and a darkgray pixel nf1 may be the shortest, and a geodesic distance between the dimgray reference pixel rf and an lightgray pixel nf2 may be the longest.
Since the geodesic distance from the dimgray reference pixel rf is 0, the convolution module may set the hierarchical weight for the dimgray reference pixel rf to a maximum value, and since the distance from the darkgray pixel nf1 is closer than the distance from the lightgray pixel nf2, the hierarchical weight for the darkgray pixel nf1 may be set higher than the hierarchical weight for the lightgray pixel nf2.
The convolution module may generate a hierarchical weight vector
A method of generating the hierarchical weight vector
The convolution module may generate a hierarchical weight vector
As described above, since the hierarchical weight of the dimgray reference pixel rf (row 2 column 2) is the maximum, in
As described above, the convolution module may generate a hierarchical weight vector
According to the present disclosure, by allocating the hierarchical weight to the hyperbolic feature vector
Meanwhile, according to the method of allocating the hierarchical weight described with reference to
may be randomly disposed according to a hierarchical property of the hyperbolic feature vector {acute over (h)}. Specifically, unlike those illustrated in
In order to solve the above-mentioned potential problems, the convolution module may rearrange a hyperbolic feature vector
Hereinafter, another method of generating the hierarchical weight vector g will be described with reference to
As described with reference to
Then, the convolution module may rearrange the hyperbolic feature vector
Referring to
At this time, the convolution module may arrange each pixel of the hyperbolic feature vector
Unlike those described above, the convolution module may rearrange each pixel in the decreasing order of the hierarchical weight, and may also arrange the pixels in the order from row 1 column 1 to row 3 column 1, from row 1 column 2 to row 3 column 2, and from row 1 column 3 to row 3 column 3.
Then, the convolution module may generate a hierarchical weight vector
Referring back to
As described above, when the allocation of the hierarchical weight and/or the generation of the hierarchical weight vector g are completed, the convolution module may convolution the hyperbolic feature vector by applying the hierarchical weight (S130). More specifically, the convolution module may first apply a hierarchical weight to the hyperbolic feature vector h and then convolution the hyperbolic feature vector
Referring to
Then, the convolution module may convolution the hyperbolic feature vector h applied with the hierarchical weight. In this case, various techniques used in the corresponding technical field may be applied to convolution, and in particular, a technique used in a spatial propagation neural network (SPN) may be used.
According to the method of applying the hierarchical weight described with reference to
The arrangement of output pixels may have a tendency according to importance, and when the tendency is applied to a convolutional neural network, learning consistency and learning accuracy are improved to establish a robust model. Referring to
Then, according to the present disclosure, it is possible to apply a weight to each pixel based on the geodesic distance between pixels in the embedded depth sample, and convolution the hyperbolic feature vector
In light of the convolution result of the present disclosure illustrated in
As described above, the present disclosure has been described with reference to the illustrated drawings, but the present disclosure is not limited to the exemplary embodiments of the present disclosure and the drawings, and it will be apparent that various modifications can be made by those skilled in the art within the scope of the technical idea of the present disclosure. Further, it is natural that even through effects according to the configuration of the present disclosure are not explicitly described while describing the exemplary embodiments of the present disclosure above, expectable effects should be recognized by the configuration.
Number | Date | Country | Kind |
---|---|---|---|
10-2021-0096529 | Jul 2021 | KR | national |