Patent Application
20230099580
This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2021-155876 filed Sep. 24, 2021.
The present invention relates to a collation device, a non-transitory computer readable medium storing a program, and a collation method.
JP6455679B discloses a processing system that has satin forming means for forming a satin pattern for identifying a part, a product, or a product having a part as a component, on a part or a product having an information display body attached that displays information related to the part, the product, or the product having the part as a component.
JP6708981B discloses a technology that generates a layer that contains minute particles on an object and has an indefinite planar shape, acquires an image of the generated layer, and extracts a feature amount that depends on a planar shape of the layer and a distribution of particles as an individual identifier of the object, from an image.
JP4930789B discloses an individual recognition device that recognizes an overlapping object in a captured image for each individual and includes means for forming a binarized image of the captured image, means for extracting an edge from the captured image, means for forming an image in which the edge is removed from the binarized image by performing difference processing of the binarized image and the edge, and means for combining the binarized images separated by the edge based on a length of the object in the image.
In a system that uniquely identifies an object by photographing a surface image of the object to acquire a collation area, and performing image collation between a pre-registered image of a random pattern of a fine pattern on a surface of the object with the collation area, in a case where a user photographs the collation area of the object by using imaging means such as a mobile terminal and collates with the registered image, in a case where the collation area exists on a printing substrate having unevenness such as a hologram, it is difficult to detect the collation area due to the unevenness of the printing substrate.
Aspects of non-limiting embodiments of the present disclosure relate to a collation device, a non-transitory computer readable medium storing a program, and a collation method that provide a technology capable of detecting a collation area even in a case where the collation area exists on a printing substrate having unevenness such as a hologram.
Aspects of certain non-limiting embodiments of the present disclosure overcome the above disadvantages and/or other disadvantages not described above. However, aspects of the non-limiting embodiments are not required to overcome the disadvantages described above, and aspects of the non-limiting embodiments of the present disclosure may not overcome any of the disadvantages described above.
According to an aspect of the present disclosure, there is provided a collation device including a processor configured to, by executing a program: (a) acquire a photographed image including a collation area provided on a printing substrate having unevenness, (b) simultaneously execute smoothing processing and shading difference enhancement processing on the photographed image, and (c) detect the collation area based on an image obtained by simultaneously executing the smoothing processing and the shading difference enhancement processing.
Exemplary embodiment(s) of the present invention will be described in detail based on the following figures, wherein:
Hereinafter, an exemplary embodiment of the present invention will be described by taking an individual identification system that uniquely identifies an object by photographing a surface image of the object and performing image collation between a registered image and a collation image as an example based on the drawings.
The individual identification system is a technology that registers an image of a part of a surface of an object, specifically about 0.1 to several mm in advance as information unique to the object and uniquely identifies that the object to be collated is the same as the registered object, that is, the object is genuine, and the information unique to the object is, for example, a random pattern by a fine pattern. A satin pattern is a specific example of the random pattern by the fine pattern. The satin pattern is not limited to a surface treatment such as frosted glass, and is a concept that includes not only the satin pattern applied by treatment processing to metal or synthetic resin (plastic or the like), but also a wrinkle pattern obtained by embossing treatment and randomly woven fiber pattern, a random fine dot pattern by printing, and a random particle distribution by printing with ink containing glitter particles. Further, the satin pattern includes not only an unintentionally formed satin pattern but also an intentionally formed satin pattern for identification or collation. In short, the satin pattern is a random pattern that is difficult to control and form. It may be said to be a kind of “artifact metrics” that optically reads such a random pattern and uses the random pattern as information.
Here, a case is assumed in which a printing substrate having unevenness such as a hologram is used as a printing substrate and a polygonal ink portion in which metal particles are dispersed is printed on the printing substrate having such unevenness to form a random pattern.
The ink portion 14 is formed by mixing metal particles such as aluminum with powder, and in a case of irradiating the ink portion with light, the light is reflected by the randomly stacked powder metal particles, and shows a pattern of shading in accordance with an intensity of the reflected light. In a case where an irradiation direction of light is changed, the random pattern of shading changes.
Generally, in a case of extracting an area having a specific shape from a photographed image, there is feature extraction processing or edge extraction processing. However, simply in the feature extraction processing or the edge extraction processing, since both unevenness of the hologram portion 12 and unevenness of the ink portion 14 are extracted as feature points, it is difficult to stably extract the shape of the ink portion 14 due to the rainbow color change of the hologram portion 12.
(a) to (d) of
(a) of
(b) of
(c) of
(d) of
(b) of
(d) of
it is difficult to extract the square of the ink portion 14 in both cases.
Therefore, in the present exemplary embodiment, the ink portion 14 formed on the hologram portion 12 by performing a series of processing on the original image, in other words, the ink portion 14 as a foreground is stably extracted from the hologram portion 12 as a background of the photographed image.
The object 10 is irradiated with a light source unit 21 such as an LED, and the light reflected from the ink portion 14 of the object 10 is photographed by the registered image photographing machine 20 to acquire a registered image 16. The registered image photographing machine 20 and the light source unit 21 can be composed of dedicated equipment for registration.
An irradiation angle φ of the irradiation light from the light source unit 21 is set to a certain fixed angle. The acquired registered image 16 is transmitted to the server computer 50 and stored in a registered image DB 50b in the server computer 50.
On the other hand, the object 10 is photographed by using a mobile terminal such as a smartphone held by the user of the collation system as the collation image photographing machine 22. The object 10 is irradiated with a light source unit 22a such as an LED mounted on a smartphone or the like, and the light reflected from the ink portion 14 of the object 10 is photographed by a camera unit 22b mounted on the smartphone or the like. An irradiation angle φ of the irradiation light from the light source unit 22a is set to be substantially the same as the angle φ which is the condition in a case where the registered image 16 is acquired. The reason is that, as described above, the random pattern of the ink portion 14 changes depending on the irradiation direction of light, so that it is necessary to set a positional relationship between the light source unit 22a, the camera unit 22b, and the object 10 to be substantially the same as a positional relationship at the time of photographing the registered image 16.
A processor of the collation image photographing machine 22 performs a series of processing on the photographed image to extract the ink portion 14 from the photographed image, further cuts out the collation image 18 from the area of the ink portion 14, and transmits the collation image 18 to the server computer 50 via the communication network. The processing of the processor of the collation image photographing machine 22 will be further described later.
The server computer 50 includes a collation unit 50a and the registered image DB 50b.
The registered image DB 50b is composed of a storage device such as a hard disk or a solid state drive (SSD), and stores an identifier ID for uniquely specifying the object 10 and the registered image 16 in association with each other.
The collation unit 50a is composed of a processor, and stores the registered image 16 received from the registered image photographing machine 20 in the registered image DB 50b in association with the ID of the object 10. Further, the collation unit 50a performs image collation between the collation image 18 received from the collation image photographing machine 22 and the registered image 16 stored in the registered image DB 50b, and outputs the collation result to the collation image photographing machine 22. Specifically, the collation unit 50a reads the registered image 16 from the registered image DB 50b, performs collation calculation with the collation image 18, and calculates a degree of similarity between the two images. For the calculation of the degree of similarity, feature amount matching by feature amount detection, template matching using image shading comparison, and the like can be used. The calculated degree of similarity is compared with a threshold, and in a case where the degree of similarity exceeds the threshold, it is determined that the images match, and in a case where the degree of similarity does not exceed the threshold, it is determined that the images do not match. The collation unit 50a transmits the collation result to the collation image photographing machine 22 via the communication network.
In the image collation, there is an error rate due to fluctuations in the input of the registered image photographing machine 20 or the collation image photographing machine 22 to an image sensor, quantization error, and the like. The error rate consists of two, a false rejection rate, which is the probability of determining false even though it is true, and a false acceptance rate, which is the probability of determining true even though it is false. The two are in a trade-off relationship, and in a case where one decreases, the other increases. Therefore, a threshold is set so that the loss in the application target of the collation determination is the smallest.
Note that a plurality of the registered images 16 may be acquired by changing the irradiation direction of light and registered in the registered image DB 50b of the server computer 50, and the image collation between the plurality of registered images 16 and the collation image 18 may be performed.
The processor 22c reads out an application program stored in the ROM 22d, executes a series of processing by using the RAM 22e as a working memory, extracts the ink portion 14 from the photographed image photographed by the camera unit 22b, and further cuts out the collation image 18. The processor 22c transmits the cut-out collation image 18 to the server computer 50 via the communication I/F 22h. Further, the processor 22c receives the collation result from the server computer 50 via the communication I/F 22h.
The input unit 22f is composed of a keyboard, a touch switch, or the like, and the user operates the input unit 22f to start the application program.
The output unit 22g is composed of a liquid crystal display, an organic EL display, or the like, and displays a preview image in a case where the object 10 is photographed. Further, in a case where the ink portion 14 is extracted by the processor 22c, the output unit 22g may display a notification indicating that the ink portion 14 is extracted by a control command from the processor 22c. Further, the output unit 22g may display a guide in a case where photographing the object 10 by the control command from the processor 22c. The guide is, for example, a guide for making the irradiation angle of the irradiation light from the light source unit 22a a fixed angle cp. Further, the output unit 22g displays the collation result received from the server computer 50 by the control command from the processor 22c. The collation result is either “match” or “mismatch”, but other messages related to the collation may be displayed.
The purpose of the processing flowchart is to acquire coordinates of the four vertices P1 to P4 of the ink portion 14 of the square (quadrangle) from the photographed image, and the processing is substantially classified into three processing of binarized image generation processing (S1), rectangular edge extraction processing (S2), and vertex coordinate estimation processing (S3).
Binarized Image Generation Processing
First, the binarized image generation processing (S1) will be described.
In the processing, first, smoothing processing and shading difference enhancement processing are simultaneously executed on the original image (S101). As described above, the shape of the ink portion 14 is blurred by simply performing the smoothing processing on the original image. Further, although the unevenness of the ink portion 14 is emphasized by simply performing the shading difference enhancement processing on the original image, the unevenness of the hologram portion 12 is also emphasized at the same time, so that the ink portion 14 cannot be extracted.
Therefore, in the present exemplary embodiment, the smoothing processing and the shading difference enhancement processing are simultaneously executed on the original image to remove the unevenness of the hologram portion 12 and the ink portion 14, and the ink portion 14 is identified from the hologram portion 12. Specifically, a mean-shift filter can be used for simultaneous execution of the smoothing processing and the shading difference enhancement processing.
(a) of
Note that although there is a filtering method that retains the edges and performs smoothing processing, such as a bilateral filter, the inventors have confirmed that the noise in the hologram portion 12 and the ink portion 14 cannot be removed by the method. By using the mean-shift filter, the edges are retained and smoothed for each color by using the color difference between the hologram portion 12 and the ink portion 14, and the noise can be removed without losing the edges.
Condition:|x-xc|≤sp,|y-yc|≤sp,∥(r,g,b)−(rc,gc,bc)∥≤sr
Then, in a case where the above condition is satisfied,
by setting (x, y, r, g, b)=(xg, yg, rc, gc, bc),
the center of gravity is searched again. The above center of gravity search processing is repeatedly executed (NO in S202).
Then, a color space distance c and the number of repetitions n are set in advance, it is determined whether or not the following condition is satisfied, and in a case where the condition is satisfied, the processing ends (YES in S202).
Condition: the number of repetitions n is satisfied,
or
|x−xc|+|y-yc|+(r-rc)2+(g-gc)2+(b-bc)2<ε.
After the center of gravity search processing ends, smoothing is performed with the value of the center of gravity in the color space (S203). That is, after the polar search ends, each pixel in the space is set as a center of gravity value of the color space. The edges are then clarified by using the Gaussian pyramid and the threshold sr.
The mean-shift filter 24 performs the smoothing processing by using a distance difference in the color space, which is an effective smoothing processing in a case where there is a difference in the color space distance between the foreground and the background, and therefore, is an effective processing for the original image 11 in which an achromatic ink portion 14 exists in the foreground and a chromatic hologram portion 12 exists in the background.
In the mean-shift filter 24, the performance of the smoothing processing and the shading difference enhancement processing can be controlled by using a color space radius sr and a pixel space radius sp as major parameters. Therefore, by adjusting the parameters, a ratio of the smoothing processing and the shading difference enhancement processing can be adjusted. Specifically,
(1) since a search range of pixels smoothed (filled) by the pixel space radius sp is designated,
in a case where sp is large→search range can be adjusted to be wide, and
in a case where sp is small→search range can be adjusted to be narrow.
Note that in a case where the sp is set too large, it takes a long time to processing, for example, so it is desirable to take this into consideration.
(2) Since the range of similar colors to be filled in the same color is determined by the color space radius sr,
in a case where sr is large→it can be adjusted so that even slightly different colors are recognized as the same color, and
in a case where sr is small→it can be adjusted so that similar colors are recognized as the same color.
Therefore, for example, it is desirable to adjust the values of the parameters sr and sp as follows depending on whether the printing substrate is the hologram portion 12 or other cases, for example, paper. In a case where the printing substrate is the hologram portion 12, since there is a color difference between the silver ink portion 14 and the rainbow-colored hologram portion 12, the color space radius sr is set relatively large, and the pixel space radius sp is set to a desired value in consideration of the processing time and the effect. In a case where the printing substrate is paper, since the color difference between the silver ink portion 14 and the paper is small, the color space radius sr is set relatively small and the pixel space radius sp is set relatively small.
(a) and (b) of
(sp, sr)=(10, 30), and
the shape of the ink portion 14 is extracted.
On the other hand, (c) and (d) of
(sp, sr)=(5, 10) and,
similarly, the shape of the ink portion 14 is extracted. In a case where the printing substrate is the paper 15, both sp and sr are set to be relatively small as compared with the case where the printing substrate is the hologram portion 12. In other words, in a case where the printing substrate is the hologram portion 12, both sp and sr are set relatively large as compared with the case where the printing substrate is the paper 15, and two parameters sp and sr are variably set in accordance with the printing substrate.
As described above, the processor 22c simultaneously executes the smoothing processing and the shading difference enhancement processing on the original image photographed with the mean-shift filter 24.
Referring back to
In a case where the ink portion 14 is photographed by the collation image photographing machine 22 such as a smartphone, the color of the hologram portion 12 around the ink portion 14 changes in a case where an irradiation position of the light source unit 22a changes. That is, in a case where the color of the hologram portion 12 changes and there is no sufficient difference in the color space distance from the ink portion 14, the foreground and the background are assimilated, and there may be portions where the shading difference enhancement is insufficient only by the processing of S101. Therefore, by further executing the shading difference enhancement processing, the shape of the ink portion 14 is extracted more stably.
Specifically, the processed image 13 in S101 is decomposed into RGB, and the shading difference enhancement processing is executed in each RGB color space. This means flattening of an in-image brightness histogram. Then, in order to extract an edge gradient, a Sobel filter for each of the vertical and horizontal directions is applied to each RGB color image. Note that since a gradient value calculated by the Sobel filter is not eight bits (256 gradations), this may be normalized to eight bits. The normalization method is processing of taking an absolute value of a gradient image and replacing all the pixel values of 255 or more with 255. As a result, the edge gradient can be acquired independently of disturbance noise.
Referring back to
In a case where a rough shape of the ink portion 14 is extracted in S102, noise is generated at a boundary between the white color and the light blue color of the hologram portion 12. In particular, since the gradients of the white color and the light blue color in an R space image are large, noise such as an edge is generated in a case where the Sobel filter is applied. Therefore, the noise is removed by using the HSV color space. Specifically, (1) the processed image 13 in S101 is HSV decomposed (2) S image is binarized (3) vertical and horizontal Sobel filter is applied to S binary image (4) black-and-white inverted binary image of vertical and horizontal Sobel image and H image are OR-synthesized.
Referring back to
Rectangular Edge Extraction Processing
Next, the rectangular edge extraction processing will be described.
After creating the binarized image in S104, the sides of the polygon constituting the square ink portion 14 are acquired from the binarized image (S105). Specifically, this is edge extraction processing using a stochastic Hough transform. Note that the stochastic Hough transform is an optimization of the Hough transform, and instead of calculating using all the pixels, points sufficient for straight line detection are randomly selected from the image and calculated. A (non-stochastic) Hough transform can also be used in S104. However, parameter tuning is difficult, and there is a drawback that the sensitivity is too good for the rectangular edges of the binarized image.
After acquiring the sides of the polygon (S105), the processing of removing extra sides is executed (S106). That is, straight lines that are not rectangular edges (sides) are removed from the straight lines extracted by the stochastic Hough transform. Specifically, a method of removing a straight line having an inclination of a fixed value or more, and a long straight line with respect to the size of the collation area 14, removing a straight line of which an angle in which vertical and horizontal lines intersect within a fixed angle, or removing a straight line in contact with an image frame and the like is used. In addition to this, extra sides may be removed by extracting edges with a rectangular hue by using a color space.
Vertex Coordinate Estimation Processing
Referring back to
In the processing, the centroid coordinates of the intersections of the sides are calculated from the image obtained by removing the extra sides in S106 (S107). That is, instead of the intersections consisting of each side, the centroid coordinates of an intersection group within a certain vicinity are calculated. Although the intersections of vertical and horizontal straight lines are calculated for the processed image 23 by addressing a one-dimensional simultaneous equation, since an edge width of the binarized image after OR synthesis is two to three pixels, a plurality of straight lines are extracted for the same edge by the stochastic Hough transform. Therefore, there are a plurality of intersections in the vicinity of certain coordinates. Since these intersections are likely to indicate the same vertex, the centroid coordinates of the intersection group are acquired, and the centroid coordinates are redefined as the vertices of the shape of the ink portion 14.
First, a plurality of intersections in a certain vicinity are dilated and combined into one (S301). The dilation processing is processing in which in a case where there is a white pixel in peripheral pixels of a certain pixel, the pixel is converted into the white pixel thereby sequentially expanding the white pixel. Next, labeling is performed on each intersection set that has been dilated (S302). Then, the centroid coordinates of each labeled intersection set are calculated (S303). In a case where the centroid coordinates are calculated as described above, the calculated centroid coordinates are set as the vertex candidates (S304).
Since the square ink portion 14 has four vertices P1 to P4 (refer to
Then, it is determined whether or not all the vertices of the ink portion 14 have been acquired (S108). In the square ink portion 14, it is determined that all the vertices have been acquired in a case where the four vertices P1 to P4 are acquired. In a case where all the vertices have not been acquired (NO in S108), it means that all the sides of the ink portion 14 have not been extracted, so the complementary processing of the disappeared side is executed next (S109).
In the complementary processing of the side, it is determined whether or not three sides constituting the square have been extracted from the square ink portion 14. Normally, in a case where the ink portion 14 is printed on the hologram portion 12 as a printing substrate, in a case where the red of background of the hologram portion 12 is covered with the red of the foreground, extraction of the sides may fail. In short, it is a case where a color space distance difference between the background and the foreground is small. Therefore, it is first determined whether or not the three sides have been extracted. The selection of the three sides can be estimated from the known shape characteristics of the ink portion 14, that is, the length or the position of the edges.
In a case where the three sides are extracted, a length x of the side having no opposite side among the three sides is calculated from the centroid coordinates calculated in S107. Then, a new parallel side is drawn at a portion separated by the length x of the side. Specifically, it is assumed that the four sides constituting the square are a, b, c, and d, a and c are opposite sides, and b and d are opposite sides, and in a case where only three sides of a, b, and c are extracted, a side parallel to b is drawn at a position separated by x from b to be d.
As parallel sides separated by x from b, it is possible to estimate a total of two, one on each of the both sides of b, and since one of the sides does not exist in the image, the side d can be uniquely drawn. This complements the disappeared sides.
Note that after the complementary processing of the disappeared sides, the threshold may be lowered for the binarized image obtained in S104, and the stochastic Hough transform may be executed again to reacquire the side, and the side obtained in this way and the side obtained by complementing in S109 may be integrated, and the processing may be transferred to the vertex coordinate estimation processing (S3) again.
In a case where the coordinates of the four vertices P1 to P4 of the ink portion 14 are acquired as described above, the processor 22c cuts out the collation image 18 with reference to the coordinates of these four vertices and transmits the collation image 18 to the server computer 50. The processor 22c converts the resolution into a predetermined size and then transmits the image to the server computer 50, as the collation image 18. In a case where transmitting to the server computer 50, a collation request is made by attaching the collation image 18 of which the resolution has been converted.
In the present exemplary embodiment, the square ink portion 14 has been described as an example, but the present exemplary embodiment is not limited to the square, and can be applied to any polygon.
In the embodiments above, the term “processor” refers to hardware in a broad sense. Examples of the processor include general processors (e.g., CPU: Central Processing Unit) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, and programmable logic device). In the embodiments above, the term “processor” is broad enough to encompass one processor or plural processors in collaboration which are located physically apart from each other but may work cooperatively. The order of operations of the processor is not limited to one described in the embodiments above, and may be changed.
Further, in the present exemplary embodiment, although the processor 22c of the collation image photographing machine 22 such as a smartphone executes the processing shown in
Further, in the present exemplary embodiment, although a case where the ink portion 14 is printed on the hologram portion 12 as the printing substrate, that is, a combination of the chromatic background and the achromatic foreground has been described, on the contrary, the processing of the present exemplary embodiment can be applied to a combination of the achromatic background and the chromatic foreground. A basic principle of the mean-shift filter 24 is to extract the shape of the foreground by using the color space distance difference between the achromatic color and the chromatic color, and the principle can be applied to a combination of the achromatic background and the chromatic foreground as well. As the combination of the achromatic background and the chromatic foreground, for example, there is a combination of the paper 15 as the printing substrate and the chromatic ink portion 14.
Further, in the present exemplary embodiment, in a case where the color space distance difference between the hologram portion 12 and the ink portion 14 is relatively small and the mean-shift filter 24 cannot sufficiently extract the shape of the ink portion 14, a known Canny method may be used in a complementary manner.
That is, the edge obtained by the stochastic Hough transform is masked for the binarized image obtained in S104.
Next, assuming that the straight line on the blue side has been extracted, the half of the blue side of the image is masked. This utilizes a fact that the mean-shift filtering is robust to the blue side and vulnerable to the red side, and the Canny method is vulnerable to the blue side and robust to the red side.
Next, the edge is extracted by the Canny method. Since the edge extracted by the Canny method is drawn with one pixel, a straight line may not be formed by the stochastic Hough transform, so the edge extracted by the Canny method is dilated. As a result, the edge of about three pixels is extracted, and the edge can be extracted even by stochastic Hough transform.
The followings are examples of methods that can be complementarily used in a case where the shape extraction by the mean-shift filter 24 is not sufficient.
Any one of these methods, or a plurality of these methods can be used in combination.
The foregoing description of the exemplary embodiments of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations will be apparent to practitioners skilled in the art. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, thereby enabling others skilled in the art to understand the invention for various embodiments and with the various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-155876 | Sep 2021 | JP | national |
