Patent Grant
10930037
1. Field of the Invention
The present invention relates to an image processing device for displaying an object detected from an input picture image, and more particularly, relates to an image processing device for detecting positions of images of multiple objects shown in a picture image, and displaying a detection result in order to allow an operator to check whether the image of the detected object is correct.
2. Description of the Related Art
When an image of a particular object is detected from a picture image in a visual field of an image-capturing apparatus by using an image processing device, in general, feature values are compared between a reference information indicating an object (which is generally referred to as a model pattern, a template, or the like) and an input picture image obtained by the image-capturing apparatus, and when the degree of match is more than a designated level (threshold value), the object is determined to have been successfully detected.
However, even when the degree of match is more than the threshold value, the image of the object is not necessarily correctly detected in reality. Therefore, in order to adjust a detection parameter such as a threshold value for the degree of match, an operator may visually check whether the object is correctly detected or not. In order to check whether an object is correctly detected or not, a method of plotting an origin of the model pattern, a rectangle enclosing the model pattern, and a shape of the model pattern at the detected position on the input picture image is often employed.
Further, not only when the object is detected from the input picture image but also when the object in the input picture image is inspected, it is necessary for the operator to visually check whether a result of the correctness and incorrectness determination is appropriate or not in the stage of the parameter adjustment. In this case, it is desired to display a determination result in an easy-to-understand manner by, e.g., attaching a color to a defect portion and enclosing an object, i.e., a defective piece, in a different color.
Machine learning for improving an object detection probability and statistical processing for improving a model pattern may be preferably performed on the basis of a detection result and a correctness and incorrectness determination result. In a technique called supervised learning of machine learning, input data and a label thereof (OK/NG or a type) are given. In order to perform supervised learning on the basis of a detection result and a correctness and incorrectness determination result, a check is performed to determine whether the detection result and the result of the correctness and incorrectness determination are correct or not, and when there is an incorrect result, the label is corrected with a button, a text box, and the like. In the statistical processing, processing is performed on the image of the object of the correct determination result, and a threshold value for determining correctness and incorrectness is calculated by using the determination result of the correctness and incorrectness. In this case, learning is considered to include machine learning and statistical processing.
Japanese Unexamined Patent Publication (Kokai) No. 2007-114843 describes a correctness and incorrectness determination for allowing an operator to input a correctness and incorrectness determination result and setting a threshold value for performing correctness and incorrectness determination with a picture image and the correctness and incorrectness determination result. Selection of a picture image is performed by selecting a picture image file name, and the correctness and incorrectness determination result cannot be checked unless images are opened one by one. A single correctness and incorrectness determination result is considered to be present for a picture image, and it is not expected to determine correctness and incorrectness of multiple objects in a picture image.
International Publication No. 2001/041068 describes a defect inspection system in which, when a parameter of defect extraction is changed, defect extraction is performed again on all the stored picture images, and the correctness and incorrectness determination as well as the thumbnail picture images are displayed as a list. When a defect determination is newly performed, portions determined to have multiple defects are shown on the picture image. In this state, in order to enlarge a region determined to be a defect and register the defect to a defect dictionary, it may be preferable to click a defect portion on the picture image. Therefore, multiple defects may not be enlarged and displayed at a time. In addition, it is troublesome to perform a work of registering a defect.
Japanese Unexamined Patent Publication (Kokai) No. 2013-142558 describes an external appearance inspection apparatus in which, when a picture image, which is to be a non-defective piece, is newly added to a picture image group with which correctness and incorrectness are configured, the external appearance inspection apparatus calculates a threshold value for determination using the picture image group of the non-defective piece, and checks whether the defective piece is determined to be a defective piece with the threshold value. In this patent application, picture images are displayed one by one to allow correction of correctness and incorrectness, but it is not easy to check correctness and incorrectness for many picture images. Determination for multiple portions in a single picture image is not taken into consideration.
As explained above, when a detection result and a correctness and incorrectness determination result are checked, a method of plotting the detection result and the correctness and incorrectness determination result on a picture image is often employed.
However, this method involves the following problems.
It is an object of the invention to provide an image processing device for displaying an object detected from an input picture image in such a manner that a detection result and a correctness and incorrectness determination result can checked easily.
An image processing device includes an object detection unit for detecting one or more images of objects from an input picture image in which the objects have been captured, on the basis of a model pattern of the object, and a detection result display unit for graphically superimposing and displaying a detection result on each image thus detected. The detection result display unit includes a first frame for displaying the entire input picture image, and a second frame for listing and displaying one or more partial picture images each including an image detected from the input picture image. On the input picture image displayed in the first frame, detection results of all the detected images are superimposed and displayed, and on the partial picture image displayed in the second frame, a detection result of an image corresponding to each partial picture image is superimposed and displayed.
The present invention will be understood more clearly by referring to the following accompanying drawings:
An image processing device for displaying an object detected from an input picture image will be described below with reference to the drawings. However, it should be understood that the invention is not limited to the drawings or embodiments described below.
In the following explanation in the embodiments, the following reference symbols are used as follows. In order to clarify the explanation, the reference symbols will be explained first.
NP: the number of feature points constituting a model pattern
P_i: the i-th feature point of the model pattern (i is 1 to NP)
NI: the number of input picture images
I_j: the j-th input picture image (j is 1 to NI)
NT_j: the number of images of objects detected from the j-th input picture image I_j
T_jg: an image of the g-th object detected from the j-th input picture image I_j (g is 1 to NT_j)
L1_jg: first label of the image of the g-th object detected from the j-th input picture image I_j
L2_j: second label of the j-th input picture image I_j
As illustrated in
The visual sensor 11 is connected to the image processing device 10 with a communication cable. The visual sensor 11 provides captured picture image data to the image processing device 10. The operation board 12 is connected to the image processing device 10 with a communication cable. The operation board 12 is used to operate a screen displayed on the display apparatus 13. The display apparatus 13 is connected to the image processing device 10 with a communication cable. The display apparatus 13 displays a picture image captured by the visual sensor 11, a detection result, and the like. It should be noted that the visual sensor 11, the operation board 12, and the display apparatus 13 may be integrally provided in the image processing device 10. The display apparatus 13 is a display device having a CRT, a liquid crystal panel, or the like. The operation board 12 may be a mouse and a keyboard, or may be a touch panel on the display apparatus 13.
As illustrated in
In this first embodiment, it is assumed that a camera is used as the visual sensor 11, and the visual sensor 11 outputs a grayscale picture image in the explanation. The camera is an electronic camera having an imaging element such as a CCD (Charge Coupled Device), and is a well-known photodetector having a function of detecting a two-dimensional picture image on its image capturing plane (on a CCD array plane) by capturing an image. It should be noted that a two-dimensional coordinate system on an image capturing plane will be hereinafter referred to as an image coordinate system.
As illustrated in
As illustrated in
In accordance with the processing performed by the learning unit 22, the contents of the operation and processing performed by the operator with the image processing device are different. In this case, first, a case where the learning unit 22 performs statistical processing will be explained as an example.
In step S101, a model pattern used for detection of an object is generated.
The model pattern according to the first embodiment is constituted by multiple feature points. Various things can be used as the feature points, but in the first embodiment, an edge point is used as a feature point. The edge point is a point where a luminance gradient is high in a picture image, and can be used in order to obtain a contour shape of an object. A method for extracting an edge point is well-known. The edge point may be obtained with any granularity, but in this case, a single edge point is extracted for a single pixel.
The feature point is not limited to the edge point, and, for example, a feature point such as SIFT may be used. A method for extracting a SIFT feature point from a picture image is well known as described in “Object Recognition from Local Scale-Invariant Features”, David G. Lowe, Proc. of the International Conference on Computer Vision, Corfu (September 1999) and the like.
The feature point includes at least a position. When a direction of a luminance gradient of an edge point is defined as a posture of the feature point, the position as well as the position posture of the feature point can be defined. In the first embodiment, the feature point is considered to have a position and a posture.
The origin of the model pattern coordinate system may be defined in any way. For example, any given point is selected from feature points constituting the model pattern, and that point may be defined as the origin, and the centroid of all the feature points constituting the model pattern may be defined as the origin.
The posture of the model pattern coordinate system (axial direction) may also be defined in any way. For example, in the picture image in which the model pattern is generated, the picture image coordinate system and the model pattern coordinate system may be defined to be in parallel, or any two points may be selected from the feature points constituting the model pattern and a direction extending from one of the two points to the other may be defined to be an X axis direction.
As illustrated in
In step S102, one or more input picture images I_j (j=1 to NI) including an image of an object are obtained. The input picture image is obtained by arranging an object in the visual field of the visual sensor 11 and capturing a picture image. The picture image is preferably captured in the same environment as the actual production line. A large number of picture images of the object that are captured in the actual production line and saved may be used. Multiple objects may be shown in a single input picture image.
In step S103, the object detection unit 21 compares the model pattern with the input picture image I_j for each of the input picture images I_j (j=1 to NI), and performs detection of the object. The detection result is associated with the input picture image I_j and stored in the detection result storage unit 33. A result obtained from a series of processing up to execution of the following learning process is considered to be stored in the detection result storage unit 33. There are various methods for detecting an object with the object detection unit 21, but, for example, Generalized Hough Transform described in “Generalizing the Hough Transform to Detect Arbitrary Shapes”, D. H. Ballard, Pattern Recognition Vol. 13, No. 2, pp. 111-122, 1981, RANSAC described in “Random sample consensus: a paradigm for model fitting with applications to picture image analysis and automated cartography”, Martin A. Fischler, et al., Communications of the Association for Computing Machinery Vol. 24 No. 6 Pages 381-395, June 1981, and the like can be used as the well-known method. In the present embodiment, an image of an object of which degree of match is less than a threshold value but of which degree of match is within a previously determined range can also be added as a detection result. For example, when the degree of match is defined as 0 to 100, up to a degree of match obtained by subtracting 10 from the threshold value for the degree of match is tolerated. As described above, even with a detection result of which degree of match is less than the threshold value, a label can be changed as a correct detection result in a later step.
NT_j images of objects are assumed to have been detected from the input picture image I_j as a result of detection. The detected image is denoted as T_jg (g=1 to NT_j), and the detected position of the image T_jg is denoted as R_Tjg. The detected position R_Tjg is a position posture of the image T_jg of the object that is seen from the picture image coordinate system, i.e., a homogeneous transformation matrix representing a position posture of the model pattern coordinate system that is seen from the picture image coordinate system when the model pattern is superimposed on the image T_jg, and is expressed by the following equation.
For example, when the object does not incline with respect to the optical axis of the camera, and only congruence transformation may be considered as a movement of an image of an object shown in a picture image, a00 to a12 are as follows.
a00=cos θ
a01=−sin θ
a02=x
a10=sin θ
a11=cos θ
a12=y
However, (x, y) is a position on a picture image, and θ is a rotational movement amount on the picture image.
When the object does not incline with respect to the optical axis of the camera but the distance between the object and the camera is not constant, the size of the image of the object shown in the picture image changes in accordance with the distance, and therefore, the movement of the image of the object shown in the picture image is a similarity transformation. In this case, a00 to a12 are as follows.
a00=s*cos θ
a01=−s*sin θ
a02=x
a10=s*sin θ
a11=s*cos θ
a12=y
However, s denotes a ratio between the size of the taught model pattern and the size of the image T_jg of the object.
The detected position R_Tjg and the input picture image I_j are made into a pair and stored in the detection result storage unit 33.
On the basis of the detected position R_Tjg of the image T_jg (j=1 to NI, g=1 to NT_j) of the object detected from each input picture image I_j (j=1 to NI), the position posture of the feature point P_i constituting the model pattern is transformed, and is superimposed on the image of the object on the picture image.
For the sake of explanation, the position posture of each feature point P_i constituting the model pattern is expressed by a homogeneous transformation matrix R_Pi. R_Pi can be described as follows.
However, t_Pi=(tx_Pi, ty_Pi) is the position of P_i in the model pattern coordinate system, and v_Pi (vx_Pi, vy_Pi) is the posture of P_i in the model pattern coordinate system.
In this case, it should be noted that the position posture R_Pi of the feature point P_i constituting the model pattern is expressed in the model pattern coordinate system.
Where the position posture of the feature point P_i which is seen from the picture image coordinate system is denoted as R_Pi′ when the model pattern is superimposed on the image T_jg of the object shown in the input picture image I_j, R_Pi′ is expressed as follows by using the position posture R_Pi of the feature point P_i which is seen from the model pattern coordinate system and the detected position R_Tjg of the image T_jg which is seen from the picture image coordinate system.
R_Pi′=R_Tjg*R_Pi
For the sake of explanation below, the position of P_i which is seen from the picture image coordinate system is denoted as t_Pi′, and the posture of P_i which is seen from the picture image coordinate system is denoted as v_Pi′.
In this case, the first label L1_jg (j=1 to NI, g=1 to NT_j) is given to the image T_jg of each object. In the first embodiment, those that can be selected as the label of the image of the object are two options, i.e., “OK” and “NG”. In the first embodiment, from among the images of the detected objects, the first labels L1_jg of the images of all the objects of which degree of match is equal to or more than the threshold value are changed to “OK”. From among the images of the detected objects, the first labels L1_jg of the images of the objects of which degree of match is less than a threshold value are changed to “NG”.
In step S104, the user checks the image T_jg (j=1 to NI, g=1 to NT_j) of the object detected from each input picture image I_j (j=1 to NI), and corrects the label. The user checks the image of the object having “OK” attached thereto, and when the user determines that it is not the object, the user changes the first label to “NG”, and the corrected label is stored. Further, the user checks the image of the object having “NG” attached thereto, and when the user determines that it is the object to be detected, the user changes the first label to “OK”, and the corrected label is stored.
In the first embodiment, the user goes back and forth between the input picture image list screen displayed by the input image display unit 42 and the detection result list screen displayed by the detection result display unit 41 to check the image T_jg (j=1 to NI, g=1 to NT_j) of the object.
The input picture image list screen and the detection result list screen are displayed on the display apparatus. The two screens may be displayed side by side, or displayed by switching the screens. In the first embodiment, a method for displaying the two screens by switching the screens will be explained.
In the input picture image list screen, all the input picture images I_j (j=1 to NI) are listed and displayed as illustrated in
When all the picture images cannot be displayed, a scroll bar is displayed. The user can view all the picture images by scrolling the screen with the scroll bar. The scroll bar may be shown vertically or horizontally.
A procedure for plotting a model pattern on an image of an object shown in an input picture image will be explained. Plotting of the model pattern may be preferably performed in accordance with the following procedure.
(1) The processing of (2) shown below is performed on all the input picture images I_j (j=1 to NI).
(2) The processing of (2-1) to (2-2) shown below is performed on the images T_jg (g=1 to NT_j) of all the objects detected from the input picture image I_j.
(2-1) The positions R_Pi′ of all the feature points P_i (i=1 to NP) of the model pattern seen from the picture image coordinate system are calculated on the basis of the position R_Tjg of the image of the object.
(2-2) A point is plotted at the position R_Pi′ of the input picture image I_j. When the first label L1_jg is “OK”, the point is plotted in green. When the first label L1_jg is “NG”, the point is plotted in red. The feature point P_i may not be plotted as a point, and instead, it may be plotted as a line segment. Therefore, the first label L1_jg is expressed as a series of feature points of the object displayed in green or red.
The second label will be explained.
The second label L2_j is determined as follows on the basis of the first label L1_jg (g=1 to NT_j).
The correction button for the second label is displayed as follows.
When the correction button for the second label is clicked, the second label L2_j is toggled between “OK” and “NG”. Accordingly, the first labels L1_jg of the images of all the objects in the same picture image are also changed to the same one as L2_j. In synchronization with the change in the first label L1_jg and the second label L2_j, the screen is also updated.
When the user clicks the input picture image I_j in the input picture image list screen, a detection result list screen displaying a list of detection results for that input picture image is displayed.
As illustrated in
In the first frame, the whole picture image is displayed.
A partial picture image is displayed in the second frame.
1. A model pattern is plotted on the picture of the detected object of the input picture image. To do this, the following processings (1-1) to (1-2) are to be performed on the image T_jg of the object.
(1-1) The position R_Pi′ of all the feature points P_i (i=1 to NP) of the model pattern seen from the picture image coordinate system are calculated on the basis of the position R_Tjg of the image of the object.
(1-2) A point is plotted at the position R_Pi′ of the input picture image I_j. When the first label L1_jg is “OK”, the point is plotted in green. When the first label L1_jg is “NG”, the point is plotted in red. The feature point P_i may not be plotted as a point, and instead, it may be plotted as a line segment.
2. A maximum value tx_Pi_max and a minimum value tx_Pi_min in X direction and a maximum value ty_Pi_max and a minimum value ty_Pi_min in Y direction of the position t_Pi′=(tx_Pi, ty_Pi) of all R_Pi′ (i=1 to NP) that have been calculated are derived.
3. Where a shape in which a partial picture image is cropped from the input picture image is a rectangle, the rectangle is defined by the position at the upper left apex and the position at the upper right apex. Where the margin is α, the position of the upper left apex can be calculated as (tx_Pi_min−α, ty_Pi_min−α), and the position of the lower right apex can be calculated as (tx_Pi_max+α, ty_Pi_max+α). The margin a may be a value determined in advance, or may be calculated from the size of the picture image.
4. A picture image is cropped from the input picture image with the calculated rectangle, and the picture image is adopted as a partial picture image.
When the partial picture image is selected by the partial picture image selection unit 51, the partial picture image is made into a selection state to be displayed in a highlighted manner. For example, when the partial picture image is clicked, the partial picture image is displayed so that it is distinguished from other partial picture images, i.e., the partial picture image is displayed in a highlighted manner. In order to display the partial picture image in a highlighted manner, for example, the partial picture image may be enclosed by a frame line in a bright color. The image T_jg of the object corresponding to the partial image is also displayed in a highlighted manner in the whole picture image. In order to display the whole picture image in a highlighted manner, the image of the object may also be enclosed by a bright frame line. The shape of the frame line may be the same as the rectangle in which the partial picture image is cropped. Alternatively, it may be displayed in a highlighted manner by changing the color in which the model pattern is plotted. Further, when the whole picture image is zoomed in, and only a portion of the whole picture image is displayed, the whole picture image may be automatically scrolled so that the operator can check the image of the object displayed in a highlighted manner.
As illustrated in
As illustrated in
In the example of
When the previous picture image button is pressed, a previous input picture image I_j−1, which is a picture image previous to the input picture image I_j, and detection results T_j−1g (g=1 to NT_j−1) are displayed.
When the subsequent picture image button is pressed, a subsequent input picture image I_j+1, which is a picture image subsequent to the input picture image I_j, and detection results T_j+1g (g=1 to NT_j+1) are displayed.
When an integer value x is input into the text box for the index of the picture image, the input picture image I_x and the detection results T_xg (g=1 to NT_x) are displayed.
Accordingly, another input picture image and the detection results for the picture image can be displayed on the detection result list screen without returning to the input picture image list screen.
The arrangement of the first frame and the second frame may not be not only a side by side arrangement but also a vertical arrangement. The first frame and the second frame may also be arranged as illustrated in
In step S105, multiple data sets including the image T_jg (j=1 to NI, g=g=1 to NT_j) of the detected objects and the first labels L1_jg are adopted as learning data, and the learning unit 22 performs machine learning or statistical processing. The processing performed by the learning unit 22 including the case of the second embodiment will be explained later.
The following case can be considered as a modification of the image processing device according to the first embodiment.
1. A rectangle enclosing a model pattern is defined in advance. This rectangle may be automatically defined when a model pattern is taught, or may be defined manually by the operator. The rectangle is defined by the position posture of an apex U1 at a left end of the rectangle, and a width U_W, and a height U_H. The position posture of the apex U1 which is seen from the model pattern coordinate system is defined with a homogeneous transformation matrix R_U1.
2. A position posture R_U1′ of the apex U1 which is seen from the picture image coordinate system is calculated on the basis of the position R_Tjg of the image of the object.
3. A partial picture image is cropped by using the position posture R_U1′ of the apex U1 in the picture image coordinate system, the width U_W, and the height U_H.
In this case, the image of the object on the partial picture image is displayed with the same posture. It is to be understood that a shape in which a partial picture image is cropped may not be a rectangle.
The image processing device according to the second embodiment includes the same hardware configuration as the image processing device according to the first embodiment, and a portion of the processing is different from that of the first embodiment. In the explanation about the second embodiment, first, for example, a case where the learning unit 22 performs statistical processing will be explained.
In the screen of
In step S201, the same processing as step S101 of the first embodiment is performed.
In step S202, an object is placed within a visual field of the visual sensor 11, and the user presses the SNAP button to capture an input picture image. The image of the object shown in the input picture image I_j preferably includes a variation that may cause a problem in the detection. When, as a result of checking the input picture image I_j, the user determines that the input picture image I_j is not preferable as learning data, the user may press the SNAP button again to capture the input picture image I_j again. The captured picture image is displayed as the whole picture image in the first frame. At this point in time, the object is not detected, and therefore, a model pattern is not plotted on the image of the object in the whole picture image, and the partial picture image is not displayed either. At this time, only the first frame may be displayed, and the second frame may not be displayed.
In step S203, when the FIND button is pressed, an object is detected from the input picture image I_j. The processing of the detection may be performed in the same manner as step S103 of the first embodiment. However, in this case, a detection is made from only a single input picture image I_j.
In step S204, the images T_jg (g=1 to NT_j) of the objects detected from the input picture image I_j are listed and displayed in a detection result list screen.
The whole picture image and the partial picture image may be displayed in the same manner as the first embodiment. At the right side of the first frame, the SNAP button and the FIND button are hidden, and an ADD button and a DISCARD button are added. The operator checks the whole picture image and the partial picture image displayed on the detection result list screen, and like the first embodiment, the user checks whether there is a problem in the detection result or not.
Further, in step S204, like the detection result list screen of the first embodiment, the first label is allowed to be corrected.
In this case, the user checks the image of the object having “OK” attached thereto, and when the user determines that it is not the object, the user changes the first label to “NG”, and the corrected label is stored. Further, the user checks the image of the object having “NG” attached thereto, and when the user determines that it is the object to be detected, the user changes the first label to “OK”, and the corrected label is stored.
In step S205, the user determines whether the images T_jg (g=1 to NT_j) of the objects are to be added to the learning data, and when the images T_jg (g=1 to NT_j) of the objects are determined to be added to the learning data, the user presses the ADD button to associate the images T_jg (g=1 to NT_j) and the first labels L1_jg (g=1 to NT_j) of the objects with the input picture image I_j, and adds them to the learning data. When the images T_jg (g=1 to NT_j) of the objects are determined not to be added to the learning data, the user presses the DISCARD button to discard the input picture image I_j and the images T_jg and the first labels L1_jg of the objects. In any of the cases where the user presses the ADD button and the DISCARD button, the processing is started from step S202 again.
When sufficient learning data has been added, the user presses the FINISH button and the processing proceeds to step S206.
In this case, before proceeding to S206, a step of using the input picture image list screen and the detection result list screen to check and correct the first labels L1_jg (j=1 to NI, g=1 to NT_j) of the images T_jg (j=1 to NI, g=1 to NT_j) of the objects corresponding to all the input picture images I_j (j=1 to NI) added to the learning data may be added just like step S104 of the first embodiment.
In step S206, on the basis of the collected learning data, just like step S105 of the first embodiment, multiple data sets including the images T_jg (j=1 to NI, g=1 to NT_j) and the first labels L1_jg (j=1 to NI, g=1 to NT_j) of the objects are adopted as learning data, and the learning unit 22 performs the statistical processing.
Hereinafter, the statistical processing performed by the learning unit 22 according to the first and second embodiments will be explained.
The learning unit 22 performs learning so that the images T_jg (j=1 to NI, g=1 to NT_j) of the objects are correctly determined to be objects that correspond to the model pattern. In other words, the learning unit 22 performs learning so that correct determination can be made as to whether an image of an object is included or not when a partial picture image that is considered to include an image of an object is input. Conventionally, the object detection unit 21 detects an object from the input picture image, so that a certain level of precision has been obtained, but as a result of learning by the learning unit 22, the precision of the detection is further improved. In this case, the object detection unit 21 makes a detection, and a partial picture image that is considered to be suitable for being adopted as a learning target and that include an image of an object of which first label is “OK” is stored to the learning data storage unit 32. When the operation processing performance of the learning unit 22 is sufficiently high, an input picture image may be input instead of a partial picture image for detecting objects. The learning unit 22 performs statistical processing on multiple partial picture images stored in the learning data storage unit 32 to improve the model pattern.
The applicant of the present application has disclosed a technique for improving the model pattern by performing statistical processing on multiple images of objects detected on the basis of a model pattern in Japanese Patent Application No. 2015-218357, and, for example, the learning unit 22 may be realized with a technique using statistical processing disclosed in Japanese Patent Application No. 2015-218357. However, the statistical processing executed by the learning unit 22 is not limited thereto.
The first and second embodiments in which the learning unit 22 executes statistical processing have been hereinabove explained. Subsequently, a case where the learning unit 22 performs machine learning (supervised learning) will be explained.
The learning unit 22 performs machine learning to improve a probability that the object detection unit 21 detects an object. In this case, the correct detection means that an object that is truly an object is correctly determined to be an object, an object that is not an object is correctly determined not to be an object, and a detection failure of failing to detect an object even though it is an object does not occur, and a false detection of determining an object that is actually not an object is an object does not occur. In order to perform such learning, it is preferable that partial picture images (“OK” “OK”) that are determined by the image processing device 10 to be images of objects in an input picture image and for which the user checks that the determination is correct, partial picture images (“OK” “NG”) that are determined by the image processing device 10 to be images of objects in an input picture image but for which the user checks that the determination is incorrect, partial picture images (“NG” “OK”) that are determined by the image processing device 10 not to be images of objects in an input picture image but for which the user checks that the determination is incorrect, and partial picture images (“NG” “NG”) that are determined by the image processing device 10 not to be images of objects in an input picture image and for which the user checks that the determination is correct are of similar ratios as learning data in order to efficiently perform the learning. Therefore, in the first and second embodiments in which the learning unit 22 performs the statistical processing, a partial picture image having a history in which a first label is changed from “NG” to “OK” and a partial picture image having a history in which a first label is changed from “OK” to “NG” are associated with each history and stored to the learning data storage unit 32. These are the partial picture image (“NG” “OK”) and the partial picture image (“OK” “NG”). Further, the operator selects partial picture images suitable for learning data from among the partial picture images (“OK” “OK”) that are determined to be images of objects in an input picture image and for which the determination is maintained and the partial picture images (“NG” “NG”) that are determined not to be images of objects in an input picture image and for which the determination is maintained, and stores them as the partial picture image (“OK” “OK”) and the partial picture image (“NG” “NG”) into the learning data storage unit 32. But the embodiment is not limited thereto, and partial picture images may be randomly collected and used as learning data. The partial picture images of “NG” “NG” may be used as learning data.
The learning unit 22 has a function of extracting useful rules, knowledgeable equations, determination references, and the like existing in a set of input data by means of analysis, outputting a determination result thereof, and performing learning (machine learning) of knowledge. There are various techniques for machine learning, but when roughly divided, the techniques can be divided into, for example, “supervised learning”, “unsupervised learning” and “reinforced learning”, but in this case the “supervised learning” is used. Further, in order to achieve these techniques, there is a technique called “deep learning” for learning extraction of a feature value itself. For such machine learning (machine learning apparatus 20), a general-purpose calculator or a processor may be used, but when GPGPU (General-Purpose computing on Graphics Processing Units), a large scale PC cluster, and the like may be applied, so that processing can be performed with a higher speed.
In the supervised learning, many sets of data including any given input and a result (label) are given to the learning unit 22, so that features in the data sets are learned, and a model for estimating a result from an input, i.e., a relationship thereof is recursively obtained. When this supervised learning is applied to the embodiment, for example, the learning unit 22 can be realized by using an algorithm such as a neural network.
A neural network can be used as an approximation algorithm of a value function in supervised learning.
As illustrated in
y=fk(Σ
The three-layer neural network constituted by combining the neurons as illustrated in
The neurons N11 to N13 output z11 to z13, respectively. In
The neurons N21 and N22 output z21 and z22, respectively. In
Finally, the neurons N31 to N33 output the result y1 to the result y3, respectively. Operations of the neural network include a learning mode and a value prediction mode. For example, in the learning mode, the weight W is learned by using the learning data set, and by using the parameter, a behavior determination of a robot is performed in the prediction mode. For the sake of convenience, it is described as a prediction, but it is to be understood that various tasks, e.g., detection, classification, and inference can be performed.
In this case, data obtained by actually operating the robot in the prediction mode may be immediately learned and reflected in a subsequent behavior (online learning), and collective learning by using a data group collected in advance may be performed, and thereafter the detection mode may be continuously performed with the same parameter (batch learning). Alternatively, in an intermediate manner, every time a certain amount of data is accumulated, the learning mode may be interposed.
The weights w1 to w3 can be learned through backpropagation. It should be noted that information about an error enters from the right-hand side and flows to the left-hand side. The backpropagation is a technique for adjusting (learning) a weight for each neuron so as to reduce the difference between the output y and the true output y (supervisor) when the input x is input.
In such a neural network, the number of layers can be increased to three or more layers (which is referred to as deep learning). An arithmetic apparatus performing feature extraction of inputs in a stepwise manner and recursively deriving a result can be automatically obtained from only the supervisor data.
The learning unit 22 includes a state quantity observation unit 61, a neural network 62, and a neural network (NN) update control unit 63. As described above, the learning unit 22 is realized with, for example, GPGPU
(General-Purpose Computing on Graphics Processing Units).
The state quantity observation unit 61 receives, from the learning data storage unit 32, a data set including a partial picture image and first labels of images of objects included therein. The learning data storage unit 32 stores an extremely large number of data sets, and transmits the data sets to the state quantity observation unit 61 in order.
The neural network 62 is a neural network for performing deep learning, and calculates a determination result (“OK” “NG”) in response to a partial picture image given by the state quantity observation unit 61. In the neural network 62, the weight for each neuron can be adjusted (learned) by the NN update control unit 63.
The NN update control unit 63 compares a determination result (“OK” “NG”) calculated by the neural network 62 in accordance with the partial picture image given by the state quantity observation unit 61 and the first label (supervisor data) of the partial picture image, and performs update control for changing the weight for each neuron of the neural network 62 so as to increase a probability that the comparison result matches.
It should be noted that the contents processed by the learning unit 22 are not limited to the above contents. In general, in the supervised machine learning, a function of mapping an input to an output (label) corresponding thereto is learned. The first and second embodiments disclose a unit for inputting an input picture image and an image of an object detected, and quickly correcting a label which is an output thereof. The method for learning a function for mapping from the input and the output may be any methods.
The embodiments have been hereinabove explained, but the detection result list screen has the following features.
The above display screen has the following advantages.
Further, the operator may want to check detection results or determination results for multiple picture images. For example, this includes the following cases.
When the operator checks detection results for multiple picture images as described above, it is effective to use a method in which reduced picture images of multiple picture images are arranged and displayed on an input picture image list screen, detection results are plotted on the picture images, and when the operator checks further into details, the operator selects a picture image to transition to an input picture image list screen in which the operator checks a detection result for each picture image explained above.
Further, a configuration described in the claims and an effect thereof are as follows.
An image processing device according to the invention includes an object detection unit for detecting one or more images of objects from an input picture image in which the objects have been captured, on the basis of a model pattern of the object, and a detection result display unit for graphically superimposing and displaying a detection result on each image thus detected. The detection result display unit includes a first frame for displaying the entire input picture image, and a second frame for listing and displaying one or more partial picture images each including an image detected from the input picture image. In the input picture image displayed in the first frame, a detection result is superimposed and displayed on all the detected images, and in the partial picture image displayed in the second frame, a detection result of an image corresponding to each partial picture image is superimposed and displayed.
According to this configuration, in the first frame, the following points can be checked in the whole picture image.
In the second frame, the user can perform checking focusing on an image of an object which is a target without being annoyed by images of other objects in the same picture image.
The first frame and the second frame are displayed in the same window at a time, so that the image of the object can be checked while making use of the above advantages at the same time. Accordingly, the images of multiple objects shown in the picture image can be checked in a short time.
As described in International Publication No. 2001/041068, a method for displaying a whole picture image in the first frame, and displaying a detection result in a table format in the second frame may be considered. However, in this method, it is difficult to find a relationship between a row of a table expressing a detection result and an image of an object on the picture image. As compared with checking a detection result by using characters in the table format, the method of seeing the partial picture image allows the user to more intuitively check the image of the object.
The detection result display unit further superimposes and displays, on the partial picture image displayed in the second frame, first label information stored in association with an image corresponding to each partial picture image, and allows an operator to change the first label information on the basis of an operation performed by the operator.
When a unit for operating a label is provided on the partial picture image in the second frame, it is easier to find the relationship between the image of the object and the operation unit for the label. The labels can be corrected without performing an operation of selecting an object. For example, in the method described in International Publication 2001/041068, it is necessary to correct a label after selecting an object. There may be another method including highlighting, on a whole picture image, an image of an object corresponding to a detection result when a detection result is displayed in a table format and a row of the table is selected, and providing a unit for correcting a label on a row of the table. In this method, however, it is necessary to select a row of a table in order to check an picture image. There may be still another method of providing a unit for operating a label on a whole picture image in a first frame, but in this method, when images of objects are overlapping or adjacent to each other, it is difficult to find a correspondence relationship between the operation unit and the image of the object.
The detection result display unit displays picture images in the first frame and the second frame with a setting magnification rate for each of them.
First, in the first frame, the entire image of an object may be configured to be enlarged and displayed in a whole picture image, and in the second frame, images of objects may be configured to be reduced and displayed. In this case, the user can check the image of the object in detail by displaying the image in an enlarged manner in the first frame, and the user can roughly check the entire image of each object in the second frame. Subsequently, in the second frame, an image of each object may be configured to be displayed in an enlarged manner, and in the first frame, images of all the objects may be configured to be displayed in a reduced manner. In this case, in the second frame, the user can check an image of an object in detail by enlarging and displaying the image of the object. In the first image, the user can find which position in the whole picture image the object is located. As described above, the user checks the image of the object at a time with different magnification rates for the same object, so that the user can check the image of the object in a shorter time.
The detection result display unit includes a partial picture image selection unit for selecting one of a plurality of the partial picture images displayed in the second frame, and in the input picture image displayed in the first frame, a detection result of an image of an object corresponding to the partial picture image selected by the partial picture image selection unit (51) is displayed in a highlighted manner.
According to this configuration, the user can easily check a correspondence relationship between an image of an object displayed in the whole picture image in the first frame and an image of an object displayed in the partial picture image in the second frame.
The detection result display unit superimposes, on the input picture image displayed in the first frame, the first label information corresponding to each of the one or more images detected from the input picture image.
According to this configuration, even in the whole picture image in the first frame, the user can check information about the label. This is useful when it is difficult to check the image of the object with only the partial picture image. For example, when the image of the object is showing a portion of an item, it may be difficult to check the positional relationship of the image of the object with respect to the entire item. In such a case, the user can easily check the image of the object by displaying label information in the whole picture image in the first frame.
The image processing device includes a detection result storage unit associating a plurality of input picture images with detection results of one or more images of the objects detected from each of the plurality of input picture images and storing them, and an input picture image selection unit for selecting one of the plurality of input picture images, and the detection result display unit displays the one of the input picture images selected by the input picture image selection unit and a detection result stored in the detection result storage unit in association with the one of the input picture images.
According to this configuration, the user can check multiple picture images on the detection result display unit without going back and forth between the input picture image display unit and the detection result display unit. For example, a subsequent picture image button is prepared as an input picture image selection unit on the detection result display unit, so that the user can successively change picture images in the detection result display unit.
The image processing device includes an input picture image display unit listing and displaying the plurality of input picture images stored in the detection result storage unit upon superimposing one or more images and detection results of the objects detected from the input picture images. The detection result display unit displays the input picture image selected by the input picture image selection unit, and enables switching between the input picture image display unit and the detection result display unit on the basis of an operation performed by an operator.
With the input picture image display unit, detection results for a plurality of picture images are listed and displayed, so that the user can guess a picture image which seems to have a problem, and when the user wants to check the picture image further into details, the user can check only the detection result for that picture image with the detection result display unit. As a result, the user can intuitively proceed to a screen for checking a detection for a particular picture image from among many picture images.
The detection result storage unit associates second label information with a plurality of input picture images and stores them, and the input picture image display unit further superimposes and displays the second label stored in association with each of the plurality of input picture images and allows an operator to change the second label information, and when the operator changes the second label information of one of the input picture images, the first label information respectively corresponding to the images of the plurality of objects stored in association with the input picture image are collectively changed.
According to this configuration, when a detection result can be checked by just checking each picture image with the input picture image display unit, the user can correct a label of a detection result without moving to the detection result display screen. For example, when the image of the object is displayed in a sufficiently large size with the input picture image display unit, the user does not have to check it in the detection result display screen. However, if a plurality of detection results are displayed in the picture image, this method can be used only when all the detection results are changed to the same label. When only a single detection result is displayed in the picture image, the user is likely to check it with only the input picture image display unit. In this case, since there is only a single label, it is not necessary to move to the input picture image display unit in order to correct each label. With such a function, no matter whether there are a single detection result or a plurality of detection results in a picture image, the same method can be used to cope with the issue.
The image processing device includes a learning data storage unit adding first label information to each of the detected images, associating them with each other to make learning data, and storing the learning data, and a learning unit performing machine learning or statistical processing on the basis of the learning data.
According to this configuration, a detection or correctness and incorrectness determination is made on an image of an object, so that the accuracy of adding the first label information is improved.
In the display achieved with the image processing device according to the present invention, there is an advantage in that a detection result and a correctness and incorrectness determination result about an object detected from an input picture image can be easily checked.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2016-034774 | Feb 2016 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4642813 | Wilder | Feb 1987 | A |
| 4783829 | Miyakawa | Nov 1988 | A |
| 5033099 | Yamada | Jul 1991 | A |
| 5577166 | Mizuno | Nov 1996 | A |
| 5583771 | Lynch | Dec 1996 | A |
| 5845048 | Masumoto | Dec 1998 | A |
| 5960125 | Michael | Sep 1999 | A |
| 5978521 | Wallack | Nov 1999 | A |
| 5995652 | Chiu | Nov 1999 | A |
| 6005959 | Mohan | Dec 1999 | A |
| 6031930 | Bacus | Feb 2000 | A |
| 6141033 | Michael | Oct 2000 | A |
| 6233365 | Teruhiko | May 2001 | B1 |
| 6396941 | Bacus | May 2002 | B1 |
| 6421458 | Michael | Jul 2002 | B2 |
| 6546137 | Lai | Apr 2003 | B1 |
| 6728582 | Wallack | Apr 2004 | B1 |
| 6771808 | Wallack | Aug 2004 | B1 |
| 6807305 | Rajagopal | Oct 2004 | B2 |
| 6920180 | Yamane | Jul 2005 | B1 |
| 6920241 | Dutta-Choudhury | Jul 2005 | B1 |
| 6944331 | Schmidt | Sep 2005 | B2 |
| 7158677 | Wenzel | Jan 2007 | B2 |
| 7340089 | Nair | Mar 2008 | B2 |
| 7421097 | Hamza | Sep 2008 | B2 |
| 7903883 | Zitnick, III | Mar 2011 | B2 |
| 7925070 | Sumida | Apr 2011 | B2 |
| 7936928 | Wenzel | May 2011 | B2 |
| 7982779 | Saeki | Jul 2011 | B2 |
| 8095237 | Habibi | Jan 2012 | B2 |
| 8103085 | Zadeh | Jan 2012 | B1 |
| 8111904 | Wallack | Feb 2012 | B2 |
| 8184912 | Hayasaki | May 2012 | B2 |
| 8380342 | Ban | Feb 2013 | B2 |
| 8600192 | Liu | Dec 2013 | B2 |
| 8620060 | Takayama | Dec 2013 | B2 |
| 8626686 | Rhodes | Jan 2014 | B1 |
| 8825452 | Shono | Sep 2014 | B2 |
| 8929608 | Takizawa | Jan 2015 | B2 |
| 9044858 | Ito | Jun 2015 | B2 |
| 9050722 | Satou | Jun 2015 | B2 |
| 9060114 | Takizawa | Jun 2015 | B2 |
| 9076035 | Tanaka | Jul 2015 | B2 |
| 9079310 | Kumiya | Jul 2015 | B2 |
| 9092851 | Marugame | Jul 2015 | B2 |
| 9124873 | Liu | Sep 2015 | B2 |
| 9156162 | Suzuki | Oct 2015 | B2 |
| 9177375 | Mizutani | Nov 2015 | B2 |
| 9305216 | Mishra | Apr 2016 | B1 |
| 9334096 | Luciano, Jr. | May 2016 | B2 |
| 9346166 | Gotou | May 2016 | B2 |
| 9434070 | Satou | Sep 2016 | B2 |
| 9494528 | Matsuda | Nov 2016 | B2 |
| 9508192 | Minato | Nov 2016 | B2 |
| 9524442 | Ijiri | Dec 2016 | B2 |
| 9529945 | Atohira | Dec 2016 | B2 |
| 9563825 | Shen | Feb 2017 | B2 |
| 9595095 | Aiso | Mar 2017 | B2 |
| 9638910 | Oba | May 2017 | B2 |
| 9643796 | Koga | May 2017 | B2 |
| 9684940 | Kimoto | Jun 2017 | B2 |
| 9721184 | Yamazaki | Aug 2017 | B2 |
| 9767564 | Bostick | Sep 2017 | B2 |
| 9769428 | Kanno | Sep 2017 | B2 |
| 9779296 | Ma | Oct 2017 | B1 |
| 9789517 | Doublet | Oct 2017 | B2 |
| 9813607 | Kalevo | Nov 2017 | B2 |
| 9844881 | Payton | Dec 2017 | B2 |
| 9844882 | Takizawa | Dec 2017 | B2 |
| 9886646 | Fukui | Feb 2018 | B2 |
| 9892344 | Tran | Feb 2018 | B1 |
| 9895810 | Nishi | Feb 2018 | B2 |
| 9898846 | Borras | Feb 2018 | B1 |
| 9930248 | Bayani | Mar 2018 | B2 |
| 9959451 | Suenaga | May 2018 | B2 |
| 9996770 | Ishii | Jun 2018 | B2 |
| 10055805 | Satou | Aug 2018 | B2 |
| 10096103 | Sagisaka | Oct 2018 | B2 |
| 10131051 | Goyal | Nov 2018 | B1 |
| 10140773 | Short | Nov 2018 | B2 |
| 10176382 | Owechko | Jan 2019 | B1 |
| 10181187 | Kubo | Jan 2019 | B2 |
| 10184789 | Sasaki | Jan 2019 | B2 |
| 10187593 | Holmes | Jan 2019 | B2 |
| 10190873 | Yamagami | Jan 2019 | B1 |
| 10201900 | Ishige | Feb 2019 | B2 |
| 10203666 | Takahashi | Feb 2019 | B2 |
| 10232512 | Ooba | Mar 2019 | B2 |
| 10235797 | Sheffield | Mar 2019 | B1 |
| 10262406 | Sakaguchi | Apr 2019 | B2 |
| 10311596 | Inazumi | Jun 2019 | B2 |
| 10416086 | Kido | Sep 2019 | B2 |
| 10424058 | Sagisaka | Sep 2019 | B2 |
| 10425560 | Kida | Sep 2019 | B2 |
| 10518417 | Wada | Dec 2019 | B2 |
| 10525598 | Takahashi | Jan 2020 | B2 |
| 10596698 | Oota | Mar 2020 | B2 |
| 10603797 | Ozaki | Mar 2020 | B2 |
| 10614565 | Namiki | Apr 2020 | B2 |
| 10664754 | Gotou | May 2020 | B2 |
| 10664767 | Takigawa | May 2020 | B2 |
| 10692018 | Ooba | Jun 2020 | B2 |
| 10712288 | Okajima | Jul 2020 | B2 |
| 10717196 | Yamazaki | Jul 2020 | B2 |
| 10737385 | Shibasaki | Aug 2020 | B2 |
| 20010004089 | Gleis | Jun 2001 | A1 |
| 20010050999 | Bacus | Dec 2001 | A1 |
| 20020052877 | Okamoto | May 2002 | A1 |
| 20020102011 | Bacus | Aug 2002 | A1 |
| 20020159641 | Whitney | Oct 2002 | A1 |
| 20020186882 | Cotman | Dec 2002 | A1 |
| 20030174891 | Wenzel | Sep 2003 | A1 |
| 20030194133 | Wenzel | Oct 2003 | A1 |
| 20030198388 | Wenzel | Oct 2003 | A1 |
| 20030210262 | Gahm | Nov 2003 | A1 |
| 20040037467 | Wenzel | Feb 2004 | A1 |
| 20040133291 | Nakayama | Jul 2004 | A1 |
| 20040146195 | Fukagawa | Jul 2004 | A1 |
| 20040175764 | Nishiyama | Sep 2004 | A1 |
| 20040184657 | Lin | Sep 2004 | A1 |
| 20040247181 | Wenzel | Dec 2004 | A1 |
| 20050111738 | Iizuka | May 2005 | A1 |
| 20050147287 | Sakai | Jul 2005 | A1 |
| 20050178949 | Saeki | Aug 2005 | A1 |
| 20050226489 | Beach | Oct 2005 | A1 |
| 20060002604 | Sakai | Jan 2006 | A1 |
| 20060039601 | Nair | Feb 2006 | A1 |
| 20060088202 | Venkatachalam | Apr 2006 | A1 |
| 20060161391 | Inaba | Jul 2006 | A1 |
| 20060203821 | Mizusawa | Sep 2006 | A1 |
| 20060268259 | Park | Nov 2006 | A1 |
| 20060285752 | Fujieda | Dec 2006 | A1 |
| 20060291719 | Ikeda | Dec 2006 | A1 |
| 20070014467 | Bryll | Jan 2007 | A1 |
| 20070058054 | Kagayama | Mar 2007 | A1 |
| 20070076944 | Bryll | Apr 2007 | A1 |
| 20070081714 | Wallack | Apr 2007 | A1 |
| 20070120844 | Beikirch | May 2007 | A1 |
| 20070177787 | Maeda | Aug 2007 | A1 |
| 20070242856 | Suzuki | Oct 2007 | A1 |
| 20080069477 | Engels | Mar 2008 | A1 |
| 20080082213 | Ban | Apr 2008 | A1 |
| 20080166053 | Furuhashi | Jul 2008 | A1 |
| 20080216075 | Ogasawara | Sep 2008 | A1 |
| 20080219516 | Suzuki | Sep 2008 | A1 |
| 20080267499 | Deischinger | Oct 2008 | A1 |
| 20080281797 | Hori | Nov 2008 | A1 |
| 20080298672 | Wallack | Dec 2008 | A1 |
| 20090033655 | Boca | Feb 2009 | A1 |
| 20090041340 | Suzuki | Feb 2009 | A1 |
| 20090060315 | Harris | Mar 2009 | A1 |
| 20090154812 | Schmitt | Jun 2009 | A1 |
| 20090202139 | Toyoda | Aug 2009 | A1 |
| 20090208090 | Nishiura | Aug 2009 | A1 |
| 20090268076 | Kawamura | Oct 2009 | A1 |
| 20090291512 | Izuha | Nov 2009 | A1 |
| 20090306816 | Champel | Dec 2009 | A1 |
| 20090316980 | Hanada | Dec 2009 | A1 |
| 20100004778 | Arimatsu | Jan 2010 | A1 |
| 20100061602 | Yamada | Mar 2010 | A1 |
| 20100091362 | Isozaki | Apr 2010 | A1 |
| 20100156896 | Ichimura | Jun 2010 | A1 |
| 20100166294 | Marrion | Jul 2010 | A1 |
| 20100214936 | Ito | Aug 2010 | A1 |
| 20100223219 | Kato | Sep 2010 | A1 |
| 20100224000 | Komatsu | Sep 2010 | A1 |
| 20100226561 | Fujikawa | Sep 2010 | A1 |
| 20100226563 | Takayama | Sep 2010 | A1 |
| 20100238229 | Ito | Sep 2010 | A1 |
| 20100239124 | Hazeyama | Sep 2010 | A1 |
| 20100266208 | Downing | Oct 2010 | A1 |
| 20100310043 | Shimada | Dec 2010 | A1 |
| 20100322489 | Tizhoosh | Dec 2010 | A1 |
| 20110075123 | Nagamori | Mar 2011 | A1 |
| 20110090359 | Sagawa | Apr 2011 | A1 |
| 20110128367 | Yoshioka | Jun 2011 | A1 |
| 20110182515 | Iwai | Jul 2011 | A1 |
| 20110182516 | Iwai | Jul 2011 | A1 |
| 20110188734 | Tsuchiya | Aug 2011 | A1 |
| 20110254764 | Kimoto | Oct 2011 | A1 |
| 20110254861 | Emura | Oct 2011 | A1 |
| 20110262043 | Sato | Oct 2011 | A1 |
| 20110274362 | Isomae | Nov 2011 | A1 |
| 20120027286 | Xu | Feb 2012 | A1 |
| 20120027307 | Naruse | Feb 2012 | A1 |
| 20120045115 | Dong | Feb 2012 | A1 |
| 20120070089 | Yamada | Mar 2012 | A1 |
| 20120131529 | Hayakawa | May 2012 | A1 |
| 20120188283 | Ohashi | Jul 2012 | A1 |
| 20120200596 | Gotou | Aug 2012 | A1 |
| 20120201448 | Nammoto | Aug 2012 | A1 |
| 20120219188 | Kurz | Aug 2012 | A1 |
| 20120250544 | Sasaki | Oct 2012 | A1 |
| 20120296592 | Luciano, Jr. | Nov 2012 | A1 |
| 20120301860 | Yamashita | Nov 2012 | A1 |
| 20120320094 | Ruddle | Dec 2012 | A1 |
| 20130010137 | Kawai | Jan 2013 | A1 |
| 20130038901 | Shimadu | Feb 2013 | A1 |
| 20130114898 | Taguchi | May 2013 | A1 |
| 20130116900 | Yuyama | May 2013 | A1 |
| 20130135463 | Aratani | May 2013 | A1 |
| 20130163887 | Nakano | Jun 2013 | A1 |
| 20130170732 | Gotou | Jul 2013 | A1 |
| 20130177232 | Hirano | Jul 2013 | A1 |
| 20130177250 | Yamamoto | Jul 2013 | A1 |
| 20130187954 | Saito | Jul 2013 | A1 |
| 20130188038 | Tanimoto | Jul 2013 | A1 |
| 20130238124 | Suzuki | Sep 2013 | A1 |
| 20130245828 | Tateno | Sep 2013 | A1 |
| 20130249952 | Kusakabe | Sep 2013 | A1 |
| 20130251266 | Nakagome | Sep 2013 | A1 |
| 20130301900 | Kiyuna | Nov 2013 | A1 |
| 20130315456 | Marugame | Nov 2013 | A1 |
| 20130322737 | Murakami | Dec 2013 | A1 |
| 20130342676 | Amano | Dec 2013 | A1 |
| 20140078498 | Ikushima | Mar 2014 | A1 |
| 20140091522 | Kelly | Apr 2014 | A1 |
| 20140098229 | Lu | Apr 2014 | A1 |
| 20140099017 | Tsai | Apr 2014 | A1 |
| 20140152794 | Takahashi | Jun 2014 | A1 |
| 20140168259 | Yamaguchi | Jun 2014 | A1 |
| 20140219504 | Haraguchi | Aug 2014 | A1 |
| 20140233807 | Ejima | Aug 2014 | A1 |
| 20140236349 | Bae | Aug 2014 | A1 |
| 20140240488 | Kanou | Aug 2014 | A1 |
| 20140241623 | Wang | Aug 2014 | A1 |
| 20140285825 | Matsumoto | Sep 2014 | A1 |
| 20140286565 | Ikenaga | Sep 2014 | A1 |
| 20140292813 | Takayama | Oct 2014 | A1 |
| 20140292814 | Tsujimoto | Oct 2014 | A1 |
| 20140294292 | Aoba | Oct 2014 | A1 |
| 20140301632 | Ikeda | Oct 2014 | A1 |
| 20140301634 | Ishii | Oct 2014 | A1 |
| 20140301648 | Kato | Oct 2014 | A1 |
| 20140306992 | Tsujimoto | Oct 2014 | A1 |
| 20140314300 | Kaufman | Oct 2014 | A1 |
| 20140314302 | Minato | Oct 2014 | A1 |
| 20140327687 | Murakami | Nov 2014 | A1 |
| 20140339374 | Mian | Nov 2014 | A1 |
| 20150003678 | Watanabe | Jan 2015 | A1 |
| 20150054855 | Sato | Feb 2015 | A1 |
| 20150055823 | Kotake | Feb 2015 | A1 |
| 20150062381 | Na | Mar 2015 | A1 |
| 20150160650 | Scelsi | Jun 2015 | A1 |
| 20150186422 | Sumitomo | Jul 2015 | A1 |
| 20150186753 | Horita | Jul 2015 | A1 |
| 20150213328 | Mase | Jul 2015 | A1 |
| 20150262012 | Kim | Sep 2015 | A1 |
| 20150262414 | Minato | Sep 2015 | A1 |
| 20150294189 | Benhimane | Oct 2015 | A1 |
| 20150345936 | Sumitomo | Dec 2015 | A1 |
| 20150355103 | Ando | Dec 2015 | A1 |
| 20150355104 | Matsuda | Dec 2015 | A1 |
| 20150356710 | Kunze | Dec 2015 | A1 |
| 20150358602 | Mayumi | Dec 2015 | A1 |
| 20160001445 | Setsuda | Jan 2016 | A1 |
| 20160005156 | Wang | Jan 2016 | A1 |
| 20160005161 | Aiso | Jan 2016 | A1 |
| 20160012277 | Tate | Jan 2016 | A1 |
| 20160019706 | Otomaru | Jan 2016 | A1 |
| 20160021306 | Cho | Jan 2016 | A1 |
| 20160025781 | Kumeta | Jan 2016 | A1 |
| 20160059419 | Suzuki | Mar 2016 | A1 |
| 20160065861 | Steinberg | Mar 2016 | A1 |
| 20160070985 | Sashida | Mar 2016 | A1 |
| 20160104046 | Doettling | Apr 2016 | A1 |
| 20160114925 | Yuyama | Apr 2016 | A1 |
| 20160203593 | Henkemeyer et al. | Jul 2016 | A1 |
| 20160229445 | Sasaki | Aug 2016 | A1 |
| 20160234429 | Cho | Aug 2016 | A1 |
| 20160253468 | Osawa | Sep 2016 | A1 |
| 20160295171 | Van Den Brink | Oct 2016 | A1 |
| 20160301903 | Gamada | Oct 2016 | A1 |
| 20160321511 | Abhau | Nov 2016 | A1 |
| 20160357406 | Lee | Dec 2016 | A1 |
| 20160370791 | Revanur | Dec 2016 | A1 |
| 20170004385 | Aoba | Jan 2017 | A1 |
| 20170004386 | Palacio | Jan 2017 | A1 |
| 20170017862 | Konishi | Jan 2017 | A1 |
| 20170018116 | Sun | Jan 2017 | A1 |
| 20170032177 | Suenaga | Feb 2017 | A1 |
| 20170046841 | Bostick | Feb 2017 | A1 |
| 20170061253 | Burgos | Mar 2017 | A1 |
| 20170069077 | Zhang | Mar 2017 | A1 |
| 20170076468 | Krauss | Mar 2017 | A1 |
| 20170092015 | McCann | Mar 2017 | A1 |
| 20170099200 | Ellenbogen | Apr 2017 | A1 |
| 20170103124 | Hassan | Apr 2017 | A1 |
| 20170106540 | Watanabe | Apr 2017 | A1 |
| 20170116743 | Matsumoto | Apr 2017 | A1 |
| 20170132451 | Namiki | May 2017 | A1 |
| 20170140091 | Okuno | May 2017 | A1 |
| 20170140521 | Sakaguchi | May 2017 | A1 |
| 20170142340 | Kishi | May 2017 | A1 |
| 20170148153 | Nakao | May 2017 | A1 |
| 20170148154 | Nakao | May 2017 | A1 |
| 20170169276 | Agaian | Jun 2017 | A1 |
| 20170193296 | Duong | Jul 2017 | A1 |
| 20170206676 | Nakazato | Jul 2017 | A1 |
| 20170236262 | Ohnuki | Aug 2017 | A1 |
| 20170243077 | Fukui | Aug 2017 | A1 |
| 20170249766 | Namiki | Aug 2017 | A1 |
| 20170252924 | Vijayanarasimhan | Sep 2017 | A1 |
| 20170256081 | Ohki | Sep 2017 | A1 |
| 20170274534 | Takahashi | Sep 2017 | A1 |
| 20170286810 | Shigenaka | Oct 2017 | A1 |
| 20170304732 | Velic | Oct 2017 | A1 |
| 20170305589 | Yuyama | Oct 2017 | A1 |
| 20170343989 | Itakura | Nov 2017 | A1 |
| 20170346996 | Kida | Nov 2017 | A1 |
| 20170351904 | Kyusojin | Dec 2017 | A1 |
| 20170352145 | Dhagat | Dec 2017 | A1 |
| 20170355078 | Ur | Dec 2017 | A1 |
| 20170358069 | Sagisaka | Dec 2017 | A1 |
| 20170358070 | Sagisaka | Dec 2017 | A1 |
| 20170358071 | Yamaoka | Dec 2017 | A1 |
| 20170372526 | Groten | Dec 2017 | A1 |
| 20180004806 | Ohashi | Jan 2018 | A1 |
| 20180005424 | Niinuma | Jan 2018 | A1 |
| 20180013950 | Steinberg | Jan 2018 | A1 |
| 20180025252 | Konishi | Jan 2018 | A1 |
| 20180046759 | Barral | Feb 2018 | A1 |
| 20180047208 | Marin | Feb 2018 | A1 |
| 20180068168 | Miyakoshi | Mar 2018 | A1 |
| 20180068480 | Cincotti | Mar 2018 | A1 |
| 20180089763 | Okazaki | Mar 2018 | A1 |
| 20180096244 | Mallinson | Apr 2018 | A1 |
| 20180099408 | Shibata | Apr 2018 | A1 |
| 20180121721 | Garcia | May 2018 | A1 |
| 20180137689 | Eastwood | May 2018 | A1 |
| 20180150899 | Waldron | May 2018 | A1 |
| 20180197311 | Takizawa | Jul 2018 | A1 |
| 20180211373 | Stoppa | Jul 2018 | A1 |
| 20180250822 | Shimodaira | Sep 2018 | A1 |
| 20180250823 | Shimodaira | Sep 2018 | A1 |
| 20180250826 | Jiang | Sep 2018 | A1 |
| 20180252647 | Levitz | Sep 2018 | A1 |
| 20180260628 | Namiki | Sep 2018 | A1 |
| 20180260661 | Konishi | Sep 2018 | A1 |
| 20180268257 | Ren | Sep 2018 | A1 |
| 20180285698 | Yamada | Oct 2018 | A1 |
| 20180314877 | Srivastava | Nov 2018 | A1 |
| 20180322623 | Memo | Nov 2018 | A1 |
| 20180328789 | Kido | Nov 2018 | A1 |
| 20180330489 | Kido | Nov 2018 | A1 |
| 20180330490 | Kido | Nov 2018 | A1 |
| 20180336407 | Namiki | Nov 2018 | A1 |
| 20180336479 | Guttmann | Nov 2018 | A1 |
| 20180342077 | Tsuno | Nov 2018 | A1 |
| 20180345485 | Sinnet | Dec 2018 | A1 |
| 20180348144 | Ando | Dec 2018 | A1 |
| 20180350060 | Nakao | Dec 2018 | A1 |
| 20180357519 | Jin | Dec 2018 | A1 |
| 20180361589 | Paquin | Dec 2018 | A1 |
| 20180370027 | Oota | Dec 2018 | A1 |
| 20190005600 | Hazeyama | Jan 2019 | A1 |
| 20190012579 | Namiki | Jan 2019 | A1 |
| 20190019045 | Ogura | Jan 2019 | A1 |
| 20190026917 | Liao | Jan 2019 | A1 |
| 20190034762 | Hashimoto | Jan 2019 | A1 |
| 20190035067 | Yamamoto | Jan 2019 | A1 |
| 20190042896 | Sabripour | Feb 2019 | A1 |
| 20190058887 | Wang | Feb 2019 | A1 |
| 20190061151 | Namiki | Feb 2019 | A1 |
| 20190065823 | Srivastava | Feb 2019 | A1 |
| 20190065901 | Amato | Feb 2019 | A1 |
| 20190066333 | Furihata | Feb 2019 | A1 |
| 20190077015 | Shibasaki | Mar 2019 | A1 |
| 20190080468 | Tsai | Mar 2019 | A1 |
| 20190084011 | Imai | Mar 2019 | A1 |
| 20190087942 | Ma | Mar 2019 | A1 |
| 20190095749 | Konishi | Mar 2019 | A1 |
| 20190095763 | Arilla | Mar 2019 | A1 |
| 20190096055 | Namiki | Mar 2019 | A1 |
| 20190096083 | Arano | Mar 2019 | A1 |
| 20190096135 | Dal Mutto | Mar 2019 | A1 |
| 20190099142 | Sato | Apr 2019 | A1 |
| 20190108159 | Donahue | Apr 2019 | A1 |
| 20190108396 | Dal Mutto | Apr 2019 | A1 |
| 20190112134 | Ooba | Apr 2019 | A1 |
| 20190114804 | Sundaresan | Apr 2019 | A1 |
| 20190114941 | Shimaoka | Apr 2019 | A1 |
| 20190116049 | Ittogi | Apr 2019 | A1 |
| 20190122173 | Souder | Apr 2019 | A1 |
| 20190126467 | Kawamoto | May 2019 | A1 |
| 20190130555 | Chang | May 2019 | A1 |
| 20190130560 | Horowitz | May 2019 | A1 |
| 20190137410 | Valkonen | May 2019 | A1 |
| 20190138786 | Trenholm | May 2019 | A1 |
| 20190139212 | Hanzawa | May 2019 | A1 |
| 20190139213 | Kurita | May 2019 | A1 |
| 20190147305 | Lu | May 2019 | A1 |
| 20190147586 | Ikeda | May 2019 | A1 |
| 20190156474 | Watanabe | May 2019 | A1 |
| 20190164047 | ter Haar Romenij | May 2019 | A1 |
| 20190171914 | Zlotnick | Jun 2019 | A1 |
| 20190171915 | Reicher | Jun 2019 | A1 |
| 20190188451 | D'Ercoli | Jun 2019 | A1 |
| 20190188840 | Kwon | Jun 2019 | A1 |
| 20190188845 | Tamai | Jun 2019 | A1 |
| 20190188846 | Tamai | Jun 2019 | A1 |
| 20190188866 | Mehrseresht | Jun 2019 | A1 |
| 20190206084 | Noble | Jul 2019 | A1 |
| 20190220966 | Nakamura | Jul 2019 | A1 |
| 20190251349 | Duerksen | Aug 2019 | A1 |
| 20190270200 | Sakai | Sep 2019 | A1 |
| 20190286946 | Uemura | Sep 2019 | A1 |
| 20190287235 | Ikeda | Sep 2019 | A1 |
| 20190295244 | Adachi | Sep 2019 | A1 |
| 20190299405 | Warashina | Oct 2019 | A1 |
| 20190308325 | Higo | Oct 2019 | A1 |
| 20190311470 | Lee | Oct 2019 | A1 |
| 20190334897 | Anzai | Oct 2019 | A1 |
| 20190337152 | Homberg | Nov 2019 | A1 |
| 20190362480 | Diao | Nov 2019 | A1 |
| 20190362486 | Diao | Nov 2019 | A1 |
| 20190378242 | Zhang | Dec 2019 | A1 |
| 20190378259 | Hayashi | Dec 2019 | A1 |
| 20200005421 | Sato | Jan 2020 | A1 |
| 20200005422 | Subramanian | Jan 2020 | A1 |
| 20200013190 | Li | Jan 2020 | A1 |
| 20200019807 | Ma | Jan 2020 | A1 |
| 20200047337 | Williams | Feb 2020 | A1 |
| 20200058161 | Remine | Feb 2020 | A1 |
| 20200074666 | Yamamoto | Mar 2020 | A1 |
| 20200075141 | Iwamura | Mar 2020 | A1 |
| 20200082524 | Hiraoka | Mar 2020 | A1 |
| 20200104993 | Hong, II | Apr 2020 | A1 |
| 20200125887 | Nishii | Apr 2020 | A1 |
| 20200125889 | Okamoto | Apr 2020 | A1 |
| 20200125894 | Maeda | Apr 2020 | A1 |
| 20200126204 | Watanabe | Apr 2020 | A1 |
| 20200134377 | Attorre | Apr 2020 | A1 |
| 20200146635 | Wang | May 2020 | A1 |
| 20200147805 | Takahashi | May 2020 | A1 |
| 20200149871 | Kohlmann | May 2020 | A1 |
| 20200156254 | Li | May 2020 | A1 |
| 20200167905 | Bian | May 2020 | A1 |
| 20200186533 | Takagi | Jun 2020 | A1 |
| 20200193623 | Liu | Jun 2020 | A1 |
| 20200210688 | Xu | Jul 2020 | A1 |
| 20200250801 | He | Aug 2020 | A1 |
| 20200279359 | Kimura | Sep 2020 | A1 |
| 20200311490 | Lee | Oct 2020 | A1 |
| 20200320748 | Levinshtein | Oct 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 104104861 | Oct 2014 | CN |
| 105279514 | Jan 2016 | CN |
| 2796918 | Oct 2014 | EP |
| 11067853 | Mar 1999 | JP |
| 2002071574 | Mar 2002 | JP |
| 2007-114843 | May 2007 | JP |
| 2010085145 | Apr 2010 | JP |
| 2012122730 | Jun 2012 | JP |
| 2013-142558 | Jul 2013 | JP |
| 2013148361 | Aug 2013 | JP |
| 2014044268 | Mar 2014 | JP |
| WO 2001041068 | Jun 2001 | WO |
| 2013145202 | Oct 2013 | WO |
| 2015036261 | Mar 2015 | WO |
| Entry |
|---|
| “Object Recognition from Local Scale-Invariant Features,” David G. Lowe, (Sep. 1999), pp. 1-8. |
| “Generalizing the Hough Transform to Detect Arbitrary Shapes”, D.H. Ballard, Pattern Recognition, vol. 13, No. 2, pp. 111-122, 1981. |
| “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography”, Martin A. Fischler, et al., Communications of the Association for Computing Machinery vol. 24 No. 6, pp. 381-395, Jun. 1981. |
| Luh, T.C., “Character Recognition Example (III): Training a Simple NN for classification, Jul. 7, 2008,” 12 pages. |
| “Image Segmentation Tutorial,” Apr. 3, 2015, 20 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20170249766 A1 | Aug 2017 | US |
