Patent Application
20240029387
The application claims priority from Chinese Patent Application No. 202010881849.X, filed Aug. 28, 2020, entitled “Image Recognition Method, Electronic Device and Readable Storage Medium”, all of which are incorporated herein by reference in their entirety.
The present invention relates to the field of image recognition of medical equipment, and more particularly to an image recognition method, an electronic device, and a readable storage medium.
Gastrointestinal motility refers to the ability of normal gastrointestinal peristalsis to help complete the digestion and absorption of food. Poor gastrointestinal motility may result in an indigestion.
In the prior art, the strength of gastrointestinal motility is generally identified by gastrointestinal markers. Specifically, after a user swallows markers of different shapes in several times, the positions of the markers are determined through images obtained by X-ray imaging, and then the strength of gastrointestinal motility is determined.
In the prior art, the positions and types of the markers in a X-ray image are usually determined by manually observing the image. However, for markers of different shapes swallowed in different times, the size displayed on the X-ray image is small, and there are many types, so it is difficult to accurately count the positions and quantities of various markers by manual observation, and the gastrointestinal motility of a subject cannot be determined.
The present invention provides an image recognition method, an electronic device, and a readable storage medium.
In order to achieve one of the above-mentioned objects of the present invention, an embodiment of the present invention provides an image recognition method. The method comprises: segmenting an original image into a plurality of unit images having the same predetermined size, where a plurality of markers are distributed in the original image;
In an embodiment of the present invention, in the step of segmenting the original image into the plurality of unit images having the same predetermined size, the method further comprises:
In an embodiment of the present invention, the method for building the neural network model comprises: extracting at least one feature layer by using convolutional neural networks corresponding to each unit image:
In an embodiment of the present invention, the method further comprises: performing pooling layer processing on the unit images for a plurality of times according to the types and size of the markers to obtain the corresponding feature layers.
In an embodiment of the present invention, in the process of performing pooling layer processing on the unit images for the plurality of times according to the types and size of the markers to obtain the corresponding feature layers, the method comprises:
In an embodiment of the present invention, the method further comprises: setting c2=4, and the offset parameters for adjusting the anchor boxes comprise width offset values and height offset values of an upper left corner, a width scaling factor and a height scaling factor.
In an embodiment of the present invention, determining whether the markers in the frames are the same marker according to the coordinate values of the two adjacent detection frames comprises:
In an embodiment of the present invention, determining whether the markers in the boxes are the same marker according to the coordinate values of the two adjacent detection frames comprises:
In an embodiment of the present invention, merging the two detection frames comprises:
It is another object of the present invention to provide an electronic device. In an embodiment, the electronic device comprises a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the image recognition method.
It is still another object of the present invention to provide a computer-readable storage medium. In an embodiment, the computer-readable storage medium stores a computer program and the computer program is executed by the processor to implement the steps of the image recognition method as described above.
According to all aspects of the present invention, the advantages over the prior art are that: with the image recognition method, the electronic device and the computer-readable storage medium of the present invention, the original image is automatically processed by a neural network model to add detection frames, and further, the detection frames labeled repeatedly are merged to improve the accuracy of image marking, so that the types and positions of markers in the image are effectively recognized, and the gastrointestinal motility of a subject is accurately determined.
The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. However, the embodiments are not intended to limit the present invention, and the structural, method, or functional changes made by those skilled in the art in accordance with the embodiments are included in the scope of the present invention.
Referring to
In an embodiment of the present invention, the original image is usually large in size, and in order to improve the accuracy of recognition, it is necessary to segment the image before the recognition of the image, and restore the image according to the segmentation sequence after the recognition of the image.
In step S1, in the process of segmenting the original image into a plurality of unit images with the same predetermined size, the image may be segmented in sequence starting from the edges and corners of the original image, or the image may be segmented as required on the basis of any point in the original image after the point is selected. Correspondingly, the method further comprises: if a size of any unit image is less than the predetermined size in the segmentation process, complementing edge pixel values to the original image before the unit images are formed, or complementing edge pixel values to the unit images with a size less than the predetermined size after the unit images are formed, so that the size of each unit image is the same as the predetermined size.
It should be noted that, in a specific application process, the original image may be segmented first, and if the size of the finally formed unit image is less than the predetermined size, the edge pixel values are complemented only for the finally formed unit image. It is also possible to calculate in advance whether the current original image can be completely segmented according to the predetermined size before the image segmentation, and if it cannot be completely segmented, the edge pixel values are complemented to the original image before the segmentation. Usually, the pixel value of the complementing position can be set specifically as needed, e.g., set to 0, which is not further described here.
Preferably, after segmentation, the unit image is a square in shape. And in an embodiment of the present invention, the size of the unit image is 320*320, and the unit is a pixel.
In step S2, the method for building the neural network model comprises: extracting at least one feature layer by using convolutional neural networks (CNN) corresponding to each unit image.
In an embodiment of the present invention, in the process of extracting the feature layer, p convolution kernels of m*m are configured to convolution predictors of anchor boxes to process the unit images, so as to predict the type and the position of the markers. Where, m is an odd positive integer. The anchor box is a preset rectangular box with different aspect ratios. p=(c1+c2)*k, where c1 represents the number of types of markers, k represents the number of anchor boxes, and c2 represents the number of offset parameters for adjusting the anchor boxes. The detection frame is obtained by changing the size of the anchor box.
Further, the unit images are subjected to pooling layer processing for a plurality of times according to the types and size of the markers to obtain the corresponding feature layers; that is, the types and size of the markers determine the number and size of the feature layers. Specifically, the size of the markers on the original image is pre-divided to determine the number and size of the feature layers.
Preferably, in the process of performing pooling layer processing on the unit images for a plurality of times according to the types and size of the markers to obtain the corresponding feature layers, the method specifically comprises: before performing pooling layer processing on a unit image each time, performing convolution layer processing on the unit image at least once, and the sizes of convolution kernels are the same.
The convolution layer processing refers to dimension reduction and feature extraction on an input image through convolution operations. The pooling layer processing is used to reduce the spatial size of the image.
In order to facilitate understanding, a specific example is described below for reference.
Referring to
Referring to
Preferably, in a specific example of the present invention, a 3*3 convolution is configured to process the image, that is, m=3 is set. There are three types of markers to be recognized, that is, c1=3 is set, which are dot type, O-ring type and tri-chamber type. The anchor box is a plurality of bounding boxes with different size and aspect ratios generated by taking any pixel point as a center. The offset parameters for adjusting the anchor box specifically comprise: width offset values and height offset values of an upper left corner, width scaling factors and height scaling factors, that is, c2=4 is set.
Specifically, the output of c1 is a one-dimensional array, denoted result c1[3]. Where result c1[0], result c1[1], and result c1[2] are three elements of the array, each representing the probability of a type that the marker in the anchor box may have. In a specific example, result c1[0] represents the probability that the marker in the anchor box is of a dot type, result c1[1] represents the probability that the marker in the anchor box is of an O-ring type, result c1[2] represents the probability that the marker in the anchor box is of a tri-chamber type, and the value ranges of the three elements are all from 0 to 1. The type of the marker in the anchor box is determined by the maximum value of the three elements. For example, when the value of result_c1[0] is the maximum among the three elements, the marker in the corresponding anchor box is of dot type.
The output of c2 is a one-dimensional array, denoted result c2[4]. Where, result c2[0], result c2[1], result_c2[2] and result c2[3] are four elements of the array, which respectively represent the width offset value of the upper left corner of the anchor box, the height offset value of the upper left corner of the anchor box, the width scaling factor of the anchor box and the height scaling factor of the anchor box, and their values range from 0 to 1. By the output of c2, the size of the anchor box is adjusted to form a detection frame.
Through the above steps, the types and positions of the markers are preliminarily determined.
In addition, it should be noted that, when the neural network model is initially built, the unit images for training are first manually labeled, and the labeling content is the type information of the marker, which includes: a detection frame enclosing each marker, and a coordinate value of the upper left corner and a coordinate value of the lower right corner corresponding to the detection frame. Then, the unlabeled unit images are input to the initially built neural network model for prediction. The closer the result of prediction is to the result of manual labeling, the higher the detection accuracy of the neural network model is, and when the ratio of the result of prediction to the result of manual labeling is greater than a preset ratio, the neural network model can be normally applied. In this process, when the ratio of the result of prediction to the result of manual labeling is not greater than the preset ratio, the neural network model needs to be adjusted until the requirement is met. Thus, the neural network model is trained through the above contents, so that the result of prediction is more accurate. In this process, when the ratio of the result of prediction to the result of manual labeling is not greater than the preset ratio, the neural network model needs to be adjusted until the requirement is met. Thus, the neural network model is trained through the above contents, so that the result of prediction is more accurate. In the process of constructing the neural network, the detection accuracy of the neural network model can be evaluated by Intersection over Union (IOU), which is not further described here.
In step S3, a neural network model is configured to add detection frames to the markers in the unit images to form a pre-output image as shown in
Correspondingly, in order to solve the problem, in the preferred embodiment of the present invention, multiple detection frames of the same marker need to be merged, so that only one detection frame corresponding to each marker in the finally output image is uniquely recognized, which is conducive to the statistics and position determination of the markers, thereby accurately determining the gastrointestinal motility of the subject.
Specifically, step S4 specifically comprises: determining the type of the marker in each detection frame according to the probability of the type of the marker, and if the markers in two adjacent detection frames are of the same type, determining whether the markers in the frames are the same marker according to the coordinate values of the two adjacent detection frames.
Further, determining whether the markers in the frames are the same marker according to the coordinate values of the two adjacent detection frames comprises:
that is, abs(rectangles[i+1][xL]−rectangles[i][xL])<n1,
and (rectangles[i+1][yL]−rectangles[i][yL])<n2 are met at the same time;
In addition, the step of determining whether the markers in the frames are the same marker according to the coordinate values of the two adjacent detection frames further comprises: determining whether the markers selected in two adjacent detection frames in the vertically stitched pre-detection unit image are the same marker;
that is, abs(rectangles[j+1][xL]−rectangles[j][xL])<n3,
and abs(rectangles[j+1][yL]−rectangles[j][yL])<n4 are met at the same time;
Further, after determining that the markers selected in the two detection frames used in current calculation are the same marker, the two detection frames are merged, and the merging method specifically comprises: comparing the coordinate values of the upper left corners of the two detection frames currently used for calculation, and respectively taking the minimum values of the horizontal coordinate and the vertical coordinate as the coordinate values of the upper left corner of the merged detection frame; comparing the coordinate values of the lower right corners of the two detection frames, and respectively taking the maximum values of the horizontal coordinate and the vertical coordinate as the coordinate values of the lower right corner of the merged detection frame.
Correspondingly, the coordinates (xaL, yaL) of the upper left corner and the coordinates (xaR, yaR) of the lower right corner of the detection frame after the horizontal merging are respectively:
x
aL=min(rectangles[i+1][xL],rectangles[i][xL]),
y
aL=min(rectangles[i+1][yL],rectangles[i][yL]),
x
aR=max(rectangles[i+1][xR],rectangles[i][xR]),
y
aR=max(rectangles[i+1][yR],rectangles[i][yR]),
where, min( ) represents the minimum value, and max( ) represents the maximum value; xR and yR represent the horizontal coordinate value and the vertical coordinate value of the lower right corner of the detection frame, respectively.
Correspondingly, the coordinates of the upper left corner (xbL,ybL) and the coordinates of the lower right corner (xbR,ybR) of the detection frame after vertical merging are respectively:
x
bL=min(rectangles[j+1][xL],rectangles[j][xL]),
y
bL=min(rectangles[j+1][yL],rectangles[j][yL]),
x
bR=max(rectangles[j+1][xR],rectangles[j][xR]),
y
bR=max(rectangles[j+1][yR],rectangles[j][yR]).
It should be noted that, for each pre-output image, it is necessary to sequentially determine whether the markers in two adjacent detection frames are the same marker in the horizontal direction and the vertical direction respectively. The sequence of horizontal merging and vertical merging is not limited. It may be horizontal merging first and then vertical merging, or vertical merging first and then horizontal merging. The sequence of horizontal merging and vertical merging may not affect the final output result. In addition, in the above examples, the feature values are described as the coordinate value of the upper left corner and the coordinate value of the lower right corner of each detection frame. In practical application, the feature value may be selected as the same coordinate value having the same position on each detection frame. For example, the feature values are the coordinate value of the lower left corner and the coordinate value of the upper right corner, and the transformation of the feature values can not affect the final output result. The methods of selecting the feature values at different positions in the detection frame are all included in the protection scope of the application, and are not further described herein.
Referring to
Further, after the detection frames are recognized and merged, an image is output, and the output image has a unique detection frame corresponding to each marker. In this case, the type and position of the marker can be determined by the label and position of the detection frame in the image, and the locations of different types of markers in the gastrointestinal tract can be determined, so that the gastrointestinal motility of the subject can be determined.
Further, the present invention provides an electronic device. In an embodiment, the electronic device comprises a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the image recognition method.
Further, the present invention provides a computer-readable storage medium. In an embodiment, the computer-readable storage medium stores a computer program and the computer program is executed by the processor to implement the steps of the image recognition method as described above.
To sum up, with the image recognition method, the electronic device and the computer-readable storage medium of the present invention, the original image is automatically processed by a neural network model to add detection frames, and further, the detection frames labeled repeatedly are merged to improve the accuracy of image marking, so that the types and positions of markers in the image are effectively recognized, the distribution of the markers in the gastrointestinal tract can be confirmed, and the gastrointestinal motility of the subject is accurately determined.
It should be understood that, although the description is described in terms of embodiments, not every embodiment merely comprises an independent technical solution. The description is presented in this way only for the sake of clarity, those skilled in the art should have the description as a whole, and the technical solutions in each embodiment may also be combined as appropriate to form other embodiments that can be understood by those skilled in the art.
The series of detailed descriptions set forth above are only specific descriptions of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. On the contrary, many modifications and variations are possible within the scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202010881849.X | Aug 2020 | CN | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CN2021/112777 | 8/16/2021 | WO |
