Patent Application
20240032819
The present application relates to the technical field of medical recognition, and in particular to a method, apparatus and system for recognizing a tremor symptom, a recognition terminal and a storage medium.
Tremor is a symptom caused by neurological diseases such as Parkinson's disease, essential tremor and dystonia, which is characterized by involuntary rhythmic shaking of a certain part of the body. Tremor delivers a significant impact on a person's quality of life, affecting activities of daily living such as eating, drinking, writing, and even speaking. Accurate and objective quantification of the tremor symptom is essential for effective diagnosis, treatment, and recognition of illness.
Currently, methods for recognizing a tremor symptom include subjective rating scales, invasive electrophysiological techniques, and accelerometer-based devices. However, existing methods for recognizing a tremor symptom have limitations in terms of accuracy, convenience, and a patient experience feeling. For example, subjective rating scales rely on the patient's self-assessment, which may be influenced by anxiety, prejudice, or poor memory. Invasive electrophysiological techniques require electrodes to be inserted into the body, which may cause discomfort and risk of infection. Accelerometer-based devices measure tremor from motion of the devices, which may be affected by factors other than tremor, including, but not limited to, tremor, motion artifacts.
In summary, there is an urgent need for a method for recognizing a tremor symptom, which has high accuracy, high convenience, and a good patient experience feeling.
Based on this, it is necessary to provide a method, apparatus and system for recognizing a tremor symptom, which have high accuracy, high convenience, and a good patient experience feeling, a recognition terminal and a storage medium.
A method for recognizing a tremor symptom includes:
In one example, the method further includes: training a set of images to be learned by means of a convolutional neural network model to obtain a convolutional neural network regression device, where all images to be learned in the set of images to be learned each include a spiral graph drawn by a patient.
In one example, the convolutional neural network model is based on a ResNet-18 backbone network, the ResNet-18 backbone network is composed of 18 parameterized layers, and the ResNet-18 backbone network includes a convolutional layer and a full connection layer, where an output layer of the full connection layer performs regression analysis to evaluate accuracy.
In one example, before the training a set of images to be learned by means of a convolutional neural network model to obtain a convolutional neural network regression device, the method further includes:
In one example, the preprocessing each of the images to be learned in the set of images to be learned specifically includes:
In one example, the method further includes:
In one example, the performing enhancement processing on each of the images to be learned and the image to be recognized specifically includes:
In one example, the method further includes:
An apparatus for recognizing a tremor symptom includes:
A recognition terminal includes a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
A system for recognizing a tremor symptom includes: a recognition terminal and at least one user terminal that execute the above method for recognizing a tremor symptom, where the recognition terminal and the at least one user terminal communicate by means of a network connection.
The recognition terminal is configured to train an image set to be learned by means of a convolutional neural network model to obtain a convolutional neural network regression device, where all images to be learned in the image set to be learned each include a spiral graph drawn by a patient.
The user terminal is configured to acquire the image to be recognized which includes a spiral graph, and send the image to be recognized to the recognition terminal.
The recognition terminal is further configured to receive an image to be recognized which is uploaded by the user terminal, where the image to be recognized includes a spiral graph used for recognizing whether a drawing person has a tremor state or not and evaluating a tremor level, is configured to take the image to be recognized as an input value of a pre-trained convolutional neural network regression device to obtain a tremor level, and is configured to send the tremor level to the user terminal.
The user terminal is further configured to receive and display the tremor level.
According to the above method, apparatus and system for recognizing a tremor symptom, the recognition terminal and the storage medium, the image to be recognized which is uploaded by the user terminal is received, the image to be recognized includes the spiral graph used for recognizing whether the drawing person has the tremor state or not and evaluating the tremor level, and the image to be recognized is taken as the input value of the pre-trained convolutional neural network regression device to obtain the tremor level. Compared with existing methods, the present application recognizes tremor by means of images, which is non-invasive and patient-friendly. Since the model is a deep learning algorithm trained on a large spiral image dataset, results of evaluating the severity of essential tremor are accurate and consistent.
For making the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail below in conjunction with the accompanying drawings and examples of the present application. It should be understood that the particular examples described herein are merely illustrative of the present application and are not intended to limit the present application.
For a method for recognizing a tremor symptom provided by the present application, tremor in the present application specifically refers to essential tremor, which may be applied to an application environment as shown in
The user terminal is configured to acquire the image to be recognized which includes a spiral graph, and send the image to be recognized to the recognition terminal.
The recognition terminal is further configured to receive an image to be recognized which is uploaded by the user terminal, where the image to be recognized includes a spiral graph used for recognizing whether a drawing person has a tremor state or not and evaluating a tremor level, is configured to take the image to be recognized as an input value of a pre-trained convolutional neural network regression device to obtain a tremor level, and is configured to send the tremor level to the user terminal.
The user terminal is further configured to receive and display the tremor level.
In an example, as shown in
S201, receive an image to be recognized which is uploaded by a user terminal, where the image to be recognized includes a spiral graph used for recognizing whether a drawing person has a tremor state or not and evaluating a tremor level.
In this example, the image to be recognized includes a spiral graph used for recognizing whether a drawing person has a tremor state or not and evaluating a tremor level, where the drawing person is a person who may have a tremor state. The drawing person may be instructed to use a pen or pencil for drawing completion on a piece of paper, and the drawer is instructed to draw a spiral from a center of the paper to an edge of the paper, which can be completed in about 30-60 seconds. An image to be recognized is acquired by photographing a spiral graph on the paper by means of a camera of the user terminal, and the image to be recognized is uploaded to the recognition terminal by the user terminal by means of the network. Or the image to be recognized can also be acquired by directly taking a picture by the recognition terminal equipped with a camera.
Preferably, before the step S202, the method further includes:
All the images to be learned in the set of images to be learned each include a spiral graph drawn by a patient, and the patient refers to a person who is diagnosed to have a tremor state. The convolutional neural network model is based on a ResNet-18 backbone network, and the ResNet-18 backbone network is composed of 18 parameterized layers and includes a convolutional layer and a full connection layer, where an output layer of the full connection layer performs regression analysis to evaluate accuracy, and the convolutional layer is responsible for extracting features from the input image. Specifically, the features of the input image include the shape, amplitude and frequency of the spiral graph, an activation layer introduces nonlinear changes into the network, a pooling layer is configured to reduce spatial resolution of the feature graph, and the full connection layer is configured to perform prediction according to the extracted features. Specifically, the output layer is adjusted from a predicted classification label to a single variable that predicts the severity of the essential tremor. The convolutional layer is slightly adjusted so as to recognize a pattern in the spiral graph, thereby accurately determining the severity of the tremor state. A modified architecture is trained on a large data set of the spiral graph to learn the mode of the hand tremor and quantify the severity thereof the hand tremor, thereby generating the convolutional neural network regression device, and taking the image including the spiral graph as the input value. The ResNet-18 architecture has the capability of learning complex features, and is very suitable for quantifying the severity of the essential tremor symptom from the spiral graph.
Specifically, regression analysis is performed by the fully connected output layer of the ResNet-18 architecture. The output layer is modified from the predicted classification label to a continuous value that predicts a predicted tremor level representing the image of the input spiral graph. Then, the predicted tremor level is compared to a true tremor level to evaluate the accuracy of the model. Regression analysis is a critical component of the convolutional neural network model because it enables the algorithm to quantize the severity of the essential tremor symptom on a continuous scale (rather than a classification scale). In this way, the severity of the essential tremor symptom can be evaluated more accurately, and more detailed information is provided for medical professionals, thereby making a judicious decision on a treatment scheme. Preferably, two indicators can be applied to evaluate internal test performance, and the indicators include a root-mean-square error (RMSE) and a Pearson correlation coefficient. The root-mean-square error is used for quantifying difference between the predicted tremor level and the true level, and a method for calculating the indicator is to average the difference between the predicted level and the true level of each image, and then take the root. The lower the root-mean-square error is, the closer the predicted level is to the true level, indicating that the higher the accuracy of the model is. In addition, the Pearson correlation coefficient is also calculated to evaluate a linear relationship between the predicted tremor level and the true level, a value range of the indicator is −1-1, where 1 represents a strong positive correlation, −1 represents a strong negative correlation, and 0 represents an irrelevant condition. The higher Pearson correlation coefficient indicates the predicted tremor level is highly related to the true level. The combination of the root-mean-square error and the Pearson correlation coefficient provides a comprehensive evaluation of the internal test performance. During internal testing, the two indicators of a test data set are calculated and used as measurement criteria for the algorithm accuracy. As a result, the algorithm has a lower root-mean-square error (0.31) and a higher Pearson correlation coefficient (0.95), indicating that the algorithm has good performance in predicting the tremor level of the image of the spiral graph.
S202, take the image to be recognized as an input value of a pre-trained convolutional neural network regression device to obtain a tremor level.
In this example, the set of images to be learned is trained by the convolutional neural network model to obtain the pre-trained convolutional neural network regression device, the image to be recognized is taken as the input value of the pre-trained convolutional neural network regression device, and a continuous output value of the convolutional neural network regression device is converted into a classified tremor level. The tremor level represents the severity of the essential tremor, and specifically the level is represents by means of the digitals of 0-4, where level 0 represents no tremor, level 4 represents the most serious tremor, and the larger the level value from 1 to 4 is, the more serious the tremor is. As shown in
S203, send the tremor level to the user terminal.
In this example, the recognition terminal sends the tremor level to the user terminal, and at the same time, saves information such as the tremor level and a user ID associated with the corresponding user terminal into a database, and the user terminal displays the tremor level, as shown in
Preferably, after the step S203, the method further includes: generate a report of a tremor level according to the tremor level and the historical tremor level associated with the ID of the user terminal. At the same time, the tremor level is saved in the database, such that a medical worker can track changes of the tremor severity over time.
In the above method for recognizing a tremor symptom, the image to be recognized which is uploaded by the user terminal is received, the image to be recognized includes the spiral graph used for recognizing whether the drawing person has the tremor state or not and evaluating the tremor level, and the image to be recognized is taken as the input value of the pre-trained convolutional neural network regression device to obtain the tremor level. Compared with existing methods, the present application recognizes tremor by means of images, which is non-invasive and patient-friendly. Since the model is a deep learning algorithm trained on a large spiral image dataset, results of evaluating the severity of essential tremor are accurate and consistent.
In an example, the method further includes:
Specifically, the step of preprocessing each of the images to be learned in the set of images to be learned specifically includes:
According to the example of the present invention, the preprocessing step is crucial for ensuring the consistency of the input images and accurately reflecting the tremor severity of an individual, and the model itself can also more accurately evaluate the tremor severity and provide reliable results by normalizing the input images and expanding the scale of the training data set.
In an example, the method further includes:
Specifically, the step of performing enhancement processing on each of the images to be learned and the image to be recognized specifically includes:
According to the example of the present invention, image enhancement is combined with the above preprocessing step for use, the quality and accuracy of input data of the deep learning model can be improved, and the model can more accurately evaluate the severity of tremor and provide reliable results for the patient with essential tremor by reducing blurring caused by tremor and other image quality influence.
It should be understood that although the various steps in the flowchart of
In an example, as shown in
The image receiving module 61 is configured to receive an image to be recognized which is uploaded by a user terminal, and the image to be recognized includes a spiral graph used for recognizing whether a drawing person has a tremor state or not and evaluating a tremor level.
The image recognition module 62 is configured to take the image to be recognized as an input value of a pre-trained convolutional neural network regression device to obtain a tremor level.
The result feedback module 63 is configured to send the tremor level to the user terminal.
Furthermore, the apparatus further includes:
Preferably, the convolutional neural network model is based on a ResNet-18 backbone network, the ResNet-18 backbone network is composed of 18 parameterized layers, and includes a convolutional layer and a full connection layer, where an output layer of the full connection layer performs regression analysis to evaluate accuracy.
Furthermore, the apparatus further includes:
Specifically, the image preprocessing module is specifically configured to crop each of the images to be learned, where each of the images to be learned subjected to cropping only retains a spiral graphic portion; to adjust a size of each of the images to be learned subjected to cropping according to preset resolution; to normalize each of the images to be learned subjected to size adjustment by means of histogram equalization or contrast stretching; to convert each of the images to be learned subjected to normalization into a grayscale image; and to augment each of the images to be learned subjected to grayscale, where the augmented manner includes: any one or a combination of more of random rotating, symmetrical flipping, scaling, perspective, changing brightness of images, contrast, saturation and hue, and inverting colors of given images.
Furthermore, the apparatus further includes:
Specifically, the image enhancement module is configured to deblur each of the images to be learned and the image to be recognized by using a non-blind deblurring algorithm, a Wiener filtering method or a bilateral filtering method; or/and denoise each of the images to be learned and the image to be recognized by means of a median filter, an adaptive Wiener filter, a non-local self-similarity model, a sparse model, a gradient model, or a Markov random field model; or/and
Furthermore, the apparatus further includes:
For the specific definition of the apparatus for recognizing a tremor symptom, reference may be made to the definition of the method for recognizing a tremor symptom above, which is not repeated herein. Each module in the above apparatus for recognizing a tremor symptom may be implemented wholly or partially by means of software, hardware, or a combination thereof. The above modules may be embedded in or independent of a processor in a recognition terminal in a hardware form, or may be stored in a memory in the recognition terminal in a software form, such that the processor may call and execute operations corresponding to the above modules.
In an example, a recognition terminal is provided, and the recognition terminal may be a server, and an internal structural diagram thereof may be as shown in
In an example, a recognition terminal is provided, the recognition terminal may be a mobile terminal, and an internal structural diagram thereof may be as shown in
Those skilled in the art may understand that the structure shown in
In an example, a recognition terminal is provided and includes a memory and a processor, where the memory stores a computer program, and when executing the computer program, the processor implements the following steps:
In an example, a computer-readable storage medium is provided, a computer program is stored on the storage medium, and when the computer program is executed by a processor, the following steps are implemented:
Those skilled in the art may understand that implementation of all or some procedures in the methods of the above examples may be accomplished by instructing related hardware by means of a computer program. The computer program may be stored in a non-volatile computer-readable storage medium, and when the computer program is executed, the procedures of the examples in the above methods may be included. Any reference to a memory, storage, a database, or other media used in the examples provided by the present application may include non-volatile and/or volatile memory. The non-volatile memory may include a read only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM) or an external cache memory. By way of illustration and not limitation, RAM is available in many forms such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDRSDRAM), an enhanced SDRAM (ESDRAM), a Synchlink DRAM (SLDRAM), a Rambus direct RAM (RDRAM), a direct Rambus dynamic RAM (DRDRAM), and a Rambus dynamic RAM (RDRAM), etc.
Various technical features of the embodiments mentioned above may be arbitrarily combined. To simplify description, all possible combinations of the various features of the embodiments mentioned above are not described. However, if only the combinations of these technical features do not conflict, they shall be considered to be within the scope of description of the present invention.
The embodiments mentioned above are merely several embodiments of the present application, and are specifically described in details, but cannot be interpreted as limiting the scope of the patent for the invention as a result. It shall be noted that for those of ordinary skill in the field, they may make several transformations and improvements on the premise of not deviating from the conception of the present application, and these transformations and improvements shall fall within the scope of protection of the application. Hence, the scope of protection of the patent for the present application shall be subject to the appended claims.
