Patent Application
20210374576
This application claims priority to Chinese Patent Application No. 202010473438.7, filed on May 29, 2020, which is hereby incorporated by reference in its entirety.
The present application relates to the technical field of computers, in particular to the field of artificial intelligence. The present application can be applied to the field of knowledge graphs.
The existing manners to verify a medical fact are mainly as follows: One is to verify it through manual searching and labeling; another is to extract a fact occurring in a medical document by manually pre-configuring a text template or a part-of-speech template, and compare the extracted fact with the fact to be verified to complete the verification.
In order to solve at least one problem in the existing technology, a medical fact verification method and apparatus, an electronic device, and a storage medium are provided according to embodiments of the application.
In a first aspect, a medical fact verification method is provided according to an embodiment of the application, including:
In a second aspect, a medical fact verification apparatus is provided according to an embodiment of the application, including:
In a third aspect, an electronic device is provided according to an embodiment of the application, including:
In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided according to an embodiment of the application, wherein the computer instructions cause a computer to perform the method of any embodiment of the first aspect.
Other effects of the above alternatives will be described below in connection with specific embodiments.
The drawings are included to provide a better understanding of the solution and are not to be construed as limiting the present application, wherein:
The exemplary embodiments of the application will be described below in combination with drawings, including various details of the embodiments of the application to facilitate understanding, which should be considered as exemplary only. Therefore, those of ordinary skill in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Likewise, descriptions of well-known functions and structures are omitted in the following description for clarity and conciseness.
The embodiment of the present application provides A medical fact verification method is provided according to an embodiment of the present application, which can be applied to an electronic device, and the electronic device can have data processing functions such as numerical calculation, logic calculation, and data storage. Referring to
S101, acquiring a medical fact to be verified and candidate evidence, wherein the medical fact to be verified includes a target entity, a target attribute and a target attribute value;
S102, inputting the target entity, target attribute value and candidate evidence into an attribute decision model to obtain a decision attribute;
S103, inputting the target entity, the target attribute value and the candidate evidence into a relevancy decision model to obtain a relevancy of the candidate evidence in a case that the target attribute and the decision attribute are the same, and
S104, determining that the medical fact to be verified is correct in a case that the relevancy of the candidate evidence accords with a preset condition.
In an embodiment of the present application, each medical fact may be represented in the form of an SPO triplet, S representing an entity, P representing an attribute, and O representing an attribute value. Taking a medical fact <measles, symptoms, skin maculopapules> as an example, the entity S is measles, the attribute P is symptoms, and the attribute value O is skin maculopapules.
Correspondingly, the processing of S101-S103 may be configured for processing the medical fact to be verified at this time, and may be configured for processing different medical facts to be verified at different times. An entity, an attribute and an attribute value in each medical fact to be verified are correspondingly referred to as a target entity, a target attribute and a target attribute value in the present application.
Alternatively, the attribute in the medical fact may include at least one of a clinical feature, an etiology and a pathology, a therapeutic regimen, a recommended medication, a complication, and a drug effect.
Alternatively, the candidate evidence is candidate evidence that verifies whether the medical fact is correct, and the candidate evidence may be retrieved from a designated medical database based on the medical fact to be verified. The designated medical database may store various types of authoritative medical materials, including books, magazines, papers, etc.
The embodiment can be configured for constructing a medical knowledge graph. In the process of constructing the medical knowledge graph, a medical fact such as <measles, symptoms, skin maculopapules> are extracted by a machine, and candidate evidence can be retrieved from a designated medical document library according to the medical fact to be verified. The verification of the medical fact is completed through the verification method provided by S101-S104, if the verification is correct, the medical fact is formally determined to be added into the medical knowledge graph, meanwhile, the relevancy of the candidate evidence can be configured for determining the corresponding supporting evidence, which is conducive to improving the accuracy of medical graph data.
In the embodiment, for the medical fact to be verified and the candidate evidence, firstly, an attribute corresponding to a target entity and a target attribute value described by the candidate evidence are decided through an attribute decision model to obtain a decision attribute; if the decision attribute accords with the target attribute, the relevancy of the candidate evidence with respect to the target entity and the target attribute value is decided through a relevancy decision model; and when the relevancy of the candidate evidence accords with a condition, the medical fact is verified to be correct.
According to the embodiment of the application, through the attribute decision model and the relevancy decision model, a dual decision of the attribute and relevancy decision can be completed; the medical fact can be verified to be correct in a case that the attribute described by the candidate evidence accords with the target attribute and the relevancy accords with the condition, which can strengthen the correlation decision of the medical fact and the candidate evidence, improve the stringency of the verification result, and better meet the requirements of medical professional data processing; moreover, manual labeling or manual defined rules are not needed, reducing labor cost, and more suitable for large-scale data processing.
In an embodiment, referring to
In an embodiment, referring to
According to the embodiment, when the attribute decision model decides that the attribute does not accord, it is decided that the candidate evidence cannot verify that the medical fact to be verified is correct, and the verification of the current candidate evidence is stopped, which effectively improves the calculation efficiency; and remarkably improves the verification efficiency especially in processing large-scale medical professional data.
In an embodiment, referring to
S102, inputting the target entity, the target attribute value and the candidate evidence into an attribute decision model to obtain a decision attribute, includes:
According to the embodiment, the attribute decision model adopts a structure with a natural language processing model and a classifier. Features are extracted from the entity, the attribute value and the candidate evidence firstly, then classification is performed on the basis of the features so as to decide the attribute to which they belong. The structure is simple, and the attribute decision can be realized.
The structure of the attribute decision model given by the above-mentioned embodiment is an alternative mode, and in other embodiments, a person skilled in the art could also realize the embodiments of deciding attributes based on target entities, target attribute values and candidate evidence through a structure with other models within the scope of the embodiment of the present application.
Optionally, the first natural language processing model adopts an enhanced representation from knowledge integration (ERNIE). In other alternatives, a BERT model may be used as the first natural language processing model.
Optionally, the first classifier adopts a Softmax classifier. It is also within the scope of the embodiments of the present application that other classifiers are selected to complete the same implementation of processing the analyzed feature vector for classification based on the natural language processing model to determine the corresponding attributes.
Alternatively, referring to
In an embodiment, the attribute decision model adopted in S102 is established by:
In the embodiment, the first natural language processing model that is pre-trained through the medical corpus is adopted, and the training of the attribute decision model can be realized through fine adjustment, namely a small amount of sample data is adopted for training, which greatly reduces the quantity requirement on the sample data, and reduces the cost of labeling the sample data manually.
In an embodiment, referring to
Correspondingly, the inputting the target entity, the target attribute value and the candidate evidence into the relevancy decision model to obtain a relevancy of the candidate evidence in S103 includes:
According to the embodiment, on the basis of adopting the natural language processing model and the classifier, the data output by the natural language processing model is split into the feature vector of the entity and the candidate evidence, and the feature vector of the attribute value and the candidate evidence, which are then processed by the two classifiers, respectively, thereby effectively strengthening the association between the candidate evidence and each of the entity and attribute value, and improving the accuracy of the relevancy.
The neurons of the output layer of the fully connected layer are connected to each neuron of the input layer. Therefore, by using the fully connected layer, the second layer feature vector of the target entity and the candidate evidence and the second layer feature vector of the target attribute value and the candidate evidence can be processed into a column item vector, facilitating the subsequent processing of the third classifier.
Optionally, the second natural language processing model adopts an ERNIE model. In other alternatives, a BERT model may be used as the first natural language processing model.
Alternatively, the second classifier and the third classifier each may adopt a Softmax classifier.
Alternatively, referring to
In addition, “X CLS” in
In an embodiment, the attribute decision model adopted in S103 is established by:
In the embodiment, the second natural language processing model that is pre-trained through the medical corpus is adopted, and the training of the relevancy decision model can be realized through fine adjustment, namely a small amount of sample data is adopted for training, which greatly reduces the quantity requirement on the sample data, and reduces the cost of labeling the sample data manually.
Alternatively, the second sample data may be obtained from known SPO triples in an existing medical knowledge base and results returned by an evidence retrieval module.
Alternatively, in the second sample data, the relevancy of the medical fact and the supporting evidence may be manually labeled.
In the embodiment, the second natural language processing model that is pre-trained through the medical corpus is adopted, and the training of the relevancy decision model can be realized through fine adjustment, namely a small amount of sample data is adopted for training, which greatly reduces the quantity requirement on the sample data, and reduces the cost of labeling the sample data manually.
In one example, the relevancy of the candidate evidence output by the relevancy decision model of S103 may be a numerical value, such as any number of interval [0, 1]. The greater the relevancy of the candidate evidence, the higher the relevancy of the candidate evidence, indicating that the candidate evidence can support the correctness of the medical fact, and the higher the probability that the medical fact is correct from the side.
Compared with other industries, the medical industry has stricter and more rigorous requirements on the overall data accuracy rate. Therefore, the attribute decision model and the relevancy decision model provided by the embodiment are ingenious in model structure, improving the accuracy rate of the verification result, and meeting the strict requirements of the medical industry on data. Moreover, according to the model provided by the embodiment of the application, through the basic features, a suitable deep learning model structure designed and the training on large-scale labeled data, high accuracy and recall rate can be obtained without depending on high-level features defined manually, and labor cost is reduced.
In an embodiment, S104 includes:
After being verified by the attribute decision model, the correctness of the medical fact can be verified if the relevancy is greater than the preset value. The decision is simple and the accuracy is high. Meanwhile, the candidate evidence with the highest relevancy is selected as the supporting evidence to provide a basis for verifying the correctness of the medical fact.
With regard to the above-mentioned S104, it is to be explained that if the relevancy of only one candidate evidence among a plurality of candidate evidence is greater than a preset threshold value, this candidate evidence with the relevancy greater than the preset value is directly considered to be the candidate evidence with the highest relevancy. In addition, if the medical fact only corresponds to one candidate evidence, if the relevancy of the candidate evidence is greater than a preset threshold value, the medical fact to be verified is verified to be correct, and the candidate evidence with the highest relevancy is used as the supporting evidence for determining that the medical fact is correct.
In other embodiments, the preset condition in S104 can also be set as other conditions, for example, the relevancy of candidate evidence exceeding a preset number is set to be greater than a preset threshold value, and the value of the preset number is greater than 1; for another example, the percentage of candidate evidence with a relevancy greater than a predetermined threshold among the plurality of candidate evidence is greater than a predetermined percentage.
In other embodiments, S104 may alternatively include selecting a plurality of candidate evidence whose relevancy ranking precedes as supporting evidence, and presenting the plurality of supporting evidence according to the relevancy ranking.
In an embodiment, the method of this embodiment further includes: if there is no relevancy of at least one candidate evidence being greater than the preset threshold value, determining that the medical fact is incorrect. No relevancy of at least one candidate evidence being greater than the preset threshold value includes the relevancy of each of the candidate evidence being less than the preset threshold value and the candidate evidence having no corresponding relevancy (i.e. the decision attributes obtained in S102 are all different from the target attributes).
The above S101-S104 are described in detail below by way of an example:
In S101, a medical fact to be verified and candidate evidence are obtained, wherein
The candidate evidence is that “measles is a viral infectious disease caused by measles virus, and belongs to Category B infectious disease among the notifiable infectious diseases in China. The main clinical manifestations of measles include fever, cough, runny nose and other catarrhal symptoms and conjunctivitis, and the characteristic manifestations of measles are Koplik spots and skin maculopapules”.
In S102, the target entity “measles”, the target attribute value “skin maculopapules”, and the candidate evidence are put into the attribute decision model to obtain a decision attribute “symptoms” corresponding to the “measles” and the “skin maculopapules”.
Specifically, referring to
In S103, because the target attribute “symptom” and the decision attribute “symptom” are the same, the target entity “measles” and the target attribute value “skin maculopapules” are input into the relevancy decision model to obtain a relevancy of the candidate evidence with respect to the target entity “measles” and the target attribute value “skin maculopapules”, and the relevancy of the candidate evidence is assumed to be 0.8.
Specifically, referring to
In S104, assuming that the preset condition is that the relevancy is greater than 0.7, and since 0.8>0.7, the relevancy 0.8 of the candidate evidence accords with the preset condition and the medical fact <measles, symptoms and skin maculopapules> to be verified is determined to be correct, and the candidate evidence can be used as supporting evidence for determining that <measles, symptoms and skin maculopapules> is correct.
An example of the verification process of one candidate evidence is given above. For the case where there are multiple candidate evidence, such as candidate evidence A, candidate evidence B, and candidate evidence C, similarly, the relevancies of candidate evidence A, candidate evidence B and candidate evidence C can be solved by S101-step S104, and the relevancies obtained are 0.3, 0.75, 0.8 in order. Because there is candidate evidence with a relevancy greater than 0.7, the medical fact can be verified to be tenable, and meanwhile, candidate evidence C with the highest relevancy can be selected to serve as the supporting evidence.
The following is an example of an output medical fact verification result, specifically:
“S”: “measles”,
“P”: “symptoms”,
“O”: “skin maculopapules”,
“label”: “1”,
“evidence”: “section V Measles.
Measles is a viral infectious disease caused by measles virus, and belongs to Category B infectious disease among the notifiable infectious diseases in China. The main clinical manifestations of measles include fever, cough, runny nose and other catarrhal symptoms and conjunctivitis, and the characteristic manifestations of measles are Koplik spots and skin maculopapules”.
Label indicates the verification result of the medical fact, label=1 indicates that the verification result is correct, and label=0 indicates that the verification result is wrong; and evidence represents supporting evidence determining that the medical fact is correct. Therefore, in the above example, the verification result is correct for the medical fact SPO <measles, symptoms, skin maculopapules> to be verified, and the above-mentioned evidence field is selected from the 8th edition of Infectious Diseases as the supporting evidence for determining that the medical fact is correct.
The method realized by the embodiment of the present application is a medical fact verification method based on a pre-training language model, and effectively improves the effect problem of fact verification on medical data. The method provided by the embodiment of the present application has at least one of the following advantages:
1. it has strong versatility and can deal with a large and wide range of medical fact verification issues;
2. the labor cost is low, mainly embodied in two aspects: firstly, for a new fact type, a new document set and a new expression mode, an extraction rule does not need to be redefined manually, and a correct result can be given according to the generalization of the model itself; secondly, the model is established in a mode of combining pre-training and fine adjustment, which reduces the requirements for the number of labeled samples, thereby reducing the cost of manually labeled samples; and
3. compared with a general fact verification method, the embodiment of the present application can be suitable for medical fact verification, and has strict data requirements, bringing certain effect improvement on medical data.
Correspondingly, the embodiment of the present application also provides a medical fact verification apparatus, and the included various modules thereof can be carried or arranged in the hardware of the electronic device, for example, the memory of the computer can carry the various modules of the device, to enable the central processing unit (CPU) of the computer to run the various modules in the memory.
Referring to
In an embodiment, referring to
In an embodiment, the attribute decision model includes a first natural language processing model and a first classifier.
Referring to
In an embodiment, the attribute decision model is established by:
In an embodiment, the relevancy decision model includes a second natural language processing model, two second classifiers, a fully connected layer, and a third classifier;
Referring to
In an embodiment, the relevancy decision model is established by:
In an embodiment, referring to
For the functions of the modules in the apparatus in the embodiments of the present application, reference may be made to the corresponding descriptions in the foregoing method, and details are not described herein again.
An electronic device and a readable storage medium are provided according to embodiments of the application.
As shown in
As shown in
The memory 1002 is a non-transitory computer-readable storage medium provided herein. Wherein the memory stores instructions executable by at least one processor to cause the at least one processor to perform the medical fact verification method provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for enabling a computer to perform the medical fact verification method provided herein.
The memory 1002, as a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, e.g. program instructions/modules corresponding to methods for medical fact verification in embodiments of the present application (such as the first acquisition module 501, the first decision module 502, the second decision module 503, and the second decision module 504 shown in
The memory 1002 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; and the storage data area may store data created according to use of the electronic device for the medical fact verification method, etc. In addition, the memory 1002 may include a high speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 1002 optionally includes memories remotely located with respect to the processor 1001, which may be connected via a network to the electronic device for the medical fact verification method. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof
The electronic device may further include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003, and the output device 1004 may be connected by a bus or otherwise, as exemplified in
The input device 1003 may receive input numeric or character information and generate key signal inputs related to user settings and functional controls of an electronic device for medical fact verification, such as touch screens, keypads, mice, track pads, touch pads, pointing sticks, one or more mouse buttons, track balls, joysticks, and other input devices. The output device 1004 may include a display apparatus, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., vibration motor), etc. The display apparatus may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display apparatus may be a touch screen.
Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitries, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implementing in one or more computer programs, which can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor capable of receiving data and instructions from, and transmit data and instructions to, a memory system, at least one input device, and at least one output device.
These computing programs (also referred to as programs, software, software applications, or codes) include machine instructions of programmable processors, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, and/or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
The systems and techniques described herein may be implemented in a computing system that includes a background component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which the user may interact with embodiments of the systems and techniques described herein), or in a computing system that includes any combination of such background component, middleware component, or front-end component. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through the communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other.
According to the technical scheme of the embodiment of the application, by the adoption of the attribute decision model and the relevancy decision model, the attribute and the relevancy decisions are sequentially completed, so that the correct technical means for verifying the medical fact can be realized in the case that the attribute described by the candidate evidence accords with the target attribute and the relevancy accords with the condition, solving the technical problem of high cost caused by manual verification in the existing technology, and reducing the labor cost; and the method is more suitable for large-scale data processing.
It will be appreciated that the various forms of flows, reordering, adding or removing steps shown above may be used. For example, the steps recited in the present application may be performed in parallel, sequentially or may be performed in a different order, so long as the desired results of the technical solutions disclosed in the present application can be achieved, and no limitation is made herein.
The above description only relates to specific embodiments of the present application, but the scope of protection of the present application is not limited thereto, and any of those skilled in the art can readily contemplate various changes or replacements within the technical scope of the present application. All these changes or replacements should be covered by the scope of protection of the present application. Therefore, the scope of protection of the present application should be determined by the scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202010473438.7 | May 2020 | CN | national |
