Patent Application
20230055094
This disclosure is directed to a method and system for facilitating the annotation of transcribed audio or audio-visual recordings of medical encounters.
Conversations between patients and medical practitioners such as doctors and nurses and their conversations are often recorded. The record of the conversation, and a transcript, are part of the patient's medical record. The transcript can be created by a speech-to-text converter or created by a trained (human) medical transcriptionist listening to the recording.
A transcript without any annotation is of limited usefulness when it is reviewed by the physician, as they have to pore over many lines or pages of the transcript to find relevant information or understand the relatedness of different comments in the transcript.
Additionally, a collection of transcripts of medical encounters can be used to train machine learning models. Training a machine learning model requires a large amount of high quality training examples, i.e., labelled data. There is a need in the art for methods for facilitating the generation of transcripts of medical encounters that are annotated, that is, relevant words or phrases are highlighted and associated with medical concepts and grouped as being related to each other. This disclosure meets that need.
In a first aspect, a method of facilitating annotation of a recording of a medical practitioner-patient conversation is disclosed. The method includes a step of generating a display of the transcribed audio recording (i.e., transcript), for example on the display of a workstation used by a human (“scribe labeler”) who is performing the annotation. A tool is provided for highlighting spans of text in the transcript consisting of one or more words. The tools can be simple mouse or keyboard shortcuts for selecting or highlighting one or more words.
The method further includes a step of providing a tool for assigning a label to the highlighted spans of text. The tool includes a feature for searching through a set of predefined labels available for assignment to the highlighted span of text. For example, when the scribe labeler highlights a word such as “stomachache” in the transcript a window pops up where the user can search through available labels, e.g. by scrolling or using a search tool. The labels encode medical entities (such as symptoms, medications, lab results, etc.) and attributes of the medical entities (e.g., severity, location, frequency, time of onset of a symptom entity).
In this document, the term “medical entities” is intended to refer to categories of discrete medical topics, such as symptoms, medications, lab results, vital signs, chief complaint, medical imaging, conditions, medical equipment, and so forth. The medical entities are predefined to be relevant to the context of the labelling task, and so in this case in one embodiment they could consist of the following list: medications, procedures, symptoms, vitals, conditions, social history, medical conditions, surgery, imaging, provider, vaccine, reproductive history, examination, and medical equipment. The medical entities could be structured in a hierarchical manner, such as the medical entity “medication” could be in the form of “medication:allergy” where “allergy” is a type or subclass of the overall class “medication.” As another example, the medical entity “symptom” could be structured in a hierarchical manner of symptoms for different parts of the body, such as “symptom:eyes”, “symptom:neurological”, etc.
The term “attributes of the medical entities” simply means some descriptive property or characteristic of the medical entity, such as for example the medical entity “medical equipment” may have an attribute of “patient's actual use” meaning that the patient is currently using a piece of medical equipment. As another example, a symptom medical entity may have an attribute of “onset.” A label of “symptom/onset” would be used as an annotation when there is word or phrase in the transcript indicating when the patient first started experiencing the symptom. As another example, a label of “medical equipment/regularly” would be used as an annotation when there is a word or phrase in the transcript indicating the patient used some piece of medical equipment regularly, with “regularly” being the attribute of the medical entity “medical equipment.”
The method further includes a step of providing a tool for grouping related highlighted spans of texts. The tool could be for example a combination of mouse clicks or keyboard shortcuts to establish the grouping. The groupings allow medical entities associated with labels assigned to the highlighted spans of text to be associated as a group. For example, in a conversation in which a patient describes a sharp chest pain that started last week, the text “sharp”, “chest pain” and “last week” would be highlighted and labeled with symptom labels and attributes of severity, location, and time of onset, respectively and grouped together as all being related to each other.
In another aspect, a system is disclosed for facilitating annotation of a recording of a medical practitioner-patient conversation. The system includes a) an interface displaying a transcript of the recording; b) a tool for highlighting spans of text in the transcript consisting of one or more words; c) a tool for assigning a label to the highlighted spans of text, wherein the tool includes a feature enabling searching through predetermined labels available for assignment to the highlighted span of text, and wherein the labels encode medical entities and attributes of the medical entities; and d) a tool for creating groupings of related highlighted spans of texts.
The methods and systems are applicable to other types of transcripts, in which a set of predefined labels are created, e.g., by an operator, which are designed to be relevant to the annotation task at hand and the labels are associated with entities and attributes relevant to the transcript and annotation task. The tools of this disclosure are used in the same manner in these other possible implementations, such as for example transcripts of legal proceedings, such as deposition or trial, or transcripts of hearings before administrative bodies, such a city council, Congress, State Legislature, etc.
This disclosure is directed to methods and systems for facilitating annotations of recordings of medical encounters, i.e., conversations between patients and medical practitioners such as doctors or nurses. The recordings could be audio or audio-visual recordings. The recordings are transcribed into written form. The transcripts could be generated by trained medical transcriptionists, that is by hand, or by the use of speech to text converters, which are known in the art. The output of the system is an annotated version of the transcript in which relevant medical information (i.e., spans of text, such as individual words or groups of words) in the text are labeled (i.e., tagged as being associated with medical entities and attributes of such entities), and grouped to express relatedness between the labelled text.
At step 104, after the required patient consents are obtained, the patient consults with the medical practitioner and a recording, either audio or audio-visual, is obtained and stored in digital format.
At step 106, a written transcript of the recording is obtained, either by a trained transcriptionist or by use of a speech-to-text converter. The transcript is preferably accompanied by a time indexing, in which the words spoken in the transcript, or lines of text, are associated with elapsed time of the recording, as will be illustrated subsequently.
At step 108, an annotation of the transcript is performed by the scribe labeler in the manner described and explained in the subsequent figures. The annotations include the assignment of labels to spans of text in the transcript and groupings of spans of text to indicate their relatedness. In step 108 a display of the transcribed audio recording is generated, for example on the display of a workstation used by the scribe labeler. See
A tool is also provided for grouping related highlighted spans of texts. The groupings allow medical entities associated with labels to be grouped together. For example, in a conversation in which a patient describes a sharp chest pain that started last week, the text “sharp”, “chest pain” and “last week” would be highlighted and labeled with symptom labels and attributes of severity, location, and time of onset, and grouped together, as they are all related to a single medical condition of the patient. This tool can consist of keyboard and/or or mouse action, as explained below.
The system may include a pre-labeler 110, shown in more detail in
As a result of the annotation step 108 an annotated transcript file 112 is created, which consists of the transcript as well as annotations in the form of labelled or tagged spans of text (words or phrases) and groupings of the tagged spans of text. The annotated transcript file is in digital form, with the annotations and groupings in the file as metadata or otherwise. The annotated transcript file 112 is then added to the patient's electronic health record (EHR) 114 or supplied to a machine learning model training system 116. The machine learning model training system 116 may, for example, be a system for training a machine learning model to automatically annotate transcripts of medical encounters. Alternatively, the machine learning model may use the annotated transcript as well as other data in the patient health record, for not only the individual patient, but also a multitude of other patients, to generate predictions of future medical events for example as described in the pending U.S. provisional application Ser. No. 62/538,112 filed Jul. 28, 2017, the content of which is incorporated by reference herein. The EHR 114 may be provided to the system 116 as indicated by the dashed line 114.
The annotated transcript file 112 may be fed back into the pre-labeler to enable further training the machine learning pre-labeler 110, as indicated by the dashed line 120. This aspect will be described in further detail later.
The interface provides a tool for text highlighting. In particular, mouse and keyboard shortcuts make highlighting spans of text easy. For example, a user can double click on a given word and the word is automatically highlighted on the display. Only words can be highlighted, not individual characters, reducing errors and increasing annotation speed. Other tools could be used for highlighting, such as by click and drag techniques with a mouse, a keyboard stroke (such as by putting the cursor over the word and hitting a particular key such as H, or CTRL-H), or a combination keyboard stroke and mouse action.
In the example of
The scribe labeler does the same process and uses the same tools to highlight the span of text “three days”, assign a label of medical entity “SymAttr” and attribute “duration” (“Symattr/duration”) to the highlighted span of text “three days” and this additional annotation shows up in the Table of annotations 410.
The scribe labeler then proceeds to highlight the span of text “upper left”, 412. The scribe labeler again uses the tool 402 to ascribe a label to the span of text “upper left.” Again this could be done using the tools described in
It will be appreciated that the search tool 402 could pop up when the scribe labeler is taking action to highlight a span of text, and disappear after the label has been assigned, or alternatively it could be a persistent feature of the user interface during annotating.
As noted previously, the user interface of
The search tool 402 of
The text grouping as shown in
The pre-labelling system 110 of
Additionally, the NER can be trained from annotated medical encounter transcripts. A NER model can also be trained from a hybrid of data sources, which may include medical and clinical text books, annotated transcripts from doctor-patient conversations, and clinical documentation contained in anonymized electronic health records of a multitude of patients. The NER model may further be trained from feedback of the annotation of the transcript as performed in
As shown in
The system and method of this disclosure has several advantages. In many natural language processing text annotation tools, relationships between must be identified in an explicit and cumbersome manner. In contrast, in this disclosure the labels (including predefined labels relevant to the annotation task) and labelling and groupings tools permit such relationships to be readily specified. The user can quickly search for labels by means of the search tools as shown in the Figures and select them with simple user interface action such as a click of a mouse. Moreover, groupings of conceptually or causally related highlighted spans of text can be created very quickly with simple user interface actions using a keyboard, mouse, or combination thereof, as explained above.
While the illustrated embodiment has described an interface and tools for assisting in labeling transcripts of medical encounters, the principles of this disclosure could be applied to other situations. In particular, a predefined list of labels is generated for entities and attributes of those entities, e.g., listing all the possible categories or classes of words of interest in a transcript and attributes associated with each of the categories or classes, analogous to the attributes of medical entities. The user interface actions described above would generally be performed in the same way, that is the scribe labeler would read the transcript and highlight words or other spans of text that are relevant to the annotation task, using simple user interface tools, and then tools would be enabled by which the scribe labeler could search through the available labels and assign them to the highlighted spans of text.
Additionally, grouping tools are provided to form groups of related highlighted spans of text. The result is an annotated transcript. The methods could have usefulness in other types of transcripts, such as deposition or trial transcripts in the context of the legal profession, hearing transcripts of testimony of governmental bodies, etc.
An example of a list of labels for use in annotation of medical transcripts is set forth below in Table 1. It will be understood of course that variation from the list is possible and that in other contexts other labels will be defined. In the list, Entity 1 is a medical entity and Entity 2 is either a subcategory of the medical entity of Entity 1 or an attribute of the medical entity, and Entity 3 is either an attribute of the medical entity or a further subcategory of the medical entity of Entity 1 in a hierarchical schema.
This application is a continuation of U.S. patent application Ser. No. 16/612,879, filed Nov. 12, 2019, which is a national stage application under 35 U.S.C. § 371 of International Application No. PCT/US2017/057640, filed Oct. 20, 2017, each of which are incorporated herein by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16612879 | Nov 2019 | US |
| Child | 17978790 | US |
