Patent Application
20170091161
The present application generally relates to information technology, and, more particularly, to annotator management techniques.
Annotators are commonly implemented to extract and mark words and/or phrases from unstructured content. However, challenges arise in annotation techniques due, for example, to inaccurate entity annotators and/or inaccurate relationship annotators, which can arise with changing scenarios and/or outlier scenarios.
In one embodiment of the present invention, techniques for updating annotator collections using run traces are provided. An exemplary computer-implemented method can include steps of generating one or more alternate versions of one or more document annotators selected from a set of multiple document annotators; executing, on one or more document data sets, (i) one or more document annotators from the set of multiple document annotators and (ii) the one or more alternate versions to generate log information for each document annotator in the set and each alternate version of the one or more alternate versions; and outputting an instruction to modify, based on the generated log information for each document annotator in the set and each alternate version, at least one document annotator from the set with at least one alternate version from the one or more alternate versions.
In another embodiment of the invention, an exemplary computer-implemented method can include steps of generating one or more alternate versions of one or more document annotators selected from a set of multiple document annotators; executing, on one or more document data sets, (i) one or more document annotators from the set of multiple document annotators and (ii) the one or more alternate versions to generate log information for each document annotator in the set and each alternate version of the one or more alternate versions; analyzing the generated log information for each document annotator in the set and each alternate version to generate a score for each of the one or more alternate versions based on each of one or more relationships established between (i) each of the one or more alternate versions and (ii) one or more document annotators from the set of multiple document annotators; and outputting an instruction to modify, based on the score generated for each of the one or more alternate versions, at least one document annotator from the set with at least one alternate version from the one or more alternate versions.
Another embodiment of the invention or elements thereof can be implemented in the form of an article of manufacture tangibly embodying computer readable instructions which, when implemented, cause a computer to carry out a plurality of method steps, as described herein. Furthermore, another embodiment of the invention or elements thereof can be implemented in the form of an apparatus including a memory and at least one processor that is coupled to the memory and configured to perform noted method steps. Yet further, another embodiment of the invention or elements thereof can be implemented in the form of means for carrying out the method steps described herein, or elements thereof; the means can include hardware module(s) or a combination of hardware and software modules, wherein the software modules are stored in a tangible computer-readable storage medium (or multiple such media).
These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.
As described herein, one or more embodiments of the present invention includes updating annotator collections using run traces. As detailed herein, annotators are often built once and used multiple times, with varying degrees of accuracy. Accordingly, at least one embodiment of the invention includes leveraging information derived from annotator traces on data sets to improve the accuracy of the annotators. Leveraging traces of annotators to generate recommendations for updating annotators can include, as further detailed herein, generalizing patterns in entity annotators as well as generalizing patterns in relationship annotators.
As used herein, an entity pattern can, for example, be in the form of a regular expression that can match occurrences of one or more desired entities. As an example, a phone number entity pattern could look like “1-[0-9]{3}-[0-9]{3}-[0-9]{4},” which indicates that a phone number is anything that starts with a “1” and has a hyphen followed by 3 numeric digits, followed by hyphen, followed by 3 numeric digits, followed by a hyphen, followed by 4 numeric digits. A relationship pattern can typically join two entity patterns by a relationship. By way of illustration, an example “NamePhone” relationship pattern can include the following: “<Name> is available at <Phone>,” wherein <Name> and <Phone> denote matches of Name and Phone patterns. This Name-Phone annotator can be thought of as connecting a person's name with his/her phone number.
One or more embodiments of the invention include generating and/or providing a system for recommending and/or outputting changes to one or more annotators determined by leveraging execution traces on a document data set (that is, annotator traces). Such an embodiment can include implementing an enhanced annotator logger component that uses pattern generalization to expand the set of annotators that are subsequently run to generate log information. Additionally, such an embodiment can include implementing a recommendation engine that analyzes log information generated by the enhanced annotator logger component to identify generalizations to improve annotator coverage.
As detailed herein, the enhanced annotator logger component 106 computes generalizations for each entity pattern and relationship pattern from among the annotators in the DB 104. By way of example, Phone.p1 (wherein “Phone” is the name of the annotator and “p1” is the particular pattern used by the annotator)=“[0-9\-( )]{8+}” could be generalized to:
p1.gen1=“[0-9\-( )]{7+}” (generalized to include shorter patterns); and/or
p1.gen2=“[0-9\-( )A-Z]{8+}” (generalized to include alphabets).
The generalizations can be computed via one of multiple ways. For example, whenever there is an element in a pattern that matches one of the nodes in a given taxonomy, a generalization is to replace the element by the pattern at its parent node. By way of additional example, whenever there is an element referred to as a quantifier (that is, a specification of the number of times the preceding sub-pattern in the pattern should occur), the element can be generalized to reduce or increase the number therein. Additionally, the set of generalizations available for a pattern can include the set of generalizations available through either of the above-noted options/examples.
By way of additional example, a relationship pattern “[Name] is available at [Phone]” could be generalized to “[Name] (token)? is available at [Phone].” As used herein, a “(token)” represents any word (commonly also including white-space on either side), and “(token)?” refers to a short-hand to represent 0 or 1 occurrence of “(token).” Additionally, such generalizations as noted above may be performed using a taxonomy.
The enhanced annotator logger component 106 also runs the annotators (obtained from DB 104) and their respective generalizations (computed by the enhanced annotator logger component 106), and logs the firings of the entity annotators and relationship annotators. As used herein, “firings” of an annotator refer to “matches” of a pattern used by the annotator. For relationship annotators, in addition to the actual entities, generalized pattern statistics can also be recorded. For example, in addition to recording that “NamePhone” did not fire with Phone.p1, the enhanced annotator logger component 106 can also record whether “NamePhone” fired with Phone.p1.gen1. The additional pattern statistics can aid in the assessment of the benefit of generalizations; in particular, such statistics can be used by the recommendation engine 110 to generate recommendations.
Log information output by the enhanced annotator logger component 106 can include, for example, information such as the following:
Entity E1 fired on Doc1 (Document 1), Offset: 4
Entity E2 fired on Doc1, Offset: 7
Relationship R1 failed on (E1, E2) for Doc1.R1
Reason: Found “XY” while looking for “XZ|XYZ.”
As also depicted in
generalizing E1.p1 to E1.p1.gen1 would result in 20% more hits for R1 and 10% more hits for R2 (wherein “E1” is a first entity, “p1” is a first pattern, “gen1” is a first generalization, and “R1” and “R2” are a first relationship annotator and a second relationship annotator, respectively); and
generalizing R3.p1 to R3.p1.gen2 would result in 25% more hits for R3 (wherein “R3” is a third relationship annotator, and “gen2” is a second generalization).
Accordingly, the generalizations of the above-noted annotators' expression would enable the annotators to increase their respective firings as indicated above.
In at least one embodiment of the invention, for every relationship annotator, Ri, the enhanced annotator logger component 106 runs the relationship annotator (that is, the relationship annotator is applied to a set of documents) and outputs the results. The results can include patterns, and as used herein, Ri.patterns=a set of patterns associated with Ri and associated with each entity that Ri uses. For every pattern p in Ri.patterns, and for every generalization p.g of p, the enhanced annotator logger component 106 can run Ri by substituting p.g for p, and collect statistics (such as a percentage increase in firings, for example,) and output log information.
Additionally, in at least one embodiment of the invention, input for the recommendation engine 110 can include a generalization threshold, g (for example, 10%). In one or more embodiments of the invention, a generalization threshold can be a configurable system parameter. Also, for every relationship annotator Ri, the recommendation engine 110 can collect the number of firings as Ri.f. For every generalization variant of Ri (Ri.j)that resulted in the number firings being greater than (the number of firings of Ri.f*(100+g)/100), the recommendation engine 110 adds the pattern generalization (that is, delta(Ri, Ri.j)) to the results. Further, the recommendation engine 110 additionally collects all results, groups the results by the annotator to which the generalization belongs, and outputs the results in the form of one or more suggestions/recommendations 112.
Accordingly, by way of illustration, an example embodiment of the invention can be carried out via system architecture such as depicted in
Based on this input, the example embodiment of the invention can include outputting a list of possible generalizations of E, with scores (scored using information from R) corresponding to each of the generalizations.
More specifically, at least one example embodiment of the invention can include identifying candidate generalizations for E as (G1, G2, . . . Gk) (computed as detailed above, for example). Such an example embodiment can also include implementing a component (such as the enhanced annotator logger component 106 depicted in
At least one embodiment of the invention additionally includes implementing a component (such as the recommendation engine 110 depicted in
While the above example embodiment describes only one entity annotator being generalized for purposes of illustration, it is to be appreciated that one or more embodiments of the invention can incorporate multiple entity annotators being generalized. For example, an example embodiment of the invention can include incorporating a combination such as [E1*, E2*, R], wherein E1 and E2 are distinct entity annotators, each separately being generalized.
Another example embodiment of the invention can include incorporating a combination such as [E*, R1, R2], wherein feedback from two relationship annotators is used to generalize E. Such an embodiment requires the recommendation engine 110 to merge evidences from each relationship annotator (that is, from R1 and from R2). An additional example embodiment of the invention can include incorporating a combination such as [E1, E2, R*], wherein the relationship annotator (R) is generalized using feedback from the each of the two entity annotators (that is, E1 and E2). Yet another example embodiment of the invention can include incorporating a combination such as [E, R1, R2*], wherein feedback from one entity annotator (E) and a first relationship annotator (R1) is used to generalize a second relationship annotator (R2). In one or more embodiments of the invention, all such possible combinations (such as [E, R1, R2*], [E1, E2, R*], etc.) can be explored. Alternatively, at least one embodiment of the invention can include pre-defining a set of possible combinations such that only those combinations are explored.
At least one embodiment of the invention (such as the example embodiment depicted in
In another example, at least one embodiment of the invention can be implemented within the context of deriving knowledge from medical records. In such a context, an example embodiment of the invention can be implemented in connection with text documents from medical transcriptionists and other textual medical reports. Such documents can include relations such as, for example, “The patient was diagnosed with X and prescribed drug Y because Z was found to be ineffective.” In such an example scenario, an example embodiment of the invention can be implemented wherein the annotators in question can include “Disease,” Drug,” “DrugFoundIneffectiveforDisease” and “DrugPrescribedforDisease.” Note that the last two are relationship annotators linking matches of drugs and diseases. Such information identified across a multitude of medical reports can, for example, provide cues for a drug research company that is trying to prioritize directions for drug discovery. In accordance with the descriptions herein, an example embodiment of the invention can be utilized to improve and tune the annotators used so as to increase performance in identifying relations and entities.
Step 204 includes executing, on one or more document data sets, (i) one or more document annotators from the set of multiple document annotators and (ii) the one or more alternate versions to generate log information for each document annotator in the set and each alternate version of the one or more alternate versions, wherein said executing is carried out by the enhanced annotator component executing on the at least one computing device and communicatively linked to a source for the one or more document data sets. Additionally, in at least one embodiment of the invention, each of the one or more alternate versions of one or more document annotators can include a generalization of one of the one or more document annotators.
In at least one embodiment of the invention, the executing step can include executing (i) one or more document annotators, from the set of multiple document annotators, that identify one or more entities, and (ii) one or more alternate versions of one or more document annotators, from a set of multiple document annotators, that identify one or more relationships between two or more entities. Alternatively, in at least one embodiment of the invention, the executing step can include executing (i) one or more document annotators, from the set of multiple document annotators, that identify one or more relationships between two or more entities, and (ii) one or more alternate versions of one or more document annotators, from a set of multiple document annotators, that identify one or more entities.
Step 206 includes outputting an instruction to modify, based on the generated log information for each document annotator in the set and each alternate version, at least one document annotator from the set with at least one alternate version from the one or more alternate versions, wherein said outputting is carried out by a recommendation engine executing on the at least one computing device and communicatively linked to the enhanced annotator component.
The techniques depicted in
Also, an additional embodiment of the invention includes generating one or more alternate versions of one or more document annotators selected from a set of multiple document annotators, wherein said generating is carried out by an enhanced annotator component executing on at least one computing device and communicatively linked to a database storing the set of multiple document annotators. Such an embodiment also includes executing, on one or more document data sets, (i) one or more document annotators from the set of multiple document annotators and (ii) the one or more alternate versions to generate log information for each document annotator in the set and each alternate version of the one or more alternate versions, wherein said executing is carried out by the enhanced annotator component executing on the at least one computing device and communicatively linked to a source for the one or more document data sets. Further, such an embodiment includes analyzing the generated log information for each document annotator in the set and each alternate version to generate a score for each of the one or more alternate versions based on each of one or more relationships established between (i) each of the one or more alternate versions and (ii) one or more document annotators from the set of multiple document annotators, wherein said analyzing is carried out by a to recommendation engine executing on the at least one computing device and communicatively linked to the enhanced annotator component. Additionally, such an embodiment includes outputting an instruction to modify, based on the score generated for each of the one or more alternate versions, at least one document annotator from the set with at least one alternate version from the one or more alternate versions, wherein said outputting is carried out by the recommendation engine executing on the at least one computing device.
The techniques depicted in
Additionally, the techniques depicted in
An embodiment of the invention or elements thereof can be implemented in the form of an apparatus including a memory and at least one processor that is coupled to the memory and configured to perform exemplary method steps.
Additionally, an embodiment of the present invention can make use of software running on a computer or workstation. With reference to
Accordingly, computer software including instructions or code for performing the methodologies of the invention, as described herein, may be stored in associated memory devices (for example, ROM, fixed or removable memory) and, when ready to be utilized, loaded in part or in whole (for example, into RAM) and implemented by a CPU. Such software could include, but is not limited to, firmware, resident software, microcode, and the like.
A data processing system suitable for storing and/or executing program code will include at least one processor 302 coupled directly or indirectly to memory elements 304 through a system bus 310. The memory elements can include local memory employed during actual implementation of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during implementation.
Input/output or I/O devices (including, but not limited to, keyboards 308, displays 306, pointing devices, and the like) can be coupled to the system either directly (such as via bus 310) or through intervening I/O controllers (omitted for clarity).
Network adapters such as network interface 314 may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems and Ethernet cards are just a few of the currently available types of network adapters.
As used herein, including the claims, a “server” includes a physical data processing system (for example, system 312 as shown in
The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out embodiments of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform one or more embodiments of the present invention.
Embodiments of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
It should be noted that any of the methods described herein can include an additional step of providing a system comprising distinct software modules embodied on a computer readable storage medium; the modules can include, for example, any or all of the components detailed herein. The method steps can then be carried out using the distinct software modules and/or sub-modules of the system, as described above, executing on a hardware processor 302. Further, a computer program product can include a computer-readable storage medium with code adapted to be implemented to carry out at least one method step described herein, including the provision of the system with the distinct software modules.
In any case, it should be understood that the components illustrated herein may be implemented in various forms of hardware, software, or combinations thereof, for example, application specific integrated circuit(s) (ASICS), functional circuitry, an appropriately programmed digital computer with associated memory, and the like. Given the teachings of the invention provided herein, one of ordinary skill in the related art will be able to contemplate other implementations of the components of the invention.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of another feature, step, operation, element, component, and/or group thereof.
At least one embodiment of the present invention may provide a beneficial effect such as, for example, using annotator traces to generate recommendations to alter annotators.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
