Patent Application
20240127146
None.
The present disclosure is directed to systems and methods for optimizing translation management systems including the use of data analytics and machine learning.
It should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Past translation management systems receive from a client a document for translation, translates the document, and provide the translated document back to the client. A translation project manager makes decisions on the document content domain to select a translator having experience working on this type of a document and the requisite performance quality and speed. Further, given the client criteria and budget and potentially the requirement for special processing (security), the project manager will try to select a translator/reviewer to meet the client's quality, cost requirements, and special needs. However, a project manager may have tens of thousands of translators from which to choose, potentially located around the world. A document manager may have personal knowledge of a small number of translators and reviews and their history. Thus, it is impossible for a manager to efficiently pick from what can be tens of thousands of translators. Prior document management systems have not taken advantage of knowledge retained by the translation management system to pick the best translator and recommend services needed for efficient translation.
Another challenge facing translation management systems, is working with documents that have PII (Personal Identity Information) or other sensitive information such as credit card numbers. A project manager may not know that a document contains sensitive or PII information given the time needed to review a document for such information. The document may require additional processing steps (redaction) or be processed in a secure environment. Not knowing these issues often disrupts the translation process, effect the cost, who can work on the project, and the time required for project completion.
What is needed are systems and methods for automatically ranking translators for a translation job and recommending services that the job might require.
According to various embodiments, the present technology is directed to a computer implemented translation decision assistant. The method begins with receiving a document for translation(s) or otherwise referred to as a translation job. Also received is the limitation for cost, time, and quality of the translation job. The document is sent to a trained Translation Decision Assistant (TDA) system. The TDA, determines a value for the document based on business metrics and cost, quality and schedule indicated by the customer. The document is then analyzed to determine insights from the document. These insights can include the language, complexity, statistics, whether the document contains named entities including personal identifiable information, and the domain of the document. These insights are evaluated against prior document translation jobs using three techniques to generate ranked resources. The evaluation can include a similarity evaluation, a domain matching evaluation, and a statistical matching evaluation. These evaluations result in multiple groups of ranked resources. These groups are aggregated together based on a voting algorithm or can used a machine learning algorithm trained on historical human choices and outcomes. The resources are fed into a constraint filter utilizing hard and soft constraints. These constraints can include certification, languages, permissions quality, availability, cost, and service level of the translators. The constraint filter provides three outputs with resources ranked by cost, time, and quality. The decision filter allocates among these ranked resources based on the value of the document and the original customer choice to prioritize cost, quality or turnaround time.
The accompanying drawings, where like reference numerals refer to identical or functionally similar elements throughout the separate views, together with the detailed description below, are incorporated in and form part of the specification, and serve to further illustrate embodiments of concepts that include the claimed disclosure and explain various principles and advantages of those embodiments.
The methods and systems disclosed herein have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
The following detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show illustrations in accordance with example embodiments of the Translation Decision Assistant (TDA). These example embodiments, which are also referred to herein as “examples,” are described in enough detail to enable those skilled in the art to practice the present subject matter. The embodiments can be combined, other embodiments can be utilized, or structural, logical and electrical changes can be made without departing from the scope of what is claimed. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope is defined by the appended claims and their equivalents.
One or more embodiments of the present disclosure include methods of and systems for assisting in decisions for translation tasks based on the document to be translated, information about resources to be used, constraints, and other limitations. These tasks include selection of translator(s) and/or reviewer(s), a machine translation engine, a terminology database, choosing a service level, prioritizing quality/cost/turnaround time, and choosing a processing path or flow. Based on generated insights regarding a document are input into up to three methods of ranking translators. These include a machine learning “Similarity” evaluation, a “Domain Matching” evaluation, and a “Statistical Matching” evaluation of the document insights. The output of the Similarity, Domain, and Statistical evaluation are input into an Aggregation learning machine. The Aggregation learning machine outputs ranked translators, computer supplied translation methods, and workflow steps. These ranked outputs are inputs to a “Constraint Solver” process. The aggregation learning machine can be implemented as a neural network.
The constraint solver utilizes soft and hard constraints to provide the options of ranking by cost, quality, and time. The hard constraints include translator certification, languages, and translator permissions. The soft constraints include a translator's history of general quality determined from the document insights step, availability, costs, and service levels.
The statistics and complexity can include but are not limited to lengths of words and sentences, types of sentences used, mix of word, verbs. The “named entities” can include personal information, phone number, account information, or classification that may need redaction or to be translated by a person with special clearances, access, and vetting. The domains refer to the subject area of the document. Examples of these domains are medical, science, technical, and business documents.
Identified service attributes can include whether a document needs to be processed in a special environment, and whether redactions are needed to remove personal or sensitive information.
Referring to
In step 120, a content value is quantified. The quantification can be from the customer or from business rules and usage. The term “value”, also referred to as “job value”, describes how impactful the content is likely to be to the customer's business. Value is determined based on the document content and the customer's business, but the concept of ‘Value’ is important as it will determine the needed output and outcome, the job service level required, and accordingly the cost of the job. The value is different from the job quality which can be determined by measuring the number of errors and error severity.
Value can be measured in a number of ways and can be part of setting up of a system implementing the method disclosed herein. Determination of the “value” can include one or more of the business performance of the subject, the type, the risk, the longevity, and the visibility of the content.
The “business performance of the subject” is the related subject of the content for translation. An example of the business performance of the subject is a new product release and the market to which the product is to be released which relates to the amount of business revenue that will be generated from the product release. Release into a small market will have a smaller value than a release into a larger market.
The “type” is how the translation content is to be used. A marketing banner will have higher Value than a knowledge base item such as an entry into a database. For example, a translation of the intent and emotional connection of a marketing message vs accuracy of a knowledge base item would have different values.
Acceptable Risk is the tolerance to poor content performance. Low tolerance would indicate a higher Value. Content covered by regulation carries a higher impact of translation content failure. Thus, regulations are an example of a high risk, low tolerance, and thus would contribute to high value and subsequent service selection.
Longevity of the content relates to how long the content will be relevant. A social media post or internal CEO update will have a shorter life than a knowledge base item.
The “visibility” of the content relates to how broadly the content is expected to be distributed and seen. A website homepage will have more visibility than content stored in a knowledge database.
How these variables are combined, weighted, and mixed to generate a “value” measure can be implemented through a template for many translation jobs. Alternatively, a custom equation can be used. Universally, the “value” measure for a piece of content will be calculated, and that output from this function in the invention will be incorporated into the determination of the services which should be applied for translation.
In step 130, document insights are generated by the CI. The insights can include but are not limited to language(s), named entities, domain, complexity, and personally identifiable information (PII). The PII and named entities are information that may need to be redacted from the document or require special handling. PII information can include but is not limited to names, emails, credit card numbers, and phone numbers. The domain is the document's characteristics including but not limited to medical, business, and scientific documents.
In step 140, the generated insights are sent or provided to the evaluator models. There can be up to three evaluator models but the invention contemplates more than three models.
Referring to
In step 210, a trained Similarity Evaluation Machine (SEV) performs an evaluation of content insights. The SEV is a learning machine that has been trained with job history data and content insights. The job history data can include the translator and their timeline, and job scoring. Job scoring can include automated, internally generated, and customer generate information regarding quality, cost, and time. The SEV utilizes dynamic clustering of jobs using content insights and the content value and job history data of the translators and machine translation engines. The clustering method can be modeling of topics using n-grams forming a non-exclusive grouping formed by determining representing labels formed in advance of the clustering of jobs. Alternatively, the clustering method can utilize a bisecting k-mean algorithm resulting in an exclusive association with a cluster. Upon forming a cluster, metrics on the grouped jobs where the document/job under evaluation is evaluated to generate an assessment of translators, machine translation engines, translation memories, and terminology. Additionally, a schedule and quality estimation can be generated. The assessment can be ranked, include a schedule, and used to identify resources with related schedule and quality estimations.
In step 220, a “Domain Matching” evaluation is performed on the content insights. For domain matching, a set of prespecified domains are selected according to one or more taxonomies. Historical job content is classified according to one of the prespecified domains according to the taxonomies. The document under evaluation, the current job, is matched to one historical content domains. The match can be based on proportional domain certainty matching. From this match, metrics and resources can be extracted from historical jobs with the same domain matching. These include translators, timelines, and job scoring as described above. From this a list of resources, translators, machine translation engines, translation memories, terminology, and quality estimations are generated. This can be a ranked list.
In step 230, a “Statistical Matching” evaluation is performed on the content insights. This evaluation applies dynamic clustering of historical jobs based on the content characteristics/insights of the historical jobs and the value evaluations and specific terminology. For clustering, the method applies a statistical weighting algorithm finding the group of historical jobs having most commonalities with the document or job under evaluation. After identifying the group, metrics on the job are collected from this group. From these metrics, a list of resources, translators, machine translation engines, translation memories, terminology, and quality estimations are generated.
In step 240, the results of the evaluations are input into an aggregation process which is a trained learning machine which can include a neural network. In one embodiment, a score is generated for each evaluation and the outputs for each evaluation are ranked. In another embodiment, the ranking algorithm gives more weight to the evaluation that has the most commonality between evaluation results. In one embodiment, the aggregation process identifies common resources (translators, machine translation engines, translation memories, terminology) and a determination is made if the common resources can form another proposal. The output of this step is an ordered list of resources for the translation job including translators, machine translation engines, translation memories, terminology sets with a related schedule and quality estimation.
In step 250, a constraint filter is applied to ranked resources identified in the aggregation step 240. Where the step 240 may have identified a ranking of translators that would best meet the constraints of cost, time, and quality, these resources may not be available at the needed time. The constraint filter applies external constraints to the identified resources.
Two types of constraints may be applied to the external resources identified by the aggregation step 240. These are hard constraints and soft constraints. The hard constraints include certifications, languages, and permissions of translators. The soft constraints include quality, availability, cost, and service levels. The constraint filter can output three ranked resource outputs. One output can be ranked based on cost, one based on quality, and one based on time. These outputs can be sent to a decision filtering step 260.
The Translation Management System (TMS) implements a security model which provides configurable models in which one can define who has access and what kind of resources or translation projects to which a translator has access. The permissions can apply to both people and to informational resources.
For example, a translator, even if it would be best deemed a good match for a job by previous steps 210, 220, 230, and 240, the translator might not be eligible to work or participate on the job by the TMS security configuration. An example of a reason preventing a translator to participate on a job is that the translation project is being run in a certain area of a company where only specific people are allowed to work on the job. Another example is that the project has a security level for which a translator does not have the required clearances.
The same permissions restrictions can apply to resources. For example, a translation memory or terminology or a machine translation engine could be deemed usable for the job by previous analysis but is not accessible based on a security constraint. These constraints or restrictions can be that a resource is stored in an electronic location with restricted access while the translation job could be triggered as part of another location in the company which does not have visibility into the location where that resource exists.
A benefit of not applying security constraints before the earlier steps 210-240 is that using the information and machine processing can determine potentially useful resources to solve a job, even if they are under security restrictions. While the translation job may be subject to those restrictions which can be enforced, the information provided by applying the analysis steps of 210-240 to the resources that are security constrained can provide important information related to system governance and improved perspective on situations where the system can be improved. Potentially the security constraints are too rigid and need to be reviewed, and thereby unlocking more business benefits.
In step 260, decision filtering is applied to the output of the constraint filtering. The decision filtering can be based on estimated content value and the customer's input of the importance of price, schedule, and quality. This step can be automated and based on the content value metric.
In step 270, the ranked resources results may be optionally presented to a user. The results can be a report, email, displayed on a video screen or by other manner of communication.
In step 280, the ranked resource results can be optionally modified by human intervention. These modifications can be sent to the historical job database to be used to update the learning machines.
Alternatively, the ranked resources may be used directly by the TMS to assign and route to the best matched resources for the task based on customer preference for cost, quality or turnaround time.
Referring to
Translation memory resources are part of a system which records previous translations. An example of a previous translation is a translation of English to French. These translations or which can be referred to as “memories” can be reused in future translations either in their entirety if the translation is found in the same context or partially reused if the text resembles the original one to some degree.
It's a common problem in the translation industry that such memories can encompass tens of thousands or more of memories. The process of identifying a memory that would be useful in a translation can be cumbersome or problematic. By using the below disclosed methods of “Similarity” machine learning/AI, “Domain Learning” and “Statistical Matching”, the disclosed invention provides an improved method to utilize “translation memories”.
The TDA is comprised of a number of components. These include a value computation component 320, a content insights component 325, an evaluation component 330, a job history database 340, and aggregation component 350, a constraint solver 360, a resources database 370, and decision filter 380. The valuation software component can include a Similarity 332 software component, a domain matching evaluator software component 334, and a statistical matching 336 component. The operation of each of these components is described above within the method description of
The components provided in the computer system 400 of
The processing for various embodiments may be implemented in software that is cloud-based. The computer system 400 may be implemented as a cloud-based computing environment. In other embodiments, the computer system 400 may itself include a cloud-based computing environment. Thus, the computer system 400, when configured as a computing cloud, may include pluralities of computing devices in various forms, as will be described in greater detail below.
In general, a cloud-based computing environment is a resource that typically combines the computational power of a large grouping of processors (such as within web servers) and/or that combines the storage capacity of a large grouping of computer memories or storage devices.
The cloud may be formed, for example, by a network of web servers that comprise a plurality of computing devices, such as the computer system 400, with each server (or at least a plurality thereof) providing processor and/or storage resources. These servers may manage workloads provided by multiple users (e.g., cloud resource customers or other users).
The example computer system 400 includes a processor or multiple processors 405 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), Digital Signal Processor, Neural Processor Unit (NPU) or any combination thereof), and a main memory 410 and static memory 415, which communicate with each other via a bus 420. The computer system 400 may further include a video display 435 (e.g., a liquid crystal display (LCD)). The computer system 400 may also include an alpha-numeric input device(s) 430 (e.g., a keyboard), a cursor control device (e.g., a mouse), a voice recognition or biometric verification unit (not shown), a drive unit 437 (also referred to as disk drive unit), a signal generation device 440 (e.g., a speaker), and a network interface device 445. The computer system 400 may further include a data encryption module (not shown) to encrypt data.
The drive unit 437 includes a computer or machine-readable medium 550 on which is stored one or more sets of instructions and data structures (e.g., instructions 555) embodying or utilizing any one or more of the methodologies or functions described herein. The instructions 455 may also reside, completely or at least partially, within the main memory 410 and/or within static memory 415 and/or within the processors 405 during execution thereof by the computer system 400. The main memory 410, static memory 415, and the processors 405 may also constitute machine-readable media.
The instructions 455 may further be transmitted or received over a network via the network interface device 445 utilizing any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP)). While the machine-readable medium 450 is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the machine and that causes the machine to perform any one or more of the methodologies of the present application, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such a set of instructions. The term “computer-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals. Such media may also include, without limitation, hard disks, floppy disks, flash memory cards, digital video disks, random access memory (RAM), read only memory (ROM), and the like. The example embodiments described herein may be implemented in an operating environment comprising software installed on a computer, in hardware, or in a combination of software and hardware.
Not all components of the computer system 400 are required and thus portions of the computer system 400 can be removed if not needed, such as Input/Output (I/O) devices (e.g., input device(s) 430). One skilled in the art will recognize that the Internet service may be configured to provide Internet access to one or more computing devices that are coupled to the Internet service, and that the computing devices may include one or more processors, buses, memory devices, display devices, input/output devices, and the like. Furthermore, those skilled in the art may appreciate that the Internet service may be coupled to one or more databases, repositories, servers, and the like, which may be utilized in order to implement any of the embodiments of the disclosure as described herein.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present technology has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the present technology in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present technology. Exemplary embodiments were chosen and described in order to best explain the principles of the present technology and its practical application, and to enable others of ordinary skill in the art to understand the present technology for various embodiments with various modifications as are suited to the particular use contemplated.
Aspects of the present technology are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present technology. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present technology. In this regard, each block in the flowchart or block diagrams may represent a module, section, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular embodiments, procedures, techniques, etc. in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details.
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” or “according to one embodiment” (or other phrases having similar import) at various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Furthermore, depending on the context of discussion herein, a singular term may include its plural forms and a plural term may include its singular form. Similarly, a hyphenated term (e.g., “on-demand”) may be occasionally interchangeably used with its non-hyphenated version (e.g., “on demand”), a capitalized entry (e.g., “Software”) may be interchangeably used with its non-capitalized version (e.g., “software”), a plural term may be indicated with or without an apostrophe (e.g., PE's or PEs), and an italicized term (e.g., “N+1”) may be interchangeably used with its non-italicized version (e.g., “N+1”). Such occasional interchangeable uses shall not be considered inconsistent with each other.
Also, some embodiments may be described in terms of “means for” performing a task or set of tasks. It will be understood that a “means for” may be expressed herein in terms of a structure, such as a processor, a memory, an I/O device such as a camera, or combinations thereof. Alternatively, the “means for” may include an algorithm that is descriptive of a function or method step, while in yet other embodiments the “means for” is expressed in terms of a mathematical formula, prose, or as a flow chart or signal diagram.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is noted that the terms “coupled,” “connected”, “connecting,” “electrically connected,” etc., are used interchangeably herein to generally refer to the condition of being electrically/electronically connected. Similarly, a first entity is considered to be in “communication” with a second entity (or entities) when the first entity electrically sends and/or receives (whether through wireline or wireless means) information signals (whether containing data information or non-data/control information) to the second entity regardless of the type (analog or digital) of those signals. It is further noted that various figures (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale.
If any disclosures are incorporated herein by reference and such incorporated disclosures conflict in part and/or in whole with the present disclosure, then to the extent of conflict, and/or broader disclosure, and/or broader definition of terms, the present disclosure controls. If such incorporated disclosures conflict in part and/or in whole with one another, then to the extent of conflict, the later-dated disclosure controls.
While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. The descriptions are not intended to limit the scope of the invention to the particular forms set forth herein. To the contrary, the present descriptions are intended to cover such alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims and otherwise appreciated by one of ordinary skill in the art. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments.
