Patent Application
20090240539
Labor markets facilitate an efficient division of labor to perform various projects. Typically, a person who requires a project to be performed searches for and hires qualified persons to perform the project. However, the granularity at which a project may be practically divided into tasks that can then be performed by different persons (or more generally entities) has been relatively coarse. For example, the building of a house can be divided into coarse tasks such as building the foundation, framing the house, installing the roof, and so on. The division of a project into more fine-grained tasks has been limited by a variety of factors such as management overhead, skills availability, difficulty of efficiently matching buyers and sellers, issues surrounding confidentiality and trust, and so on. The limits may be especially problematic for knowledge workers (e.g., people who generate electronic documents such as scholarly articles, professional drawings, patent applications, and presentations). These knowledge workers, who are typically highly specialized, often need tasks to be performed that are outside their area of expertise. For example, a physics professor in China who is writing a scholarly article in English on the formation of black holes may not be particularly knowledgeable about English grammatical rules. To ensure that the article is free of grammatical errors, the professor needs a skilled editor to review the article. Similarly, the professor may not be familiar with drawing tools needed to make the figures of the article look professional. Unless the professor's university happens to have a skilled editor for the English language or a skilled draftsperson on staff in the physics department, it can be difficult for the professor to find the right persons to perform those tasks.
Some systems are available to help knowledge workers manage tasks. For example, a project management tool can help a manager in a company track a complex project such as generating a request for proposal or preparing a response to a request for proposal. If the knowledge workers to whom tasks are assigned are employees of the company, it can be fairly easy for the manager to assign the tasks of the project. It, however, becomes more difficult if the tasks need to be assigned to people outside the company. As another example, a workflow tool may allow a manager to specify the workflow sequence for a document. The workflow may specify that a certain junior writer is to generate the first draft, a certain senior writer is to revise and approve it, an editor is to review and edit it, a draftsperson is to generate professional drawings for it, a layout person is to format it, and so on. These tools, however, do not provide much assistance in helping a manager or knowledge worker identify who can perform a task (especially when the skill to perform the task is not readily available), how to describe the needed task, how much to pay for the task, what tasks are really needed, and so on.
A person who performs a task on many documents for many different people may use productivity tools to improve their efficiency and effectiveness. For example, a person who performs language translation (e.g., Japanese to English) may use an automated translator to generate an initial translation of a document. The translator might then manually review the translated document to correct any translation errors. Each translator may be able to improve the effectiveness of the automated translator by customizing the translation model (e.g., by adding mappings from kanji characters to possible English words and adding words to the translation dictionary). As another example, a person who performs speech-to-text translation may use an automated translator to generate an initial translation of a document and then edit the translation. The translator may train the automated translator by highlighting mistranslations and providing the correct translations (e.g., “once killed in the art” corrected to “one skilled in the art”). A difficulty with such an approach is that each person who performs a task can improve their effectiveness based only on their own experiences. In addition, a person who wants to start performing a task would need to start from a generic productivity tool and customize it over time based on their own experiences. Thus, people who currently use the productivity tool cannot effectively benefit from the experiences of others, and new people who want to start performing a task may be at such a competitive disadvantage that they simply decide not to compete.
A machine learning system is provided that uses machine learning techniques to learn models used by productivity tools based on experiences of providers who perform tasks on electronic documents for customers. In some embodiments, the machine learning system works in conjunction with a task brokerage system provided by a broker that helps customers who need tasks to be performed on documents to identify providers who can perform the requested tasks. The task brokerage system allows customers to publish their tasks and providers to discover the published tasks. The discovery process may match providers to customers based on criteria such as reputation, pricing, and availability.
The machine learning system learns models to assist providers in processing documents of customers. Providers may use various productivity tools that use the learned models to assist in performing tasks on target documents of customers. Each productivity tool may have a model, a mapping, a dictionary, and/or other list of parameters, generally referred to as a model, that can be customized to improve the effectiveness of the productivity tools. The machine learning system may initially train models based on demographic and other relevant information of customers and training data of the customers. The machine learning system may identify clusters of customers with similar demographic information using various clustering algorithms. The machine learning system may assume that customers with similar demographic information will likely benefit from similar models. To generate the models, the machine learning system collects the training data for the customers of each cluster and then trains a model for each cluster.
The machine learning system uses the models generated for the clusters to assist providers in performing tasks on target documents of customers. When a task is to be performed on a target document of a customer, the machine learning system identifies the cluster of that customer and uses a model for that cluster to assist in performing that task. A provider may then refine the result generated using the model to provide a more refined result. The machine learning system may adjust the model based on refinements to the result made by the provider to generate a refined result.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
A machine learning system is provided that uses machine learning techniques to learn models used by productivity tools based on experiences of providers who perform tasks on electronic documents for customers. In some embodiments, the machine learning system works in conjunction with a task brokerage system provided by a broker that helps customers who need tasks to be performed on documents to identify providers who can perform the requested tasks. The task brokerage system allows customers to publish their tasks and providers to discover the published tasks. The discovery process may match providers to customers based on criteria such as reputation, pricing, and availability.
A machine learning system is provided that learns models to assist providers in processing documents of customers. In some embodiments, the machine learning system works in conjunction with a task brokerage system as described in U.S. patent application Ser. No. 12/026,523. A task brokerage system provided by a broker (also referred to as a “microwork broker”) helps customers (also referred to as “microwork customers”) who need tasks (also referred to as “microtasks”) to be performed on documents to identify providers (also referred to as “microwork providers”) who can perform the requested tasks. The task brokerage system allows customers to publish their tasks and providers to discover the published tasks. The task brokerage system may also maintain reputations of the customers and providers who are “participants” in the brokering of tasks. The reputations may be derived from customer ratings of providers and provider ratings of customers. The discovery process may match providers to customers based on criteria such as reputation, pricing, and availability. The task brokerage system may provide facilities by which a customer can help ensure that a provider will preserve the confidentiality of the customer's information.
In some embodiments, providers may use various productivity tools to assist in performing tasks on target documents of customers. For example, a productivity tool may be a speech-to-text translator, a language translator (e.g., French to English), a document layout generator, a grammar checker, a drawing translator, and so on. Each of these productivity tools may have a model, a mapping, a dictionary, and/or a list of parameters, generally referred to as a model, that can be customized to improve the effectiveness of the productivity tools. For example, a speech-to-text translator may have a model representing the speaker's voice, a document layout generator may have a mapping of input formats to desired target formats (e.g., a table without borders to a table with borders), and a grammar checker may have a syntactic model. The machine learning system may learn a model based on initial training data. Many productivity tools provide a training mode in which training data can be collected and then used to generate a model. For example, a speech recognition productivity tool may ask a person to read a document and then train a model based on the corresponding acoustics of the speech. Productivity tools may use a variety of well-known learning techniques such as those based on Hidden Markov models, support vector machines, Bayesian networks, k-nearest neighbor algorithms, genetic programming, Monte Carlo methods, adaptive boosting algorithms, belief networks, decision trees, and so on.
In some embodiments, the machine learning system may initially train models based on demographic information of customers and training data of the customers. The demographic information may include the characteristics of each customer as maintained by the task brokerage system plus additional characteristics that may be useful in generating a model. For example, the characteristics of a customer that are maintained by the task brokerage system may include gender, occupation, age, home address, spoken language, and so on. The characteristics of a customer that may be useful in generating a model may include regional accent, style of writing (e.g., fiction, technical, or press release), field of specialization (e.g., medical or mathematics), nationality, and so on. The machine learning system may identify clusters of customers with similar demographic information using various clustering algorithms such as k-means clustering, hierarchical clustering, and so on. The machine learning system may assume that customers with similar demographic information will likely benefit from similar models. For example, a speech-to-text model that is customized to customers who speak a particular dialect or have a particular accent will result in a more effective translation than a more generic model. To generate the models, the machine learning system collects the training data for the customers of each cluster and then trains a model for each cluster.
The machine learning system uses the models generated for the clusters to assist providers in performing tasks on target documents of customers. When a provider requests a task to be performed on a target document of a customer, the machine learning system identifies the cluster of that customer and uses a model for that cluster to assist in performing that task. If the customer is a new customer that has not been associated with a cluster, then the machine learning system identifies a cluster for that customer. The machine learning system may use a distance metric to compare the demographic information of the new customer with the mean demographic information of the customers of each cluster. The machine learning system then associates the new customer with the cluster with the closest distance. The machine learning system may also use a collaborative filtering technique to identify a cluster for the customer. In some embodiments, the machine learning system may use multiple models or combine models when a customer has demographic information that is similar to the customers of different clusters. The machine learning system may use different weights for the different models based on similarity between the demographic information of the customer and the demographic information of the customers of the cluster or based on confidence that a cluster is the correct cluster for the customer.
In some embodiments, the machine learning system may adjust a model based on changes to the output or result of the productivity tool made by a provider. For example, a provider may use a speech-to-text translator to generate a result, which is an initial translation of a target document for a customer. The provider may then use a word processing program to correct or refine the translation. The corrected translation is a refined result that has been modified by refinements. For example, the provider may indicate that certain text represents a mistranslation or misrecognition of the corresponding speech. The machine learning system may also identify the refinements by comparing the result generated by the productivity tool with the refined result generated by the provider to identify the differences, which may be considered to be refinements. The machine learning system uses the refinements to adjust (retrain or relearn) the model. The adjusting of the model of a cluster can be an incremental adjustment or a complete relearning based on the initial training data and the refinements. The machine learning system adjusts the model for the cluster associated with the customer. The machine learning system may also factor in the confidence it has in a customer belonging to a cluster to weight the adjustments to the model. Thus, if the machine learning system is very confident that a user belongs to a cluster, then the machine learning system may give full weight to the refinements when adjusting the model. If the machine learning system is, however, not very confident that the user belongs to a cluster, then the machine learning system may give only partial weight to the refinements when adjusting the model for that cluster. The machine learning system may also factor in modifications made by customers to the refined results provided by the providers. These modifications can be useful in assessing not only the quality of the tasks performed by the providers, but also to adjust the models. For example, if a customer modifies many of the changes made by a provider, then the machine learning system may not want to adjust the model based on the provider's refinements. Also, if the customer makes additional refinements to the result, then the machine learning system may adjust the model based on the refinements of both the provider and the customer.
The machine learning system may adjust a model based on implicit or explicit input from a participant. Implicit input includes the refinements to a document, options selected by a user (e.g., from alternative word choices), and so on. Explicit input, in contrast, is provided directly by a participant. For example, a participant may indicate that a translation that selected a first option was wrong and that the second option would have been correct. In such a case, the machine learning system may weight this explicit input more than if the input was made implicitly. The machine learning system may not know whether the implied input was to correct something that was really wrong or simply to make a stylistic change and thus does not know how heavily to weight the refinements.
The machine learning system may use a client/server architecture to make productivity tools and models available to providers to perform tasks for customers. The models can be adjusted based on the refinements provided by the providers. The machine learning system may, however, not physically publish the models to the providers, but rather only use the models internally at the server. Because the providers are allowed to use the models only via the server, a provider cannot take a model and customize it for their own needs. Rather, the machine learning system encourages providers to contribute refinements that are used to improve the model that is shared by all providers. Thus, each provider can benefit from the models that have been adjusted based on the refinements of other providers. To prevent corruption of a model intentionally by an unscrupulous provider or unintentionally by an unskilled provider, the machine learning system may implement a rating system to rate a provider's history of adjustments to a model. The machine learning system can then prevent adjustments to the model based on refinements from providers with low ratings and may even prevent those providers from participating in the task brokerage system.
In some embodiments, the task brokerage system helps encourage customers to use the task brokerage system to contact providers, rather than contract a provider directly. This encouragement may take different forms. As described above, the task brokerage system may provide a machine learning system that provides automated performing of tasks that is better than and cheaper than the automated performing that can be provided by an individual provider. Also, the task brokerage system may automatically identify tasks for a customer, publish the identified tasks, identify providers to perform the tasks, and/or assign the tasks to the identified providers. In general, participants will be encouraged to use the task brokerage system because they can take advantage of its services that are improved and customized in part from the experiences gained in the brokering of many tasks for many consumers and providers.
In some embodiments, the machine learning system may combine models when models tend to be similar or split a model when the demographics indicates separate models may be more effective. To combine models, the machine learning system may calculate the distance between a model and every other model. If the distance between a model and another model is less than a combine threshold distance, then the machine learning system combines the associated clusters and models. The combined model for the new cluster may be generated by collecting the training data and refinements for all the customers in the clusters for the models that are to be combined and training a new model based on the training data and refinements. The new model is associated with a new cluster that includes all the customers of the two old clusters. To split models, the machine learning system may generate two sub-clusters of the customers of the cluster for the model. The machine learning system may train a model for each sub-cluster based on the training data and refinements for the customers in that sub-cluster. If the distance between the two models is greater than a split threshold distance, then the machine learning system considers the sub-clusters to be two new clusters, each represented by the corresponding new model.
The task brokerage system 150 may include a participant registry 151, a published task store 152, a subscription store 153, a provider offer store 154, an assigned task store 155, and a history store 156. The participant registry may contain customer and provider profile information, which includes characteristics of the participants. The published task store contains an entry describing each task that has been published by a customer. The subscription store contains an entry for subscriptions of providers to published tasks. The subscription information can be used to notify providers when tasks are published that match the criterion of their subscription (e.g., using a publisher/subscriber model). The provider offer store contains an entry for each offer of a provider to perform a published task. The assigned task store contains a mapping of published tasks to the provider who the customer and the provider agree is to perform the task of the customer. The history store contains information describing the performance and other information about each transaction in which a provider performs a task for a customer. The task brokerage system may also include a workflow component that allows a customer to specify a sequence of tasks to be performed on the task and coordinates the performing of the tasks of the workflow.
The task brokerage system includes a machine learning system 160 that supports performing of tasks that are model-based. The machine learning system includes a generate models component 161, an apply model component 162, a combine models component 163, a split models component 164, an adjust models component 165, an identify refinements component 166, a training data store 167, and a refinement store 168. The generate models component identifies clusters of customers based on demographic information and generates an initial model for each cluster based on the training data of the customers within the cluster. The apply model component inputs a target document and an indication of a customer, identifies a cluster associated with that customer, and applies the model for that cluster to the target document to generate a result. The combine models component identifies models of clusters within a combine threshold distance and generates a combined model based on the training data and refinements of the customers of the clusters. The split models component identifies models that would more appropriately be split into two models and generates new models for the model being split. The adjust models component inputs refinements to results generated by a model and adjusts the model accordingly. The identify refinements component identifies refinements to results either by receiving refinements from providers and/or comparing results to refined results to identify differences. The training data store contains the training data of the customers. The refinement store contains the refinements made by providers to the results of target documents.
The computing device on which the machine learning system is implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives). The memory and storage devices are computer-readable media that may be encoded with computer-executable instructions that implement the machine learning system, which means a computer-readable medium that contains the instructions. In addition, the instructions, data structures, and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communication link. Various communication links may be used, such as the Internet, a local area network, a wide area network, a point-to-point dial-up connection, a cell phone network, and so on.
Embodiments of the machine learning system may be implemented in and used with various operating environments that include personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, digital cameras, network PCs, minicomputers, mainframe computers, computing environments that include any of the above systems or devices, and so on.
The machine learning system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. The data structures illustrated include logical representations of data. The actual organization of the data structures may include hash tables, indexes, trees, and so on.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. For example, the machine learning system may allow a provider to specify its customers (a group of customers) and have a model trained for those customers or models trained for clusters of those customers. Thus, the machine learning system may support provider-specific models or models that are specific to groups of providers (e.g., translators that are employees of a translation service company). The machine learning system may also provide recommendations for providers to a customer based on analysis of performance of the providers on target documents of customers with demographic information similar to that of the customer. Accordingly, the invention is not limited except as by the appended claims.
This application is related to U.S. patent application Ser. No. 12/026,523, entitled “Affordances Supporting Microwork on Documents,” and filed on Feb. 8, 2008, and U.S. patent application Ser. No. ______ (41826.8477US), entitled “Recommendation System for a Task Brokerage System,” and filed on Mar. 21, 2008, which are hereby incorporated by reference.
