METHOD, SYSTEM, AND COMPUTER-READABLE MEDIUM FOR DERIVING TASK RESULTS BY REFLECTING RELIABILITY OF WORKER PROCESSING WORK COLLECTED THROUGH CROWDSOURCING

TECHNICAL FIELD

The present invention relates to a method, a system, and a computer-readable medium for deriving task results by reflecting reliability information of a worker processing a work collected through crowdsourcing, and more particularly, to a method, a system, and a computer-readable medium for deriving task results by reflecting reliability information of a worker processing a work collected through crowdsourcing in which, upon a worker processes a work through crowdsourcing, reliability information (working or reviewing ability) for each worker is updated, and a task result for the corresponding work is derived based on the updated reliability information, so as to effectively infer the task result for the work.

BACKGROUND ART

Recently, as technologies related to artificial intelligence have been developed and various solutions using the artificial intelligence have been developed, interest in methods of collecting or building data for learning artificial intelligence is increasing. Since the artificial intelligence, especially deep learning-based artificial intelligence has the better performance when the amount of data for learning is greater and the quality of the data is higher, it is increasingly important to ensure high-quality data rather than simply ensuring data for learning.

In general, in the case of data for training artificial intelligence, labeled data is required, such as separately labeling vehicle regions in an image containing vehicles. Accordingly, in addition to simply collecting data, it is necessary to separately label the collected data through manual work and the like, and this requires a lot of resources, such as securing human power for performing labeling and time requirement for the labeling, in securing learning data.

Accordingly, methods for building data based on crowdsourcing have been proposed recently in order to efficiently ensure a large amount of labeled training data. According to the crowdsourcing, work such as data is provided to an unspecified number of workers, the workers perform a task such as labeling on the work, task results performed by the workers are reviewed by multiple reviewers, pieces of labeled data are finally established, and the workers having labeled the corresponding data are rewarded with respect to the data finally established through the review.

In addition, because quality of task results for the same work may vary depending on the ability of a worker, the role of the reviewer reviewing the task results is important in order to build quality labeled data. However, In the conventional methods for determining a task result based on a review result of a reviewer, there is a method of determining the task result by a majority vote based on the review result reviewed by a plurality of reviewers for one task result. However, it is assumed that each reviewer in the majority vote has the same review ability, the review result of a reviewer with low review ability and the review result of a reviewer with high review ability are reflected with the same weight to determine the task result, and accordingly, the reliability of determination on the task results is lowered.

In the conventional methods for solving the above problem in the majority vote, there is a method of determining the task result by using the review ability of each reviewer as a weight by calculating the review ability (reliability) based on the review results performed by multiple reviewers in the past. However, according to the method, since the review ability of the reviewer may be different depending on the type of a reviewed work, an appropriate weight cannot be applied when the type of work reviewed in the past is different from the type of work reviewed currently. In addition, even when the reviewer has no history of performing reviews in the past or the formation of a plurality of reviewers having performed reviews in the past is different from the formation of a plurality of reviewers performing current reviews, appropriate weights cannot be applied.

Therefore, the need for developing a new method, which effectively determines the task result according to review results of a plurality of reviewers reviewing task results on a work, is emerging.

DISCLOSURE
Technical Goal of the Invention

The present invention relates to a method, a system, and a computer-readable medium for deriving task results by reflecting reliability information of a worker processing a work collected through crowdsourcing, and more particularly, provides a method, a system, and a computer-readable medium for deriving task results by reflecting reliability information of a worker processing a work collected through crowdsourcing, in which upon a worker processes a work through crowdsourcing, reliability information (working or reviewing ability) for each worker is updated, and a task result for the corresponding work is derived based on the updated reliability information, so as to effectively infer the task result for the work.

Means for Resolving the Problem

In order to solve the above problems, one embodiment of the present invention provides a method of deriving task results by reflecting reliability information of workers who process works collected through crowdsourcing performed in a computing device having at least one memory and at least one processor, which includes the steps of: receiving task results of a plurality of workers for a work including a plurality of unit task; a task result inference step of deriving a first comprehensive task result for each of a plurality of unit tasks based reliability information of the workers and the task results of the workers with respect to each of the unit tasks for deriving a comprehensive task result; a reliability information updating step of updating the reliability information of each of the workers based on the first comprehensive task result and the task result of the workers; and deriving a final comprehensive task result for each of the unit tasks, based on the updated reliability information of each of the workers and the task results of the workers, wherein the task result inference step and the reliability information update step may be sequentially performed N times or more (N is a natural number greater than or equal to 1), the reliability information of the workers in the first task result inference step is determined according to a preset rule, the reliability information of the workers used in an M times of the task result inference step (M is a natural number greater than or equal to 2) corresponds to reliability information updated in an M−1 times of reliability information update step.

According to one embodiment of the present invention, in the reliability information update step, reliability information for each worker may be derived and updated to minimize an error value between the task results for each of the workers, and the first comprehensive task result for the unit task corresponding to the task result for each of the workers.

According to one embodiment of the present invention, in the reliability information update step, reliability information of each of the workers may be repeatedly updated based on the task result for each of the workers and the first comprehensive task result derived just before until the reliability information for each worker converges to a specific value.

According to one embodiment of the present invention, the reliability information of the worker may include a plurality of detailed reliability information determined according to a number of a plurality of values that may correspond to a task result for a work including the unit task.

According to one embodiment of the present invention, when the number of a plurality of values that may correspond to the task result for the work including the unit task is N, the reliability information of the worker may include detailed reliability information about the probability that the worker answers with the j-th value to the task result of the unit task corresponding to the actual i-th value (i, j is a natural number less than or equal to N), that is, total N*N detailed reliability information.

According to one embodiment of the present invention, when a value of the task result for the work including the unit task is True or False, the reliability information of the worker may include: first detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as True; second detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual truth as False; third detailed reliability information about the probability that the worker evaluates the task result of the unit task corresponding to an actual False as True; and fourth detailed reliability information on the probability that the worker evaluates the task result of the unit task corresponding to an actual False as False.

According to one embodiment of the present invention, in the task result inference step, when a value of the task result for the work including the unit task is True or False, a first value is assigned when the value of the task result is True, and a second value is assigned when the value of the task result is False, so that the following [Equation 1] is used to derive the first comprehensive task result for each of the unit tasks.

First comprehensive task result for i-th unit task=f(Σ_jreliability information_j*task result_i,j) [Equation 1]

- (where, task result_i,jis a value of a task result evaluated by the j-th worker for the i-th unit task, reliability information_jis reliability information of the j-th worker, and f is a function representing a value reflecting the reliability information_jin the task result as an interpretable comprehensive transformation value)

According to one embodiment of the present invention, in the task result inference step, when a value of the task result for the work including the unit task is True or False, a first value is assigned when the value of the task result is True, and a second value is assigned when the value of the task result is False, so that the following [Equation 2] is used to derive the first comprehensive task result for each of the unit tasks.

First comprehensive task result for i-th unit task=f(Σ_jreliability information_j*task result_i,j) [Equation 2]

- (where, task result_i,jis a value of a task result evaluated by the j-th worker for the i-th unit task, reliability information_jis values of the first detailed reliability information—the third detailed reliability information of the j-th worker, or values of the fourth detailed reliability information—the second detailed reliability information, and f is a function representing a value reflecting the reliability information_jin the task result as an interpretable comprehensive transformation value)

According to one embodiment of the present invention, in the task result inference step, the following [Equation 3] is used with respect to the task result for the work including the unit task, so that a first comprehensive task result for each of the unit tasks may be derived.

First comprehensive task result for i-th unit task=f(Σ_jreliability information_j*task result_i,j) [Equation 3]

- (where, task result_i,jis a value of a task result evaluated by the j-th worker for the i-th unit task, reliability information_jis reliability information of the j-th worker, and f is a function representing a value reflecting the reliability information_jin the task result as an interpretable comprehensive transformation value)

According to one embodiment of the present invention, in the reliability information update step, reliability information may be updated to minimize an error between the first comprehensive task result for each of the unit tasks derived through [Equation 2] in the task result inference step, and the task results for each of the unit tasks for each of the workers.

According to one embodiment of the present invention, in the reliability information update step, reliability information may be updated to minimize an error between the first comprehensive task result for each of the unit tasks derived through [Equation 3] in the task result inference step, and the task results for each of the unit tasks for each of the workers.

According to one embodiment of the present invention, the method of deriving a task result further includes: receiving test results of a plurality of workers for a plurality of initial reliability tests; and an initial reliability information deriving step of deriving initial reliability information of the workers, based on the test results of the workers, wherein, in the task result inference step, the first comprehensive task result for each of a plurality of unit tasks is derived, based on the initial reliability information for each of the workers and the task results of the workers at an initial execution.

According to one embodiment of the present invention, in the step of receiving the test result, test results, which are performed between works including the unit tasks performed by the workers, for a plurality of initial reliability tests are received.

In order to solve the above problem, one embodiment of the present invention provides a system for deriving task results by reflecting reliability information of workers processing works collected through crowdsourcing, which performs: receiving task results of a plurality of workers for a work including a plurality of unit task; a task result inference step of deriving a first comprehensive task result for each of a plurality of unit tasks, based reliability information of the workers and the task results of the workers with respect to each of the unit tasks for deriving a comprehensive task result; a reliability information updating step of updating the reliability information of each of the workers, based on the first comprehensive task result and the task result of the workers; and deriving a final comprehensive task result for each of the unit tasks, based on the updated reliability information of each of the workers and the task results of the workers, wherein the task result inference step and the reliability information update step may be sequentially performed N times or more (N is a natural number greater than or equal to 1), the reliability information of the workers in the first task result inference step is determined according to a preset rule, and the reliability information of the workers used in an M times of the task result inference step (M is a natural number greater than or equal to 2) corresponds to reliability information updated in an M−1 times of reliability information update step.

In order to solve the above problem, one embodiment of the present invention provides a computing-readable medium for implementing a method of deriving task results reflecting reliability information of workers who process works collected through crowdsourcing performed on a computing device having at least one processor and at least one memory, and the computer-readable medium stores instructions for allowing the computing device to perform the following steps including: receiving task results of a plurality of workers for a work including a plurality of unit task; a task result inference step of deriving a first comprehensive task result for each of a plurality of unit tasks, based reliability information of the workers and the task results of the workers with respect to each of the unit tasks for deriving a comprehensive task result; a reliability information updating step of updating the reliability information of each of the workers, based on the first comprehensive task result and the task result of the workers; and deriving a final comprehensive task result for each of the unit tasks, based on the updated reliability information of each of the workers and the task results of the workers, wherein the task result inference step and the reliability information update step may be sequentially performed N times or more (N is a natural number greater than or equal to 1), the reliability information of the workers in the first task result inference step is determined according to a preset rule, and the reliability information of the workers used in an M times of the task result inference step (M is a natural number greater than or equal to 2) corresponds to reliability information updated in an M−1 times of reliability information update step.

Advantageous Effects

According to one embodiment of the present invention, since reliability information is calculated based on the task results performed by a plurality of workers on the work including each unit task, the comprehensive task results for task results can be derived with weights on the reliability information (review/task ability) of the worker. Even when the worker has not previously performed the task, the reliability information can be calculated based on the currently performed task results.

According to one embodiment of the present invention, the task result inference step and the reliability information update step may be repeatedly performed to update the reliability information of the worker so that the error value between the task result for each worker and the first comprehensive task result for the unit task corresponding to the task result for each worker is minimized, so that the reliability information for accurately reflecting the task results performed by a plurality of workers can be derived.

According to one embodiment of the present invention, a plurality of initial reliability tests may be provided to the workers, and the initial reliability information for each worker may be derived based on the test results performed by the worker, so that the initial value for updating the reliability information for each worker can be effectively allocated.

According to one embodiment of the present invention, a plurality of initial reliability tests are provided to the worker between the works including the unit task performed by the worker, so that the initial reliability information can be derived while considering the worker's concentration that changes as the worker continuously performs the tasks.

DESCRIPTION OF DRAWINGS

FIG. 1 schematically shows a system for establishing data in a crowdsourcing scheme according to one embodiment of the present invention.

FIG. 2 schematically shows internal configuration of the computing device for implementing the method for deriving the review result by reflecting the reliability information of the reviewer according to one embodiment of the present invention.

FIG. 3A schematically shows unit tasks included in a work according to one embodiment of the present invention.

FIG. 3B schematically shows unit tasks included in a work according to one embodiment of the present invention.

FIG. 3C schematically shows unit tasks included in a work according to one embodiment of the present invention.

FIG. 3D schematically shows unit tasks included in a work according to one embodiment of the present invention.

FIG. 4 schematically shows detailed processes of the method for deriving the review result by reflecting the reliability information of the reviewer according to one embodiment of the present invention.

FIG. 5 schematically shows the reliability information according to one embodiment of the present invention.

FIGS. 6A and 6B schematically show a process of updating reliability information according to review results of a plurality of reviewers for task results of a plurality of unit tasks according to one embodiment of the present invention.

FIGS. 7A, 7B and 7C schematically show a process of deriving initial reliability information on a plurality of reviewers by receiving test results on a plurality of initial reliability tests of a plurality of reviewers according to one embodiment of the present invention.

FIG. 8 schematically shows internal components of the computing device according to one embodiment of the present invention.

MODE FOR INVENTION

Hereinafter, various embodiments and/or aspects will be described with reference to the drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects for the purpose of explanation. However, it will also be appreciated by a person having ordinary skill in the art that such aspect(s) may be carried out without the specific details. The following description and accompanying drawings will be set forth in detail for specific illustrative aspects among one or more aspects. However, the aspects are merely illustrative, some of various ways among principles of the various aspects may be employed, and the descriptions set forth herein are intended to include all the various aspects and equivalents thereof.

In addition, various aspects and features will be presented by a system that may include a plurality of devices, components and/or modules or the like. It will also be understood and appreciated that various systems may include additional devices, components and/or modules or the like, and/or may not include all the devices, components, modules or the like recited with reference to the drawings.

The term “embodiment”, “example”, “aspect”, “exemplification”, or the like as used herein may not be construed in that an aspect or design set forth herein is preferable or advantageous than other aspects or designs. The terms ‘unit’, ‘component’, ‘module’, ‘system’, ‘interface’ or the like used in the following generally refer to a computer-related entity, and may refer to, for example, hardware, software, or a combination of hardware and software.

In addition, the terms “include” and/or “comprise” specify the presence of the corresponding feature and/or component, but do not preclude the possibility of the presence or addition of one or more other features, components or combinations thereof. In addition, the terms including an ordinal number such as first and second may be used to describe various components, however, the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from another component. For example, the first component may be referred to as the second component without departing from the scope of the present invention, and similarly, the second component may also be referred to as the first component. The term “and/or” includes any one of a plurality of related listed items or a combination thereof.

In addition, in embodiments of the present invention, unless defined otherwise, all terms used herein including technical or scientific terms have the same meaning as commonly understood by those having ordinary skill in the art. Terms such as those defined in generally used dictionaries will be interpreted to have the meaning consistent with the meaning in the context of the related art, and will not be interpreted as an ideal or excessively formal meaning unless expressly defined in the embodiment of the present invention.

A method, a system, and a computer-readable medium for deriving task results by reflecting reliability information of a worker processing a work collected through crowdsourcing of the present invention may be used for the purpose of deriving task results for various types of tasks, such as setting a borders of objects, performed by workers based on crowdsourcing, by reflecting the reliability information of the workers.

In addition, specifically, the present invention may be used to derive task results, in the form of selecting a specific option among multiple options, on tasks performed by workers, with reflection of reliability information on the workers.

In addition, the task may refer to a task result previously performed by a primary worker on a work provided through a computing device performing the present invention, or review work performed by a secondary worker (reviewer) on a task result previously performed by the primary worker provided through an external separate computing device.

More specifically, according to the present invention, the secondary worker (reviewer) may select whether the task result performed by the primarily worker is a correct answer or not (T/F), so as to be used to derive a task result on a task to be reviewed.

In addition, hereinafter, in order to facilitate the description of the present invention, as one embodiment of the present invention, a method of derive a review result subjected to a review based on reliability information of a corresponding worker after a secondary worker (reviewer) selects whether a task result performed by a primary worker is a correct answer or not (T/F) will be described. In other words, the reviewer described below may be included in a worker performing a review belonging to a specific type of task. However, the present invention is not limited to the scope of the following description, and the present invention may be used to derive task results on various tasks performed through the above-described crowdsourcing.

FIG. 1 schematically shows a system for establishing data in a crowdsourcing scheme according to one embodiment of the present invention.

As shown in FIG. 1, the system for constructing data, preferably, labeled learning data, in a crowdsourcing manner includes: a plurality of worker terminals 2000 for performing tasks on works, a plurality of reviewer terminals 3000 for reviewing the task results performed by the workers, and a computing device 1000 communicating with the worker terminals 2000 and the reviewer terminals 3000.

The worker terminal 2000 communicates with the computing device 1000 to receive at one work on which tasks may be performed, and transmits a task result inputted by the worker on the corresponding work to the computing device 1000. In addition, the worker terminal 2000 may display an interface for displaying the work to allow the worker to perform the task on the provided work, and the worker may input task results for the work through the interface displayed on the worker terminal 2000.

In addition, when the task result is transmitted to the computing device 1000 through the worker terminal 2000, or reviews on the task result by multiple reviewers are completed after the task result is transmitted the worker may receive a predetermined reward from the computing device 1000. Specifically, the computing device 1000 may provide a predetermined reward according to the task result to an account corresponding to the worker having provided the task result, and the worker terminal 2000 may display the reward provided to the corresponding account according to an input of the worker. In addition, regarding the predetermined reward, the size of the reward may be determined according to amounts of performed tasks on the work and reviewed results on the performed task result, and accordingly, the reward may be a motivation for enabling the workers to output high-quality task results.

The reviewer terminal 3000 communicates with the computing device 1000 to receive one or more task results performed by a plurality of workers, and transmits the review results inputted by the reviewers on the corresponding task results to the computing device 1000. In addition, the reviewer terminal 3000 may display an interface for displaying the task results so that the reviewer performs the review on the provided task results, and the reviewer may input the review result according to the review on the task results via the interface displayed on the reviewer terminal 3000.

In addition, in another embodiment of the present invention, the reviewer in addition to the worker also may receive a predetermined reward from the computing device 1000 according to the review result performed by the reviewer.

Accordingly, the worker terminal 2000 and the reviewer terminal 3000 may be various types of computing devices capable of communicating with the computing device 1000, such as a smartphone or PC, to display information and receiving input from a user. In addition, the reviewer terminal 3000 and the reviewer terminal 3000 may be installed therein with a web browser capable of executing an application or web page for communicating with the computing device 1000, and the communication with the computing device 1000 may be performed by executing the application or the web page.

In addition, the application or the web page may include a separate application or a separate web page for the workers, and a separate application or a separate web page for the reviewers. Whereas, the application or the web page may include an application or web page commonly used by both of the worker and the reviewer, and different information according to an account type may be displayed upon log-in with the account type corresponding to each of the worker and the reviewer.

The computing device 1000 may communicate with a plurality of worker terminals 2000 and a plurality of reviewer terminals 3000, so as to provide work to the worker terminals 2000, thereby receiving task results and provide the task results to the reviewer terminals 3000, thereby receiving review results. In addition, a comprehensive review result, such as whether the task result is correct or not, may be derived based on the review results for the multiple task results received from the reviewer terminals 3000. This will be described in detail with reference to FIGS. 2 and 4.

In addition, the computing device 1000 may provide a predetermined reward to a corresponding worker for the task result performed by the worker, or may provide a predetermined reward to a corresponding reviewer for the review result performed by the reviewer. Although the computing device 1000 in FIG. 1 is illustrated as a single computing device 1000 that is not physically separated, the computing device 1000 may include a plurality of physically separated detailed computing devices. For example, the computing device 1000 may include a first detailed computing device (not shown) including configurations for providing works to the worker terminals 2000 to receive task results from the worker terminals 2000, providing the task results to the reviewer terminals 3000 to receive review results from the reviewer terminals 3000, and providing predetermined rewards to the workers and the reviewers, respectively, and a second detailed computing device (not shown) including a configuration for outputting comprehensive review result for the corresponding task results based on the received review results. In the above case, the first detailed computing device and the second detailed computing device may be physically separated, however, data may be exchanged through mutual communication. The computing device 1000 may be various types of data processing devices, such as a server, capable of performing communication with the worker terminals 2000 and the reviewer terminals 3000 and deriving the comprehensive review result according to review results of multiple reviewers.

Although not shown in FIG. 1, according to another embodiment of the present invention, the computing device 1000 may communicate with a data requestor terminal (not shown) and may receive works requiring labeling tasks requested by a data requestor through the data requestor terminal, and receive the labeled work from the computing device 1000 according to the comprehensive result outputted based on the review result on the task results for the corresponding work. In addition, the work having a type required by the data requester may be pre-stored in the computing device 1000, and the data requestor terminal may receive the work labeled for the pre-stored work from the computing device 1000.

As shown in FIG. 2, the computing device 1000 may include a plurality of components for implementing a method for deriving review results by reflecting reliability information of the reviewer. Specifically, in order to label and review a work, the components for communicating with a plurality of worker terminals 2000 and a plurality of reviewer terminals 3000 may include a work providing unit 1010, a task result receiving unit 1020, a task result providing unit 1030, a review result receiving unit 1040, an initial reliability test providing unit 1050, and a test result receiving unit 1060.

The work providing unit 1010 provides at least one work for performing labeling to a plurality of worker terminals 2000. Each work may include at least one unit task, the worker may input task results by performing labeling for each unit task included in the provided work. In addition, the work providing unit 1010 may provide the work previously stored in the DB 1110 of the computing device 1000 or the work received from the data requestor terminal to the worker terminals 2000.

The task result receiving unit 1020 receives, from a corresponding worker terminal 2000, a task result performed by the worker with respect to the provided work. The task result may include detailed task results on the at least one unit task included in the work, or the task result may be the task result for each of the at least one unit task included in the work. In addition, the received task result may be stored in the DB 1110 of the computing device 1000.

The task result providing unit 1030 provides the task results to a plurality of reviewer terminals 3000 in order to review the task results received from the worker terminals 2000. The reviewer may input review results after performing reviews on the provided task results.

The review result receiving unit 1040 receives the review result performed by the reviewer for the provided task result, from the reviewer terminal 3000. For example, when the task result indicates a region of a car included in an image and labels the region as a car, the review result may refer to inputting whether the corresponding region is a car.

The initial reliability test providing unit 1050 requires reliability information on each reviewer in order to derive a comprehensive review result for each unit task for the review results of the reviewers. In order to derive initial reliability information corresponding to an initial value of reliability information on each reviewer, the initial reliability test providing unit 1050 provides a plurality of initial reliability tests to a plurality of reviewer terminals 3000.

The test result receiving unit 1060 receives, from a plurality of reviewer terminals 3000, test results inputted by performing a plurality of initial reliability tests provided through the initial reliability test providing unit 1050 by a plurality of reviewers. As in the above, initial reliability information may be created for each reviewer, compared to correct answers assigned to the initial reliability tests through the test results received for each reviewer.

In addition, in another embodiment of the present invention, the configuration in which the initial reliability test providing unit 1050 provides a plurality of initial reliability tests to a plurality of reviewer terminals 3000 may be included in the task result providing unit 1030. Specifically, the task result providing unit 1030 may provide a plurality of task results and a plurality of initial reliability tests together to a plurality of reviewer terminals 3000. Accordingly, the configuration, in which the test results are received from the reviewer terminals 3000 in the above-described test result receiving unit 1060, is also included in the review result receiving unit 1040, so that the review result receiving unit 1040 may receive review results and test results on the initial reliability tests, from the reviewer terminals 3000.

In addition, the computing device 1000 may further include components for deriving the comprehensive review result for each of a plurality of unit tasks, and the corresponding component may include an initial reliability information derivation unit 1070, a review result inference unit 1080, a reliability information update unit 1090, and a final comprehensive review result derivation unit 1100.

The initial reliability information derivation unit 1070 may derive initial reliability information for each reviewer, based on the test results on each reviewer received from the above-described test result receiving unit 1060 and correct answers of the initial reliability tests. The initial reliability information for each reviewer derived from the initial reliability information derivation unit 1070 may correspond to reliability information used to initially derive the first comprehensive review result on the review results of a plurality of reviewers in the review result inference unit 1080 described later.

The review result inference unit 1080 derives the first comprehensive review result for each unit task, based on the review results performed by a plurality of reviewers for each unit task and the reliability information for each reviewer. When the first comprehensive review result is derived for the first time, the review result inference unit 1080 may derive the first comprehensive review result by using the initial reliability information for each reviewer created by the initial reliability information derivation unit 1070, and then may repeatedly derive the new first comprehensive review results by using the reliability information updated in the reliability information update unit 1090.

The reliability information update unit 1090 updates the reliability information for each reviewer, based on the first comprehensive review result for each unit task derived from the review result inference unit 1080 and the review results of a plurality of reviewers for each unit task. Based on the updated reliability information and the review results performed by the reviewers, the first comprehensive review result may be derived from the review result inference unit 1080 again, and the reliability information update unit 1090 may update the reliability information again based on the new first comprehensive review result.

The final comprehensive review result derivation unit 1100 may be updated for a predetermined number of times in the reliability information update unit 1090 and derive a final comprehensive review result for each unit task based on the reliability information finally updated and the review results performed by a plurality of reviewers for each unit task. The final comprehensive review result may be a finally labeled result for the unit task.

In addition, the configuration in which the final comprehensive review result is derived in the final comprehensive review result derivation unit 1100 may be included in the review result inference unit 1080. Specifically, the review result inference unit 1080 may derive each first comprehensive review result based on each reliability information until finally updated, and may also derive the final comprehensive review result based on the finally updated reliability information.

In addition, the computing device 1000 may further include a DB 1110 in addition to the above components. The DB 1110 may store information for constructing labeled data based on crowdsourcing. Specifically, the DB 1110 may store review result inference information that includes: worker information on each worker using a worker terminal 2000 communicating with the computing device 1000, reviewer information on each reviewer using a reviewer terminal 3000, a work on which labeling is performed, a task result performed by each worker on the work, a review result performed by each reviewer for the task result, initial reliability test information for deriving initial reliability information of the reviewer, initial reliability information of each reviewer and reliability information updated by the reliability information update unit 1090, and a first comprehensive review result and a final comprehensive review result derived by the review result inference unit 1080 and the final review comprehensive result derivation unit 1110.

In addition, the internal configuration of the computing device 1000 shown in FIG. 2 is shown as only essential components in order to easily describe the present invention, and various components such as a communication unit and a control unit may be further included in addition thereto.

In addition, the computing device 1000 may be implemented as one device that is physically separated. However, according to another embodiment of the present invention, the computing device 1000 may include the above-described one or more components in a plurality of physically separated devices, and the physically separated devices may communicate with each other to perform functions of the computing device 1000.

FIGS. 3A through 3D schematically show unit tasks included in a work according to one embodiment of the present invention.

FIGS. 3A, 3B, 3C and 3D are embodiments of an interface including a work, which may be displayed on the worker terminal 2000.

Referring to the leftmost side of FIG. 3A first, the leftmost drawing shows a requested object photographed by using a camera provided in the worker terminal 2000, For example, the task result may be an image of a calendar photographed by the worker upon a request for a task of taking a picture of a calendar. Meanwhile, with respect to the above task results, the reviewer may input a review result by inputting whether the photographed image is a calendar.

The second drawing of FIG. 3A is a view showing an interface including a work in the form of an image. The worker provided with the above work may input a task result by setting a region of a specific object included in the image. In this case, a specific object (such as a table) to be set as the region may be indicated in the interface. Meanwhile, for the corresponding task result, the reviewer may input a review result by inputting whether the region set in the image is the specific object, or by inputting whether the region of the specific object is set normally.

The third drawing of FIG. 3A, similar to the second drawing, is a view showing an interface including a work in the form of an image. The worker provided with the above work may input a task result by selecting specific objects included in the image. Likewise, a specific object (such as a vehicle) to be selected may be indicated in the interface. Meanwhile, for the corresponding task result, the reviewer may input a review result by inputting whether all the specific objects included in the image are selected, or by inputting whether the region of the selected specific object is set normally

The fourth drawing of FIG. 3A is a view showing an interface including a work in another form of an image. The worker provided with the above work may input a task result by selecting an option related to the image or directly inputting information related to the image. Meanwhile, for the corresponding task result, the reviewer may input a review result by inputting whether the selected option for the image is correct, or by inputting whether the directly inputted information is appropriate.

The drawings shown in FIG. 3B show embodiments of interfaces including text-based works. The leftmost drawing of FIG. 3B is a view showing an interface including a work in the form of an image including specific text. The worker provided with the above work may input a task result by directly inputting the text included in the image. In addition, for the corresponding task result, the reviewer may input a review result by inputting whether the text included in the image matches the text inputted by the worker.

The second drawing of FIG. 3B is a view showing an interface using one or more key words as a work. The worker provided with the above work may input a task result by inputting a sentence related to the at least one key word. In addition, for the corresponding task result, the reviewer may input a review result by inputting whether the inputted sentence is properly related to the at least one key word.

The third drawing of FIG. 3B is a view showing an interface including a work in the form of a voice obtained by converting predetermined text into voice. The worker provided with the above work may input a task result by listening to the voice and directly inputting the corresponding voice in the form of text. In addition, for the corresponding task result, the reviewer may input a review result by inputting whether the inputted text matches the voice.

The fourth drawing of FIG. 3B is a view showing another type of interface using one or more key words as a work. The worker provided with the above work may input a task result by recording a sentence related to one or more keywords in the form of voice. In addition, for the corresponding task result, the reviewer may input a review result by inputting whether the recorded voice and the one or more keywords are properly related, or whether the recording is normally conducted.

FIG. 3C is a view showing another type of interface including an image type work. Specifically, the worker provided with the work may input a task result by setting one or more feature points requested in the image. For example, in FIG. 3C, an image of a human face is provided as a work, and the worker may input a task result by setting a plurality of feature points for ‘forehead’, ‘left eyebrow’, ‘right eyebrow’, ‘left eye’, ‘right eye’, ‘nose’, ‘left chin’, ‘lip’, ‘right chin’, and ‘chin’ in the face image.

In addition, as shown in FIG. 3C, one work may include one or more unit tasks, and the worker may input a task result for each unit task. For example, in FIG. 3C, in addition to the task result for the unit task of setting the feature points as described above, the worker may input the task result for the unit task by inputting a specific age group with respect to the unit task of inputting an age group estimated from the face image. In addition, the worker may input the task result for the unit task by inputting a specific sex with respect to the unit task of inputting a sex estimated from the face image. In addition, the worker may input the task result for the unit task by inputting the objects included in the image with respect to the unit task of inputting objects included in the image.

Accordingly, one or more unit tasks may be included in one work, and the reviewer may perform a review on each task result of each unit task for the corresponding work, thereby inputting a review result for each unit task.

FIG. 3D is a view showing an interface including an image or video type work. Specifically, in the image or video (a specific frame of the video) provided as a work as shown in FIG. 3D, the worker may set a region of a main object or a specific object requested as a task by inputting a plurality of points, and may input a task result by performing labeling on the set region. In addition, for the corresponding task result, the reviewer may input a review result by inputting whether the region of the object is set normally, or whether the inputted label is correct for the set region.

In addition, in addition to selecting a specific option from two options such as True or False as described above, the review result inputted by the reviewer may include various types of review results, such as selecting a specific option from three or more options, or directly inputting text or the like, by the reviewer, for the task result.

As shown in FIG. 4, the method for deriving review results reflecting reliability information on reviewers reviewing works collected through crowdsourcing performed in the computing device 1000 having at least one memory and at least one processor includes: receiving task results of the worker for a plurality of unit task (S10); Receiving the review results of a plurality of reviewers for the task results of a plurality of unit task (S11); a review result inference step (S12) of deriving a first comprehensive review result for each of the unit tasks, based on reliability information of the reviewers and the review results of the reviewers with respect to each of the unit tasks for deriving a comprehensive review result; a reliability information update step (S13) of updating reliability information on each of the reviewers, based on the first comprehensive review result and the review results of the reviewers; and a step (S14) of deriving a final comprehensive review result for each of the unit tasks, based on the updated reliability information on each of the reviewers and the review results of the reviewers, wherein the review result inference step S12 and the reliability information update step S13 may be sequentially performed N times or more (N is a natural number greater than or equal to 1), the reliability information of the reviewers in the initial review result inference step S12 may be determined according to a preset rule, and the reliability information of the reviewers used in the M times of review result inference step S12 (M is a natural number greater than or equal to 2) may correspond to the reliability information updated in the M−1 times of reliability information update step S13.

Specifically, as described in FIG. 3, the worker performs a task for the provided work and inputs the task result to the worker terminal 2000, and the task result receiving unit 1020 of the computing device 1000 performs a step S10 of receiving the task result to receive a plurality of task results from a plurality of worker terminals 2000. In addition, the computing device 1000 provides the received task results to reviewer terminals 3000 of a plurality of reviewers for reviewing the task results, so as to enable the reviewers to review each task result through the reviewer terminal 3000 and input the review result.

In another embodiment of the present invention, the step S10 may be omitted, and the task result of the worker (primary worker) for a plurality of unit tasks may be provided through an external computing device such as a separate server.

In addition, the review result receiving unit 1040 of the computing device 1000 performs a step S11 of receiving the review result, so as to receive a plurality of review results for the task result from a plurality of review terminals 3000.

In another embodiment of the present invention, the step S11 of receiving the review result may refer to receiving task results of the worker performing a task including a review.

Next, the review result inference unit 1080 performs the review result inference step S12, thereby deriving a first comprehensive review result for each unit task, based on the reliability information for each of the reviewers having performed the reviews and the review result performed by each reviewer. In addition, the review result inference step S12 may be repeatedly performed, and the first comprehensive review result derived when the review result inference step S12 is initially performed derive a first comprehensive review result for each unit task, based on the reliability information for each reviewer determined according to the preset rule and the review result performed by each reviewer.

In order to derive the initial first comprehensive review result, the reliability information for each reviewer determined according to the preset rule may correspond to initial reliability information derived for each reviewer, based on test results for a plurality of initial reliability tests performed by each reviewer in the above-described initial reliability information derivation unit 1070. In addition, the first comprehensive review result derived from the review result inference step S12 may be used to update the previous reliability information for each reviewer in the reliability information update step S13 described later.

In another embodiment of the present invention, the review result inference step S12 may refer to a task result inference step of deriving the first comprehensive task result for each unit task, based on reliability information for each of a plurality of workers performing tasks including a review and a task result performed by each worker.

In the reliability information update step S13 performed by the reliability information update unit 1090, the first comprehensive review result for each unit task is compared with the review result for each of the reviewers for each unit task, so that the reliability information for each reviewer is updated so as to minimize an error value. In addition, the reliability information updated through the reliability information update step S13 may be used as reliability information for deriving a new first comprehensive review result in the review result inference step S12.

In other words, the first comprehensive review result derived in the review result inference step S12 may be used to update the previous reliability information in the reliability information update step S13, and the reliability information updated in the reliability information update step S13 may be used to derive a new first comprehensive review result in the review result inference step S12. Accordingly, the review result inference step S12 and the reliability information update step S13 may be performed one or more times sequentially. In the M-th (M is a natural number greater than or equal to 2) review result inference step S12, the M-th first comprehensive review result may be derived based on the reliability information updated in the (M−1)—the reliability information update step S13.

The above process may be repeated until the reliability information converges to a specific value or repeated for a preset number of times. Finally, when reliability information is updated, the step S14 of deriving a final comprehensive review result may be performed based on the reliability information.

According to another embodiment of the present invention, in the reliability information update step S13, the above-described first comprehensive task result for each unit task is compared with the task results for the workers for each unit task, so that reliability information for each worker may be updated to minimize an error value.

As described above, the final comprehensive review result derivation unit 1100 performs the step S14 of deriving the final comprehensive review result to derive the final comprehensive review result for each unit task based on the finally updated reliability information for each reviewer and the review results performed by the reviewers. Accordingly, the final comprehensive review result for each unit task derived in the step S14 of deriving the final comprehensive review result is may correspond to a result inferred as a correct answer for each unit task.

According to another embodiment of the present invention, the step S14 of deriving the final comprehensive review result may refer to the step of deriving the final comprehensive task result with respect to each of the unit tasks, based on reliability information on each of a plurality of workers having performed tasks including a plurality of updated reviews and the task results of the workers.

Accordingly, in the present invention, the reliability information of the reviewer, that is, the review ability of the reviewer is estimated based on the reviewer results currently performed by the reviewer, and the estimated review ability of the reviewer is used as a weight for estimating the correct answer (final comprehensive review result) of the corresponding unit task, so that high-quality learning data may be effectively established.

In other words, the present invention can more accurately estimate the correct answer of the task result, compared to the conventional method for determining a correct answer of the task result by a majority vote without consideration of a review ability of each reviewer, or estimating a correct answer of a current task result by estimating a review ability based on past review results of the reviewer.

FIG. 5 schematically shows the reliability information according to one embodiment of the present invention.

As shown in FIG. 5, the reliability information of the reviewer includes a plurality of detailed reliability information in which a plurality of values that may correspond to a review result for the task result of the unit task are determined according to the number.

Specifically, the reliability information of the reviewer may include a plurality of pieces of detailed reliability information, and the detailed reliability information and the number thereof may be determined according to a value of the review result which the reviewer can input, that is, according to the number of options which can be inputted as the review result. For example, the options which can be inputted as the review result may include various cases such as a review result (True or False) on whether the task result is performed normally, a review result (Male or Female) on whether a sex of a person included in an image is inputted normally, and a review result on whether a label and a region of an object included in the image are set normally (Labeling normal—region setting normal, labeling normal—region setting abnormal, labeling abnormal—region setting normal and labeling abnormal—region setting abnormal).

In addition, when there are 2 review result values for example as shown in FIG. 5, and a value of the review result for the task result of the unit task corresponds to True or False, the reliability information of the reviewer may include first detailed reliability information about the probability that the reviewer evaluates the task result of the unit task corresponding to an actual truth as True; second detailed reliability information on the probability that the reviewer evaluates the task result of the unit task corresponding to an actual truth as False; third detailed reliability information about the probability that the reviewer evaluates the task result of the unit task corresponding to an actual False as True; and fourth detailed reliability information on the probability that the reviewer evaluates the task result of the unit task corresponding to an actual False as False.

Specifically, the reviewer may input the review result by selecting one of the two options of True/False for the task result, and at least one detailed reliability information included in the reliability information of the corresponding reviewer may be determined according to a review result reviewed by the reviewer on the task result and a type of correct answer of the actual task result.

Referring to FIG. 5, when the review result has two options of True/False, the reliability information may include a total of four detailed reliability information. The detailed reliability information may include first detailed reliability information PTT about the probability that the reviewer evaluates the task result of the unit task, which actually corresponds to a True correct answer, as True, second detailed reliability information PTF about the probability that the reviewer evaluates the task result of the unit task, which actually corresponds to a True correct answer, as False, third detailed reliability information PFT about the probability that the reviewer evaluates the task result of the unit task, which actually corresponds to a False correct answer, as True, and fourth detailed reliability information PPP about the probability that the reviewer evaluates the task result of the unit task, which actually corresponds to a False correct answer, as False.

In addition, since the detailed reliability information corresponding to the probability that the reviewer correctly reviews the task result (True for True and False for False) corresponds to the first detailed reliability information PTT and the fourth detailed reliability information PFF, the first detailed reliability information PTT and the fourth detailed reliability information PFF may have the same value.

In addition, since the detailed reliability information corresponding to the probability that the reviewer incorrectly reviews the task result (False for True and True for False) corresponds to the second detailed reliability information PTF and the third detailed reliability information PFT, the second detailed reliability information PTF and the third detailed reliability information PFT may have the same value.

In addition, the sum of the first detailed reliability information PTT and the third detailed reliability information PFT may be 1. Likewise, the sum of the second detailed reliability information PTF and the fourth detailed reliability information PFF may also be 1.

Accordingly, the reliability information for each reviewer may include at least one detailed reliability information, and the detailed reliability information may be determined according to at least one option that may correspond to the review result. Meanwhile, the reliability information for each reviewer may be used to derive the first comprehensive review result and the final comprehensive review result in the step of deriving the final review result inference step S12 and the final comprehensive review result S14, and the reliability information for each reviewer may be updated until converging to a specific value in the reliability information update step S13.

FIG. 6A is a view showing review results (T or F) performed by a plurality of reviewers (reviewer 1 to reviewer j) with respect to task results of a plurality of unit tasks (unit task 1 to unit task i). FIG. 6B is a view showing a process of deriving the first comprehensive review result based on review results performed by a plurality of reviewers with respect to task results of a plurality of unit tasks and reliability information of the reviewers, and of updating reliability information according to the first comprehensive review result.

As shown in FIG. 6A, when a value of the review result for the task result of the unit task corresponds to True or False in the review result inference step S12, a first value is assigned when the value of the review result is True, and a second value is assigned when the value of the review result is False, so that the first comprehensive review result for each of a plurality of unit tasks by using the following [Equation 1].

First comprehensive task result for i-th unit task=f(Σ_jreliability information_j*task result_i,j) [Equation 1]

- (where, task result_i,jis a value of a task result evaluated by the j-th worker for the i-th unit task, reliability information_jis reliability information of the j-th worker, and f is a function representing a value reflecting the reliability information_jin the task result as an interpretable comprehensive transformation value)

Specifically, a plurality of unit tasks shown in FIG. 6A may correspond to different unit tasks, but may correspond to task results of the same type of unit task. Alternatively, a plurality of unit tasks may all correspond to the same unit task, but may correspond to task results by tasks of a plurality of different workers. Accordingly, the reliability information of the reviewer for each unit task can be equally applied.

In addition, in the review result inference step S12, the reliability information for each reviewer for each unit task and the review result for the unit task are calculated using [Equation 1], so that the first comprehensive review result may be derived for each unit task. More specifically, the first comprehensive review result for a specific unit task may correspond to a value obtained by adding, for each reviewer, all of a value (first value or second value) assigned according to the review result of the reviewer for the unit task and values of a function using the reliability information of the reviewer as a variable.

In addition, an example of function f withe reliability information as a variable may also be expressed as:

$f (a_{i}, b_{i}) = \frac{a_{i} p_{i}}{a_{i} p_{i} + b_{i} (1 - p_{i})}$

In the above Equation, pi is the probability that the review result for the task result of the i-th unit task is True, ai is the probability of getting the correct answer when the correct answer of the task result of the i-th unit task is True, and bi is the probability of getting the correct answer when the correct answer of the task result of the i-th unit task is False. In other words, ai and bi may correspond to reliability information.

More specifically, as an example for the above [Equation 1], when a value of the review result for the task result of the unit task corresponds to True or False in the review result inference step S12, a first value is assigned when the value of the review result is True, and a second value is assigned when the value of the review result is False, so that the first comprehensive review result for each of a plurality of unit tasks by using the following [Equation 2].

First comprehensive task result for i-th unit task=f(Σ_jreliability information_j*task result_i,j) [Equation 1]

- (where, task result_i,jis a value of a task result evaluated by the j-th worker for the i-th unit task, reliability information_jis values of the first detailed reliability information—the third detailed reliability information of the j-th worker, or values of the fourth detailed reliability information—the second detailed reliability information, and f is a function representing a value reflecting the reliability information_jin the task result as an interpretable comprehensive transformation value)

In other words, [Equation 2] may correspond to an Equation describing [Equation 1] in more detail. Preferably, the first value (when the review result is True) may correspond to 1, and the second value (when the review result is False) may correspond to −1. In addition, referring to the description in FIG. 5, the reliability information on reviewer j may include first detailed reliability information P_TTj, second detailed reliability information P_TFj, third detailed reliability information P_FTj, and fourth detailed reliability information P_FFj.

In addition, the following Equation may correspond to one embodiment of

$\begin{matrix} First comprehensive review result for i - th unit task = \frac{f (\sum_{j} review {result}_{i, j} * reliability {information}_{j})}{L} & [Equation 2] \end{matrix}$

- (where, review result_i,jis a value of a review result evaluated by the j-th reviewer for the i-th unit task, reliability information_jis values of the first detailed reliability information—the third detailed reliability information of the j-th reviewer, or values of the fourth detailed reliability information—the second detailed reliability information, and L is the total number of reviewers or L=Σ_jreliability information_j

When calculating the first comprehensive review result by using the above Equation for the task result for the first unit task (unit task 1) shown in FIG. 6A, the first comprehensive review result for unit task 1 may correspond to ((1*(P_TT1−P_FT1))+(−1*(P_FF2−P_TF2))+ . . . +(1*(P_TTj−P_FTj)))/j.

In the above manner, the first comprehensive review result for each unit task may be derived based on the reliability information of the reviewers and the review result of the reviewer for each unit task.

Preferably, the first comprehensive review result may correspond to information on specific options that may correspond to the review result determined according to a reference value with respect to a predetermined value calculated through [Equation 2]. For example, the reference value may be 0. When the predetermined value calculated through [Equation 2] is 0 or more, the first comprehensive review result may correspond to True, and when the predetermined value calculated through [Equation 2] is less than 0, the first comprehensive review result may correspond to False.

In addition, when the review result inference step S12 is initially performed, the reliability information of a plurality of reviewers may derive a first comprehensive review result by using the initial reliability information derived according to a preset rule, and the initial reliability information may have the same initial value for each reviewer, or correspond to initial reliability information derived based on the test results for a plurality of initial reliability tests performed by the reviewer in the reliability information update step S13 as described above.

The above-described [Equation 1] and [Equation 2] are configured to derive the first comprehensive task result for unit task in a special case in which the task result of a task including the unit task is True or False, that is, there are two options as the task result, so as to easily describe the present invention. Further, when the task result has 3 or more options, the first comprehensive task result for the unit task may be derived through the following [Equation 3].

In the task result inference step, the following [Equation 3] is used with respect to the task result for the work including the unit task, so as to derive the first comprehensive task result for each of a plurality of unit tasks.

First comprehensive task result for i-th unit task=f(Σ_jreliability information_j*task result_i,j) [Equation 3]

- (where, task result_i,jis a value of a task result evaluated by the j-th worker for the i-th unit task, reliability information_jis reliability information of the j-th worker, and f is a function representing a value reflecting the reliability information_jin the task result as an interpretable comprehensive transformation value)

For [Equation 3], in the general case where the number of task results is 3 or more, the reliability information_jsignifying the reliability information of the j-th worker may be expressed as follows.

When the number of a plurality of values that may correspond to the task result for the work including the unit task is N, the reliability information of the worker may include detailed reliability information about the probability that the worker answers with the j-th value to the task result of the unit task corresponding to the actual i-th value (i, j is a natural number less than or equal to N), that is, total N*N detailed reliability information.

In other words, for the reliability information of the worker, the number of a plurality of detailed reliability information is determined according to the number of values that may correspond to the task result, and the reliability information of the worker may be outputted based on a plurality of detailed reliability information. The worker's reliability information outputted in the above manner is used as a factor in [Equation 3] so as to finally derive the first comprehensive task result for the unit task.

Then, as shown in FIG. 6B in the reliability information update step S13, the reliability information may be updated to minimize the error between the first comprehensive review result for each of the unit tasks derived through [Equation 3], and the review result for each of the unit tasks for each of the reviewers in the review result inference step S12.

Specifically, in the reliability information update step S13, the reliability information may be updated to minimize the error between the first comprehensive review result for each unit task derived through [Equation 1] to [Equation 3], and the review result for each reviewer in the review result inference step S12 as described above. In other words, in the reliability information update step S13, the reviewers' reliability information is derived and updated for minimizing the comprehensive error between the first comprehensive review result for each of the unit tasks derived by the review result inference step S12 and the review result of each of the reviewers, thereby calculating a function or probability model that uses the total number of reviewers as a dimension or variable, so that the reliability information of the reviewer may be updated.

Accordingly, as one embodiment for updating the reliability information of the reviewer, a probability model p(z,q) for a correct answer z of each unit task corresponding to a latent variable and a reliability or review ability q of the reviewer may be created, and the probability model may be used, so that the reliability information of the reviewer may be updated.

More specifically, the probability model p(z, q) may be expressed as an observable value as in [Equation 4] described below.

$\begin{matrix} p (z, q | L, θ) \propto \prod_{j \in [M]} p (q_{j} | θ) \prod_{i \in N_{j}} p (L_{i j} | z_{i}, q_{j}) = \prod_{j \in [M]} p (q_{j} | θ) {q_{j}^{c_{j}} (1 - q_{j})}^{γ_{j} - c_{j}} & [Equation 4] \end{matrix}$

In other words, the probability model when observed data (review result) L and a parameter θ for the model are given is proportional to the product of p(q_j|θ) and p(L_ij|z_iq_j) corresponding to the observable value (j is the j-th reviewer, i is the i-th unit task). When a latent variable for maximizing a probability value of the probability model with respect to [Equation 3] is calculated, the reliability information of the reviewer may be outputted.

Preferably, an expected value of the latent variable may be calculated (E-step) as in [Equation 5] with respect to the above-mentioned [Equation 4], and an expectation maximization (EM) algorithm, which estimates (M-step) the reliability information on a reviewer using the calculated expected value, may be used, so that reliability information for each reviewer may be updated.

$\begin{matrix} E - step : μ_{i} (z_{i}) \propto \prod_{j \in M_{i}} {{\hat{q}}^{δ_{i j}} (1 - {\hat{q}}_{j})}^{1 - δ_{i j}}, & [Equation 5] \end{matrix}$

$M - step : {\hat{q}}_{j} = \frac{Σ_{i \in N_{j}} μ_{i} (L_{i j}) + α - 1}{❘ N_{j} ❘ + α + β - 2}$

The EM algorithm may use the reliability information estimated in the t-th cycle to calculate the expected value in an E-step of the (t+1)-th cycle, and the expected value calculated in the E-step of the (t+1)-th cycle may be used to estimate reliability information in the M-step of the (t+1)-th cycle, so that the E-step and the M-step may be repeatedly performed until the estimated value of reliability information converges to a specific value.

In another embodiment of the present invention, the reliability information of the reviewer may be updated by using a belief propagation algorithm for estimating a latent variable, which integrates (marginalizes) the above-mentioned [Equation 3] withe reliability q by using a graphic model to maximize a probability value of the probability model. In another embodiment of the present invention, review results of reviewers may be set as a matrix, and a Spectral Method may be used for the matrix, so that the reliability of each reviewer and the final comprehensive review result may be derived.

In addition, the reliability information updated in the reliability information update step S13 of the t-th cycle may be used to derive the first comprehensive review result in the review result inference step S12 of the (t+1)-th cycle, and the first comprehensive review result derived from the review result inference step S12 of the (t+1)-th cycle may be used to update the reliability information in the reliability information update step S13 of the (t+1) cycle. The repeated process of the review result inference step S12 and the reliability information update step S13 may be repeated until the reliability information of the reviewer converges to a specific value or may be repeated by a predetermined number of times.

The reliability information finally updated through the above process may be used to derive the final comprehensive review result for a plurality of unit tasks in the step S14 of deriving the final comprehensive review result. In the step S14 of deriving the final comprehensive review result, the final comprehensive review result for the unit task may be derived by using [Equation 1] to [Equation 3] in the same manner as the review result inference step S12.

As shown in FIG. 7A, the method for deriving a review result further includes: receiving test results of a plurality of reviewers for a plurality of initial reliability tests (S21); and an initial reliability information deriving step (S22) of deriving initial reliability information of the reviewers, based on the test results of the reviewers, wherein, in the review result inference step S12, the first comprehensive review result for each of a plurality of unit tasks may be derived based on the initial reliability information for each of the reviewers and the review results of the reviewers, at the time of initial execution.

Specifically, the initial reliability test providing unit 1050 of the computing device 1000 provides (S20) a plurality of initial reliability tests to reviewer terminals 3000 of a plurality of reviewers reviewing the task result, and each reviewer performs a test for a plurality of initial reliability tests through the corresponding reviewer terminal 3000 and inputs the test result. In addition, the test result receiving unit 1060 performs a step (S21) of receiving the test results inputted by each reviewer from the reviewer terminals 3000. Finally, the initial reliability information derivation unit 1070 derives (S22) initial reliability information for each reviewer based on the received test results for each reviewer. Accordingly, the initial reliability information for each reviewer derived from the initial reliability tests may be used as reliability information for deriving the first comprehensive review result when the review result inference step S12 is initially performed.

The content of the initial reliability test may have a separate test content different from the review on the task result in order to derive the initial reliability, however, may preferably correspond to the content similar to that of the review by the reviewer on the task result.

In addition, as one embodiment of a method for deriving initial reliability information based on test results for a plurality of initial reliability tests, each initial reliability test has a preassigned correct answer, and the test result inputted by the reviewer is compared with the correct answer for the initial reliability test, so that the initial reliability information of the reviewer may be derived.

In another embodiment of the present invention, each initial reliability test has a correct answer and a difficulty level that are preassigned, and the weight according to the difficulty level is given instead of setting each initial reliability test to the same weight, so that more accurate initial reliability information may be derived.

In addition, according to the present invention, when the initial reliability test is provided to the reviewer, the initial reliability test may be clearly stated on the reviewer terminal 3000 so that the reviewer can recognize that the process is a separate test rather than an actual review, or the initial reliability test may not be clearly stated, so that the reviewer cannot distinguish whether the process is the actual review or the initial reliability test, thereby deriving more effective initial reliability information.

In addition, according to the present invention, various methods may exist to provide the initial reliability test to the reviewer reviewing the task results, and FIGS. 7B and 7C show embodiments of the above method.

In FIG. 7B, the reviewer is allowed to perform an initial reliability test before performing a review on task results of a plurality of unit tasks. When the initial reliability test is performed in the above manner before the actual review, the reviewer performs the test in a state of high concentration, so that the initial reliability may be derived relatively higher than the reliability in the actual review process.

In the above case, it may take longer time to finally update the reliability information, or computing resources for calculating the reliability information may be required more.

Accordingly, as shown in FIG. 7C, in the step of receiving the test result, test results for a plurality of initial reliability tests performed between the task results of the unit tasks on which a plurality of reviewers perform reviews are received in order to efficiently derive the initial reliability information.

Specifically, the initial reliability test provided to the reviewer may be arranged and provided between the task results of the unit task to be actually reviewed, or some of the initial reliability tests may be provided before the actual review, and the remaining initial reliability tests may be arranged and provided between the task results of the unit task to be actually reviewed.

When the reviewer performs the review through the above configuration, the initial reliability information may be derived while considering the deterioration of concentration or condition, so that the time required for finally updating the reliability information from initial reliability information can be shortened, or the amount of computing resources used to calculate the reliability information can be reduced.

In addition, according to the present invention as shown in FIG. 7C, it is not limited to the configuration of providing one initial reliability test between a task result of the unit task and a task result of another unit task, and it may even include the configuration of providing a plurality of initial reliability tests between a task result of the unit task and a task result of another unit task.

FIG. 8 schematically shows internal components of the computing device according to one embodiment of the present invention.

The computing device 1000 shown in the above-described FIG. 1 may include components of the computing device 11000 shown in FIG. 8.

As shown in FIG. 8, the computing device 11000 may at least include at least one processor 11100, a memory 11200, a peripheral device interface 11300, an input/output subsystem (I/O subsystem) 11400, a power circuit 11500, and a communication circuit 11600. The computing device 11000 may correspond to the computing device 1000 shown in FIG. 1.

The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. The memory 11200 may include a software module, an instruction set, or other various data necessary for the operation of the computing device 11000.

The access to the memory 11200 from other components of the processor 11100 or the peripheral interface 11300, may be controlled by the processor 11100.

The peripheral interface 11300 may combine an input and/or output peripheral device of the computing device 11000 to the processor 11100 and the memory 11200. The processor 11100 may execute the software module or the instruction set stored in memory 11200, thereby performing various functions for the computing device 11000 and processing data.

The input/output subsystem may combine various input/output peripheral devices to the peripheral interface 11300. For example, the input/output subsystem may include a controller for combining the peripheral device such as monitor, keyboard, mouse, printer, or a touch screen or sensor, if needed, to the peripheral interface 11300. According to another aspect, the input/output peripheral devices may be combined to the peripheral interface 11300 without passing through the I/O subsystem.

The power circuit 11500 may provide power to all or a portion of the components of the terminal. For example, the power circuit 11500 may include a power failure detection circuit, a power converter or inverter, a power status indicator, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for generating, managing, and distributing the power.

The communication circuit 11600 may use at least one external port, thereby enabling communication with other computing devices.

Alternatively, as described above, if necessary, the communication circuit 11600 may transmit and receive an RF signal, also known as an electromagnetic signal, including RF circuitry, thereby enabling communication with other computing devices. The above embodiment of FIG. 10 is merely an example of the computing device 11000, and the computing device 11000 may have a configuration or arrangement in which some components shown in FIG. 10 are omitted, additional components not shown in FIG. 10 are further provided, or at least two components are combined. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen, a sensor or the like in addition to the components shown in FIG. 10, and the communication circuit 11600 may include a circuit for RF communication of various communication schemes (such as WiFi, 3G, LTE, Bluetooth, NFC, and Zigbee). The components that may be included in the computing device 11000 may be implemented by hardware, software, or a combination of both hardware and software which include at least one integrated circuit specialized in a signal processing or an application.

The methods according to the embodiments of the present invention may be implemented in the form of program instructions to be executed through various computing devices, thereby being recorded in a computer-readable medium. In particular, a program according to an embodiment of the present invention may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the computing device 11000 through a file provided by a file distribution system. For example, a file distribution system may include a file transmission unit (not shown) that transmits the file according to the request of the computing device 11000.

The above-mentioned device may be implemented by hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented by using at least one general purpose computer or special purpose computer, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and at least one software application executed on the operating system. In addition, the processing device may access, store, manipulate, process, and create data in response to the execution of the software. For the further understanding, some cases may have described that one processing device is used, however, it is well known by those skilled in the art that the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations, such as a parallel processor, are also possible.

The software may include a computer program, a code, and an instruction, or a combination of at least one thereof, and may configure the processing device to operate as desired, or may instruct the processing device independently or collectively. In order to be interpreted by the processor or to provide instructions or data to the processor, the software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or in a signal wave to be transmitted. The software may be distributed over computing devices connected to networks, so as to be stored or executed in a distributed manner. The software and data may be stored in at least one computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions to be executed through various computing mechanisms, thereby being recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, independently or in combination thereof. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known to those skilled in the art of computer software so as to be used. An example of the computer-readable medium includes a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specially configured to store and execute a program instruction such as ROM, RAM, and flash memory. An example of the program instruction includes a high-level language code to be executed by a computer using an interpreter or the like as well as a machine code generated by a compiler. The above hardware device may be configured to operate as at least one software module to perform the operations of the embodiments, and vise versa.

Although the above embodiments have been described with reference to the limited embodiments and drawings, however, it will be understood by those skilled in the art that various changes and modifications may be made from the above-mentioned description. For example, even though the described descriptions may be performed in an order different from the described manner, and/or the described components such as system, structure, device, and circuit may be coupled or combined in a form different from the described manner, or replaced or substituted by other components or equivalents, appropriate results may be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

METHOD, SYSTEM, AND COMPUTER-READABLE MEDIUM FOR DERIVING TASK RESULTS BY REFLECTING RELIABILITY OF WORKER PROCESSING WORK COLLECTED THROUGH CROWDSOURCING

Information

Publication Number

Date Filed

Date Published

Inventors

Original Assignees

CPC

International Classifications

Abstract

Description

Claims

Priority Claims (1)

PCT Information