Patent Application
20220309940
This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2021-049644, filed Mar. 24, 2021, the entire contents of which are incorporated herein by reference.
The present disclosure relates generally to an automatic scoring method, a server, an automatic scoring system and a recording medium.
Online classes have been introduced in recent years. In the online classes, for example, examinations are also conducted online. A system capable of automatic scoring for the examinations has been proposed (Jpn. Pat. Appln. KOKAI Publication No. 2019-61189).
An automatic scoring method by processor of an aspect includes receiving information including an answer to a question, scoring the answer according to one or more evaluation items preset for the question, and visually distinguishing a result of the scoring in accordance with an evaluation item that has matched the answer.
One embodiment of the present disclosure will be described with reference to the drawings.
The server 10 includes a processor 11, a ROM 12, a RAM 13, a storage 14 and a communication module 15. These are connected to each other via a system bus 19.
The processor 11 may be an integrated circuit including a central processing unit (CPU) and the like. The ROM 12 records information for use in operating the processor 11 and the like. The RAM 13 is a main storage device to operate the processor 11 and the like. The storage 14 stores various programs such as server control programs used in the processor 11 and arithmetic operation programs for performing various arithmetic operations, parameters and the like. The server control programs include an automatic scoring program. The processor 11 controls the operation of the server 10 in accordance with the programs stored in the storage 14. As the processor 11, a processor other than the CPU, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) and a graphic processing unit (GPU) may be used. The communication module 15 includes a circuit that communicates with an external communication network such as the network 30.
The terminals 20a and 20b may be electronic devices such as a personal computer (PC), a tablet and a smartphone. The terminals 20a and 20b may also be scientific calculators having a communication function. The terminal 20a is operated by an examination marker such as a teacher. The terminal 20b is operated by an examinee such as a student. The configuration of the terminal 20a will be described on the assumption that the terminal 20a has the same configuration as that of the terminal 20b. In the following descriptions, the marker may be distinguished as a first user from the examinee as a second user when necessary.
The terminal 20a includes a CPU 21, a ROM 22, a RAM 23, a storage 24, an input device 25, a display 26 and a communication module 27. These are connected to each other via a system bus 29. Note that the terminals 20a and 20b do not necessarily have the same configuration.
The CPU 21 is a processor that controls various operations of the terminal 20a. The ROM 22 records a start program or the like of the terminal 20a. The RAM 23 is a main storage device for the CPU 21 and the like. The storage 24 stores various programs such as a terminal control program used in the CPU 21, parameters and the like. The CPU 21 controls the operation of the terminal 20a by executing various programs in response to an input signal from the input device 25 and a reception signal from the communication module 27. The programs may be downloaded from a web server (not shown) into the storage 24 via the network 30 and the communication module 27. The communication module 27 includes a circuit to communicate with an external communication network such as the network 30.
The input device 25 includes a keyboard, a mouse, a touch panel and the like. In response to a user's operation performed via the input device 25, a signal indicating the contents of the user's operation is input to the CPU 21 via the system bus 29.
The display 26 is, for example, a liquid crystal display and an organic EL display. The display 26 may be provided integrally with the terminal 20a or may be provided separately from the terminal 20a. Various images are displayed on the display 26.
As one example, a user designates the address of the server 10 in a web browser running on the terminal 20a. At this time, a display screen for a web application stored in the server 10 is displayed on the web browser on the terminal 20a. A request is issued to the server 10 in response to an operation to be performed through the input device 25 on the display screen. This operation includes, for example, an operation related to the scoring of an examination taken by the user of the terminal 20b. The server 10 performs a process corresponding to the request and returns a result of the process to the terminal 20a as a response. In response from the server 10, the terminal 20a makes displays or the like which corresponds to the user's operation. The system 1 thus achieves a function as a web application for examinations and the like based on a program running on the web browser of the terminal 20a and an arithmetic operation program of the server 10. Similarly, the system 1 achieves a function as a web application for examinations and the like based on a program running on the web browser of the terminal 20b and an arithmetic operation program of the server 10.
The web application can be used in, for example, mathematics classes in school education where information and communication technology (ICT) is increasingly developing.
The screen 26a displayed on the display 26 of the terminal 26a includes an upper stage area 100a and a lower stage area 100b. The upper stage area 100a is displayed on the upper side of the screen 26a. The upper stage area 100a is narrower than the lower stage area 100b. A new paper preparation icon 100c is displayed in the upper stage area 100a. An answer paper preparation button 100d is displayed in the upper stage area 100a. The lower stage area 100b is located below the upper stage area 100a in the screen 26a. Hereinafter, the lower stage area 100b will also be referred to as paper 100. Various types of “sticky note” 101 may be displayed on the paper 100. The sticky note 101 is a display area for displaying various items of information concerning the web application. For example, the sticky note 101 includes a mathematical sticky note for creating a numerical expression, a graph sticky note for creating a graph, a table sticky note for creating a table, a figure sticky note for creating a figure, and a comment sticky note for making comments, and the like. The sticky note 101 may be a floating object. The floating object is an object (display body) to be displayed on the screen, and at least the display position thereof can be changed in response to user operation.
In the present embodiment, the system 1 can create various types of sticky note starting from a blank sheet of paper 100. Note that the same paper 100 can be displayed on the terminal 20b as that on the terminal 20a. However, the answer paper preparation button 100d may not be displayed on the paper 100 of the terminal 20b.
Below is a description of a flow of a series of steps in the system 1.
The scoring criteria information includes information such as “question name,” “answer type,” “model answer,” “scoring criteria” and “comments.” The “question name,” “answer type,” “model answer,” “scoring criteria” and “comments” can be stored for each question. If there are three questions, the “question name,” “answer type,” “model answer,” “scoring criteria” and “comments” can be stored three by three.
The “question name” is information representing the name of a question. The “question name” is uniquely assigned to each question in the same examination. The “question name” may be assigned by the first user entering a text or may be selected by the first user from among several alternatives.
The “answer type” is information of a type of sticky note that can be used to answer the corresponding question. When the “answer type” is “a mathematical expression” in a mathematical question, it is set such that a sticky note other than a mathematical expression sticky note cannot be used to answer the corresponding question.
The “model answer” is information of a model answer as one of information items of evaluation items assumed by the first user such as a teacher. When the answer of the second user such as a student matches the model answer described in the “model answer,” the answer is determined to be a correct answer.
The “scoring criteria” is information on evaluation items that serve as a criterion for automatic scoring. Specifically, the “scoring criteria” is information on evaluation items such as how many scores are given to the second user such as a student as reference scores when the answer of the second user matches the model answer, how many scores are deducted from the reference scores when the answer does not match the model answer but is mathematically equivalent, and how many scores are added when the answer does not match the model answer and is not mathematically equivalent, and can freely be described by the first user such as a teacher. “Mathematically equivalent” means that the answer is equivalent to the model answer even if it differs from the model answer only in its expression form, and indicates that the answer matches any evaluation item other than the model answer. For example, when the model answer is in the decimal format of “0.25” and the second user's answer is in the fractional format of “¼,” the latter answer is determined to be “mathematically equivalent” to the former answer. The “scoring criteria” may be described, for example, by combining an evaluation expression and a variable for evaluating whether an answer defined in advance in a web application is mathematically equivalent to the model answer. In addition, exclusion conditions may be described as the “scoring criteria.” An answer that meets the exclusion conditions may be determined to be neither identical nor mathematically equivalent to the model answer. The format of information described in the “scoring criteria” is not limited to a specific one as long as the format can be used for automatic scoring in the server 10.
The “comments” are text information of comments added by a teacher and the like. The entry of “comments” may be omitted.
The process of
In step S1, the first user such as a teacher who is preparing an answer paper operates the input device 25 of the terminal 20a to select the new paper preparation icon 100c. In step S2, the terminal 20a transmits a request to prepare new paper 100 to the server 10. In step S3, the server 10 prepares new paper 100 and transmits it to the terminal 20a.
In step S4, the first user performs an operation of adding a sticky note as required. This operation may be, for example, an operation of selecting a sticky note additional icon (not shown) displayed on the new paper 100 or an operation of selecting a sticky note additional item from the menu displayed by right clicking the mouse. After adding the sticky note, the first user enters a supplementary comment or the like of the question in the sticky note as necessary. In step S5, the terminal 20a transmits the sticky note information, which is input by the first user, to the server 10. The server 10 associates the information of the sticky note with the paper 100. The sticky note information includes, for example, information such as positional information of the sticky note on the paper 100 and information such as text written onto the sticky note. The process of steps S4 to S5 may be repeated until the first user selects an answer paper preparation button 100d in step S6.
In step S6, the first user finishes writing supplementary comments or the like onto the sticky note and then operates the input device 25 to select the answer paper preparation button 100d. In step S7, the terminal 20a transmits an answer paper preparation request to the server 10. In step S8, the server 10 generates a dialog for preparing an answer paper and transmits the generated dialog to the terminal 20a. The terminal 20a displays the received dialog on the paper 100.
In step S9, the first user sets a scoring criteria. Specifically, the first user selects a scoring criteria information file in the dialog 201. After that, the first user selects the “Next” button 201d. In step S10, the terminal 20a transmits to the server 10 information of the selected scoring criteria information file such as the name of the scoring criteria information file. In step S11, the server 10 adds scoring criteria information to the paper 100 based on the information of the selected scoring criteria information file. Specifically, the server 10 associates the scoring criteria information file selected by the first user with the paper 100. After that, the server 10 transmits a dialog for the next setting to the terminal 20a. The terminal 20a displays the received dialog on the paper 100.
The server 10 transmits a dialog 202 for setting an answer deadline shown in
In step S12, the first user sets the answer deadline in the dialog 202. After that, the first user selects the “Next” button 202c. In step S13, the terminal 20a transmits information of the set answer deadline information to the server 10. In step S14, the server 10 adds the information of the answer deadline to the paper 100. Specifically, the server 10 associates the information of the answer deadline with the paper 100. After that, the server 10 generates a URL unique to each paper 100. Then, the server 10 transmits a dialog containing the generated URL to the terminal 20a. The terminal 20a displays the received dialog on the paper 100.
The server 10 transmits to the terminal 20a a dialog 203 for notifying the terminal 20a of the completion of preparation of the answer paper shown in
In step S15, the first user confirms the URL, copies the URL as necessary, and distributes electronic mail to the terminal 20b of the corresponding second user. The electronic mail may be distributed by the server 10. After these processes, the first user selects the close button 203c. In step S16, the terminal 20a transmits to the server 10 a notification of the end of the process of preparing the answer paper. Thus, the server 10 recognizes the end of the process. Upon completion of the process, the display of the paper 100 on the terminal 20a may be ended.
In step S101, the second user such as a student operates the terminal 20b to supply the web browser with the URL designated in advance by the first user such as a teacher via email or the like. In step S102, the terminal 20b gains access to the server 10 according to the URL. Note that a login process such as input of an ID and a password may be performed to gain access to the URL.
The server 10 determines whether the status of paper of the URL is a status error. For example, the server 10 determines whether the answer deadline set in the paper of the URL has expired. The server 10 also determines whether the submission status of the second user to the paper of the URL is “submitted.” As will be described later, when the second user answers a question and “submits” the answer, the server 10 sets the submission status of the second user from “not submitted” to “submitted.” The server 10 determines the submission status of the second user who has gained access to the server 10. When the answer deadline has expired or when the submission status of the second user is “submitted,” the server 10 determines that the status of paper of the URL is a status error. If the server 10 determines the status as a status error, it notifies the terminal 20b of the error in step S103, and then terminates the process. This error includes, for example, a message indicating that no answer can be made to the paper of the URL. On the other hand, when the answer deadline has not expired and the submission status of the second user is “not submitted,” the server 10 determines that the status of paper of the URL is not a status error. If the server 10 determines that it is not a status error, it transmits the paper of the URL to the terminal 20b in step S104. The terminal 20b displays the paper on the web browser.
The second user can add a sticky note for each question to the answer paper 300. The second user can prepare a sticky note for the answer when a question is answered. In
The user can perform various input operations corresponding to the added sticky note, such as preparing a numerical expression, preparing a graph, preparing a table and preparing a figure. The user can also cause the server 10 to execute the calculation of a numerical expression. The result of the calculation of the numerical expression is displayed as the answer to a question, for example, at the lower right of the sticky note. In addition, the sticky notes 301, 302 and 303 include check columns 301a, 302a and 303a. For example, when the sticky note 303 is selected, a checkmark is displayed in the corresponding check column 303a. Not only a checkmark is displayed, but a sticky note that is selected and a sticky note that is not selected may be distinguished by another display method or the like. Each sticky note may be provided with an answer label. When the second user selects a drop-down list provided in an answer, a list of question names available for the corresponding sticky note is displayed. The second user selects a desired question name from the list and thus the sticky note is provided with an answer label. In the example shown in
In step S105, the second user adds a sticky note to the paper 300 to answer a question. In step S106, the terminal 20b transmits the added sticky note information to the server 10. The server 10 associates information of the sticky note with the paper 300 for each second user. The sticky note information includes, for example, identification information and positional information of each sticky note on the paper 300 and information of an answer entered in the sticky note. In step S107, the second user provides an answer label for the sticky note. In step S108, the terminal 20b transmits information of an answer label to the server 10. The server 10 associates information of the answer label with information of the sticky note for each second user. The information of the answer label includes, for example, information of a question name represented by the answer label. The process of steps S105 to S108 may be repeated until the second user selects the answer submission button 300c in step S109.
The second user prepares a sticky note, writes an answer, and provides the sticky note with an answer label. In step 109 that is a desired timing of submission of the answer, the second user selects the answer submission button 300c. In step S110, the terminal 20b transmits to the server 10 a notification that an answer is submitted.
The server 10 determines whether there is an unanswered question by totaling the information of answer labels associated with the paper 300 indicating that the answer is submitted. When the server 10 has determined that there is an unanswered question, it transmits a warning dialog to the terminal 20b in step S111. The warning dialog includes, for example, a message for confirming whether the submission process can be continued even though there is an unsubmitted answer. The warning dialog may also include a button for accepting an operation to continue submitting an answer and a button for accepting an operation to cancel submitting an answer. In this case, the server 10 may determine whether to continue or cancel the process of submitting an answer according to a button selected by the operation of the input device 25 of the second user. When the server 10 has determined that the answer submission process is to be continued, the process proceeds to step S112. If it has determined that the answer submission process is canceled, the process returns to step S105. If it has determined that there is no unanswered question, the process proceeds to step S112.
In step S112, the server 10 changes a submission status of the second user who is the user of the terminal 20b requesting the submission of the answer to the paper 300 to “Submitted.” Then, the server 10 generates a URL unique to each second user associated with the paper 300 that has been submitted. Then, the server 10 transmits a dialog containing the generated URL to the terminal 20b. The terminal 20b displays the received dialog on the paper 300. Note that the dialog displayed on the terminal 20b in the process of step S112 is similar to the dialog shown in
In step S113, the server 10 copies the URL and distributes electronic mail to the terminal 20a of the first user. The first user can confirm the paper 300 answered by the second user, using the URL transmitted in step 113, for example, for scoring.
In step S201, the first user such as a teacher operates the terminal 20a to supply the web browser with the URL designated in advance from the server 10 via email or the like. In step S202, the terminal 20a gains access to the server 10 according to the URL. Note that a login process such as input of an ID and a password may be performed to gain access to the URL.
In step S202, the server 10 calls the paper 300 of the input URL. Then, the server 10 extracts the second user's answer from the information of a sticky note associated with the paper 300. After that, the server 10 compares the second user's answer with the scoring criteria information associated with the paper 300 to make automatic scoring. When the automatic scoring is completed, the server 10 transmits paper including the result of the automatic scoring to the terminal 20a. The terminal 20a displays the received paper on the web browser.
In step S302, the server 10 acquires a question name of a question label and a reference score corresponding to the question name from the scoring criteria information, and gives the reference score to the second user's score in the question label. As described above, the reference score is a score given when the second user's answer is correct.
In step S303, the server 10 determines whether the selected second user's answer matches a model answer in the corresponding question name. When the server 10 determines in step S303 that the second user's answer matches the model answer, the process proceeds to step S304. If it is determined in step S303 that the answer by the second user does not match the exemplary answer, the process proceeds to step S305.
In step S304, the server 10 sets the background color of a sticky note of the corresponding question label to “green.” The process proceeds to step S310. The background color is the background color of a sticky note included in the paper displayed in the terminal 20a as a result of the automatic scoring. The background color is changed according to the result of the automatic scoring based on the scoring criteria information. That is, when the second user's answer matches the model answer, the background color is set to “green.” Note that the reference score is given as the second user's score of the corresponding question label.
In step S305, based on the scoring criteria information, the server 10 determines whether the second user's answer is mathematically equivalent to the model answer. In step S305, when the second user's answer matches the exclusion conditions described in the scoring criteria information, the server 10 may determine that the second user's answer is not mathematically equivalent to the model answer. In step S305, when the server 10 determines that the second user's answer is mathematically equivalent to the model answer, the process proceeds to step S306. In step S305, when the server 10 determines that the second user's answer is not mathematically equivalent to the model answer, the process proceeds to step S308.
In step S306, the server 10 subtracts the score of the corresponding question label from the reference score according to the evaluation items of the scoring criteria.
In step S307, the server 10 sets the background color of a sticky note of the corresponding question label to “yellow.” After that, the process proceeds to step S310. That is, when the second user's answer does not match the model answer but is mathematically equivalent thereto, the background color is set to “yellow.”
In step S308, the server 10 sets the score of the corresponding question label to 0. In step S308, the server 10 may add scores from 0 depending on the evaluation items of the scoring criteria.
In step S309, the server 10 sets the background color of a sticky note of the corresponding question label to “red.” After that, the process proceeds to step S310. That is, when the second user's answer is wrong, the background color is set to “red.”
In step S310, the server 10 determines whether to end the automatic scoring. For example, when the scoring for all the question labels is completed, the server determines that the automatic scoring is ended. When the server 10 determines in step S310 that the automatic scoring is not ended, the process returns to step S301. When the server 10 determines in step S310 that the automatic scoring is ended, the process proceeds to step S311.
In step S311, the server 10 calculates the total of scores given for each question label.
In step S312, the server 10 prepares paper of results of the automatic scoring. Then, the server 10 ends the process of
On the paper 400 of the automatic scoring results, sticky notes for answers added to the answer paper 300 by the user and their corresponding sticky notes are displayed. In
A background color is set for each of the sticky notes 401, 402 and 403. The background color is set according to the result of automatic scoring. As described above, the “green” background color is set to the sticky notes when the answer matches the model answer, the “yellow” background color is set thereto when the answer does not match the model answer but is mathematically equivalent to the model answer, and the “red” background color is set thereto when the answer does not match the model answer or is not mathematically equivalent thereto.
In addition, a scoring criteria sticky note 404 is displayed on the paper 400 of the result of automatic scoring. The scoring criteria sticky note 404 is a list of evaluation items of the scoring criteria to each of the questions associated with the corresponding answer paper 300.
The scoring criteria sticky note 404 includes a model answer 404a, an evaluation item 404b and a total score 404c. The model answer 404a is a “model answer” recorded in the scoring criteria information about the corresponding question. The evaluation item 404b is a text representing each of the evaluation items of the “scoring criteria” recorded in the scoring criteria information about the corresponding question. The total score 404c are the total scores for each question calculated as a result of the automatic scoring. The numerical value displayed at the total score 404c can be corrected by the first user.
In the scoring criteria sticky note 404, the field of the model answer 404a is colored according to the result of the automatic scoring. For example, when the answer matches the model answer, the field of the model answer 404a is colored “green.” If the answer does not match the model answer but is mathematically equivalent to the model answer, the field of the model answer 404a is colored “yellow.” For example, if the answer matches an evaluation item other than the model answer, it is determined to be mathematically equivalent thereto. If the answer does not match the model answer or is not mathematically equivalent thereto, the field of the model answer 404a is colored “red.” That is, the field of the model answer 404a is colored in the same color as the background color of each of the sticky notes 401, 402 and 403. Note that the color scheme shown here is an example and may be changed as appropriate.
Let us now return to
The first user selects the scoring completion button 400c in step S208 when the scoring result is desired by the first user. In step S209, the terminal 20a transmits a notification of completion of the scoring to the server 10. Accordingly, the server 10 recognizes the end of the process. Upon completion of the process, the display of the paper 400 on the terminal 20a may be ended. The paper 400 for which scoring is completed may be set so as not to be edited.
As described above, according to the present embodiment, the answers submitted by the second user such as a student are automatically scored according to the scoring criteria information including predetermined evaluation items. As a result of the automatic scoring, in accordance with which of the evaluation items the answers match, an answer which matches the model answer, an answer which does not match the model answer but is substantially equivalent to the model answer, and an answer which does not match the model answer and is not substantially equivalent thereto are displayed separately from one another. Thus, the first user such as a teacher can intuitively recognize a type of scoring criteria based on which the automatic scoring for each question is performed. In particular, questions colored “yellow” are scored based on evaluation items prepared in advance based on the first user's idea of scoring, rather than on an absolute criteria of whether or not the answers to the questions match the model answers. Thus, the first user has only to check mainly the questions colored “yellow.” Therefore, the first user's burden of confirming the result of automatic scoring by visual observation is reduced and so is the error of the confirmation, with the result that the quality of automatic scoring is improved.
[Modification]
A modification to the embodiment will be described below. The embodiment is directed to an example of automatic scoring for answers to mathematics questions. In contrast, the method of the modification may be applied to automatic scoring for answers to various types of question other than mathematics questions.
On the paper 500 of the automatic scoring results, an answer sticky note added to the answer paper 300 by the user and its corresponding sticky note are displayed.
The background color of the sticky note 501 is set according to the result of automatic scoring. For example, the background color of “green” is set to a sticky note when the answer matches the model answer, the background color of “yellow” is set to a sticky note when the answer semantically matches the model answer but has a problem in the expression form, and the background color of “red” is set to a sticky note when the answer does not semantically match the model answer. The problem in the expression form is answers including errors, though the answers are assumed to have the same meaning as the model answers, such as “words to be written in small letter contain capital letters,” “verbs have different forms” and “words contain misspellings” as shown in
In the modification, the field of an evaluation item that has matched an answer is colored. For example,
In the embodiment, the server 10 performs automatic scoring according to the scoring criteria prepared by the first user and colors the answers according to the results of the automatic scoring. On the other hand, the server 10 may perform automatic scoring by, for example, artificial intelligence that have learned the scoring criteria prepared by the first user. In this case, too, the server 10 differentially displays the answers on the display according to the automatic scoring results.
Also, in the embodiment, the background colors of sticky notes vary among an answer that matches the model answer, an answer that does not match the model answer but is substantially equivalent thereto, and an answer that does not match the model answer or is not substantially equivalent thereto. However, these answers have only to be displayed so as to be distinguished visually by the first user. For example, the display may be made to change the shape of the frame of a sticky tag, to change the color of the frame, or to change the thickness of the frame.
Also, in the embodiment, the background colors of sticky notes vary among an answer that matches the model answer, an answer that does not match the model answer but is substantially equivalent thereto, and an answer that does not match the model answer or is not substantially equivalent thereto. In contrast, the background color of a sticky note may be colored only for the answer that does not match the model answer but is substantially equivalent thereto.
The present disclosure is not limited to the embodiment or modification described above, but their structural elements can be modified in different ways without departing from the spirit of the invention when the disclosure is reduced to practice. The embodiment and modification can possibly be combined as appropriate, and an advantageous effect can be obtained from the combination. The embodiment includes inventions in various stages, and various inventions can be extracted from appropriate combinations of structural elements of the embodiment. Even though some of the structural elements are deleted from the embodiment, if the problem is to be solved by the invention and the advantageous effects are obtained, a configuration from which the structural elements are deleted can be extracted as an invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2021-049644 | Mar 2021 | JP | national |
