Patent Application
20160180727
Some grading systems utilize intelligent character recognition (ICR) to convert hand-written or hand-marked student work into a digital format. The student work is typically scanned and the images are sent to a classification engine which determines their most probable meaning. However, ICR is a challenging task often requiring teachers to manually confirm the ICR results before assigning the student a final evaluation. This manual confirmation is labor and time consuming.
In addition, to avoid false positives, existing grading systems typically do not classify an answer as correct unless the probability associated with the answer exceeds a certain threshold value. However, this often leads to an unclear understanding of a student's (or a group of students) understanding of the underlying subject matter due to an excess of false negatives.
This disclosure is not limited to the particular systems, methodologies or protocols described, as these may vary. The terminology used in this description is for the purpose of describing the particular versions or embodiments only, and is not intended to limit the scope.
As used in this document, the singular forms “a,” “an,” and “the” include plural reference unless the context clearly dictates otherwise. Unless defined otherwise, all technical and scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art. All publications mentioned in this document are incorporated by reference. All sizes recited in this document are by way of example only, and the invention is not limited to structures having the specific sizes or dimension recited below. As used herein, the term “comprising” means “including, but not limited to.”
In an embodiment, a system for scoring an assessment may include a computing device having an image capture device and an image recognition engine, and a computer-readable storage medium. The computer-readable storage medium may have one or more programming instructions that, when executed, cause the computing device to capture, by the image capture device, an image of a completed assessment. The completed assessment may include one or more handwritten responses of a student. The computer-readable storage medium may have one or more programming instructions that, when executed, cause the computing device to, for each question of the completed assessment, parse, by the image recognition engine, the image to identify the question and the handwritten response associated with the question, determine a correct answer to the question, perform, by the image recognition engine, an image recognition analysis on the assessment to determine a confidence value associated with the determined correct answer for the question, determine a question score for the question based on the determined confidence value and a point value associated with the correct answer, and determine a total score for the assessment by summing the determined question scores. The computer-readable storage medium may have one or more programming instructions that, when executed, cause the computing device to assign the total score to the student, and generate a report comprising one or more of the following: one or more of the question scores, one or more of the confidence values, one or more of the point values, and the total score.
In an embodiment, a method of scoring an assessment may include capturing, by an image capture device, an image of a completed assessment that includes one or more handwritten responses of a student, and for each question of the completed assessment, parsing, by an image recognition engine, the image to identify the question and the handwritten response associated with the question, determining a correct answer to the question, performing, by the image recognition engine, an image recognition analysis on the assessment to determine a confidence value associated with the determined correct answer for the question, determining a question score for the question based on the determined confidence value and a point value associated with the correct answer, and determining a total score for the assessment by summing the determined question scores. The method may include assigning the total score to the student, and generating a report comprising one or more of the following: one or more of the question scores, one or more of the confidence values, one or more of the point values, and the total score.
The following terms shall have, for purposes of this application, the respective meanings set forth below:
An “assessment” refers to an instrument for testing one or more student skills that requires one or more handwritten answers. An assessment may be a quiz, a test, an essay, or other type of evaluation. In an embodiment, an assessment may be an instrument embodied on physical media, such as, for example, paper.
A “computing device” or “electronic device” refers to a device that includes a processor and non-transitory, computer-readable memory. The memory may contain programming instructions that, when executed by the processor, cause the computing device to perform one or more operations according to the programming instructions. As used in this description, a “computing device” or “electronic device” may be a single device, or any number of devices having one or more processors that communicate with each other and share data and/or instructions. Examples of computing devices or electronic devices include, without limitation, personal computers, servers, mainframes, gaming systems, televisions, and portable electronic devices such as smartphones, personal digital assistants, cameras, tablet computers, laptop computers, media players and the like.
An “image capture device” refers to image sensing hardware, logic and/or circuitry that is capable of optically viewing an object, such as an assessment or other document, and converting an interpretation of that object into one or more electronic signals. Examples of an image capture devices include without limitation, cameras, scanners and/or the like.
An “image recognition engine” refers to hardware, logic, memory and/or circuitry that is capable of parsing an image and converting an interpretation of at least a portion of the image content into one or more electronic signals for analysis.
Grading engines typically operate on a question-by-question level. That is, did a student answer a specific question correctly or incorrectly? However, educators may also be interested in understanding the answers at a higher level. For example, an educator may want insight into: (1) a student's overall score on an exam; (2) whether a class as a whole is mastering a subject area; and (3) whether students had difficulty answering one question in particular.
In an embodiment, a client computing device 102a-N may be used by an educator to access, view, change, modify, update and/or enter one or more student assessment results. A client computing device 102a-N may include, without limitation, a laptop computer, a desktop computer, a tablet, a mobile device and/or the like.
An assessment computing device 104 may be a computing device configured to receive and/or process one or more student assessments, and may include, without limitation, a laptop computer, a desktop computer, a tablet, a mobile device and/or the like. As illustrated by
The image capture device 108 may be in communication with the image recognition engine 110. The image capture device 108 may provide a captured image as input to the image recognition engine. An image capture device 108 may be a device configured to capture an image of an assessment such as, for example, a camera, a scanner and/or the like.
In an embodiment, an image recognition engine 110 may be comprised of logic 112, circuitry 114 and/or memory 116. The logic 112 and/or circuitry 114 of the image recognition engine 110 may cause the image recognition engine to parse a received image to identify an assessment question or response, or perform image recognition analysis on the captured image as described in more detail below. Image recognition analysis may be used by an image recognition engine to determine a likelihood that one or more assessment responses are correct, incorrect and/or the like. Memory 116 may be used to store received images, determined likelihoods and/or other information. Examples of memory may be read only memory (ROM), random access memory (RAM) and/or another tangible, non-transitory computer-readable medium.
As illustrated by
In an embodiment, the assessment may be provided to a student, and the student may complete 202 the assessment. A student may complete 202 at least a portion of the assessment by providing a handwritten answer for at least a portion of the assessment. For instance, an assessment may evaluate a student's math skills by asking the student to complete 202 certain mathematical equations. A student may complete 202 this assessment by writing answers to the equations on the assessment.
In an embodiment, the assessment may be provided as input to an educational assessment system. An educational assessment system may be a software application executing on or hosted by one or more computing devices that grades or otherwise evaluates one or more assessments. An image capture device of an educational assessment system may capture 204 an image of a completed assessment. For instance, an educational assessment system may capture 204 an image of a completed assessment through scanning, taking a picture and/or any other capturing technique.
In various embodiments, the system may parse 206 a captured image to identify a question and/or a handwritten response associated with the question. An image recognition engine may be used to parse a captured image to identify a question and/or a response to a question.
In an embodiment, the system may identify 208 a correct answer for one more questions of an assessment. A system may identify 208 one or more correct answers using an answer key associated with an assessment, receiving correct answers to one or more questions from an educator, or otherwise accessing one or more correct answers for the assessment by, for example, retrieving such answers from a computer-readable storage medium.
The educational assessment system may perform 210 an image recognition analysis, such as Intelligent Character Recognition (ICR), on a received completed assessment. In various embodiments, an image recognition engine of the system may perform image recognition analysis. In an embodiment, the image recognition analysis may be used to analyze or more of a student's written answers.
In various embodiments, the image recognition analysis may be performed to an assessment to generate one or more confidence values associated with one or more possible answers. A confidence value may reflect a likelihood that a specific handwritten answer is the correct answer. For example, the possible answer outcomes for a math assessment may be integers between 0 and 9. Table 1 illustrates example confidence information corresponding to Question 1 (Q1) of the assessment illustrated by
In an embodiment, a confidence value may be generated for each potential answer. For example, in arithmetic in which answers range from 0 to 9, a confidence value for each potential character may be generated using one or more models, statistical approaches and/or the like. For example, one or more confidence values may be generated using one or more neural networks, Bayesian methods and/or the like. For instance, the system may identify a response to an assessment question, and may use the identified response as input to a neural network. The output of the neural network may be a confidence value associated with one or more possible answers for the question.
As an example,
In various embodiments, a neural network, and therefore a confidence value, may be tailored to a particular student. For example, a grading system may have access to historical assessment information for a student, such as, for example, past handwriting samples, indications of academic ability of the student in one or more areas, such as, for example, grades and/or the like.
Table 1 also illustrates a point award policy associated with each possible answer. A point award policy may indicate a number of points or other score to award to a student for the corresponding answer. In the example illustrated by Table 1, the correct answer is ‘7’ and is worth a total of 5 points. The other answers are assigned a value of zero points, indicating that no partial credit is given for wrong answers.
In an embodiment, a point award policy may be defined by an educator. For instance, an educator may specify the point award policy for a question, an assessment and/or the like in connection with creating an assessment. In an embodiment, a point award policy may be specific to a question, an assessment and/or the like.
In an embodiment, the system may determine 212 a question score for one or more questions of an assessment. A question score may refer to a score for an individual question. To avoid false positives, existing grading systems typically do not classify an answer as correct unless the likelihood probability associated with the answer exceeds a certain threshold value. However, this often leads to an unclear understanding of a student's (or a group of students) understanding of the underlying subject matter and often underestimates the true score.
The system may determine 212 a score for one or more questions based on the confidence value associated with the correct answer and a point value associated with the correct answer.
As an example, a student may complete the assessment of
7×1=7 Q1
2×4=6 Q2
3×3=9 Q3
The confidence values for the correct answers to Q2 and Q3 may be illustrated by Tables 2 and 3, respectively:
Because the likelihood probability associated with the correct answer for Q2 and Q3 may be below the threshold value, existing grading systems may classify each of these answers as incorrect and award no points for these answers. As such, a known grading system may award a total of 5 points out of 15 points to the student, when the student's true score is 10 points out of 15 points.
The described system may determine a question score for one or more questions by multiplying the confidence value associated with the correct answer by the point value associated with the correct answer. For example, referring to the question corresponding to Table 1, a question value may be determined by multiplying the confidence value associated with the answer ‘7’ (0.98) by the total points associated with the answer ‘7’ (5) to yield a question score of 4.9 points out of five possible points.
As another example, referring to Q2 and Table 2, a question score may be determined by multiplying the confidence value associated with the answer ‘8’ (0.15) by the total points associated with the answer ‘8’ (5) to yield a question score of 0.75 points out of five possible points.
As yet another example, referring to Q3 and Table 3, a question score may be determined by multiplying the confidence value associated with the answer ‘9’ (0.85) by the total points associated with the answer ‘9’ (5) to yield a question score of 4.25 points out of five possible points.
In an embodiment, partial credit may be available for one or more questions. For example, Table 4 illustrates example confidence values and point values for a question where partial credit is available and the correct answer is ‘4.’
As illustrated by Table 4, a correct answer of ‘4’ is worth 5 points. But an incorrect answer of ‘9’ is worth 2 points. When partial credit is available for a question, the system may determine a question score for the question by, for each answer for which full or partial credit is given, multiplying the confidence value and the point values associated with a particular answer, and then summing the values. For instance, with respect to the question corresponding to Table 4, a question score may be determined by: (0.75*5)+(0.25*2)=3.75+0.50=4.25 points out of 5 total points.
In certain embodiments, the system may determine 214 a total score for an assessment. A total score may be determined 214 by summing the question scores associated with one or more questions of an assessment. For example, if an assessment includes Q1, Q2 and Q3 as described above, the total score for the assessment may be equal to 9.9 points out of 15 points (i.e., 4.9 points+0.75 points+4.25 points).
In an embodiment, the system may assign 216 the total score to the corresponding student. The system may assign 216 the total score to the student by causing one or more records associated with the student and/or the assessment to reflect the total score, notifying the student and/or the educator of the total score, sending the total score to the student and/or the educator and/or the like.
In an embodiment, the system may determine 218 a group score for an assessment. A group score may be a score associated with a plurality of students such as, for example, a class, a subgroup of a class, a school and/or the like. A group score may be an average of the total scores of the students in the group. For instance, Table 5 illustrates example total scores for students in a class who completed the example assessment above.
The group score associated with the group illustrated by Table 5 may be ((9.90+12.20+10.20)/3)=10.77.
In various embodiments, the system may generate 220 a report. A report may include one or more of a confidence value and/or point award policy for one or more questions, a question score for one or more questions and/or students, a total score for one or more students, a group score and/or the like. The report may include one or more graphs, charts or other visual representations.
In an embodiment, the system may present 222 a report to an educator. The report may be presented 222 to an educator via a graphical user interface, email, and/or the like. The educator may have an opportunity to change or override one or more scores illustrated in a report.
A controller 520 interfaces with one or more optional non-transitory computer-readable storage media 525 to the system bus 500. These storage media 525 may include, for example, an external or internal DVD drive, a CD ROM drive, a hard drive, flash memory, a USB drive or the like. As indicated previously, these various drives and controllers are optional devices.
Program instructions, software or interactive modules for providing the interface and performing any querying or analysis associated with one or more data sets may be stored in the ROM 510 and/or the RAM 515. Optionally, the program instructions may be stored on a tangible, non-transitory computer-readable medium such as a compact disk, a digital disk, flash memory, a memory card, a USB drive, an optical disc storage medium and/or other recording medium.
An optional display interface 530 may permit information from the bus 500 to be displayed on the display 535 in audio, visual, graphic or alphanumeric format. Communication with external devices, such as a printing device, may occur using various communication ports 540. A communication port 540 may be attached to a communications network, such as the Internet or an intranet.
The hardware may also include an interface 545 which allows for receipt of data from input devices such as a keyboard 550 or other input device 555 such as a mouse, a joystick, a touch screen, a remote control, a pointing device, a video input device and/or an audio input device.
It will be appreciated that the various above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications or combinations of systems and applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.
