The present invention relates generally to educational assessment design, and relates more particularly to systems for constructing evidence-centered designed educational assessment items.
An educational assessment is a special kind of evidentiary argument that requires making sense of complex data to draw inferences or conclusions. Specifically, an educational assessment is a way of gathering information (e.g., in the form of particular things that students say, do, or make under particular circumstances) to make inferences about what students know, can do, or have accomplished or learned.
Assessment task designers are often required to use evidence-centered design (ECD) documents as resources for constructing assessment tasks. ECD documents are typically voluminous and contain complex content and associations, making them difficult to work with under the constraints that many designers face (e.g., tight deadlines, requirements for large numbers of tasks, high technical quality standards, etc.). Thus, it is difficult to scale ECD resources in a manner that makes them easily used by writers of assessment items.
A method for guiding a user in a construction of an assessment item includes identifying a requirement of the assessment item, filtering a plurality of documents in accordance with the requirement to produce a set of relevant documents, presenting a workflow for use in constructing the assessment item, wherein the workflow is based on the set of relevant documents, and presenting a set of questions for use in validating, by the user, that a proposed assessment item meets the requirement.
A computer readable storage device according to another embodiment of the invention contains an executable program for guiding a user in a construction of an assessment item, wherein when the program is executed, the program causes a processor to performs steps including identifying requirements of the assessment item, filtering a plurality of documents in accordance with the requirements to produce a set of relevant documents, presenting a workflow for use in constructing the assessment item, wherein the workflow is based on the set of relevant documents, and presenting a set of questions for use in validating, by the user, that a proposed assessment item meets the requirement.
A system for guiding a user in a construction of an assessment item includes a database that relates potential assessment targets to potential observations that facilitate evaluation of the potential assessment targets, a processor for filtering data contained in the database in response to an identification of an assessment target and a design choice associated with the assessment item, and a user interface for leading a user through a workflow that incorporates data that is identified as being relevant to a design, creation, and validation of the assessment item based on the filtering.
The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
To facilitate understanding, identical reference numerals have sometimes been used to designate elements common to multiple figures.
The present invention relates to a method and apparatus (e.g., a “wizard”) for guiding the construction and validation of assessment items. Embodiments of the invention filter and organize relevant evidence-centered design (ECD) documentation into a streamlined guide that task designers can use when developing assessment items. The guide includes an ordered sequence of questions that directs the writer of an assessment item through the development process by simplifying complex associations within the ECD documentation and by posing reflective questions that are intended to improve the technical quality of the resulting assessment item.
Within the context of the present invention, an “assessment item” is a article (e.g., a question) that is designed to gather information useful in evaluating a student's knowledge, skill, or ability (i.e., what he knows, can do, or has accomplished or learned). A series of assessment items is referred to as an “assessment task.” An “assessment target” refers to a collection of standards, task models, or constructs related to particular knowledge, skills, or abilities possessed by the student and about which the assessment item is designed to gather evidence.
The user endpoint device 102 may be any type of endpoint device such as a desktop computer or a mobile endpoint device such as a cellular telephone, a smart phone, a tablet computer, a laptop computer, a netbook, an ultrabook, a portable media device (e.g., an MP3 player), a gaming console, a portable gaming device, or the like. It should be noted that although only one user endpoint device 102 is illustrated in
The network 104 may be any type of communications network, such as for example, an Internet Protocol (IP) network (e.g., an IP Multimedia Subsystem (IMS) network, an asynchronous transfer mode (ATM) network, a wireless network, a cellular network (e.g., 2G, 3G, and the like), a long term evolution (LTE) network, and the like). It should be noted that an IP network is broadly defined as a network that uses Internet Protocol to exchange data packets. Additional exemplary IP networks include Voice over IP (VoIP) networks, Service over IP (SoIP) networks, and the like.
In one embodiment, the server 106 is an application server that hosts an application (e.g., a “wizard”) for guiding the construction and validation of assessment items. This application is a separate application from that in which the writer actually creates the assessment item. The server 106 may comprise a general purpose computer as illustrated in
The database 108 is a repository for evidence-centered design (ECD) documentation. In one embodiment, the database 108 is a relational database that relates potential assessment targets to potential observations that facilitate their evaluation (e.g., an assessment target concerning fractions is related to one or more observations that allow one to determine whether a student has mastered fractions). In a further embodiment, the data contained in the database is customized for an entity with whom the writer of the assessment item is associated. In addition, the database 108 may contain state variables for users of the system 100. Although only a single database 108 is illustrated, it will be appreciated that the system may include multiple databases (e.g., discrete databases for relational data and state variables).
The method 200 begins in step 202. In step 204, the server 106 identifies a requirement of the assessment item. The requirement may comprise, for example, an assessment target (e.g., a particular item of knowledge, skill, or ability) about which the assessment is intended to gather evidence. Alternatively or in addition, the requirement may comprise a design choice associated with the assessment item (e.g., the format of the assessment item (multiple choice, true/false, open ended, etc.), a rubric for use in evaluating a response to the assessment item, a resource to be used in connection with responding to the assessment item (a passage of text, an image, etc.), or a grade or age level to which the assessment item is targeted). Multiple requirements may be identified in step 204.
In one embodiment, the requirement is identified explicitly by the writer of the assessment item (who is not necessarily the assessment designer), via the user interface on the user endpoint device 102. For instance, the user interface may present a list of potential assessment targets to the writer, and the server 106 may subsequently receive a signal from the user endpoint device 102 indicating that the writer has selected an assessment target from the list.
For instance, step 204 may be embodied in a drill-down process that starts by assuming that the writer of the assessment item has been provided with an assignment that corresponds to a particular task model. The writer may enter this task model via the user interface in order to be presented with the number and types of assessment items that are required. In one embodiment, the user interface may display each potential assessment item with a hyperlink that may be clicked for further information (e.g., a list of focal knowledge, skills, or abilities that may be selected).
In step 206, the server 106 filters the documents in the database 108, in accordance with the identified requirement, in order to produce a set of documents that is relevant to the requirement. For instance, based on the selection of a particular knowledge, skill, or ability, the server 106 may identify a set of potential observations that facilitate evaluation of the selected knowledge, skill, or ability. The documents may include characteristic features (e.g., types of tasks, response options, particular text to be included) of items or tasks designed to assess the selected knowledge, skill, or ability.
In step 208, the user interface presents a workflow for use in constructing the assessment item, where the workflow is based on the set of documents produced in step 206. In one embodiment, the workflow comprises an ordered sequence of prompts relating to the requirement identified in step 204. A prompt may focus the item writer's attention to elements of a potential assessment item, and may be presented as a question and a set of corresponding response options. In a further embodiment, the workflow is based on a stored set of assessment targets and associated instructions created by an individual (e.g., an assessment designer) who is responsible for setting an overall coverage goal for the assessment item. The workflow may be based on evidence-centered design principles. Using this workflow as a guide, the writer may develop a potential assessment item in a separate application.
In step 210, the user interface presents a set of questions for use in validating that a potential assessment item proposed by the designer will meet the requirement identified in step 204. The writer of the assessment item will review the potential assessment item, using the set of questions as a guide to verify that the potential assessment item is appropriate. For instance, the set of questions may ask the writer to confirm that the criteria of certain prompts presented in the workflow in step 208 are met. It is noted that the set of questions alone is typically not enough to entirely validate the potential assessment item, but review of the potential assessment item in accordance with the set of questions is part of the validation process.
The method 200 ends in step 212.
It should be noted that although not explicitly specified, one or more steps of the methods described herein may include a storing, displaying and/or outputting step as required for a particular application. In other words, any data, records, fields, and/or intermediate results discussed in the methods can be stored, displayed, and/or outputted to another device as required for a particular application. Furthermore, steps or blocks in the accompanying Figures that recite a determining operation or involve a decision, do not necessarily require that both branches of the determining operation be practiced. In other words, one of the branches of the determining operation can be deemed as an optional step. In addition, unless stated otherwise, the steps of the methods described herein do not necessarily need to be performed in the order illustrated in the Figures. For instance, some of the described steps may be performed in parallel (e.g., substantially simultaneously), even though they may be illustrated in an ordered sequence.
Alternatively, the wizard module 405 can be represented by one or more software applications (or even a combination of software and hardware, e.g., using Application Specific Integrated Circuits (ASIC)), where the software is loaded from a storage medium (e.g., I/O devices 406) and operated by the processor 402 in the memory 404 of the general purpose computing device 400. Thus, in one embodiment, the wizard module 405 for guiding the construction and validation of assessment items described herein with reference to the preceding Figures can be stored on a non-transitory or tangible computer readable medium or carrier (e.g., RAM, magnetic or optical drive or diskette, and the like).
Although various embodiments which incorporate the teachings of the present invention have been shown and described in detail herein, those skilled in the art can readily devise many other varied embodiments that still incorporate these teachings.
Number | Date | Country | |
---|---|---|---|
61805682 | Mar 2013 | US |