System and method for generating and providing educational exercises

Abstract

A system and method for generating and administering classroom exercises to a student. One or more root questions is generated, where each root question has a stem, a plurality of right answers, and a plurality of wrong answers. A multiple-choice question instance is created by combining the stem with a set of answer choices that are created from one of the right answers and one or more of the wrong answers. The questions are administered to a student. Preferably, the sets of questions based on the same root questions are administered one or more times to the student—e.g., a student may be required to work three sets of question instances based on the same set of root questions, or may be required to continue working new sets of question instances until all of the questions in one set are answered correctly.

Description

FIELD OF THE INVENTION

[0002] The present invention relates generally to the field of computer-assisted education. More particularly, the invention provides systems and methods for assisting in the teaching of a subject to a student by the use of computer-generated exercises.

BACKGROUND OF THE INVENTION

[0003] The cost of education has risen, relative to inflation, faster than almost any other industry. For example, in the past 45 years, tuition at major colleges, relative to the cost of sending a letter across the United States by air, has increased fivefold. It has increased 500 times relative to the cost of a cross-country phone call. Computerization has had little effect on this cost. There are some on-line courses to take, and there are some activities in the corporate-retraining arena. But nothing yet has reduced significantly the cost of college education.

[0004] While computers have generally found their way into a wide field of endeavors—with the result being nearly universal cost-reduction—the use of computers in education has some limits. Residential college education cannot be replaced entirely by impersonal, on-line courses. The social aspects of college, including learning in groups, informal teacher-student discussions, and so on, are too valuable and important. Savings, therefore, must come from elimination of redundancy. At any time, there are hundreds of courses in a given subject matter (e.g., database systems, differential equations, conversational French) that are being offered at the numerous colleges in the United States (or in the rest of the world). They differ mainly in small ways. For instance, they have similar, yet distinct, homework assignments, each requiring substantial time to design, debug, and grade.

[0005] Generating and grading homework assignments is a task that can be automated. Individualized help, group discussions, and other aspects of a course that require personal attention of an instructor cannot be easily automated. By automating those tasks that can be replicated efficiently, the course staff is free to do the tasks that cannot easily be automated, and efficiency is achieved.

[0006] Certain types of automated systems for generating and/or grading assignments exist. Electronic grading of multiple-choice problems has existed for years. However, such systems suffer from the problem that the questions are entirely generated by human effort, so, to avoid the enormous effort of creating individual questions for each student, all students are generally given the same set of questions, which raises the possibility of answer sharing.

[0007] In view of the foregoing, there is a need for a system that overcomes the drawbacks of the prior art.

SUMMARY OF THE INVENTION

[0008] The present invention provides systems and methods for teaching a subject to a student by use of computer-generated exercises. Source information is provided from which questions may be generated. The source information is used to generate questions for the student to answer, where the questions sample the student's knowledge of the subject being taught More specifically, knowledge of the answer to one particular problem that can be solved using the subject being taught. One or more sets of the questions may be presented to the student for the student to answer. The student's performance on the questions is evaluated. Preferably, if the student's performance on a set of questions does not meet a defined standard, then new sets of questions based on the same source information may be repeatedly administered to the student, until the student's performance does meet a standard. (Alternatively, the system may administer an arbitrary number of sets of questions to the student, irrespective of the student's performance on previous sets of questions.) Because a vast number of questions can be generated automatically based on the source information, it is possible to ask a student to work an underlying problem and answer questions about it repeatedly, without incurring the problems that inhere in repeatedly giving the student the exact same set of questions, and without incurring the additional human labor required to generate a large number of different new questions.

[0009] In a preferred embodiment, the questions to be administered to the student are multiple-choice questions. In this case, the source information may comprise a question stem, a set of right answers, and a set of wrong answers, and the system may generate questions by combining the stem, one right answer, and one or more wrong answers, to produce a multiple-choice question. The source information may be generated by a class instructor, or may be provided in the form of prepackaged information that is commercially distributed as a teaching tool. In greater generality, the source information may comprise any type of template for a question, and it will be appreciated that the combination of a stem, right answers, and wrong answers that form the building blocks for multiple-choice questions is merely one embodiment of the above-mentioned source information.

[0010] Optionally, answer explanations may be provided with the answer choices—especially the wrong answer choices—which can be displayed to the student after the student answers the questions in order to assist the student in learning the subject matter. These answer explanations can be generated by the class instructor, or can be included with prepackaged questions.

[0011] In a preferred embodiment, the student is asked to answer all questions in a set correctly, and is given new sets of similar questions (i.e., new question instances based on the same underlying source information) to answer until the student answers all questions in a single set correctly. Preferably, there a limit is imposed as to the number of, and/or frequency with which, a student may take a set of questions.

[0012] Other features of the invention are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The foregoing summary, as well as the following detailed description of preferred embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the invention, there is shown in the drawings example constructions of the invention; however, the invention is not limited to the specific methods and instrumentalities disclosed. In the drawings:

[0014] FIG. 1 is a block diagram of source information, and the derivation of exercise questions therefrom, in accordance with aspects of the invention;

[0015] FIG. 2 is a block diagram of an example type of source information from which exercise questions can be derived;

[0016] FIG. 3 is a block diagram of a system that presents exercise questions to a student in accordance with aspects of the invention;

[0017] FIG. 4 is a flow diagram of an example process for generate and administering a teaching exercise in accordance with aspects of the invention; and

[0018] FIG. 5 is an example product in which aspects of the invention may be embodied.

DETAILED DESCRIPTION OF THE INVENTION

[0019] The present invention provides systems and methods for generating and administering a classroom exercise. A root question constitutes source information from which questions for the exercise may be derived. A set of one or more question instances derived from the root question(s) is administered to the student, and the student's performance on the question instances is evaluated. Optionally, plural sets of question instances may be administered to the student, or sets of question instances may be recurrently administered to the student until the student's performance meets a defined standard.

[0020] Mechanisms are derived below for implementing the above-described system, and certain variations thereon.

[0021] Derivation of Questions from Source Information

[0022] FIG. 1 is a block diagram showing how questions, which are to be administered to a student, may be derived from an underlying set of source information. Source information 102 constitutes some type of template, or other general information, from which individual question instances 104(1), 104(2), . . . , 104(n) may be derived.

[0023] The invention is not limited to any particular form of source information 102 or questions 104(1) through 104(n), although a preferred embodiment of source information 102 is described below in connection with FIG. 2. In one example, source information 102 may comprise a problem template from which a large number of specific question instances can be derived—e.g., source information 102 may comprise the general question template: “Compute the number of square yards of carpet needed to cover a rectangular room whose length is $x feet and whose width is $y feet.” In this case, a large number of questions can be created from this type of source information 102 by substituting different values for $x and $y. In this example, source information 102 may also include the logic needed to compute the answer (e.g., the logic needed to compute the value $x times $y divided by nine), so that a student's answer to the question can be evaluated. Thus, an example of a question instance derived from this source information is “Compute the number of square yards of carpet needed to cover a rectangular room whose length is 9 feet and whose width is 3 feet.” The formula for deriving a question instance from source information 102 (e.g., randomly select two real numbers and substitute them for the variables $x and $y) is an example of a “rule.”

[0024] As another example, source information 102 may comprise the question template: “What is the capital of $state?” This example of source information 102 would also comprise a list of states that could be substituted for the variable $state. Thus, based on this source information 102, a derived question instance could be “What is the capital of Alaska?” (In this case, the “rule” for deriving question instances from source information 102 would require that one of the listed states be substituted for the variable $state.) Moreover, since there is no mathematical logic that can be used to compute the name of a capital from the name of a state, source information 102 would also include information that correlates each state with its capital, so that a student's answer to any derived question instance could be evaluated.

[0025] In a preferred embodiment, source information 102 comprises information from which multiple-choice questions can be derived. Although the invention is not limited to systems that use multiple-choice questions, it should be noted that multiple-choice questions are advantageous in systems that automate the generation and grading of questions, because they can be derived according to a simple formula or “rule” (e.g., generate a question instance with one correct choice and three incorrect choices), and answers thereto can be evaluated unambiguously (i.e., the student's answer of choice “A” is unambiguous, whereas answers to the state capital question described above could have ambiguities due to misspelling, capitalization, abbreviation, etc.). Thus, in the case of multiple-choice questions, source information 102 could take the form shown in FIG. 2.

[0026] In the example of FIG. 2 source information 102 comprises the building blocks from which multiple-choice questions may be derived. Thus, source information 102 comprises a stem 202, a set of correct answer choices 204, and a set of incorrect answer choices 206. As is known in the relevant art, the “stem” of a multiple-choice question is the preamble that sets up the problem and asks the student to pick the correct answer from a set of possible answers; a correct answer choice is a legitimate answer to the question posed by the stem; and an incorrect answer choice is an answer that does not answer correctly the question posed by the stem. Thus, an example of a multiple choice question is:

[0027] 1. Which of the following cities is a state capital in the United States?

[0028] A. Jacksonville, Fla.

[0029] B. Tacoma, Wash.

[0030] C. Pierre, S.D.

[0031] D. Eugene, Oreg.

[0032] In this example, “Which of the following cities is a state capital in the United States?” is the stem 202; “Pierre, S.D.” is an example of a correct answer choice 204; and “Jacksonville, Fla.,” “Tacoma, Wash.,” and “Eugene, Oreg.” are examples of incorrect answer choices 206. The entire question—i.e., stem, correct and incorrect answer choices, and letters before the answer choices—is an example of a “question instance” that can be derived from the source information.

[0033] 61 Preferably, answer choices 206 comprise a plurality of possible correct answers 204(1), 204(2), . . . 204(i) that could be used in multiple-choice questions. Thus, in the state-capital example answer choices 206 could include fifty correct answer choices (Montgomery, Ala.; Juneau, Ark.; Phoenix, Ariz.; . . . ; Cheyenne, Wyo.). If the lesson to which the question relates is on the capitals of the fifty states, this type of source information 102 is sufficient to generate questions that will sample the student's knowledge of the entire lesson. Similarly, there is preferably a plurality of incorrect answer choices 206(1), 206(2), . . . , 206(j). Since a question instance typically has two to four incorrect answer choices for every correct one, source information 102 preferably includes more incorrect answer choices 206 than correct answer choices 204, although this is not a requirement. Generating good multiple choice questions merely requires a set of plausible incorrect choices that is large enough to keep their recurrence small over a reasonably-sized sample of question instances (since a high rate of recurrence could help a student to identify incorrect choices without learning the underlying material simply by remembering which answer choices have previously appeared).

[0034] Source information of the type depicted in FIG. 2, from which a multiple-choice question may be derived, may be referred to herein as a “root question.” In general, a root question is a multiple-choice question that has several right answers and many wrong answers.

[0035] 81 In a preferred embodiment, source information 102 may, optionally, include explanations 208 of the answer choices, which can be presented to the student as a learning tool after he or she has answered the question (or after he or she has answered all of the questions in a set of questions). For example, the answer choice “Tacoma, Wash.” may be associated with an explanation such as “Tacoma is a city in Washington, but is not the capital of that state. The capital of Washington is Olympia.” Thus, if a student had chosen “A” as the answer to the above question, he or she could be presented with this explanation of the wrong answer, which could assist in the student's learning of the material. Correct answer choices can also be associated with explanations 208 (e.g., “Correct. Pierre is the capital city of South Dakota.”), although, depending on the nature of the material, the correct answer choices may be more self-explanatory explanatory than the incorrect answer choices, and an explanation of the correct answers may be less relevant than an explanation of the incorrect answers.

[0036] While state capitals provide a convenient example of a root question, the techniques of the present invention can also be applied to more complex subject matter, such as college-level science or engineering. For example, in a course on database systems, a typical homework question posed to a class may present two (or more) relations (e.g., R1 and R2), and then may ask the student to perform some operation on the relations such as: “Calculate the natural join of R1 and R2”. Conventionally, a homework assignment would ask the student to write the relation that is the natural join of R1 and R2, and the answer would be graded by a human teaching assistant. However, a root question can be constructed for this type of problem that effectively requires the student to compute the natural join of R1 and R2, but is much easier to grade. For example, an instructor (or some other root question writer) can generate a root question with the following stem: “Which of the following tuples is in the natural join of R1 and R2?” Correct answer choices would then be tuples that are actually members of the natural join of R1 and R2, and incorrect answer choices can comprise tuples that are not in the join. Preferably, the incorrect answers can be constructed in a manner that reflects common errors that students make when calculating joins. If the incorrect answer choices are skillfully constructed, then for a student to reliably answer the question correctly should effectively require the student to compute the join in order to determine which tuples are members, thereby providing a similar incentive for the student to work the underlying problem as there would be if the problem had been presented in a long-answer form. In a preferred embodiment, the student could be asked a set of plural questions relating to the same underlying facts (e.g., three separate questions, each of which asks a student to identify a tuple that is in the join), and his or her performance could be evaluated under a standard that requires the student to get all the questions correct. Thus, it is unlikely that a student would be able to get all of the answers right by guessing and without understanding the underlying problem, so—in terms of teaching the subject matter and ensuring that the student understands the problem—such a set of multiple-choice questions approaches the reliability of a long answer problem, but with a greatly reduced grading cost.

[0037] Example System for Administering Questions

[0038] FIG. 3 shows an example system that administers exercise questions. Exercise questions are preferably administered to a student using a general purpose computer 302. The components and capabilities of a general-purpose computer are generally known in the art, and thus are not described in detail herein.

[0039] Computer 302 is loaded with various types of data and software: question instances 104, a program that implements a user interface 306 for presenting the question instances to the student, and grading logic 308 that evaluates the students answers and preferably determines whether the student's performance on the question instances meets the applicable standards (e.g., complete correctness on a set of questions, etc.). Question instances 104 may have been pre-derived from source information 102 (shown in FIGS. 1 and 2) and then pre-loaded onto the computer 302 that student 304 uses. Alternatively, computer 302 may be loaded with source information 102 itself, together with logic that derives question instances 104 from the source information. It will be appreciated that there are various ways to cause question instances 104 to exist on computer 302 at the time that question instances 104 are administered to student 304 (e.g., stand-alone workstation that contains all logic and data needed to derive the question instances, client-server architecture where the server generates the question instances and delivers the question instances to computer 302 for administration to the student, web server that delivers content to a browser at the student workstation, etc.), and the invention is not limited to any particular architecture or design.

[0040] Student 304 uses computer 302 to answer a set of questions. User interface logic 306 causes computer 302 to present question instances 104 (or some subset thereof) to student 304. Student 304, in turn, provides answers to these question instances. Grading logic 308 determines whether student 304's answers to the question instances are correct or incorrect, and, preferably, evaluates the student's overall performance on a set of question instances. Computer 302 may optionally be loaded with a program and/or data that provides answer explanations to the students (preferably only explanations for those questions that the student got wrong), as more particularly described above.

[0041] Example Process(es) for Creating and Administering Questions

[0042] FIG. 4 is a flow diagram showing an example process for generating and administering questions in accordance with aspects of the invention.

[0043] Initially, source information (e.g., a root question) is created (402), from which one or more question instances may be derived. Depending on the nature of the exercise(s) to be administered, plural sets of source information may be created. For example, in the non-limiting case where the exercise is used as part of a database systems course, an exercise may comprise a set of questions on various aspects of database systems (e.g., binary operations, query language syntax, etc.), and different root questions (or other source information) may be used to create question instances on these different aspects. Thus, there would be one set of source information that asks (in the stem) for a student to identify tuples that are in the natural join of two relations, along with sets of correct and incorrect answer choices for that problem. Another set of source information might contain a stem that asks a student to identify a syntactically correct query in SQL, and would contain different sets of correct and incorrect answer choices appropriate for that stem. There is no limit to the number of different sets of source information that may be created.

[0044] After the source information has been created, question instances are derived from the source information (404). As described above, the manner in which a question instance is derived from source information depends on the nature of the source information itself. In the case of multiple-choice questions that are derived from root questions, the nature of the derivation is more particularly described above in connection with FIG. 2.

[0045] Once one or more question instances have been derived, the question instances may be presented to a student (406). The student interacts with a computer to read and answer the question instances, and the student provides his or her answers to the computer that is being used to administer the questions. The student's answers are then evaluated (406) to determine whether the answers are right or wrong, and, optionally, to determine whether the student's overall performance on a set of questions meets a defined standard. In one embodiment, the student may be asked to complete plural sets of questions, in which case the process returns to 406 to present a new set of questions to the student, and this process repeats until the number of sets of questions that the student is to answer has been reached.

[0046] If the student's performance is being evaluated to determine whether that performance meets a defined standard (e.g., whether the student answers all question instances in a set correctly), then it is determined (410) whether the student's performance meets that standard. As noted above, an exercise may comprise the administration of plural sets of questions to the student without regard to the student's performance on any given set (e.g., a homework assignment may require the student to work three sets of questions). In one variation, however, the student is asked to repeatedly take new sets of question instances (based on different sets of questions derived from the source information) until the student's performance on an entire set meets the defined standard. If the questions are being administered in this way, then a determination at 410 that the student has met the standard may constitute the end of the student's exercise, while failure to meet the standard returns the process to 406 to present a new set of questions instances (possibly preceded by choice explanations for the student's incorrect answers on the last set of questions), and this cycle of administering questions and evaluating the student's performance under the standard may be repeated until the student does meet the standard.

[0047] In the case where the repetition of sets of questions is based on the student's performance on prior sets, the process of FIG. 4 optionally may impose limits on the repeated taking of sets of questions by a student. For example, the student may be limited to taking the exercise n times, may be limited to taking the exercise n times per day, or may be required to wait a minimum amount of time before retaking an exercise.

[0048] The dotted line in FIG. 4 represents a division between the acts that generate question instances and the acts that administer question instances to a student. Typically, an instructor will perform the acts above the line, and a student will perform and/or participate in acts below the line. Typically, these sets of acts will take place on different machines (e.g., the instructor's machine and the student's machine). However, it should be noted that the invention is not limited to any particular architectural, physical, or temporal distribution of the acts shown in FIG. 4. For example, blocks 402 and 404 may occur at a commercial question provider who is in the business of providing root questions and software to administer them. In another example, the commercial question provider generates the source information at block 402, but the derivation of questions is performed either at a local instructor's machine or at the student's machine. In another example, the generation of source information may be performed by a class instructor, and the derivation of questions from that source information may be performed either at the instructor's machine or at the student's machine. There is no limit to the way in which the different acts depicted in FIG. 4 may be distributed.

[0049] Example Exercise Scenario

[0050] The following in an example scenario in which the techniques and mechanisms described herein may be used for a class exercise:

[0051] 1. An assignment consisting of several questions is assigned to the class. Each student in the class is allowed to take the assignment as many times as they wish, and only a student's last score counts. Thus, there is an incentive for each student to answer sets of questions until he or she scores 100% on the assignment. In a preferred embodiment, an assignment includes 4-5 questions. Too few questions allow a student to guess randomly and eventually get a perfect score; too many questions create too much risk that the student will accidentally answer a question incorrectly and will have to start over again.

[0052] 2. A student opens the assignment the first time. The software that administers the assignment gives the student the questions in random order (to make it harder for students to share information about, e.g., “question number 3”). The system also chooses at random one correct answer and three wrong answers, and presents those in random order. The student may be able to answer all of the questions, but, more likely, he or she will have trouble with one or more, so they do the best they can and submit their answers.

[0053] 3. The student studies the material needed to answer the questions with which he or she is having trouble, off-line. For example, if the student doesn't know how to compute a join, the student could reread the material on that topic, attend office hours, or send an E-mail to a human teaching assistant. The student can then compute the entire join, as if the question had been the long-answer problem “compute this join.”

[0054] 4. The student opens the assignment a second time, this time prepared with the answers to the general questions for which they will receive new, random instances. For example, if the student has correctly computed the join of R1 and R2, then the student should be able to identify that tuple from among the plural answer choices.

[0055] Example Product Incorporating Programs/Data for Using Exercise Questions

[0056] FIG. 5 shows an example product that may embody some or all of the features of the invention. Product 500 comprises a teaching tool (which is depicted herein as textbook 502), and also includes data and/or programs that can be used to administer exercise questions relating to the subject matter of the teaching tool. (These data and programs are depicted, by way of example and not limitation, as a computer-readable medium 504 (e.g., CD-ROM), or an electronic download 506.) The data and/or programs may include source information (e.g., root questions) from which question instances may be derived and administered together with the software necessary to administer the questions; or, the data/programs may include pre-derived question instances that are based on source information together with the necessary administration software; or, the data/programs may include software that allows an instructor to create root questions and derive questions instances therefrom, and to administer those exercises based on the question instances to students. The programs/data included in product 500 may incorporate any of the techniques, systems, or methods described above.

[0057] In one example embodiment, a distributor of educational materials may package a teaching tool (e.g., textbook) together with data/programs for administering classroom exercises based on the subject matter of the textbook. For example, a textbook may come with a CD-ROM affixed thereto, or may come with a token that permits download of the necessary programs/data over the Internet.

[0058] It is noted that the foregoing examples have been provided merely for the purpose of explanation and are in no way to be construed as limiting of the present invention. While the invention has been described with reference to various embodiments, it is understood that the words which have been used herein are words of description and illustration, rather than words of limitations. Further, although the invention has been described herein with reference to particular means, materials and embodiments, the invention is not intended to be limited to the particulars disclosed herein; rather, the invention extends to all functionally equivalent structures, methods and uses, such as are within the scope of the appended claims. Those skilled in the art, having the benefit of the teachings of this specification, may effect numerous modifications thereto and changes may be made without departing from the scope and spirit of the invention in its aspects.

Claims

1. A method of teaching a subject matter comprising: administering a first set of one or more question instances to a student, said first set of one or more question instances being based on: (1) source information that relates to the subject matter, and (2) one or more rules that govern derivation of question instances from said source information; evaluating said student's performance on said first set of one or more question instances; and if said student's performance on said first set of one or more question instances does not meet a standard, then repeating said administering and evaluating acts until said student's performance meets said standard, wherein each repetition of said administering and evaluating steps is performed with a new set of question instances that is different from said first set of question instances and different from any new set of question instances that has been already been administered to said student, and wherein each new set of question instances is based on: (1) source information that relates to the subject matter, and (2) one or more rules that govern derivation of question instances from said source information.

2. The method of claim 1, wherein said source information comprises a stem, a set of correct answers to said stem, and a set of incorrect answers to said stem, and wherein said one or more rules comprises a rule that a question instance is formed by combining said stem, one of said correct answers, and a predetermined number of incorrect answer.

3. The method of claim 1, wherein said standard comprises requiring said student to answer every question instance in a set of question instances correctly.

4. The method of claim 1, further comprising: if said student's performance does not meet said standard, then providing an explanation relating to a question instance that said student answered incorrectly.

5. The method of claim 1, further comprising: imposing a limit on the repetition of said administering act.

6. The method of claim 5, wherein said limit comprises a maximum number of times that said administering act can be performed.

7. The method of claim 5, wherein said limit comprises a maximum frequency, with respect to time, that said administering act can be performed.

8. A computer-readable medium encoded with information comprising: source data from which question instances relating to a subject matter are derivable; one or more rules that govern derivation of said question instances from said source data; and software that uses said one or more rules to derive question instances from said source data, and that administers one or more sets of derived question instances to a student.

9. The computer-readable medium of claim 8, wherein said source data comprises a stem, a set of correct answers to said stem, and a set of incorrect answers to said stem, and wherein said one or more rules comprises a rule that a question instance is formed by combining said stem, one of said correct answers, and a predetermined number of said incorrect answers.

10. The computer-readable medium of claim 8, wherein said software evaluates said student's performance on the one or more sets of derived question instances, and provides explanations relating to at least those question instances that said student answers incorrectly.

11. The computer-readable medium of claim 8, wherein said software further evaluates said student's performance on a set of the derived question instances, and that administers said one or more sets by acts comprising: administering a first one of said sets of derived question instances to said student; evaluating said student's performance on said first one of said sets of derived question instances; and if said student's performance on said first, one of said sets of derived question instances does not meet a standard, then recurrently administering one or more new sets of derived question instances until said student's performance on a new set of derived question instances meets said standard.

12. The computer-readable medium of claim 11, wherein said standard comprises a requirement that said student answer every question instance in a set of derived question instances correctly.

13. The computer-readable medium of claim 11, wherein said software imposes a limit on the recurrence of administering sets of derived question instances.

14. The computer-readable medium of claim 13, wherein said limit comprises a maximum number of times that administration of sets of question instances can recur.

15. The computer-readable medium of claim 13, wherein said limit comprises a maximum frequency, with respect to time, that administration of sets of question instances can recur.

16. The computer-readable medium of claim 8, wherein said software provides an interface through which a teacher can perform acts comprising any one or more of the following: adding new source data for use by said software to derive new question instances; changing a standard, wherein said standard measures said student's performance on a set of derived question instances; and changing a limit, wherein said limit governs recurrence on administration of sets of derived question instances.

17. A method of creating a student exercise comprising: creating base information that comprises or implies a set or range of facts; deriving a first set of question instances based on said set or range of facts; presenting said first set of question instances to a student; obtaining said student's answers to said first set of question instances; evaluating said student's performance on said first set of question instances; and if said student's performance on said first set of question instances does not meet said standard, then performing the following: deriving a new set of question instances based on said set or range of facts; performing said presenting, obtaining, and evaluating acts with said new set of question instances in place of said first set of questions, until said student's performance on a new set of question instances meets said standard.

18. The method of claim 17, wherein said base information comprises a stem, a set of correct answers to said stem, and a set of incorrect answers to said stem, and wherein said deriving acts are performed by combining said stem, one of said correct answers, and a predetermined number of said incorrect answers.

19. The method of claim 17, wherein said standard comprises a requirement that said student answer correctly every question instance in either said first set of question instances or in one of the new sets of question instances.

20. The method of claim 17, further comprising: imposing a limit on the presenting of sets of question instances to said student.

21. The method of claim 20, wherein said limit comprises a maximum number of sets of question instances that may be presented to said student.

22. The method of claim 20, wherein said limit comprises a maximum frequency, with respect to time, that sets of question instances may be presented to said student.

23. The method of claim 17, further comprising: providing explanation of answers to said sets of question instances to said student.

24. The method of claim 23, further comprising: receiving said explanations of said answers as input from an instructor.

25. A computer-readable medium encoded with computer-executable instructions to perform a method of providing an exercise to a student, the method comprising: deriving a plurality of question instances from a set of source information, said source information comprising or implying a set or range of facts, said questions being answerable with knowledge about said set or range of facts; presenting a first set of said plurality of question instances to a student; receiving said student's answers to said first set of said plurality of question instances; if said student's performance on said plurality of question instances does not satisfy a standard, then repeatedly presenting different sets of said plurality of question instances until said student's performance on a presented set satisfies said standard.

26. The computer-readable medium of claim 25, wherein said source information comprises a stem, a set of correct answers to said stem, and a set of incorrect answers to said stem, and wherein said question instances comprise said stem, one of said correct answers, and a predetermined number of said incorrect answers.

27. The computer-readable medium of claim 25, wherein said standard comprises a requirement that said student answer all question instances in a set of question instances correctly.

28. The computer-readable medium of claim 25, further comprising: providing explanations of question instances that said student has answered incorrectly.

29. The computer-readable medium of claim 25, further comprising: enforcing a limit on the presentation of sets of question instances to students, wherein the limit comprises one or more of the following: a limit on the number of sets of question instances that may be presented to said student; and a limit on the frequency, with respect to time, with which sets of question instances may be presented to said student.

30. A system for teaching a subject matter comprising: a teaching tool that describes the subject matter; data that comprises: source information relating to the subject matter that comprises or implies a set or range of facts about which question instances can be derived; software that performs acts comprising: deriving, from said source information, a set of question instances that are answerable with knowledge of said set of facts; presenting said set of question instances to a student; receiving said student's answers to said set of question instances; and evaluating said student's performance on said set of question instances to determine whether said student's performance meets a standard.

31. The system of claim 30, wherein said source information comprises a stem, a set of correct answers to said stem, and a set of incorrect answers to said stem, and wherein said deriving act comprises combining said stem, one of said correct answers, and a predetermined number of said incorrect answers.

32. The system of claim 30, wherein said teaching tool comprises a book about said subject matter.

33. The system of claim 30, wherein said data is provided on a computer-readable medium that accompanies said teaching tool.

34. The system of claim 30, wherein said software performs acts further comprising: if said student's performance on said set of question instances does not meet said standard, then repeatedly presenting different sets of question instances derived from said source information to said student until said student's performance on a set of question instances meets said standard.

35. The system of claim 34, wherein said software performs acts further comprising: imposing a limit on the repeated presentation of different sets of question instances to said student, wherein said limit comprises one or more of the following: a maximum number of sets of question instances that may be presented to said student; and a maximum frequency, with respect to time, with which different sets of question instances may be presented to said student.

36. The system of claim 30, wherein said standard comprises a requirement that said student answer every question instance in a set of question instances correctly.