NSF Award
2302686
Timely assessment of students' learning is crucial for addressing their needs. However, current assessment methods such as multiple-choice or calculation tests may not truly reveal students’ internal thinking processes, because students may guess answers or follow the step-by-step procedures of examples without conceptual understanding. Existing research shows that asking students to write down justifications for their answers, and plan and reflection on their learning can promote them to develop deeper conceptual understanding and apply cognitive and metacognitive strategies in their learning. Nonetheless, this approach is not widely adopted due to the time-consuming nature of assessing free-text responses. This project aims to develop a rubrics-based automatic assessment tool to identify individual students' misconceptions and learning deficiencies from their free-text responses, allowing instructors and students to instantly adjust their teaching and learning, and eventually enabling personalized instructions tailored to individual learners’ needs, which particularly benefits students at HBCUs. The project will also provide the base for delivering research experience for both undergraduates and teachers and outreach various audiences, including public school students, increasing the literacy of the public on artificial intelligence.<br/><br/>This research proposes innovative strategies to fine-tune and calibrate the pre-trained language model, through self-supervised learning and few-shot learning, for automatic classification of texts based on user-specified rubrics, with a particular focus on learners' cognitive and metacognitive processes. The proposed model integrates three novel attributes for improving similarity comparison between texts and rubric keywords: (1) across-attention between compared texts for increasing the sensitivity of comparison; (2) joint embedding of words, phrases, and sentences for improving the accuracy of comparison; (3) incorporation of thematic relevance for enhancing the breadth of comparison. Given the assessment rubrics, the model can classify texts to reveal students’ thinking or other traits in multiple finer granular perspectives. It is flexible for adding new assessment perspectives and more transparent than the overall assessment, and allows involving human judgment, leading to more reliable assessment acceptable for practice, and advancing knowledge on adapting pre-trained language models for personalized instructions. The dataset collected from diverse students and specific strategies will be adopted to mitigate the potential biases of the proposed model.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
