Combining Automated and Human Classification Decisions

open_access
Open Access
Author:
Kumar, Suraj
Area of Honors:
Computer Science
Degree:
Bachelor of Science
Document Type:
Thesis
Thesis Supervisors:
Rebecca Jane Passonneau, Thesis Supervisor
Sencun Zhu, Thesis Honors Advisor
Keywords:
automated grading
machine learning
natural language processing
classification
deferral
neural networks
student assessment
Abstract:
This thesis investigates how relational neural models and selective deferral strategies can improve the reliability of automated short-answer grading (ASAG). While most ASAG systems focus on overall accuracy, this work looks more closely at how models behave on borderline responses, how their errors can be interpreted, and how they might work alongside human graders. Using the I-STUDIO dataset, we compare two graph-based models—SFRN and AsRRN—on both accuracy and patterns of disagreement. We use selective prediction with a logistic regression confidence model and apply a joint training strategy (JTSP) that helps the system decide when to pass questions to a human. Custom scripts analyze misclassified responses by type, confidence level, and language features. Results show that both models achieve comparable overall accuracy (~82%) but diverge systematically on low-confidence inputs. The JTSP framework improves deferral accuracy while limiting the number of human-labeled cases, achieving a deferral rate under 3% with accuracy gains above 2%. Additionally, we test cross-domain generalization by applying the learned deferral policy to the MNLI dataset. Overall, combining model confidence with learned deferral seems to make grading more reliable and better suited for shared decision-making between models and humans.