Experiments with Manual and Automated Content Analysis of Student Writing
Open Access
Author:
Rubin, Noah
Area of Honors:
Computer Science
Degree:
Bachelor of Science
Document Type:
Thesis
Thesis Supervisors:
Rebecca Jane Passonneau, Thesis Supervisor Danfeng Zhang, Thesis Honors Advisor
Keywords:
summary annotation nlp
Abstract:
The task of analyzing and grading the content of student writing presents many challenges and questions. We present some of those questions here and discuss rigorous and objective methods to answer them. We discuss the task of grading student-written summaries and introduce the pyramid annotation method to perform this task, as well as the PyrEval algorithm that automates it. We also discuss the efficacy of applying this method to tasks that are not strictly summaries. We consider the task of determining the level of agreement among multiple graders and introduce Krippendorff's alpha, a metric that quantifies this agreement. Finally, we consider the task of constructing vectors that encode word meanings and discuss and compare two algorithms that accomplish this task in different ways to ultimately determine which is better suited for the task of grading student summaries.
