NLP Annotation Tools for Manual Markup of Content in Short Summaries and Essays

Open Access
Author:
Driban, Alexander
Area of Honors:
Computer Science
Degree:
Bachelor of Science
Document Type:
Thesis
Thesis Supervisors:
  • Rebecca Jane Passonneau, Thesis Supervisor
  • John Joseph Hannan, Honors Advisor
Keywords:
  • Natural Language Processing
  • Application Development
  • Argumentation Mining
  • Summarization
  • Annotation
  • Education
Abstract:
Automated evaluation of students’ reading and writing skills could enable teachers to more efficiently assess student abilities. One important skill is mastery of content: a student’s ability to understand reading material and demonstrate their understanding through short summaries or essays about what they have read. Student mastery of content can be evaluated by comparing students’ written summaries to those written by a “wise crowd,” considered to be a gold standard of content mastery. There are several automated methods for building the wise crowd content model (called a pyramid) and scoring student summaries, such as PyrEval. These methods have been tested against manual methods for accuracy using DUCView, a tool for performing and collecting manual content annotations of summaries. However, PyrEval and DUCView are only suitable for simple summaries. I have developed a new tool called SEAView, using the DUCView source code as a starting point, for content annotation of essays that have a special format. This special format includes a summary in the header of the essay, followed by a body that makes an argument. DUCView was last updated in 2005, and the original Java source code has since been lost. However, the DUCView JAR file has been decompiled using two decompilers to recover the code. I have made modifications to the DUCView source code and created this new tool using the decompiled Java code. The manual annotations created using this tool will be used for developing, training, and testing machine learned models to performed automated annotations. It will also be used to study the relationship between summary and essay content, and for scoring essays of this format according to their content.