NLP Annotation Tools for Manual Markup of Content in Short Summaries and Essays

Open Access
- Author:
- Driban, Alexander
- Area of Honors:
- Computer Science
- Degree:
- Bachelor of Science
- Document Type:
- Thesis
- Thesis Supervisors:
- Rebecca Jane Passonneau, Thesis Supervisor
Dr. John Joseph Hannan, Thesis Honors Advisor - Keywords:
- Natural Language Processing
Application Development
Argumentation Mining
Summarization
Annotation
Education - Abstract:
- Automated evaluation of students’ reading and writing skills could enable teachers to more efficiently assess student abilities. One important skill is mastery of content: a student’s ability to understand reading material and demonstrate their understanding through short summaries or essays about what they have read. Student mastery of content can be evaluated by comparing students’ written summaries to those written by a “wise crowd,” considered to be a gold standard of content mastery. There are several automated methods for building the wise crowd content model (called a pyramid) and scoring student summaries, such as PyrEval. These methods have been tested against manual methods for accuracy using DUCView, a tool for performing and collecting manual content annotations of summaries. However, PyrEval and DUCView are only suitable for simple summaries. I have developed a new tool called SEAView, using the DUCView source code as a starting point, for content annotation of essays that have a special format. This special format includes a summary in the header of the essay, followed by a body that makes an argument. DUCView was last updated in 2005, and the original Java source code has since been lost. However, the DUCView JAR file has been decompiled using two decompilers to recover the code. I have made modifications to the DUCView source code and created this new tool using the decompiled Java code. The manual annotations created using this tool will be used for developing, training, and testing machine learned models to performed automated annotations. It will also be used to study the relationship between summary and essay content, and for scoring essays of this format according to their content.