An Empirical Assessment of Generative AI in Police Narrative Creation

open_access
Open Access
Author:
Nagiub, Maya
Area of Honors:
Data Science
Degree:
Bachelor of Science
Document Type:
Thesis
Thesis Supervisors:
Dr. Dana Calacci, Thesis Supervisor
John Yen, Thesis Honors Advisor
Keywords:
AI-generated police reports
generative AI
factual accuracy
hallu- cination
body-worn cameras
accountability
FActScore
Abstract:
The use of generative artificial intelligence in police narrative writing has raised widespread concerns regarding accuracy, bias, and accountability. This study provides an empirical evaluation of AI-generated police reports by assessing factual correctness and narrative quality. A dataset of 80 body-worn camera videos was collected and transcribed using automatic speech recognition. AI generated police report narratives were then produced by prompting Gemini, and then were decomposed into atomic facts, which were evaluated by human coders for accurate, inaccurate, and unsupported content. Narrative quality was assessed using an eight question Likert scale framework measuring structural completeness and procedural adequacy. The model achieved a mean factual accuracy rate of 0.742, with an average hallucination rate of 0.034 and an average inaccuracy rate of 0.223. Narrative quality averaged 2.61 out of 4, indicating moderate structural completeness. Performance varied substantially across reports. These findings suggest that the primary risk of AI-generated police reports lies in the distortion and incomplete representation of real events. In the absence of transparency and audit mechanisms, such errors may go undetected, raising concerns about accountability and potential impacts on legal outcomes.