Automated Assessment of Idea Inspiration in Human-AI Co-Creativity

open_access
Open Access
Author:
Gonzalez, Mackenzie
Area of Honors:
Psychology
Degree:
Bachelor of Science
Document Type:
Thesis
Thesis Supervisors:
Roger Beaty, Thesis Supervisor
Rick Gilmore, Thesis Honors Advisor
Keywords:
Creativity
Artificial Intelligence
Idea inspiration
Automated assessment
Abstract:
The rapid integration of generative artificial intelligence (GenAI) has transformed how individuals generate and evaluate creative ideas. Despite growing interest in human–AI co-creativity, little research has examined how to reliably identify the source of ideas produced in these collaborations. This study investigated whether a large language model (LLM) can accurately detect the inspiration source of human–AI co-created ideas. Participants completed two creativity tasks, an engineering design task and a story title generation task, with optional AI assistance. An LLM (GPT-5) assigned inspiration labels based on participant responses and chat transcripts, and its performance, along with the participant self-ratings, was compared to expert judgments. Results showed that self-ratings had weak agreement with expert evaluations, particularly for open-ended design tasks. In contrast, both zero-shot and few-shot LLM conditions demonstrated moderate to good agreement, with zero-shot prompting slightly outperforming few-shot in categorical accuracy. Performance was higher for the title generation task than for the engineering design task. These findings suggest that LLMs can effectively automate the assessment of idea inspiration in human–AI collaboration, while highlighting challenges in detecting AI influence in complex tasks.