This study examines the impact of a Generative Artificial Intelligence (GenAI)-powered formative assessment tool that employs Socratic questioning to support student learning in a large, fully online undergraduate-level computing course. Employing a quasi-experimental, mixed-methods design, the team investigated participants’ engagement patterns, the influence of user experience on engagement, and impacts on both perceived and actual learning outcomes. The team collected data from system logs, user experience surveys, perceived engagement and learning gains, student reflections, and course performance data. Results indicated that participants consistently reported high levels of affective, behavioral, and cognitive engagement, and that positive user experiences were strongly linked to these outcomes and to perceived learning gains. Quantitative analysis further revealed that students who engaged with the GenAI tool experienced significant gains in their quiz scores compared to those who did not, particularly benefiting students with lower baseline achievement.

Additionally, thematic analysis of qualitative feedback revealed substantial perceived improvements in higher-order thinking skills, including problem-solving, critical thinking, and self-reflection. Their findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills. As higher education institutions expand the integration of GenAI in the curriculum, this dialogic, GenAI-powered assessment tool can offer a scalable strategy to promote meaningful student learning outcomes.