AI assistance in peer feedback contexts leads students to rely on AI rather than learn from it, potentially undermining student agency in self-regulated learning.

Objective: The main goal of this study was to investigate how AI-powered educational technologies impact students' agency and ability to self-regulate their learning, specifically examining whether students learn from AI assistance or merely rely on it without developing independent skills.

Methods: The researchers conducted a randomized controlled experiment with 1,625 undergraduate students across 10 courses from various disciplines at a four-year public university. The study used a peer review system (RiPPLE) where students evaluated learning resources created by peers. For the first four weeks, all students received AI-powered prompts that utilized techniques like rule-based suggestion detection, semantic similarity, and comparison with previous comments to enhance their feedback submissions. For the following four weeks, students were randomly divided into four groups:

• Control group (AI): Continued receiving AI prompts
• Experiment 1 (NR): No longer received AI prompts
• Experiment 2 (SR): Received self-monitoring checklists instead of AI prompts
• Experiment 3 (SAI): Received both AI prompts and self-monitoring checklists

The researchers measured several key metrics including the rate of flagged reviews, similarity to previous comments, relatedness to resources under review, comment length, time spent on reviews, and helpfulness ratings.

Key Findings:

• Students who lost AI assistance (NR group) showed significant decreases in feedback quality compared to those who continued receiving AI support, with higher rates of flagged reviews (+17%), more similarity to previous comments (+4%), less relatedness to resources under review (-4%), and shorter comments (-4 words on average).
• Self-monitoring checklists (SR group) helped fill some of the gap left by removing AI assistance but were not as effective as AI assistance alone, showing higher flag rates (+12%) and lower relatedness scores (-3%) compared to the AI group.
• Combining AI assistance with self-monitoring strategies (SAI group) did not yield significant advantages over AI assistance alone, suggesting that when a stronger form of assistance is present, the contribution of a weaker approach may be overshadowed.
• Students generally tended to rely on AI assistance rather than learning from it, as evidenced by the deterioration in performance when support was removed.

Implications: The study reveals important insights into the complex relationship between AI and student agency in educational contexts. While AI-powered tools can effectively scaffold learning and improve performance, there's a risk they may foster dependency rather than developing students' self-regulation skills. This highlights the need for a balanced approach between leveraging AI capabilities and fostering student agency. As AI becomes increasingly integrated into education, careful consideration must be given to designing systems that support rather than supplant student agency.

Limitations: The experiment was limited to the first eight weeks of the semester, making it difficult to assess longer-term impacts. The study focused on one specific educational context (peer feedback) and relied primarily on quantitative measures without including qualitative data from interviews or open-ended questions with students and instructors. Additionally, course content, teaching methods, student motivation, and instructor preferences could have influenced the outcomes.

Future Directions: Future research should explore the long-term impacts of AI assistance on student agency across different educational contexts and with larger participant groups. More investigation is needed into the design of hybrid human-AI systems that enhance student agency rather than diminish it. The study also calls for exploring potential challenges posed by AI-generated feedback, such as data-based biases and new conflicts in learning environments, and for replicating this study with other learning tools beyond peer feedback.

Title and Authors: "Impact of AI assistance on student agency" by Ali Darvishi, Hassan Khosravi, Shazia Sadiq, Dragan Gašević, and George Siemens

Published On: November 30, 2023

Published By: Computers & Education (Elsevier)