K-12 students can learn effective AI prompting skills through hands-on practice with immediate AI-generated feedback, significantly boosting their confidence in using AI for educational purposes.
Objective: The main goal of this study was to design, implement, and evaluate a web-based instructional module that teaches "prompting literacy" to secondary school students (grades 6-12). The researchers aimed to equip students with the skills to effectively communicate with AI chatbots for learning purposes, focusing on three core learning objectives: understanding AI's capacity for supporting learning, identifying appropriate contexts for AI use in education, and developing effective prompt formation skills. The study sought to address the growing need for AI literacy education in K-12 settings, particularly as AI tools become increasingly integrated into daily life and educational environments.
Methods: The research employed a comprehensive two-study approach involving 111 students across 11 authentic secondary education classrooms in East Asia. The instructional module featured three hypothetical learning scenarios spanning biology, geography, and mathematics subjects. Students engaged in a four-step practice pipeline: scenario introduction, prompt creation and AI interaction, receiving AI-generated responses, and obtaining automated feedback through an AI auto-grader. The module utilized OpenAI's GPT-4o for both generating responses and providing automated grading based on six key dimensions including relevance, clarity of purpose, conciseness, background context, request elaboration, and avoiding direct answer-seeking. The evaluation methodology incorporated pre- and post-tests, Likert-scale confidence surveys, and qualitative feedback collection. Study 1 focused on measuring AI auto-grader accuracy, student learning outcomes, and user experience, while Study 2 refined the assessment approach by replacing multiple-choice questions with True/False and open-ended questions to better capture student understanding and skills.
Key Findings: The AI-based auto-grader demonstrated impressive accuracy, achieving an average of 92% accuracy across all grading dimensions when compared to human evaluators, with particularly high performance in relevance (98%), background context (96%), and elaboration (90%) categories. Students showed significant improvement in incorporating background context into their prompts, with performance increasing significantly from the first to third practice question (p = .039). Most notably, student confidence in using AI for learning increased by 10.4% on average following the instructional intervention (p < .001). Qualitatively, 87% of students reported learning valuable AI-related knowledge, including understanding appropriate AI use contexts and effective prompting strategies. However, the study revealed a ceiling effect in pre-test scores, indicating that while students could conceptually identify good prompts, their actual prompt-writing abilities varied considerably. The assessment iteration in Study 2 showed that True/False and open-ended questions provided better measurement capabilities than multiple-choice questions, with 60% of open-ended questions and 30% of True/False questions meeting quality assessment criteria.
Implications: This research makes significant contributions to the emerging field of AI literacy education by providing one of the first empirically-validated instructional frameworks specifically designed for K-12 prompting literacy. The findings suggest that scenario-based, hands-on learning approaches combined with immediate AI-generated feedback can effectively teach students to interact more skillfully with AI systems. The study demonstrates the feasibility of using AI systems themselves as educational tools for teaching AI literacy, creating a scalable model for widespread implementation. The research also highlights the importance of moving beyond passive AI literacy instruction toward active, experiential learning that develops practical skills students can apply across various academic contexts. The successful implementation of AI auto-grading for prompt evaluation opens possibilities for scalable, personalized feedback in AI literacy education.
Limitations: The study acknowledges several important limitations. The sample size was relatively small (111 students) and geographically concentrated in East Asia, potentially limiting generalizability to other cultural and educational contexts. The research lacked comparison with alternative AI literacy instruction methods, making it difficult to assess the relative effectiveness of this specific approach. Technical limitations emerged in the AI auto-grader's performance on certain dimensions, particularly in interpreting prompt intentions and distinguishing between direct and indirect answer-seeking. Students also reported practical challenges including slow AI response times, limited subject diversity in scenarios, and basic computer skills barriers that hindered some participants' engagement with the platform. The study's duration was limited, preventing assessment of long-term retention and skill transfer.
Future Directions: The researchers propose several important avenues for future investigation. Scaling the approach to larger, more diverse populations will enable more robust statistical analyses using advanced psychometric models like Item Response Theory. Comparative studies evaluating this approach against other AI literacy instruction methods are essential for establishing best practices. Future research should explore expanding scenario diversity to include more subjects and contexts that align with varied student interests and backgrounds. Long-term longitudinal studies are needed to assess skill retention and transfer across different learning contexts. Additionally, the researchers suggest investigating the integration of this prompting literacy instruction with existing curriculum frameworks and exploring how these skills develop as AI technologies continue to evolve.
Title and Authors: "Learning to Use AI for Learning: How Can We Effectively Teach and Measure Prompting Literacy for K–12 Students?" by Ruiwei Xiao, Xinying Hou, Ying-Jui Tseng, Hsuan Nieu, Guanze Liao, John Stamper, and Kenneth R. Koedinger.
Published On: August 19, 2025
Published By: arXiv (arXiv:2508.13962v1 [cs.HC])