This study successfully developed and validated an 8-dimensional AI competency framework and SAICS scale that enables educators to systematically assess K-12 students' confidence in using AI ethically and productively for learning.
Objective: The main goal of this study was to propose a comprehensive AI competency framework for K-12 students from teachers' perspectives and develop and validate the Student AI Competency Self-Efficacy (SAICS) scale. The researchers aimed to address the gap in reliable instruments for measuring student AI competency, moving beyond existing frameworks designed for higher education or workplace settings to create age-appropriate assessment tools for younger learners.
Methods: The study employed a rigorous two-stage mixed-methods approach. Stage 1 involved a three-round Delphi study with 29 experienced AI teachers from Hong Kong who met strict selection criteria including undergraduate or graduate degrees in relevant fields, at least 3 years of AI teaching experience, and completion of 12+ hours of professional development in AI education. These experts evaluated and refined dimensions and items through iterative rounds until reaching 75% agreement consensus. Stage 2 involved quantitative validation with 448 students aged 12-17 from six Hong Kong secondary schools across diverse socioeconomic backgrounds. The researchers used confirmatory factor analysis (CFA) and model comparisons to validate the scale's reliability and examine gender differences in AI competency self-efficacy.
Key Findings: The research yielded several significant outcomes. The final framework comprises eight distinct dimensions of AI competency: Interdisciplinary Learning with AI (ILAI), Assessment with AI (ASAI), Decision Making with AI (DMAI), Data, Ethics, and AI (DEAI), Designing AI (DGAI), Multimedia Creation with AI (MCAI), Human-Centric Learning (HCLG), and Confidence with AI (CFAI). Each dimension contains four validated items, resulting in a 32-item SAICS scale. The validation analysis demonstrated excellent psychometric properties with Cronbach's alpha values ≥0.89 for all dimensions, factor loadings ranging from 0.76 to 0.96, and strong model fit indices (CFI=0.95, RMSEA=0.056). Importantly, the scale showed measurement invariance across gender groups, indicating consistent reliability for both male and female students. The framework emphasizes students' capacity to use AI ethically, safely, and productively in their learning, incorporating contemporary concerns about generative AI technologies like ChatGPT.
Implications: This research makes substantial contributions to AI education by providing the first validated framework specifically designed for K-12 students based on teacher expertise rather than academic or professional perspectives. The SAICS scale enables researchers to design and evaluate AI education interventions, helps teachers develop appropriate learning objectives for AI-based activities, and provides policymakers with tools to establish national AI competency standards. The framework addresses critical contemporary issues including ethical AI use, multimedia creation capabilities, and human-centric learning approaches that ensure students maintain agency in their learning process rather than becoming overly dependent on AI tools.
Limitations: The study acknowledges several important limitations. The framework focuses primarily on AI competency without comprehensively incorporating related literacies such as data, mathematics, science, and computational thinking. The scale lacks discipline-specific validation and focuses solely on general learning competency rather than subject-specific applications. The research examined only gender differences without considering other potentially influential factors like culture and socioeconomic background. Most significantly, the study was conducted exclusively with Chinese students in Hong Kong, potentially limiting generalizability to other cultural contexts and educational systems.
Future Directions: The researchers outline several critical areas for future investigation. They recommend developing comprehensive frameworks that integrate AI competency with related literacies, conducting discipline-specific validation studies across different subject areas, and examining the impact of cultural and socioeconomic factors on AI competency development. Additionally, they emphasize the need for adaptation and validation in diverse international contexts including the Middle East, South Asia, United States, Europe, United Kingdom, and Australia to ensure cross-cultural applicability and effectiveness.
Title and Authors: "Validating student AI competency self-efficacy (SAICS) scale and its framework" by Thomas K. F. Chiu, Murat Çoban, Ismaila Temitayo Sanusi, and Musa Adekunle Ayanwale.
Published On: May 12, 2025 (accepted), May 27, 2025 (published online)
Published By: Education Tech Research Dev (Educational Technology Research and Development journal), Springer