An AI-driven predictive model using XGBoost with L2 regularization, SMOTER, and Active Learning successfully predicts optimal Mathematics Education Technology (MET) usage duration, with model-guided implementation significantly outperforming traditional methods in middle school mathematics instruction.

Objective

The primary goal of this study was to develop and validate an AI-based predictive model to optimize the effectiveness of Mathematics Education Technology (MET) in K-12 classrooms, specifically focusing on determining optimal usage duration across different instructional contexts. The researchers aimed to address the inconsistent effectiveness of MET by creating a data-driven framework that could guide teachers—particularly novice educators—in making informed decisions about when and how long to use educational technology for specific teaching objectives and content areas.

Methods

The study employed a comprehensive methodology integrating meta-analysis with machine learning. Researchers collected data from 423 publications on MET effectiveness in Chinese K-12 mathematics education, covering 38,163 students. Using the PRISMA framework, they systematically identified studies meeting six criteria, including experimental or quasi-experimental design, control and experimental groups, and measurable mathematics achievement outcomes.

Eleven potential predictor variables were identified: mathematics topics, three teaching content categories, mathematical abilities, grade level, educational assessment methods, teaching methods, technology type, sample size, and MET usage duration. The researchers calculated Hedges' g effect sizes for each study and conducted extensive subgroup analyses to examine how moderators influenced MET effectiveness.

Nine AI-driven predictive models were developed and tested under three optimization frameworks: base model, SMOTER with L2 regularization, and SMOTER with Active Learning and L2 regularization. The models included Linear Regression, Bayesian Regression, Support Vector Regression, Decision Tree, Random Forest, Gradient Boosting, Artificial Neural Network, K-Nearest Neighbor, and eXtreme Gradient Boosting (XGBoost). Performance was evaluated using R², MAE, MSE, and RMSE metrics with 5-fold cross-validation.

The best-performing model underwent hyperparameter optimization using Particle Swarm Optimization, and SHAP (Shapley Additive Explanations) values were calculated to assess feature importance. Finally, a controlled experiment with eight middle school classes in southeastern China validated the model's practical applicability, comparing model-guided MET implementation against traditional approaches and a comparison control group.

Key Findings

The meta-analysis revealed that MET has a significant positive effect on K-12 mathematics learning with an overall effect size of g=0.677 (p<0.001). Publication bias tests confirmed the dataset's reliability, while subgroup analyses showed that effectiveness varied significantly across moderators. Foundation mathematical concepts yielded larger effects than advanced concepts, primary school students benefited more than high school students, and problem-solving abilities showed the strongest gains among mathematical competencies.

Among the nine predictive models tested, XGBoost consistently outperformed others across all three optimization frameworks. The final PSO-optimized XGBoost model achieved R²=0.842, MAE=0.084, MSE=0.021, and RMSE=0.146, representing substantial improvement over the base model. SHAP analysis revealed that MET usage duration was the most influential predictor (48.5% importance), followed by mathematics topics (9.5%) and teaching content (8.6%).

Crucially, the analysis demonstrated an inverted U-shaped relationship between MET usage duration and effectiveness: both insufficient and excessive use reduced benefits, while moderate usage yielded optimal outcomes. The controlled experiment validated these findings, showing that model-guided MET (18 teaching periods) significantly outperformed both the control group (no MET) and the indirect control group (14 periods based on teacher experience) with p-values all less than 0.001. The effect sizes for the six experimental groups ranged from 0.727 to 0.764, all exceeding the meta-analytic average of 0.677.

Implications

This research establishes a replicable framework for conducting predictive learning analytics when large-scale educational experiments are impractical. By demonstrating that meta-analysis can serve as a viable data mining approach for AI model development, the study opens new possibilities for evidence-based educational technology implementation. The finding that usage duration follows an inverted U-shaped relationship with effectiveness challenges the "more is better" assumption and provides educators with actionable guidance for calibrating technology integration.

For practitioners, the model offers data-driven recommendations that can help bridge the experience gap between novice and veteran teachers, promoting educational equity. The framework also highlights the importance of context-sensitive implementation, showing that optimal MET use varies by content type, grade level, and instructional objectives.

Limitations

The study's limitations include its relatively small sample size of 59 participants and focus solely on Chinese educational contexts, which may limit generalizability to other cultural and educational settings. The different delivery formats (intensive 2.5-day training for inservice teachers versus 6-week course for preservice teachers) introduced potential confounding variables. Both participant groups had limited opportunities to apply their learning in authentic teaching contexts, and the study relied exclusively on quantitative data without qualitative insights into user experiences or implementation challenges.

Additionally, the ethics component of AI knowledge did not show significant improvement across either group, suggesting inadequate coverage of critical topics like fairness, transparency, and accountability. The external validation, while positive, was limited to a single middle school context, and the model's R² on external data was relatively small, indicating room for improvement in prediction accuracy.

Future Directions

Future research should expand data collection to include broader geographic regions and employ advanced systematic search algorithms to capture more comprehensive datasets. Studies should explore process-oriented assessment methods beyond test scores to provide fuller pictures of student progress. Researchers should also investigate the model's applicability across diverse educational contexts with larger, more varied participant groups and longer follow-up periods to assess sustained impact.

Additional directions include developing methodologies to assess publication bias more comprehensively, investigating individualized MET interventions to prevent educational outcome polarization, and expanding the framework to subjects beyond mathematics. Qualitative research methods should complement quantitative approaches to better understand implementation mechanisms and contextual factors influencing MET effectiveness.

Title and Authors

Title: "AI-driven predictive models for optimizing mathematics education technology: Enhancing decision-making through educational data mining and meta-analysis"

Authors: Aneng He, Wenwen Yuan, Lai Soon Lee, and Tian Tian

Published On

The article was received on May 29, 2025, and accepted on September 30, 2025. It was published online on October 16, 2025.

Published By

The article was published in Smart Learning Environments (https://doi.org/10.1186/s40561-025-00415-z), an open-access journal. The article is licensed under a Creative Commons Attribution 4.0 International License.