Large language models like ChatGPT improve overall writing quality for college students but create new socioeconomic inequalities in educational benefits, with high-SES students better leveraging these tools to overcome language barriers.

Objective: The main goal of this study was to examine the real-world impacts of large language models (LLMs) like ChatGPT on educational equity by analyzing how these tools affect academic writing quality gaps between linguistically advantaged and disadvantaged college students from different socioeconomic backgrounds.

Methods: Researchers analyzed 1,140,328 academic writing submissions from 16,791 college students across 2,391 courses between 2021 and 2024 at a public, minority-serving institution in the US. They examined two post-LLM phases: Phase 1 (January to June 2023) when most people were still exploring the technology, and Phase 2 (October 2023 to March 2024) after a period of adjustment and training. Writing quality was measured through three composite indices: readability, lexical diversity, and syntactic complexity. Each index was constructed by averaging standardized scores of established linguistic measures. The researchers conducted linear regression analyses to assess changes in writing quality across different student groups based on linguistic background (international vs. domestic students, non-English vs. English home language speakers, lower vs. higher entering writing scores) and socioeconomic status (low vs. high family income, first-generation vs. continuing-generation college students).

Key Findings:

• Overall writing quality gradually increased following LLM availability, with Phase 2 showing substantial improvements (0.086 to 0.162 standard deviations) across all language proficiency indices.
• The writing quality gaps between linguistically advantaged and disadvantaged students became increasingly narrower over time, with disadvantaged students showing significantly larger improvements.
• In Phase 1, linguistically disadvantaged students improved by 0.044 standard deviations more than their advantaged peers on average; in Phase 2, this differential improvement increased to 0.088 standard deviations.
• However, among linguistically disadvantaged students, those from higher socioeconomic status backgrounds experienced greater writing improvements than their low-SES peers.
• Students from low-income families experienced 0.025 standard deviations less writing improvement on average than their higher-income peers; first-generation college students experienced 0.015 standard deviations less improvement than continuing-generation students.
• These socioeconomic gaps in writing improvement existed in Phase 1 and persisted in Phase 2, suggesting that high-SES students were better able to leverage LLM tools to overcome language barriers.

Implications: This study provides evidence of complex patterns of digital inequalities in the era of LLMs and highlights the need for policy responses to carefully harness the power of LLMs for educational equity. While LLMs show potential to help close language proficiency gaps, their benefits are not uniformly distributed across socioeconomic groups. These findings align with established research on digital divides, where new technologies tend to create inequalities that mirror existing social and economic disparities. The study emphasizes the importance of researching novel technologies "in the wild" beyond controlled environments to understand the multifaceted factors that may complicate their actual impacts on educational activities and outcomes.

Limitations: The study focused specifically on academic writing submissions to forum-based assignments, which typically contribute only marginally to final course grades and are not usually assessed based on writing quality. The researchers were not able to observe significant changes in grade performance, though they suggest more prominent disparities might exist in higher-stakes writing tasks. Additionally, the study did not estimate the actual usage of LLM tools, focusing instead on observed writing outputs and their quality.

Future Directions: The researchers advocate for a balanced approach between innovation and responsibility in LLM development and implementation. They suggest that developers should embed ethical considerations into the development process from the beginning, addressing issues like stereotypes in training data. Educators should closely monitor real-world impacts of LLM applications, especially for learners from disadvantaged backgrounds. Policymakers need to establish context-specific guidelines for equitable and accountable deployment of LLMs. The researchers also call for more research on digital inequalities across a wider variety of educational activities, capturing subtleties in access, literacies, engagement, and outcomes.

Title and Authors: "Whose ChatGPT? Unveiling Real-World Educational Inequalities Introduced by Large Language Models" by Renzhe Yu, Zhen Xu, Sky CH-Wang, Richard Arum

Published On: November 2, 2024

Published By: arXiv (preprint repository)