Advancing Language Assessment with GPT: Is It Nonnative-Language Friendly?

초록

Despite the growing interest in AI for language assessment, there remains a significant research gap regarding its usefulness for assessing less proficient language skills, particularly those of learners of English as a second or foreign language (S/FL). AI models often prioritize proficient writing, neglecting the intricacies of learner language. This study addresses this gap by evaluating the capacity of AI to replicate the written performances of both L1 and L2 novice academic writers, with a focus on formulaic language. Using an AI tool based on the GPT-2 model, two AI-generated essay corpora were created to closely resemble existing L1- and L2-English corpora of argumentative essays by first-year university students. The comparative analysis reveals AI's inability to accurately replicate S/FL learner language. Notably, the AI model failed to generate typical "learner bundles," which are formulaic language patterns considered unique to or characteristic of English S/FL learners. The findings of this study indicate an urgent need for improving AI technology to more effectively represent the diverse linguistic attributes of S/FL learner language, an improvement which may play a critical role in fair and effective language assessment.

키워드