Saudi experiment measures AI impact on writing: when does editing outperform generation

While institutions race to delegate writing tasks to generative AI models, a quasi-experimental study from a Saudi university revealed that the actual leap in text efficiency occurs specifically when automatic generation features are disabled and the machine’s role is limited to constrained editing.

We examined the details of the study published in the open-access journal Heliyon and carried out by researchers Dr. Ismail, Fares al-Luhyani, Khalid Ahmad Abdul Al-Ibrahim, and Muhammad Ahmed Suleim Khasouna; it tracked the performance of sixty university students majoring in English, aged between 20 and 29, all classified as intermediate according to the Oxford placement test. The researchers divided the participants into two study groups of thirty each, which followed the same methodology under the same lecturer for twelve weeks, with identical ten essay tasks, three draft rounds per task, and the same evaluation criteria and deadlines.

The essential difference was granting the experimental group access to the Grammarly platform during drafting and revision, but with a strict design:Generative rephrasing, text generation, plagiarism detection, and citation features were disabled, and usage was limited to checking grammar, spelling, punctuation, clarity, brevity, and word choice.Before the experiment began, the students received two orientation sessions of 45 minutes each to train them to examine the system’s suggestions and not accept them automatically without aligning them with the intended meaning.

Mixed-effects linear models showed a decisive advantage for the group supported by automated editing across all four performance indicators compared with the control group; writing quality increased by 5.42 points on a twenty-point scale for the experimental group versus 1.47 points for the control, a gain of 3.95 points and a standardized effect size of 1.77. Motivation scores jumped by 13.50 points versus 4.40, benefit from feedback rose by 4.44 points versus 0.93, and self-regulated learning strategies increased by 13.90 points versus 2.23.

These figures did not stop at the questionnaires but were reflected directly in editorial behavior across successive drafts evaluated by independent judges.Seventy-seven percent of the experimental group achieved the highest levels of accurately integrating feedback without distorting meaning, compared with forty-three percent in the traditional group.Eighty percent of the experimental students addressed all feedback points compared with sixty percent, while seventy percent made revisions that went beyond superficial correction versus fifty percent of their peers, and eighty-five percent retained the improvements in subsequent drafts versus sixty-five percent.

In the semi-structured interviews, participants praised the rapid detection of errors during drafting and the tool’s ease of use, but they clearly criticized its shortcomings; some suggestions were generic or unsuitable for context in complex constructions, and the students emphasized their need for support in developing ideas and building arguments, a capability not provided by single-sentence tools. The students attributed credit to the lecturer for guiding them on when to accept, modify, or reject a suggestion entirely, keeping the human writer’s effectiveness as the primary driver.

The researchers identified clear methodological limitations for the experiment: reliance on existing study groups prevented individual random allocation, the novelty and enthusiasm for new technology may explain part of the motivation increase, self-reporting was used to measure organization and motivation, and there was no delayed follow-up test to verify the long-term sustainability of the gains.

The operational value of these results for managers of work teams or vocational training programs in the Gulf region lies in correcting the trajectory of reliance on AI tools.The common tendency to outsource report and correspondence writing to generative models empties competencies of logical structuring and critical thinking skills; the Saudi model shows that restricting the machine to rapid feedback tasks and integrating it with human supervision provides a professional capable of self-organization and argument control without becoming merely a conduit for unvetted text.