• Inyang, Oto-Obong
• Testrow, Craig Paul
• Mizouri, Arin
• Yeadon, Will
• Peach, Alex

Abstract

<jats:title>Abstract</jats:title><jats:p>The latest AI language modules can produce original, high quality full short-form (300-word) Physics essays within seconds. These technologies such as ChatGPT and davinci-003 are freely available to anyone with an internet connection. In this work, we present evidence of AI generated short-form essays achieving First-Class grades on an essay writing assessment from an accredited, current university Physics module. The assessment requires students answer five open-ended questions with a short, 300-word essay each. Fifty AI answers were generated to create ten submissions that were independently marked by five separate markers. The AI generated submissions achieved an average mark of <jats:inline-formula><jats:tex-math><?CDATA $712\%$?></jats:tex-math><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"><mml:mn>71</mml:mn><mml:mo>±</mml:mo><mml:mn>2</mml:mn><mml:mi mathvariant="normal">%</mml:mi></mml:math><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="pedacc5cfieqn1.gif" xlink:type="simple" /></jats:inline-formula>, in strong agreement with the current module average of <jats:inline-formula><jats:tex-math><?CDATA $715\%$?></jats:tex-math><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"><mml:mn>71</mml:mn><mml:mo>±</mml:mo><mml:mn>5</mml:mn><mml:mi mathvariant="normal">%</mml:mi></mml:math><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="pedacc5cfieqn2.gif" xlink:type="simple" /></jats:inline-formula>. A typical AI submission would therefore most-likely be awarded a First Class, the highest classification available at UK universities. Plagiarism detection software returned a plagiarism score between <jats:inline-formula><jats:tex-math><?CDATA $21$?></jats:tex-math><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"><mml:mn>2</mml:mn><mml:mo>±</mml:mo><mml:mn>1</mml:mn></mml:math><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="pedacc5cfieqn3.gif" xlink:type="simple" /></jats:inline-formula>% (Grammarly) and <jats:inline-formula><jats:tex-math><?CDATA $72$?></jats:tex-math><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"><mml:mn>7</mml:mn><mml:mo>±</mml:mo><mml:mn>2</mml:mn></mml:math><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="pedacc5cfieqn4.gif" xlink:type="simple" /></jats:inline-formula>% (TurnitIn). We argue that these results indicate that current natural language processing AI represent a significant threat to the fidelity of short-form essays as an assessment method in Physics courses.</jats:p>

Topics