Скачать или смотреть Comparing ChatGPT and Gemini understanding the IELTS scale 1

I offered a week ago to adapt my 9-step prompt for students to use to get feedback on their speaking, which uses the CEFR scale and the Global Scale of English to work with the IELTS scale. In many countries the IELTS scale is used more often than the CEFR scale.

Unfortunately, after experimenting with seven different variations of my 9-step prompt with various differences between steps, I began to suspect that ChatGPT had no idea about how to rate texts or create them at different points on the IELTS scale. I wanted to test this.

Daniel Tena Calderon had experimented with using Gemini to do something very similar to what I’m trying to do with my 9-step prompt, so I decided to see how the two, ChatGPT and Gemini, compared using the same transcript and the same prompt.

I asked each of the LLMs to grade the five versions on the IELTS, CEFR and GSE scales and they both asserted that they had done what they had been asked to do: produce texts that go from the original to half a level up and then one level further and one level further again.

Gemini appeared to grade the original more severely, but appeared to reach higher levels later in the sequence.

I haven’t got a tool to independently check if the LLMs were right, but I have used an average of the CEFR grades given by Text Inspector and Pearson’s Text Analyzer as a reliable way to grade texts.

Gemini was much closer to the Check levels and ChatGPT was so far out that I think it is fair to say that ChatGPT simply doesn’t understand how to grade or create texts on the IELTS scale.

Before writing this, I made a video making the same point and illustrating it with the texts created by the two LLMs. As usual, I used Clipchamp to make a couple of edits and to generate the subtitles file, which I then used to create subtitles on YouTube. I experimented with various formats of the video and subtitles and uploaded the four formats to YouTube.