How we work

Using our database enables us to expose unfair English proficiency test practices through advanced statistical methods and artificial intelligence.

When analyzing the database, we focus on two questions:

• Are there unexpected patterns from a statistical point of view?
• If so, can these patterns be replicated for several combinations?

Unexpected patterns in English proficiency tests:

If you roll a dice six times, you cannot expect to get each number once. However, if you roll the dice 600 times, you would expect each number to appear around 100 times, or the procedures may be biased. For example, the dice might be weighted to favor the number six. This simple explanation explains why we need as many test results as possible. 

A similar logic applies to English proficiency tests. But, again, one would expect a specific distribution of outcomes, and if this pattern is not given, it is time to step in and find out why. 

Replication of patterns:

We cannot say what procedure is the best, as we can only analyze the data to see if it meets the statistical expectations. 

Quick example:

By analyzing our existing 33.000 datasets, we concluded that the IELTS Writing macro skill did not meet what was expected: The Writing scores were generally lower than those of Speaking, Listening, and Reading. We then explored the reasons behind this discrepancy.

One can argue that the Writing macro skill is difficult to learn. But if that is so, we should see the same distribution of Writing for General Training and Academic tests and the same distribution for each country the test is taken. If the results for different combinations differ, the test itself is not fair because it measures other variables depending on different combinations. 

The outcome for this example was that "Writing" was more stable in the Academic modules and stable in the General Training module for one specific country. Still, there are entirely different results for some other countries. The results of the Writing macro skill appear to be arbitrary. These results assume that the tests are used to reach specific immigration goals rather than measuring English language skills. It can also be seen to exclude applicants from certain countries.

And this is what we want to avoid: If language tests are used as gatekeepers, governments have to explain this to all interested persons. Otherwise, people are expected to pay and sit for English language assessment tests time and time again. For example, we have one test candidate who sat for the IELTS test 25 times. So yes, English language assessment tests are the perfect business model for the governments and the language test providers. Still, it is often the poor and disadvantaged who pay for the tests repeatedly.