The Simpson paradox of school grading in Italy

Abstract

Data from the 2003 OECD-PISA Survey for Italy reveal a striking difference in the relationship between students’ competence (as measured by PISA score in Mathematics) and school grades across regions: a competence level granting bare sufficiency in the North yields excellence grades in the South. This has spurred a lively debate on education policy in the country, based on the inference drawn from this evidence that grading practices are excessively different in the two areas. We show in this note that this inference overlooks a Simpson paradox hidden in the data. After a more careful analysis, the above inference is seen to be wrong. The crucial omitted variable is the school-level average competence: schools with low-performing students, all over the country, inflate grades. Students in the South get higher grades simply because they are in weaker schools; grading policy is actually homogeneous across regions.