How reliable is PISA?

    A very reliable and optimum way of doing intellectual measurement on world's population has come out as PISA test or Program for International Student Assessment. The question arises, how reliable is the PISA sample ? First let me re-state the sampling method of PISA.

PISA sampling method:- PISA asks for the educational boards to present a list of schools in the country. It then selects a specific number of schools randomly. After that, up to 30 students each are selected from each school and mean scores for test takers are presented. 

On first attempt, it may seem that "two-step randomization in sampling" ensures that indeed a random person is tested. there are a large number of sampling errors that can arise due to following reasons:-

1. Non-uniformity of student population in schools across a nation:-

Let me give the example of Qatar. An expert by the name of rec1man has published Qatar school rankings on http://www.unz.com/akarlin/the-puzzle-of-indian-iq-a-country-of-gypsies-and-jews/

The PISA scores for top 30 schools are as follows:-

Science, Math, Reading scores

1. Al-Khor Indian Stream, ( GEMS ) = 566, 592, 604 = 113 IQ = Indian Hindu technicians and Engineers of NGL
2. The International School of Choueifat ( SABIS ) = 554, 562, 565 = 109 IQ = Lebanese Xtian
3. Doha College Private ( British Embassy ) = 572, 553, 563 = 109 IQ = UK
4. DPS Modern Indian School ( Delhi Public School Society ) = 552, 538, 563 = 107 IQ = Indian Hindu
5. Qatar Academy ( US educators ) = 540, 547, 562 = 107 IQ
6. American School of Doha, ( US Embassy ) = 553, 546, 559 = 108 IQ
7. Park House English ( UK ) = 568, 528, 552 = 107 IQ
8. Birla Public School = 586, 539, 549 = 108 IQ = Indian Hindu
9. Qatar Intl Private School ( UK ) = 539, 529, 540 = 105 IQ
10. Al Bayan Girls = 481, 464, 516 = Muslim Arab = 98 IQ
11. Cambridge Intl Private School = 531, 484, 514 = 101 IQ
12. Doha Modern Indian School ( Jai Gopal Jindal ) = 554, 525, 514 = 104 IQ = Indian Hindu
13. Al-Khor British Stream ( GEMS ) = 507, 505, 503 = 102 IQ
14. Dukhan English School ( UK ) = 529, 501, 500 = 102 IQ

15. Debakey High School for Health ( USA ) = 492, 467, 493 = 98 IQ

16. Qatar Canadian School = 451, 456, 491 = 95 IQ
17. MES Indian School ( Muslim Education Society ) = 484, 469, 490 = 97 IQ = Indian Muslim
18. Ideal Indian School Girls, ( Muslim ) = 481, 450, 489 = 96 IQ = Indian Muslim
19. Sudanese School = 463, 411, 488 = 93 IQ , remarkably high for black-arab mullatos
20. Al Arqam = 454, 451, 484 = 95 IQ
21. The Gulf English = 468, 448, 482 = 95 IQ
22. Philipine School = 466, 461, 480 = 96 IQ
23. Jordanian School = 446, 422, 472 = 92 IQ
24. Tunisian School = 459, 436, 463 = 93 IQ
25. Lebanese School ( Muslim ) = 444, 501, 463 = 96 IQ
26. Middle East Intl = 484, 452, 461 = 95 IQ
27. Al Andalus = 446, 397, 454 = 90 IQ
28. Ideal Indian School, boys ( Muslim ) = 462, 465, 453 = 94 IQ = Indian Muslim
29. Egyptian School = 463, 435, 434 = 92 IQ
30. American Academy = 462, 434, 434 = 92 IQ

Qatar, 153 school average = 379, 368, 372 = 81 IQ

 Though the writer is calculating PISA IQ on 500 mean, 100 standard deviation. The real IQ has to be calculated at 518 mean, 100 standard deviation. 518 is the score for White Americans which is used as a standard for calculating IQ.

Anyways, the PISA procedure of calculating the mean scores for all schools isn't very reliable. If you look at many of the top 30 schools in Qatar, they enroll 3-4x more students than poor scoring schools.

The two GEMS school in Qatar are among the biggest schools as well and have 7x more students than smallest schools. If overall PISA score for a nation is to be calculated, it has to be done by "weighing for school size".

In Singapore, 167/171 schools participated in PISA. On one hand, schools from National University of Singapore (NUS High School of mathematics) and Nanyang Girls High School which are among the top schools besides Raffles Institute, have enrollment of 980 and 1500 students. There are large number of schools which have school enrollment as high as 7000 students.

Why does it matter ?

Well, differences in school size (unaccounted by PISA) can cause up to 10-15 IQ variation in overall overall reported scores of a nation.

Take the example of Qatar school rankings. Two GEMS schools scored 592 and 505 on maths PISA ranking in the top 30/153. These schools have student population 3x higher than national average. 

But only 30 students from these schools will be considered in PISA sample. There may be a few schools where population is 5x below national average. Even in those schools, only 30 students will take part.

If I am to count these Qatar GEMS schools as 3 schools (consistent with their school enrollment size), the average maths score of Qatar will jump to 376 from 368 (1.3 IQ jump).

In many nations, bigger sized schools have better facilities, better education and will have better scores which will be under-represented in PISA as one school is counted as one (irrespective of how much students are studying there) in PISA calculations. 6-7 schools can cause a gap of 8 IQ points on PISA sample.
In many nations, bigger sized schools are reserved for the poor people and in this case the low IQ population will be under-represented in PISA's calculation.
So, PISA will show an IQ gap as high as 17 IQ points between these two kind of nations even when there isn't any real IQ gap. Just because of poorly done work. 

In case of Singapore, Nanyang Girls High school and NUS High School score 121 IQ and have 2.2x lower school enrollment than national average. Just the PISA miscalculation on these 2 schools for "not accounting school size" has caused a jump of 0.25 IQ in Singapore's overall score.

Imagine what 20 such schools can do. So, PISA does have a lot of errors in it's work. When we are talking about getting IQ sample across nations which do not differ more than 4-5 IQ, such a big miscalculation or poorly done work is extremely unreliable.

2. Sending a list of schools to PISA:- 

It is up to the government which list of schools is sent to PISA. In Qatar, 153 schools participated and has the highest percentage of population tested in any nation. In both Singapore and Qatar, all schools participated in PISA.

Imagine if Qatar had decided to send a list of top 30 schools to PISA. It would be able to show an IQ of 103 (as per rec1man's calculation) v/s 81 IQ currently.

3. Un-attending school kids in PISA:-

In many cases, large number of students do not come for PISA exam. Even if bottom 20% of school kids are absent on PISA day, it can cause a jump of 10 IQ points in school's results. 

https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf
Page 185.

In many nations, only 80% of kids that were selected by PISA gave PISA exam.

It is entirely possible for any educational administration to work out and cheat "the PISA test" and show up to 28 IQ higher than it should be.