Make your own free website on Tripod.com
University Ranking
Home

UK Resorting to Biased Ranking to Woo International Students?

Commentary on the latest 2007 THES-QS World University Ranking (released 9th Nov 2007)

Published on 19 November 2007

The THES-QS World University Ranking, compiled by the Times Higher Education Supplement (THES) and Quacquarelli Symonds (QS), both based in London, UK, is running in its fourth year now since its inception in 2004. The ranking has been widely criticized for biased and questionable methodologies even though it is frequently referred to by many universities, especially those in UK and the Asia-pacific region for institutional marketing. The aims of this article are to systematically examine the deficiencies of the THES-QS rankings and suggest possible socio-economic and/or political motives behind the flawed rankings.† We first provide a general overview of the latest 2007 ranking.

It would seem that the THES, being a respectable periodical based in London, would certainly not come out with just another irresponsible musing by self-appointed experts for personal gains or commercial profits. However, this seems to be the case, as would be shown later. One important modification to the methodology this year is that reviewers are prevented from voting for their own institutions in the "Peer Review" component. Despite promising more stringent and transparent ranking criteria, the new ranking still seems rather incredible. For example, four out of the top 10 universities are in UK (Table 1). We observe that Imperial College has steadily risen up the chart each year, from 13th in 2005 to 9th in 2006 to the current position of 5th, ahead of prestigious US universities like Caltech, Princeton and MIT (Table 2).

Table 1. Top 10 Universities in 2007 THES-QS Chart

No.

University

2006 Rank

2007 Rank

Country

1

HARVARD University

1

1

US

2

University of CAMBRIDGE

2

2

UK

3

University of OXFORD

3

2

UK

4

YALE University

4

2

US

5

Imperial College LONDON

9

5

UK

6

PRINCETON University

10

6

US

7

CALIFORNIA Institute of Technology (Caltech)

7

7

US

8

University of CHICAGO

11

7

US

9

UCL (University College LONDON)

25

9

UK

10

MASSACHUSETTS Institute of Technology (MIT)

4

10

US

Table 2. The Rise of British Universities

No.

University

2005 Rank

2006 Rank

2007 Rank

1

University of CAMBRIDGE

3

2

2

2

University of OXFORD

4

3

2

3

Imperial College LONDON

13

9

5

4

UCL (University College LONDON)

28

25

9

†We observe that many other Australian, New Zealand and UK universities have also made quantum leaps (Table 3).

Table 3. The Rise of UK, Australian and NZ Universities

No.

University

2006 Rank

2007 Rank

Jump

1

University of Adelaide

105

62

43

2

University of LANCASTER

228

147

81

3

King's College LONDON

46

24

22

4

University of Western Australia

111

64

47

5

University of BRISTOL

64

37

27

6

University of ABERDEEN

195

137

58

7

University of CANTERBURY

333

188

145

8

University of DUNDEE

238

171

67

9

CARDIFF University

141

99

42

††

We also note that three previously top-ranked US universities (MIT, UC Berkeley and Stanford) have dropped outside the Top 10. An even steeper plunge is made by Washington university which has dropped 113 positions from 46th and now falls even outside the top 150 (Table 4).

Table 4. The Fall of US Universities

No.

University

2006 Rank

2007 Rank

Plunge

1

MASSACHUSETTS Institute of Technology (MIT)

4

10

6

2

STANFORD University

6

19

13

3

University of California, BERKELEY

8

22

14

4

WASHINGTON University in St. Louis

48

161

113

††

The Previous Rankings

Previously (from 2004 to 2006), the THES-QS rankings have been criticized on a number of counts. Among its most outspoken critics were Richard Holmes from the MARA University of Technology in Malaysia and Simon Marginson from Melbourne University in Australia.†In a paper commenting on the 2004 and 2005 editions of the THES-QS Ranking, Richard Holmes mentioned the following points:[1]

  1. The sampling procedure is not explained and is very probably seriously biased, the weighting of the various components is not justified, inappropriate measures of teaching quality are used, the assessment of research achievement is biased against the humanities and social sciences, the classification of institutions is inconsistent, there are striking and implausible changes in the rankings between 2004 and 2005 and they are based in one crucial respect on regional rather than international comparisons. It is recommended that these rankings should not be the basis for the development and assessment of national and institutional policies.
  2. First of all, the rankings were not compiled by the respected THES but by a firm of consultants, the much less well-known QS Quacquarelli Symonds, although it was apparently THES that decided on the weighting to be allocated to each component. This company specializes in promoting international MBA education and executive recruitment. It does not seem to have any specialized knowledge of research and teaching in the natural and social sciences or the humanities. The London-based QS also has offices in Washington DC, Paris, Beijing, Singapore, Tokyo and Sydney, the current dominant centres of global business activity (QS Network, 2006). It does not have offices in less fortunate places like Latin America, Canada, Africa, the Middle East, Eastern Europe or South Asia.
  3. There are six universities from the Peoples Republic of China (not counting Hong Kong) but only one from Taiwan. Despite a smaller population, Australia has twice as many universities in the top 200 as Canada.
  4. The peer review section is the most questionable of all the criteria. THES rightly calls it "the core of our analysis". It constitutes 40 per cent of the weighting, down from 50 per cent in 2004, and it is the criterion for admission to the initial group of 300 universities from which the top 200 are drawn.
  5. The sampling method, as far as can be discerned from the little that we have been told, does not seem to adhere to conventional social scientific standards of quality. For consultants who claim to be able to pick academic experts who can assess the quality of universities, this is a little ironic. In 2004, according to the THES, QS asked 1,300 academics in 88 countries ?to nominate both the academic subjects and the geographical areas on which they felt able to comment? and to name the top institutions in these areas and subjects. There follows a rather puzzling comment. THES says that in 2004 additional reviewers were added to "balance nominations" in the subject areas and geographical regions (Times Higher Education Supplement, 5/11/2004). If this means that the consultants found that they did not get enough responses from specific geographical it might be acceptable. But if it means that the reviewers did not nominate universities from certain areas so that QS went and got more reviewers until they got the answers they wanted then it is another thing altogether. It is rather like continually moving the goalposts until somebody finally scores a goal. It is somewhat questionable and certainly needs some explanation.
  6. The Citations section raises many questions. Firstly, it does not appear to be a representative sample of world academic opinion. Only those academics deemed by QS to be experts are included. The panel seems to be composed of those that the consultants considered to be experts but how their expertise was determined is not stated. We are given no information at all about how the sample was selected and how the respondents were distributed within the three economic regions.It is difficult to avoid the suspicion that the peer review was based on convenience sampling, with QS simply asking those that they had come across during their consultancy activities. This would explain the presence in the top 200 of several apparently undistinguished universities from France, Australia and China[2] where the consultants have offices and the comparative scarcity of universities from Eastern Europe, Israel, Taiwan and Canada where they do not. Thus, it is probable that this section is heavily biased towards those universities that are involved in globalised education, especially graduate business training, and those in Western Europe and the Asia-Pacific region.
  7. Furthermore, the peer review is not really an international ranking. According to Martin Ince (Times Higher Education Supplement, 28/10/05) in 2005, QS repeated their procedure of 2004 and asked the academics to name "the top universities in the subject areas and the geographical regions in which they have expertise."†Chinese and Australian universities are getting good peer reviews not because they are highly regarded throughout the world, which is what an international ranking ought to mean, but because they are selected by academics in the Asia-Pacific region who have been asked to name the best universities in a specific region. Some mysteries can now be cleared up. Why are there so many more Australian than Canadian universities? Because Canadian universities had to compete with those in the US while Australian universities were being compared with those in countries like Pakistan or Myanmar. All this is rather like FIFA announcing that, instead of having a final round of the world cup, they would just count the performance of the teams in the regional rounds. So, China and Australia would do very well having scored a lot of goals playing against India or Papua New Guinea and perhaps even surpass Argentina and Italy who struggled to narrow victories against the likes of England or Spain.
  8. Recruiter Ratings: In effect, QS asks universities which companies recruit their graduates and then goes to those companies and asks them where they do their recruiting. Any social science graduate student would recognize this is not a sensible way of selecting a sample. One suspects, moreover, that this component of the rankings is composed largely, if not entirely, of companies that have had dealings with QS or universities deeply committed in one or another to the global MBA trade.
  9. International Students and Faculty: It is true, perhaps, that large numbers of international students could mean that a university has a worldwide reputation and a strong international presence among the faculty might suggest a search for the very best intellectual talent. On the other hand, it could have something to do with liberal immigration policies or, as in the United Kingdom, quirks in regulations about fees and admissions.[3]
  10. Is it just a coincidence that QS has offices in Sydney, Paris, Singapore and Beijing?[4] These are, as we have noted, dynamic economic areas, except perhaps for France, where international business education is flourishing and where QS is very active.

Indeed, Philip G. Altbach, Monan professor of higher education and director of the Center for International Higher Education at Boston College wrote: [5]

'The problem with ranking concerns the practice, not the principle. How is it possible to accurately measure a nation's academic system, or for that matter the quality of a single institution? Or of academic institutions worldwide? Many rankings resemble "popularity contests", asking groups in the academic community, especially administrators, their opinions about peer institutions. This method is especially popular among the many magazines and newspapers worldwide that rank institutions. Even the most sophisticated rankings include these peer opinions, although many more measures are also included.'

A more piercing criticism of the THES-QS ranking is given by Peter Wills, Auckland Branch President of the Association of University Staff who commented in a report:[6]

'We are fully in favour of leading staff learning from the experience of others in the international community, but we deplore the replacement of sound and thoughtful judgment by the identification of pseudo-objective measures of things that depend, in the end, on subjective values. Therefore, we seriously question the value of the Future Heads and General Staff Leaders spending their time trying to identify characteristics of excellence for the purpose of understanding international ranking systems.

We note that the Times Higher Education Supplement?s ranking of The University of Auckland rose from 67th in 2004 to 52nd in 2005. But we note also that this survey establishes its rankings by appealing to university staff, even offering financial enticements to participate (see Appendix II). Staff are likely to feel it is in their greatest interest to rank their own institution more highly than others. This means the results of the survey and any apparent change in ranking are highly questionable, and that a high ranking has no real intrinsic value in any case. We are vehemently opposed to the evaluation of the University according to the outcome of such PR competitions.'

Simon Marginson, Professor of Higher Education and Australian Professorial Fellow at the Centre for the Study of Higher Education in The University of Melbourne mentioned:[7]

'Half of the THES index is comprised by existing reputation: 40 per cent by a reputational survey of academics ("peer review"), and another 10 per cent determined by a survey of "global employers". The THES index is too easily open to manipulation as it is not specified who is surveyed or what questions are asked. By changing the recipients of the surveys, or the way the survey results are factored in, the results can be shifted markedly.'

'Results have been highly volatile. There have been many sharp rises and falls, especially in the second half of the THES top200 where small differences in metrics can generate large rankings effects. Fudan in China has oscillated between 72 and 195, RMIT in Australia between 55 and 146. In the US, Emory has risen from 173 to 56 and Purdue fell from 59 to 127.'

Anthony F.J. van Raan, Centre for Science and Technology Studies at Leiden University commented on the peer review:[8]

"It is questionable whether all the individual academics involved in such large-scale surveys can be regarded as knowledgeable experts in all those parts of the evaluated entities that really matter. As indicated above, the "cognitive distance" between evaluating person and evaluated object is becoming too large. In such cases, the "experts" will more and more tend to judge on the more general basis of established reputation, instead of their own actual knowledge (if they have!) of recent past performance. The probability that these ?experts? -who have to judge the quality of all the life sciences at all major universities in the world- are aware of important, recent breakthroughs in a specific field, decreases dramatically. This awareness, however, is precisely what a peer must have. It is also this recognition of recent past performance that forms the strength of bibliometric analysis...

Dependence of the outcomes on the choice of experts is one of the major problems. This dependence may cause biases in fields of expertise. After publication of the first THES ranking, The Sydney Morning Herald gloried the presence of not less than six Australian universities in the top 50, while the German news agency DPA bemoaned the absence of the country in the top positions (THES 2004). Six universities out of the top-50 universities counts for 12%. However, Australia contributes for only about 2.5 % to the worldwide scientific impact whereas Germany contributes around 8 % to worldwide scientific impact. Most of these Australian ?top? universities score low to very low in citations. These strange discrepancies between the results of an expert survey and bibliometric findings suggest that most probably there are strong geographical biases, particularly an Asian one, in the expert survey of THES. The expert survey of THES was produced by a London-based company specialized in MBA and graduate recruitment. This may very well cause a positive bias for universities with large institutes of economics and schools of management, and a negative bias for universities without a strong emphasis on these disciplines."

THES rebuts:

†"But the Australian universities are popular in our peer review and do especially well in our rankings of international success. They are among the world's most enthusiastic recruiters of international staff and students, with years of recruiting in Asia and beyond now visibly paying off"† - (THES 2004).

In view of the above, the following comment seems almost comical:

"...Peer review, the most trusted method for university comparison..."

- Martin Ince (THES, 2004)

To understand why such rankings have been made by a respectable magazine as Times, we must first understand the importance of higher education to a country's economy. †

Higher Education as a Source of Revenue

Each year, hundreds of thousands of international students travel overseas for higher education. Global higher education is a lucrative business and universities throughout the world know only too well how much revenue can be generated from foreign students? tuition fees alone. †In 2004-05, on average, international students paid £6,868 in tuition fees and £187.57 a week in living costs in the UK. Universities and colleges in the UK charge international students vastly inflated fees to study at both undergraduate and postgraduate level.[9] International students are charged from £4,350 for an arts-based undergraduate course at City University to £15,675 at Oxford University. Postgraduate arts courses range from £2,100 at City University to £28,850 at the London Business School, while undergraduate-level science-based courses range from £5,177 a year at Heythrop College to £17,350 a year at Imperial. And at postgraduate level, international students face fees of between £3,950 at Edge Hill University and £16,686 at the London School of Economics. However, UK is in danger of losing precious international students by universities pricing themselves out of the market.[10] Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI) in UK, says:†

"There is a real possibility that we may price ourselves out of the market. It could well be in the national interest to lower the fee charged of overseas students, in order to maximise their number. The students bring considerable benefit to the economy, not just to the universities that receive their fees. So it is worth maximising the number of them."


-
Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI), UK

According to a report published by the British Council, education is worth more to UK exports than financial services or the automotive industry.[11] A total of £28bn in 2003-04 was earned from overseas students by a sector ranging from world famous universities to small English language colleges, from independent schools to publishers and broadcasters. That figure has jumped from £23bn in 2001-02 as numbers of international students have risen, and compares with £19bn for financial services and £20bn for the automotive industry, said the report compiled by Dr Pamela Lenton, of the University of Sheffield. Martin Davidson, chief executive of the British Council, said education was vital to the UK both economically and culturally. He added: †

"... our position is vulnerable. Unless we start taking education much more seriously as a global business, we will lose out to other countries who understand the value of education to their economy much better than we do."

- Martin Davidson, chief executive of the British Council

Foreign students bring huge benefits to the country beyond their fees, such as the living expenses they pay and the fact that many stay on to work. However, Britain's reputation as a world-leading destination for international students could be under threat as new research reveals that almost 30 per cent do not think the education they receive is worth the money.[12]† Although the number of international students has increased overall in the UK, its share of the market has fallen from 16 per cent in 1998 to 11 per cent in 2004, with Chinese students in particular heading to the US instead. Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI), says:

"Competition from other countries is increasing and information on what students get for their money is increasing, so our unique selling points are being eroded, and our status in the international market is under threat. I think we are in danger of killing that golden goose."

- Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI), UK

One foreign student who came from Kenya remarked: 'People are asking whether it is worth the risk, effort and hassle. In the past, if you had the chance to study in the UK you grabbed it with both hands, but not any more. Increasingly students are looking elsewhere.'

The impact of league tables on the destinations of degree-seeking international students cannot be overemphasized.† Many students do take such league tables into account before making a decision on their destination. With well-paid jobs increasingly hard to find in a society that now has a glut of undergraduates and postgraduates, anything that can give a student a competitive edge comes at a premium and people are prepared to pay whatever it costs. Greg Philo, head of Glasgow University's Media Group, for the British Council said:[13]

"China's one-child policy means that many parents feel they have to super-educate their children. Often the entire life-savings of a family are spent on a child's education."

- Greg Philo, head of Glasgow University's Media Group, for the British Council†

It is therefore not surprising many established educational institutions throughout the First World are competing fiercely with each other to tap into Chinas enormous market. Confronted with increasingly stiff competition from other G7 nations like France and Germany which are offering an increasing number of international postgraduate programmes over the years at only a fraction of the costs, Times-QS has resorted to compiling their own international league tables to demonstrate that their own universities are still among the best in the world in an attempt to convince international students that higher education in UK is well worth the money invested. Whoever gets a sizeable share of this "International higher education pie" is assured billions in revenues each year, a tremendous boost to the country's economy.

Fierce advertising campaigns by UK and Australian universities are a common sight in Asian countries like Singapore, Malaysia and China. Advertisements titled "International MBA from an internationally recognized institution" or "Get a degree in Financial Computing in Two Years!" are not uncommon in the local newspapers and magazines in these Asian economies. We also note that numerous Diploma Mills, whose main purpose is revenue-generating have sprang out of UK over the years offering unrecognized degrees to ineligible and sometimes unsuspecting international students, the only prerequisite being that they are able to pay the tuition fees.

The 2007 Ranking

We first note that 4 out of the top 10 universities are from UK. While positions of many other universities (several prestigious ones included) fluctuated over the years, Cambridge, Oxford and Imperial?s positions are always improving. This year, Imperial College suddenly rose to Number 5 in the world, ahead of prestigious US universities like MIT, Berkeley and Caltech. Imperial does well on the 2007 THES QS rankings partly because of outstanding scores on the peer review (99 out of 100), employer review (99), international students (100) and student faculty ratio (100). As mentioned above, the peer and employer reviews are meaningless and can be ignored. According to Richard Holmes, QS might have counted both academic and research staff when computing the student faculty ratio[14], leading to Imperial?s stellar score in this criterion. We also note that the Imperial College Press is a joint venture of Imperial College and World Scientific and World Scientific is a Singapore-based publishing company whose subscription list is used by QS to construct their "peer review.

This year, University College London suddenly rose to Number 9 in the world, ahead of MIT and Stanford.† Instead of painstakingly providing reasons as to why this is wholly unjustified, we will just ask a few questions. †

  1. How many Nobel Laureates and Fields Medallists has Cambridge/Oxford/Imperial/UCL(combined?) produced compared to MIT over the last ten years?
  2. What is the yearly endowment of Cambridge/Oxford,/Imperial/UCL compared to Stanford or Berkeley?

It seems that THES-QS is capitalizing on the good reputation of UK?s established universities like Oxbridge and is trying to demonstrate (via a dubious survey) that they are still among the very best in the world, in order to attract degree-seeking international students who are increasingly choosing other destinations over UK. To be fair, Oxford and Cambridge are good, but not that good. They still lack far behind their US counterparts like MIT and Stanford, whether in terms of yearly endowments or availability of top international talents.[15] An eligible overseas student will most likely choose MIT over Cambridge or Imperial for his engineering education. We all know that US is the home to Lockheed Martin, Boeing, NASA, etc In terms of technology, we are not even sure if UK is on par with Germany or France, home to Airbus, Eurofighter, EADS, BMW, etc. In terms of number of Nobel laureates in either Physics or Chemistry, UK is ranked third, after US and Germany. Yet, not a single German university made it to the Top 50, while four UK universities made it to Top 10 and numerous more made it to Top 50.† Do you now see the absurdity of it all?

The impartially of the judge is a fundamental prerequisite in any fair competition. Yet, the managing director of QS, Nunzio Quacquarelli, said:[16]

"The rankings recognised the quality of education that UK universities offer. In an environment of increasing student mobility, the UK is putting itself forward as a top choice for students worldwide."

Conclusion

While we do applaud the efforts of organizations in bringing into light the relative standings of world universities which can be valuable to degree-seeking international students and policy makers of academic institutions, we deplore the use of league tables as means to inflate the international standings of one's own institutions via dubious ranking methodologies and unverifiable survey data, for economic gains or other political motives.

IT is difficult to understand why this notorious "THES=QS World University Ranking" has persisted to its fourth year. While there have been sporadic efforts made by individuals to illuminate the deficiencies of the rankings, a concerted and all-out effort by intellectuals from around the world to condemn its results is long since overdue. It's disheartening to see many of the undeservedly top-ranked universities have commended the THES-QS on a job well-done without first seriously examining the validity of the ranking methodologies. It is a basic tenet of Confucianism that one should not go into raptures over the praises of others without first examining the motives behind the praises or if one should deserve them. University administrators, being the de-facto models for the students and staff, should have the courage and dignity to dismiss or even condemn rankings which they know are flawed even though they are might be rated highly in such rankings.

The THES rankings have contributed nothing but serve only to mislead degree-seeking international students who need to make well-informed decisions on their destinations of study. Staff and students in overrated universities might develop a false sense of superiority and security and spend their time identifying criteria of international ranking systems for the purpose of maintaining or improving their positions instead of making any real improvements in teaching or research. †

Peddling propaganda is certainly not a means to becoming a truly world-class university.



[1] From "The THES University Rankings: Are They Really World Class?" by Richard Holmes, Asian Journal of University Education, 1/1, 1-14.

[2] I believe the list can be expanded to include UK, Singapore and Malaysia where the Times is well-known.

[3] We also note that some universities (especially those in Singapore and Hong Kong) offer plenty of scholarships to international students, especially at the postgraduate level.

[4] And of course, QS is based in London, UK.

[5] From "The Dilemmas of Ranking" by Philip G. Altbach, the Boston College Center for International Higher Education, International Higher Education, 42, Winter 2006

[6] From "Response to Review of Strategic Plan" by Peter Wills, Auckland Branch president, Association of University Staff

[7]From "Rankings ripe for misleading" by Simon Marginson, The Australian, December 06, 2006

[8]From† "Challenges in Ranking of Universities", †Anthony F.J. van Raan, First International Conference on World Class Universities, Shanghai Jiao Tong University, Shanghai, June 16-18, 2005

[9] From "Foreign students charged hugely inflated tuition fees, survey reveals" by Anthea Lipsett,

June 19, 2007, EducationGuardian.co.uk

[10] From "International student fees 'overpriced'", by Anthea Lipsett, July 12, 2007, EducationGuardian.co.uk

[11] From "Education worth more to British exports than banking" by Donald MacLeod, September 18, 2007, EducationGuardian.co.uk

[12] From "Costs deter foreign students" by Anushka Asthana, education correspondent, September 16, 2007, The Observer

[13] From "Not just a source of revenue" by John Crace, June 26, 2007, The Guardian

[15] From "Oxbridge is outclassed by rivals in America" by John Clare, Education Editor ISSUE 1427

22 April 1999, http://www.telegraph.co.uk

[16] From "UK universities rise up rankings" 8 November 2007, BBC

free stats