UK Resorting to Biased Ranking to Woo International Students?
Commentary on the latest 2007 THES-QS World University Ranking (released 9th Nov 2007)
Published on 19 November 2007
The THES-QS World University Ranking, compiled by the Times Higher Education Supplement (THES) and Quacquarelli Symonds (QS), both based in London, UK, is running in its fourth year now since its inception in 2004. The ranking has been widely criticized for biased and questionable methodologies even though it is frequently referred to by many universities, especially those in UK and the Asia-pacific region for institutional marketing. The aims of this article are to systematically examine the deficiencies of the THES-QS rankings and suggest possible socio-economic and/or political motives behind the flawed rankings. We first provide a general overview of the latest 2007 ranking.
It would seem that the THES, being a respectable periodical based in London, would certainly not come out with just another irresponsible musing by self-appointed experts for personal gains or commercial profits. However, this seems to be the case, as would be shown later, albeit in an indirect sense. One important modification to the methodology this year is that reviewers are prevented from voting for their own institutions in the “Peer Review” component. Despite promising more stringent and transparent ranking criteria, the new ranking still seems highly incredible. For example, four out of the top 10 universities are in UK (Table 1). We observe that Imperial College has steadily risen up the chart each year, from 13th in 2005 to 9th in 2006 to the current position of 5th, ahead of prestigious US universities like Caltech, Princeton and MIT (Table 2).
Table 1. Top 10 Universities in 2007 THES-QS Chart
No. |
University |
2006 Rank |
2007 Rank |
Country |
1 |
HARVARD University |
1 |
1 |
US |
2 |
University of CAMBRIDGE |
2 |
2 |
UK |
3 |
University of OXFORD |
3 |
2 |
UK |
4 |
YALE University |
4 |
2 |
US |
5 |
Imperial College LONDON |
9 |
5 |
UK |
6 |
PRINCETON University |
10 |
6 |
US |
7 |
CALIFORNIA Institute of Technology (Caltech) |
7 |
7 |
US |
8 |
University of CHICAGO |
11 |
7 |
US |
9 |
UCL (University College LONDON) |
25 |
9 |
UK |
10 |
MASSACHUSETTS Institute of Technology (MIT) |
4 |
10 |
US |
Table 2. The Rise of British Universities
No. |
University |
2005 Rank |
2006 Rank |
2007 Rank |
1 |
University of CAMBRIDGE |
3 |
2 |
2 |
2 |
University of OXFORD |
4 |
3 |
2 |
3 |
Imperial College LONDON |
13 |
9 |
5 |
4 |
UCL (University College LONDON) |
28 |
25 |
9 |
We observe that many other Australian, New Zealand and UK universities have also made quantum leaps (Table 3).
Table 3. The Rise of UK, Australian and NZ Universities
No. |
University |
2006 Rank |
2007 Rank |
Jump |
1 |
University of Adelaide |
105 |
62 |
43 |
2 |
University of LANCASTER |
228 |
147 |
81 |
3 |
King's College LONDON |
46 |
24 |
22 |
4 |
University of Western Australia |
111 |
64 |
47 |
5 |
University of BRISTOL |
64 |
37 |
27 |
6 |
University of ABERDEEN |
195 |
137 |
58 |
7 |
University of CANTERBURY |
333 |
188 |
145 |
8 |
University of DUNDEE |
238 |
171 |
67 |
9 |
CARDIFF University |
141 |
99 |
42 |
We also note that three previously top-ranked US universities (MIT, UC Berkeley and Stanford) have dropped outside the Top 10. An even steeper plunge is made by Washington university which has dropped 113 positions from 46th and now falls even outside the top 150 (Table 4).
Table 4. The Fall of US Universities
No. |
University |
2006 Rank |
2007 Rank |
Plunge |
1 |
MASSACHUSETTS Institute of Technology (MIT) |
4 |
10 |
6 |
2 |
STANFORD University |
6 |
19 |
13 |
3 |
University of California, BERKELEY |
8 |
22 |
14 |
4 |
WASHINGTON University in St. Louis |
48 |
161 |
113 |
The Previous Rankings
Previously (from 2004 to 2006), the THES-QS rankings have been criticized on a number of counts. Among its most outspoken critics were Richard Holmes from the MARA University of Technology in Malaysia and Simon Marginson from Melbourne University in Australia. In a paper commenting on the 2004 and 2005 editions of the THES-QS Ranking, Richard Holmes mentioned the following points:[1]
Indeed, Philip G. Altbach, Monan professor of higher education and director of the Center for International Higher Education at Boston College wrote: [5]
“The problem with ranking concerns the practice, not the principle. How is it possible to accurately measure a nation’s academic system, or for that matter the quality of a single institution? Or of academic institutions worldwide? Many rankings resemble “popularity contests”—asking groups in the academic community, especially administrators, their opinions about peer institutions. This method is especially popular among the many magazines and newspapers worldwide that rank institutions. Even the most sophisticated rankings include these peer opinions, although many more measures are also included.”
A more piercing criticism of the THES-QS ranking is given by Peter Wills, Auckland Branch President of the Association of University Staff who commented in a report:[6]
“We are fully
in favour of leading staff learning from the experience of others in the
international community, but we deplore the replacement of sound and thoughtful
judgment by the identification of pseudo-objective measures of things that
depend, in the end, on subjective values. Therefore, we seriously question the
value of the Future Heads and General Staff Leaders spending their time trying
to identify characteristics of excellence for the purpose of understanding
international ranking systems.
We note that the Times Higher Education Supplement’s ranking of The University of Auckland rose from 67th in 2004 to 52nd in 2005. But we note also that this survey establishes its rankings by appealing to university staff, even offering financial enticements to participate (see Appendix II). Staff are likely to feel it is in their greatest interest to rank their own institution more highly than others. This means the results of the survey and any apparent change in ranking are highly questionable, and that a high ranking has no real intrinsic value in any case. We are vehemently opposed to the evaluation of the University according to the outcome of such PR competitions.”
Simon Marginson, Professor of Higher Education and Australian Professorial Fellow at the Centre for the Study of Higher Education in The University of Melbourne mentioned:[7]
“Half of the THES index is comprised by existing reputation: 40 per cent by a reputational survey of academics (‘peer review’), and another 10 per cent determined by a survey of ‘global employers’. The THES index is too easily open to manipulation as it is not specified who is surveyed or what questions are asked. By changing the recipients of the surveys, or the way the survey results are factored in, the results can be shifted markedly.”
“Results have been highly volatile. There have been many sharp rises and falls, especially in the second half of the THES top200 where small differences in metrics can generate large rankings effects. Fudan in China has oscillated between 72 and 195, RMIT in Australia between 55 and 146. In the US, Emory has risen from 173 to 56 and Purdue fell from 59 to 127.”
Anthony F.J. van Raan, Centre for Science and Technology Studies at Leiden University commented on the peer review:[8]
"It is questionable whether all the individual academics involved in such large-scale surveys can be regarded as knowledgeable experts in all those parts of the evaluated entities that really matter. As indicated above, the ‘cognitive distance’ between evaluating person and evaluated object is becoming too large. In such cases, the ‘experts’ will more and more tend to judge on the more general basis of established reputation, instead of their own actual knowledge (if they have!) of recent past performance. The probability that these ‘experts’ -who have to judge the quality of all the life sciences at all major universities in the world- are aware of important, recent breakthroughs in a specific field, decreases dramatically. This awareness, however, is precisely what a peer must have. It is also this recognition of recent past performance that forms the strength of bibliometric analysis...
Dependence of the outcomes on the choice of experts is one of the major problems. This dependence may cause biases in fields of expertise. After publication of the first THES ranking, The Sydney Morning Herald gloried the presence of not less than six Australian universities in the top 50, while the German news agency DPA bemoaned the absence of the country in the top positions (THES 2004). Six universities out of the top-50 universities counts for 12%. However, Australia contributes for only about 2.5 % to the worldwide scientific impact whereas Germany contributes around 8 % to worldwide scientific impact. Most of these Australian ‘top’ universities score low to very low in citations. These strange discrepancies between the results of an expert survey and bibliometric findings suggest that most probably there are strong geographical biases, particularly an Asian one, in the expert survey of THES. The expert survey of THES was produced by a London-based company specialized in MBA and graduate recruitment. This may very well cause a positive bias for universities with large institutes of economics and schools of management, and a negative bias for universities without a strong emphasis on these disciplines.”
THES rebuts:
‘But the Australian universities are popular in our peer review and do especially well in our rankings of international success. They are among the world’s most enthusiastic recruiters of international staff and students, with years of recruiting in Asia and beyond now visibly paying off’ - (THES 2004).
In view of the above, the following comment seems almost comical:
"...Peer review, the most trusted method for university comparison..."
- Martin Ince (THES, 2004)
To understand why such rankings have been made by a respectable magazine as Times, we must first understand the importance of higher education to a country's economy.
Higher Education as a Source of Revenue
Each year, hundreds of thousands of international students travel overseas for higher education. Global higher education is a lucrative business and universities throughout the world know only too well how much revenue can be generated from foreign students’ tuition fees alone. In 2004-05, on average, international students paid £6,868 in tuition fees and £187.57 a week in living costs in the UK. Universities and colleges in the UK charge international students vastly inflated fees to study at both undergraduate and postgraduate level.[9] International students are charged from £4,350 for an arts-based undergraduate course at City University to £15,675 at Oxford University. Postgraduate arts courses range from £2,100 at City University to £28,850 at the London Business School, while undergraduate-level science-based courses range from £5,177 a year at Heythrop College to £17,350 a year at Imperial. And at postgraduate level, international students face fees of between £3,950 at Edge Hill University and £16,686 at the London School of Economics. However, UK is in danger of losing precious international students by universities pricing themselves out of the market.[10] Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI) in UK, says:
"There is a real possibility that we may price ourselves out of the market. It could well be in the national interest to lower the fee charged of overseas students, in order to maximise their number. The students bring considerable benefit to the economy, not just to the universities that receive their fees. So it is worth maximising the number of them."
- Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI),
UK
According to a report published by the British Council, education is worth more to UK exports than financial services or the automotive industry.[11] A total of £28bn in 2003-04 was earned from overseas students by a sector ranging from world famous universities to small English language colleges, from independent schools to publishers and broadcasters. That figure has jumped from £23bn in 2001-02 as numbers of international students have risen, and compares with £19bn for financial services and £20bn for the automotive industry, said the report compiled by Dr Pamela Lenton, of the University of Sheffield. Martin Davidson, chief executive of the British Council, said education was vital to the UK both economically and culturally. He added:
"Ö our position is vulnerable. Unless we start taking education much more seriously as a global business, we will lose out to other countries who understand the value of education to their economy much better than we do."
- Martin Davidson, chief executive of the British Council
Foreign students bring huge benefits to the country beyond their fees, such as the living expenses they pay and the fact that many stay on to work. However, Britain's reputation as a world-leading destination for international students could be under threat as new research reveals that almost 30 per cent do not think the education they receive is worth the money.[12] Although the number of international students has increased overall in the UK, its share of the market has fallen from 16 per cent in 1998 to 11 per cent in 2004, with Chinese students in particular heading to the US instead. Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI), says:
"Competition from other countries is increasing and information on what students get for their money is increasing, so our unique selling points are being eroded, and our status in the international market is under threat. I think we are in danger of killing that golden goose."
- Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI), UK
One foreign student who came from Kenya remarked: 'People are asking whether it is worth the risk, effort and hassle. In the past, if you had the chance to study in the UK you grabbed it with both hands, but not any more. Increasingly students are looking elsewhere.'
The impact of league tables on the destinations of degree-seeking international students cannot be overemphasized. Many students do take such league tables into account before making a decision on their destination. With well-paid jobs increasingly hard to find in a society that now has a glut of undergraduates and postgraduates, anything that can give a student a competitive edge comes at a premium and people are prepared to pay whatever it costs. Greg Philo, head of Glasgow University's Media Group, for the British Council said:[13]
"China's one-child policy means that many parents feel they have to super-educate their children. Often the entire life-savings of a family are spent on a child's education.”
- Greg Philo, head of Glasgow University's Media Group, for the British Council
It is therefore not surprising many established educational institutions throughout the First World are competing fiercely with each other to tap into China’s enormous market. Confronted with increasingly stiff competition from other G7 nations like France and Germany which are offering an increasing number of international postgraduate programmes over the years at only a fraction of the costs, Times-QS has resorted to compiling their own international league tables to demonstrate that their own universities are still among the best in the world in an attempt to convince international students that higher education in UK is well worth the money invested. Whoever gets a sizeable share of this ‘international higher education pie’ is assured billions in revenues each year, a tremendous boost to the country’s economy.
Fierce advertising campaigns by UK and Australian universities are a common sight in Asian countries like Singapore, Malaysia and China. Advertisements titled “International MBA from an internationally recognized institution” or “Get a degree in Financial Computing in Two Years!” are not uncommon in the local newspapers and magazines in these Asian economies. We also note that numerous Diploma Mills, whose main purpose is revenue-generating have sprang out of UK over the years offering unrecognized degrees to ineligible and sometimes unsuspecting international students, the only prerequisite being that they are able to pay the tuition fees.
The 2007 Ranking
We first note that 4 out of the top 10 universities are from UK. While positions of many other universities (several prestigious ones included) fluctuated over the years, Cambridge, Oxford and Imperial’s positions are always improving. This year, Imperial College suddenly rose to Number 5 in the world, ahead of prestigious US universities like MIT, Berkeley and Caltech. Imperial does well on the 2007 THES QS rankings partly because of outstanding scores on the peer review (99 out of 100), employer review (99), international students (100) and student faculty ratio (100). As mentioned above, the peer and employer reviews are meaningless and can be ignored. According to Richard Holmes, QS might have counted both academic and research staff when computing the student faculty ratio[14], leading to Imperial’s stellar score in this criterion. We also note that the Imperial College Press is a joint venture of Imperial College and World Scientific and World Scientific is a Singapore-based publishing company whose subscription list is used by QS to construct their "peer review. This year, University College London suddenly rose to Number 9 in the world, ahead of MIT and Stanford. Instead of painstakingly providing reasons as to why this is wholly unjustified, we will just ask a few questions.
It seems that THES-QS is capitalizing on the good reputation of UK’s established universities like Oxbridge and is trying to demonstrate (via a dubious survey) that they are still among the very best in the world, in order to attract degree-seeking international students who are increasingly choosing other destinations over UK. To be fair, Oxford and Cambridge are good, but not that good. They still lack far behind their US counterparts like MIT and Stanford, whether in terms of yearly endowments or availability of top international talents.[15] An eligible overseas student will most likely choose MIT over Cambridge or Imperial for his engineering education. We all know that US is the home to Lockheed Martin, Boeing, NASA, etc In terms of technology, we are not even sure if UK is on par with Germany or France, home to Airbus, EADS, BMW, etc. In terms of number of Nobel laureates in either Physics or Chemistry, UK is ranked third, after US and Germany[16][17]. Yet, not a single German university made it to the Top 50, while four UK universities made it to Top 10 and numerous more made it to Top 50. Do you now see the absurdity of it all?
The impartially of the judge is a fundamental prerequisite in any fair competition. Yet, the managing director of QS, Nunzio Quacquarelli, said:[18]
“The rankings recognised the quality of education that UK universities offer. In an environment of increasing student mobility, the UK is putting itself forward as a top choice for students worldwide.”
Conclusion
While we do applaud the efforts of organizations in bringing into light the relative standings of world universities which can be valuable to degree-seeking international students and policy makers of academic institutions, we deplore the use of league tables as means to inflate the international standings of one’s own institutions via dubious ranking methodologies and unverifiable survey data, for economic gains or other political motives.
IT is difficult to understand why this notorious ‘THES=QS World University Ranking’ has persisted to its fourth year. While there have been sporadic efforts made by individuals to illuminate the deficiencies of the rankings, a concerted and all-out effort by intellectuals from around the world to condemn its results is long since overdue. It’s disheartening to see many of the undeservedly top-ranked universities have commended the THES-QS on a job well-done without first seriously examining the validity of the ranking methodologies. It is a basic tenet of Confucianism that one should not go into raptures over the praises of others without first examining the motives behind the praises or if one should deserve them. University administrators, being the de-facto models for the students and staff, should have the courage and dignity to dismiss or even condemn rankings which they know are flawed even though they are might be rated highly in such rankings.
The THES rankings have contributed nothing but serve only to mislead degree-seeking international students who need to make well-informed decisions on their destinations of study. Staff and students in overrated universities might develop a false sense of superiority and security and spend their time identifying criteria of international ranking systems for the purpose of maintaining or improving their positions instead of making any real improvements in teaching or research.
Peddling propaganda is certainly not a means to becoming a truly world-class university.
[1] From “The THES University Rankings: Are They Really World Class?” by Richard Holmes, Asian Journal of University Education, 1/1, 1-14.
[2] I believe the list can be expanded to include UK, Singapore and Malaysia where the Times is well-known.
[3] We also note that some universities (especially those in Singapore and Hong Kong) offer plenty of scholarships to international students, especially at the postgraduate level.
[4] And of course, QS is based in London, UK.
[5] From “The Dilemmas of Ranking” by Philip G. Altbach, the Boston College Center for International Higher Education, International Higher Education, 42, Winter 2006
[6] From “Response to Review of Strategic Plan” by Peter Wills, Auckland Branch president, Association of University Staff
[7]From “Rankings ripe for misleading” by Simon Marginson, The Australian, December 06, 2006
[8]From “Challenges in Ranking of Universities”, Anthony F.J. van Raan, First International Conference on World Class Universities, Shanghai Jiao Tong University, Shanghai, June 16-18, 2005
[9] From “Foreign students charged hugely inflated tuition fees, survey reveals” by Anthea Lipsett,
June 19, 2007, EducationGuardian.co.uk
[10] From “International student fees 'overpriced'”, by Anthea Lipsett, July 12, 2007, EducationGuardian.co.uk
[11] From “Education worth more to British exports than banking” by Donald MacLeod, September 18, 2007, EducationGuardian.co.uk
[12] From “Costs deter foreign students” by Anushka Asthana, education correspondent, September 16, 2007, The Observer
[13] From “Not just a source of revenue” by John Crace, June 26, 2007, The Guardian
[15] From “Oxbridge is outclassed by rivals in America” by John Clare, Education Editor ISSUE 1427
22 April 1999, http://www.telegraph.co.uk
[16] From Ranking by Nations, Nobel Prize in Physics, Wikipeda
[17] From Ranking by Nations, Nobel Prize in Chemistry, Wikipedia
[18] From “UK universities rise up rankings” 8 November 2007, BBC