UK Resorting to
Biased Ranking to Woo International Students?
Commentary on the latest 2007
THES-QS World University Ranking
(released 9th Nov 2007)
Published on 19 November 2007
The
THES-QS World University Ranking, compiled by the
Times Higher Education Supplement (THES)
and
Quacquarelli Symonds (QS),
both based in London, UK, is running in its fourth year now since its inception
in 2004. The ranking has been widely criticized for biased and questionable
methodologies even though it is frequently referred to by many universities,
especially those in UK and the Asia-pacific region for institutional marketing.
The aims of this article are to systematically examine the deficiencies of the
THES-QS rankings and suggest possible socio-economic and/or political motives
behind the flawed rankings. We first provide a general overview of the latest
2007 ranking.
It would seem that the THES, being a respectable periodical
based in London, would certainly not come out with just another
irresponsible musing by self-appointed experts for personal gains or commercial
profits. However, this seems to be the case, as would be shown later.
One important modification to the methodology this year is
that reviewers are prevented from voting for their own institutions in the "Peer
Review" component. Despite promising more stringent and transparent ranking
criteria, the new ranking still seems rather incredible. For example,
four out of the top 10 universities are in UK (Table 1). We observe that
Imperial College has steadily risen up the chart each year, from 13th
in 2005 to 9th in 2006 to the current position of 5th,
ahead of prestigious US universities like Caltech, Princeton and MIT (Table 2).
Table 1. Top 10 Universities in 2007 THES-QS Chart
No. |
University |
2006 Rank |
2007 Rank |
Country |
1 |
HARVARD University |
1 |
1 |
US |
2 |
University of
CAMBRIDGE |
2 |
2 |
UK |
3 |
University of OXFORD |
3 |
2 |
UK |
4 |
YALE University |
4 |
2 |
US |
5 |
Imperial College
LONDON |
9 |
5 |
UK |
6 |
PRINCETON University |
10 |
6 |
US |
7 |
CALIFORNIA Institute
of Technology (Caltech) |
7 |
7 |
US |
8 |
University of
CHICAGO |
11 |
7 |
US |
9 |
UCL (University
College LONDON) |
25 |
9 |
UK |
10 |
MASSACHUSETTS
Institute of Technology (MIT) |
4 |
10 |
US |
Table 2. The Rise of British Universities
No. |
University |
2005 Rank |
2006 Rank |
2007 Rank |
1 |
University of
CAMBRIDGE |
3 |
2 |
2 |
2 |
University of OXFORD |
4 |
3 |
2 |
3 |
Imperial College
LONDON |
13 |
9 |
5 |
4 |
UCL (University
College LONDON) |
28 |
25 |
9 |
We observe that many other Australian, New Zealand and UK
universities have also made quantum leaps (Table 3).
Table 3. The Rise of UK, Australian and NZ Universities
No. |
University |
2006 Rank |
2007 Rank |
Jump |
1 |
University of Adelaide |
105 |
62 |
43 |
2 |
University
of LANCASTER |
228 |
147 |
81 |
3 |
King's College
LONDON |
46 |
24 |
22 |
4 |
University of Western Australia |
111 |
64 |
47 |
5 |
University
of BRISTOL |
64 |
37 |
27 |
6 |
University
of ABERDEEN |
195 |
137 |
58 |
7 |
University
of CANTERBURY |
333 |
188 |
145 |
8 |
University
of DUNDEE |
238 |
171 |
67 |
9 |
CARDIFF
University |
141 |
99 |
42 |
We also note that three previously top-ranked US universities
(MIT, UC Berkeley and Stanford) have dropped outside the Top 10. An even steeper
plunge is made by Washington university which has dropped 113 positions from
46th and now falls even outside the top 150 (Table 4).
Table 4. The Fall of US Universities
No. |
University |
2006 Rank |
2007 Rank |
Plunge |
1 |
MASSACHUSETTS Institute
of Technology (MIT) |
4 |
10 |
6 |
2 |
STANFORD
University |
6 |
19 |
13 |
3 |
University
of California, BERKELEY |
8 |
22 |
14 |
4 |
WASHINGTON
University in St. Louis |
48 |
161 |
113 |
The Previous Rankings
Previously (from 2004 to 2006), the
THES-QS rankings have been criticized on a number of counts. Among its most
outspoken critics were
Richard Holmes from the MARA
University of Technology in Malaysia and
Simon Marginson from Melbourne University in Australia. In a paper
commenting on the 2004 and 2005 editions of the THES-QS Ranking, Richard Holmes
mentioned the following points:
-
The sampling procedure is not explained and is very probably seriously biased,
the weighting of the various components is not justified, inappropriate
measures of teaching quality are used, the assessment of research achievement
is biased against the humanities and social sciences, the classification of
institutions is inconsistent, there are striking and implausible changes in
the rankings between 2004 and 2005 and they are based in one crucial respect
on regional rather than international comparisons. It is recommended that
these rankings should not be the basis for the development and assessment of
national and institutional policies.
-
First of all, the rankings were not compiled by the
respected THES but by a firm of consultants, the much less well-known QS
Quacquarelli Symonds, although it was apparently THES that decided on the
weighting to be allocated to each component. This company specializes in
promoting international MBA education and executive recruitment. It does not
seem to have any specialized knowledge of research and teaching in the natural
and social sciences or the humanities. The London-based QS also has offices in
Washington DC, Paris, Beijing, Singapore, Tokyo and Sydney, the current
dominant centres of global business activity (QS Network, 2006). It does not
have offices in less fortunate places like Latin America, Canada, Africa, the
Middle East, Eastern Europe or South Asia.
-
There are six universities from the Peoples Republic of China (not counting
Hong Kong) but only one from Taiwan. Despite a smaller population, Australia
has twice as many universities in the top 200 as Canada.
-
The peer review section is the most questionable of all the criteria.
THES rightly calls it "the core of our analysis". It constitutes 40 per cent
of the weighting, down from 50 per cent in 2004, and it is the criterion for
admission to the initial group of 300 universities from which the top 200 are
drawn.
-
The sampling method, as far as can be discerned from the little that we have
been told, does not seem to adhere to conventional social scientific standards
of quality. For consultants who claim to be able to pick academic experts who
can assess the quality of universities, this is a little ironic. In 2004,
according to the THES, QS asked 1,300 academics in 88 countries ?to nominate
both the academic subjects and the geographical areas on which they felt able
to comment? and to name the top institutions in these areas and subjects.
There follows a rather puzzling comment. THES says that in 2004 additional
reviewers were added to "balance nominations" in the subject areas and
geographical regions (Times Higher Education Supplement, 5/11/2004). If this
means that the consultants found that they did not get enough responses from
specific geographical it might be acceptable. But if it means that the
reviewers did not nominate universities from certain areas so that QS went and
got more reviewers until they got the answers they wanted then it is
another thing altogether. It is rather like continually moving the goalposts
until somebody finally scores a goal. It is somewhat questionable and
certainly needs some explanation.
-
The Citations section raises many questions. Firstly, it does not appear to be
a representative sample of world academic opinion. Only those academics deemed
by QS to be experts are included. The panel seems to be composed of those that
the consultants considered to be experts but how their expertise was
determined is not stated. We are given no information at all about how the
sample was selected and how the respondents were distributed within the three
economic regions.It is difficult to avoid the suspicion that the peer
review was based on convenience sampling, with QS simply asking those that
they had come across during their consultancy activities. This would
explain the presence in the top 200 of several apparently undistinguished
universities from France, Australia and China
where the consultants have offices and the comparative scarcity of
universities from Eastern Europe, Israel, Taiwan and Canada where they do not.
Thus, it is probable that this section is heavily biased towards those
universities that are involved in globalised education, especially graduate
business training, and those in Western Europe and the Asia-Pacific region.
-
Furthermore, the peer review is not really an international ranking. According
to Martin Ince (Times Higher Education Supplement, 28/10/05) in 2005, QS
repeated their procedure of 2004 and asked the academics to name "the top
universities in the subject areas and the geographical regions in which they
have expertise." Chinese and Australian universities are getting good
peer reviews not because they are highly regarded throughout the world, which
is what an international ranking ought to mean, but because they are selected
by academics in the Asia-Pacific region who have been asked to name the best
universities in a specific region. Some mysteries can now be cleared up.
Why are there so many more Australian than Canadian universities? Because
Canadian universities had to compete with those in the US while Australian
universities were being compared with those in countries like Pakistan or
Myanmar. All this is rather like FIFA announcing that, instead of having a
final round of the world cup, they would just count the performance of the
teams in the regional rounds. So, China and Australia would do very well
having scored a lot of goals playing against India or Papua New Guinea and
perhaps even surpass Argentina and Italy who struggled to narrow victories
against the likes of England or Spain.
-
Recruiter Ratings: In effect, QS asks universities which companies recruit
their graduates and then goes to those companies and asks them where they do
their recruiting. Any social science graduate student would recognize this is
not a sensible way of selecting a sample. One suspects, moreover, that this
component of the rankings is composed largely, if not entirely, of companies
that have had dealings with QS or universities deeply committed in one or
another to the global MBA trade.
-
International Students and Faculty: It is true, perhaps, that large numbers of
international students could mean that a university has a worldwide reputation
and a strong international presence among the faculty might suggest a search
for the very best intellectual talent. On the other hand, it could have
something to do with liberal immigration policies or, as in the United
Kingdom, quirks in regulations about fees and admissions.
-
Is it just a coincidence that QS has offices in Sydney, Paris, Singapore and
Beijing?
These are, as we have noted, dynamic economic areas, except perhaps for
France, where international business education is flourishing and where QS is
very active.
Indeed, Philip G. Altbach, Monan professor of higher education and director of
the Center for International Higher Education at Boston College wrote:
'The problem with ranking concerns the practice, not the principle. How
is it possible to accurately measure a nation's academic system, or for that
matter the quality of a single institution? Or of academic institutions
worldwide? Many rankings resemble "popularity contests", asking groups in the
academic community, especially administrators, their opinions about peer
institutions. This method is especially popular among the many magazines and
newspapers worldwide that rank institutions. Even the most sophisticated
rankings include these peer opinions, although many more measures are also
included.'
A more piercing criticism of the THES-QS ranking is given by Peter Wills,
Auckland Branch President of the Association of University Staff who commented
in a report:
'We are fully
in favour of leading staff learning from the experience of others in the
international community, but we deplore the replacement of sound and thoughtful
judgment by the identification of pseudo-objective measures of things that
depend, in the end, on subjective values. Therefore, we seriously question the
value of the Future Heads and General Staff Leaders spending their time trying
to identify characteristics of excellence for the purpose of understanding
international ranking systems.
We note that
the Times Higher Education Supplement?s ranking of The University of Auckland
rose from 67th
in 2004 to 52nd
in 2005. But we note also
that this survey establishes its rankings by appealing to university staff, even
offering financial enticements to participate (see Appendix II). Staff are
likely to feel it is in their greatest interest to rank their own institution
more highly than others. This means the results of the survey and any apparent
change in ranking are highly questionable, and that a high ranking has no real
intrinsic value in any case. We are vehemently opposed to the evaluation of the
University according to the outcome of such PR competitions.'
Simon Marginson, Professor of
Higher Education and Australian Professorial Fellow at the Centre for the Study
of Higher Education in The University of Melbourne mentioned:
'Half of the THES index is comprised
by existing
reputation: 40 per cent by a reputational survey of academics ("peer review"),
and another 10 per cent determined by a survey of "global employers". The THES
index is too easily open to manipulation as it is not specified who is surveyed
or what questions are asked. By changing the recipients of the surveys, or the
way the survey results are factored in, the results can be shifted markedly.'
'Results have been highly volatile.
There have
been many sharp rises and falls, especially in the second half of the THES
top200 where small differences in metrics can generate large rankings effects.
Fudan in China has oscillated between 72 and 195, RMIT in Australia between 55
and 146. In the US, Emory has risen from 173 to 56 and Purdue fell from 59 to
127.'
Anthony F.J. van Raan,
Centre for Science and Technology Studies at Leiden University commented on the
peer review:
"It is questionable whether all
the individual academics involved in such large-scale surveys can be regarded as
knowledgeable experts in all those parts of the evaluated entities that really
matter. As indicated above, the "cognitive distance" between evaluating person
and evaluated object is becoming too large. In such cases, the "experts" will
more and more tend to judge on the more general basis of established reputation,
instead of their own actual knowledge (if they have!) of recent past
performance. The probability that these ?experts? -who have to judge the quality
of all the life sciences at all major universities in the world- are aware of
important, recent breakthroughs in a specific field, decreases dramatically.
This awareness, however, is precisely what a peer must have. It is also this
recognition of recent past performance that forms the strength of bibliometric
analysis...
Dependence of the outcomes on
the choice of experts is one of the major problems. This dependence may cause
biases in fields of expertise. After publication of the first THES ranking, The
Sydney Morning Herald gloried the presence of not less than six Australian
universities in the top 50, while the German news agency DPA bemoaned the
absence of the country in the top positions (THES 2004). Six universities out of
the top-50 universities counts for 12%. However, Australia contributes for only
about 2.5 % to the worldwide scientific impact whereas Germany contributes
around 8 % to worldwide scientific impact. Most of these Australian ?top?
universities score low to very low in citations. These strange discrepancies
between the results of an expert survey and bibliometric findings suggest that
most probably there are strong geographical biases, particularly
an Asian one, in the expert survey of
THES. The expert survey of THES was produced by a London-based company
specialized in MBA and graduate recruitment. This may very well cause a positive
bias for universities with large institutes of economics and schools of
management, and a negative bias for universities without a strong emphasis on
these disciplines."
THES rebuts:
"But
the Australian universities are popular in our peer review and do especially
well in our rankings of international success. They are among the world's most
enthusiastic recruiters of international staff and students, with years of
recruiting in Asia and beyond now visibly paying off" - (THES 2004).
In view of the above,
the following comment seems almost comical:
"...Peer review, the most trusted method for university comparison..."
-
Martin Ince (THES, 2004)
To understand why such rankings have been made by a
respectable magazine as Times, we must first understand the importance of higher
education to a country's economy.
Higher Education as a Source of
Revenue
Each year, hundreds of thousands of
international students travel overseas for higher education. Global higher
education is a lucrative business and universities throughout the world know
only too well how much revenue can be generated from foreign students? tuition
fees alone. In 2004-05, on average, international students paid £6,868
in tuition fees and £187.57 a week in living costs in the UK. Universities and
colleges in the UK charge international students vastly inflated fees to study
at both undergraduate and postgraduate level.
International students are charged from £4,350 for an arts-based undergraduate
course at City University to £15,675 at Oxford University. Postgraduate arts
courses range from £2,100 at City University to £28,850 at the London Business
School, while undergraduate-level science-based courses range from £5,177 a year
at Heythrop College to £17,350 a year at Imperial. And at postgraduate level,
international students face fees of between £3,950 at Edge Hill University and
£16,686 at the London School of Economics. However, UK is in danger of losing
precious international students by universities pricing themselves out of the
market.
Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI)
in UK, says:
"There is a real possibility that we may
price ourselves out of the market. It could well be in the national interest to
lower the fee charged of overseas students, in order to maximise their number.
The students bring considerable benefit to the economy, not just to the
universities that receive their fees. So it is worth maximising the number of
them."
- Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI),
UK
According to a report published by the British Council, education is worth more
to UK exports than financial services or the automotive industry.
A total of £28bn in 2003-04 was earned from overseas students by a sector
ranging from world famous universities to small English language colleges, from
independent schools to publishers and broadcasters. That figure has jumped from
£23bn in 2001-02 as numbers of international students have risen, and compares
with £19bn for financial services and £20bn for the automotive industry, said
the report compiled by Dr Pamela Lenton, of the University of Sheffield. Martin
Davidson, chief executive of the British Council, said education was
vital to the UK both economically and culturally. He added:
"... our position is vulnerable. Unless we
start taking education much more seriously as a global business, we will lose
out to other countries who understand the value of education to their economy
much better than we do."
- Martin Davidson, chief executive
of the British Council
Foreign students bring huge benefits to the country beyond their fees, such as
the living expenses they pay and the fact that many stay on to work. However,
Britain's reputation as a world-leading destination for international students
could be under threat as new research reveals that almost 30 per cent do not
think the education they receive is worth the money.
Although the number of international students has increased overall in the UK,
its share of the market has fallen from 16 per cent in 1998 to 11 per cent in
2004, with Chinese students in particular heading to the US instead. Bahram
Bekhradnia, director of the Higher Education Policy Institute (HEPI),
says:
"Competition from other
countries is increasing and information on what students get for their money is
increasing, so our unique selling points are being eroded, and our status in the
international market is under threat. I think we are in danger of killing that
golden goose."
- Bahram Bekhradnia, director of the Higher Education Policy Institute (HEPI),
UK
One foreign student who came from Kenya remarked:
'People are asking whether it is worth the risk, effort and hassle. In the past,
if you had the chance to study in the UK you grabbed it with both hands, but not
any more. Increasingly students are looking elsewhere.'
The impact of league tables on the destinations of
degree-seeking international students cannot be overemphasized. Many students
do take such league tables into account before making a decision on their
destination. With well-paid jobs increasingly hard to find in a society
that now has a glut of undergraduates and postgraduates, anything that can give
a student a competitive edge comes at a premium and people are prepared to pay
whatever it costs. Greg Philo, head of Glasgow University's Media Group, for the
British Council said:
"China's
one-child policy means that many parents feel they have to super-educate their
children. Often the entire life-savings of a family are spent on a child's
education."
- Greg Philo, head of Glasgow University's Media Group, for the British Council
It is therefore not surprising many established educational
institutions throughout the First World are competing fiercely with each other
to tap into Chinas enormous market. Confronted with increasingly stiff
competition from other G7 nations like France and Germany which are offering an
increasing number of international postgraduate programmes over the years at
only a fraction of the costs, Times-QS has resorted to compiling their own
international league tables to demonstrate that their own universities are still
among the best in the world in an attempt to convince international students
that higher education in UK is well worth the money invested. Whoever gets a
sizeable share of this "International higher education pie" is assured billions
in revenues each year, a tremendous boost to the country's economy.
Fierce advertising campaigns by UK and Australian
universities are a common sight in Asian countries like Singapore, Malaysia and
China. Advertisements titled "International MBA from an internationally
recognized institution" or "Get a degree in Financial Computing in Two Years!" are not uncommon in the local newspapers and
magazines in these Asian economies.
We also note that numerous Diploma Mills, whose main purpose is
revenue-generating have sprang out of UK over the years offering unrecognized
degrees to ineligible and sometimes unsuspecting international students, the
only prerequisite being that they are able to pay the tuition fees.
The 2007 Ranking
We first note that 4 out of the top 10
universities are from UK. While positions of many other universities (several
prestigious ones included) fluctuated over the years, Cambridge, Oxford and
Imperial?s positions are always improving. This year, Imperial College suddenly
rose to Number 5 in the world, ahead of prestigious US universities like MIT,
Berkeley and Caltech. Imperial does well on
the 2007 THES QS rankings partly because of outstanding scores on the peer
review (99 out of 100), employer review (99), international students (100) and
student faculty ratio (100). As mentioned above, the
peer and employer reviews are meaningless and can be ignored. According to
Richard Holmes, QS might have counted both academic and research staff when
computing the student faculty ratio,
leading to Imperial?s stellar score in this criterion. We also note that the
Imperial College Press is a joint venture of
Imperial College and World Scientific and World Scientific is a Singapore-based
publishing company whose subscription list is used by QS to construct their
"peer review.
This year, University College London suddenly rose to Number
9 in the world, ahead of MIT and Stanford. Instead of painstakingly providing
reasons as to why this is wholly unjustified, we will just ask a few questions.
-
How many Nobel Laureates and Fields Medallists has
Cambridge/Oxford/Imperial/UCL(combined?) produced compared to MIT over the
last ten years?
-
What is the yearly endowment of Cambridge/Oxford,/Imperial/UCL
compared to Stanford or Berkeley?
It seems that THES-QS is capitalizing on the good reputation
of UK?s established universities like Oxbridge and is trying to demonstrate (via
a dubious survey) that they are still among the very best in the world,
in order to attract degree-seeking international students who are increasingly
choosing other destinations over UK. To be fair, Oxford and Cambridge are good,
but not that good. They still lack far behind their US counterparts like
MIT and Stanford, whether in terms of yearly endowments or availability of top
international talents.
An eligible overseas student will most likely choose MIT over Cambridge or
Imperial for his engineering education. We all know that US is the home to
Lockheed Martin, Boeing, NASA, etc In terms of technology, we are not even sure
if UK is on par with Germany or France, home to Airbus, Eurofighter, EADS, BMW,
etc. In terms of number of Nobel laureates in either Physics or Chemistry, UK is
ranked third, after US and Germany. Yet, not a single German university made it
to the Top 50, while four UK universities made it to Top 10 and numerous more
made it to Top 50. Do you now see the absurdity of it all?
The impartially of the judge is a fundamental prerequisite in
any fair competition. Yet, the managing director of QS, Nunzio
Quacquarelli, said:
"The rankings recognised
the quality of
education that UK universities offer. In an environment of increasing student
mobility, the UK is putting itself forward as a top choice for students
worldwide."
Conclusion
While we do applaud the efforts of
organizations in bringing into light the relative standings of world
universities which can be valuable to degree-seeking international students and
policy makers of academic institutions, we deplore the use of league tables as
means to inflate the international standings of one's own institutions via
dubious ranking methodologies and unverifiable survey data, for economic gains
or other political motives.
IT is difficult to understand why this notorious "THES=QS
World University Ranking" has persisted to its fourth year. While there have
been sporadic efforts made by individuals to illuminate the deficiencies of the
rankings, a concerted and all-out effort by intellectuals from around the world
to condemn its results is long since overdue. It's disheartening to see many of
the undeservedly top-ranked universities have commended the THES-QS on a job
well-done without first seriously examining the validity of the ranking
methodologies. It is a basic tenet of Confucianism that one should not go
into raptures over the praises of others without first examining the motives
behind the praises or if one should deserve them. University administrators,
being the de-facto models for the students and staff, should have the courage
and dignity to dismiss or even condemn rankings which they know are flawed even
though they are might be rated highly in such rankings.
The THES rankings have contributed nothing but serve only to
mislead degree-seeking international students who need to make well-informed
decisions on their destinations of study. Staff and students in overrated
universities might develop a false sense of superiority and security and spend
their time identifying criteria of international ranking systems for the
purpose of maintaining or improving their positions instead
of making any real improvements in teaching or research.
Peddling propaganda is certainly not a means to becoming a
truly world-class university.