Mixed Report Card Issued on Australian Research

In 2010, the Australian Research Council (ARC) conducted the first full evaluation of the Excellence in Research for Australia (ERA) initiative.  Its report, released on 31 January 2010, provides the outcomes of the ERA 2010 evaluations, which applies to research undertaken between 1 January 2003 and 31 December 2008.

The Report indicates that in many fields of research, Australia’s universities perform at, or above, world standards.  (The Report makes the point that ‘world standard’ is a defined indicator of quality and not, for example, a geographical or ‘average’ performance measure.)  Particular strengths include the mathematical, physical and earth sciences, areas of agricultural and veterinary sciences, areas of engineering (especially aerospace, environmental, manufacturing and resources), biotechnology, microtechnology (especially communications and nanotechnology), areas of medical and health sciences, history and archaeology.

Specific fields of research in which four or more institutions were rated as being ‘well above world standards’ include cardiovascular medicine, oncology, immunology, astronomical and space sciences, quantum physics and historical studies.

Areas of relative weakness include education, economics, commerce, social studies, legal studies, creative arts and writing, language, communication and culture.

TOP PERFORMERS

The top 10 universities, measured by the percentage of research conducted in the institution (by field of research) that is above world standard are: the University of Melbourne (80%); the Australian National University (78%); the University of Queensland (76%); the University of New South Wales (64%); the University of Sydney (58%); Monash University (57%); Adelaide University (44%); Macquarie University (36%); the University of Western Australia (35%); and the University of Wollongong (31%). 

THE ERA INITIATIVE

The ERA initiative is a key element of the Australian Government’s ten year Innovation Agenda, Powering Ideas.   Its objective is to assess research quality within Australia's higher education institutions using a combination of indicators and expert review by committees comprising experienced, internationally-recognised experts.

A media release issued by Innovation Minister Senator Kim Carr states that ‘when it comes to funding quality research it is important the Australian Government fully understands Australia’s strengths and weaknesses.’

The release goes on to quote the Minister:

The Australian Government invests billions of dollars in research each year. ERA gives the Australian taxpayer assurance that their money is being invested wisely and gives the Government a clear idea of the research areas we need to focus on for improvement and continued excellence.

The ERA 2010 National Report shows us that we have a large number of research strengths — we should be very proud of this. Our performance in areas like history, immunology and quantum physics is truly outstanding.

But the report also shows we need to do better in some areas. This is to be expected when you conduct an all encompassing assessment of a country's research effort in universities. The evidence the ERA evaluation has given us will help us find ways to improve.

‘PATENTS SEALED’

One metric contributing to the overall assessment of performance in a particular field of research is the number of patents sealed.  While this number, i.e. patents actually granted, is no doubt a better indication of the quality of the underlying research than the number of applications filed, the problem we perceive with this metric is that it lags behind the current state of research.

For example, patents sealed during the study period (2003-2008) were most likely filed during the period between 1998 and 2003, and based upon research conducted during, or prior to, this period.

Bearing this in mind, the broad fields of research in which the greatest numbers of patents were sealed were engineering, medical and health sciences, biological sciences and chemical sciences.  This is unsurprising, considering that these are substantially fields of applied research with the greatest scope for commercialisation in the short-to-medium term.  The number of patent applications  filed and sealed in these fields may therefore be more reflective of the greater potential for commercial exploitation than of any superior inherent quality of the research compared to subjects with less immediate commercial application.

COMMENT: VALUE OF THE ERA INITIATIVE

These kinds of data-heavy reports always make us a little uncomfortable.  It is obviously easy to gather, tabulate and graph data supplied by the various universities, but less straightforward to draw meaningful conclusions from the numbers.  While those in charge of the money are no doubt more comfortable with spreadsheets than the rather more untidy realities, the fact remains that using measures of past performance as a basis for future funding is a fraught enterprise.

What is required, in our view, is a balance between supporting the country’s existing research strengths, and building new knowledge, skills and expertise in areas likely to be of commercial or strategic importance in the future.

Senator Carr has been reported in the press as saying ‘‘We are putting people on notice.  I can’t justify funding second-rate research.’’  On one level, this is perfectly reasonable position, since limited funds clearly should not be ‘wasted’ on projects that will ultimately deliver no benefit to the country.  On the other hand, the definition of ‘beneficial’, in this context, is somewhat obscurely built into the ERA metrics – what the metrics reward with higher scores is what the Minister will conclude is worthy of funding.  Furthermore, some research is ‘second rate’ because it is in a developing field in this country.  No matter how great its potential importance for the country’s future, today’s ‘second rate’ research will never become ‘world class’ if it is not adequately supported.

As the saying goes, there are lies, there are damned lies, and then there are statistics – in the ERA Report some 300 pages of statistics.  The numbers themselves represent facts that are inherently neither good nor bad.  But there are undoubtedly good and bad ways of interpreting and using those numbers.  We hope that our leaders and bureaucrats are able to tell the difference.

Tags: Innovation policy, News, Public research