The experts mirror: an independent authority over “reality”?

Printable version of the article  
Send this article by mail  

How can knowledge generated by the PISA survey be converted into usable information for policy-makers? In this excerpt, we analyze how the OECD developed PISA as a credible and useful tool for the exercise of government power and/or government scrutiny.

I think a broker is a negotiator and we don’t do that, we are more like a mirror. We have a mirror; you can look at it and you can see how the world looks, if you don’t like it you don’t look at it” (Interview, OECD’s PISA Secretariat).

Downstream from the knowledge used for the development of PISA, we have noticed the existence of a new type of knowledge generated by these sources and the actors utilizing this knowledge. PISA reports incorporate, demarcate and highlight the knowledge generated by various “collectives”, while giving greater importance to and changing the nature of the knowledge generated in the abovementioned collaborative forces. The background work underlying the survey (e.g., experts preparing the framework, building the assessment instruments, and collecting, processing and analyzing the statistical data) that is detailed in the technical annex of the report along with the procedures leading to the creation of said knowledge, has been transformed into “results”, “analyses” and “implications”. In other words, the report mobilizes and concretizes the knowledge generated throughout the survey activities, but also stretches this information to lend it new meaning.

PISA management, at the OECD, is fully aware that their texts are selectively used by their audiences. This seems to be the price to pay for greater value and a larger ambition, that is to say, supply credible and understandable information to their audiences and generate learning: “(…) the selective views of the information and the policy debate are a problem that we have to live with. Selecting information is better than having no information". (…) "I must say for comparison purposes, that a lot has happened between the public debate on PISA in the year 2000 and the current situation. In 2000, I was very frustrated by the simplistic discussion on PISA. Today we really have people who take things very seriously, who look at the substance. (…) I take that as a learning process at all levels”(Interview, OECD’s PISA Secretariat).

Hence, the reality deriving from and based on the scientific methodology reflected in the mirror, is developed in line with these canons and adapted to the audience. There is a central tension in the writing of the reports: “A good report is one that tells a very clear story (…) and (…) our role is to always find an objective interpretation of the data. I could have written 10 pages for each of these indicators stating what I believe they tell me, but that’s not what we do in this report; we limit ourselves to things that we have seen. A good report offering substance should be accessible to people and understandable. Our reports probably are a bit too complicated. I think that’s the price to pay…if journalists worked on this, they would make everything much clearer and much simpler, but accuracy would be lost…[Researcher: Is it a difficult balance between…] I think that if we have to make a choice, we should always aim for accuracy, even if it makes things more complicated" (Interview, OECD’s PISA Secretariat).

We have previously demonstrated how the reports do not directly prescribe policies – insofar as they do not proclaim ‘this is what should be done’. Instead, they are guidelines for organizing the readers’ analysis, appraisals and reflections. Our final analytical incursion will now deal with this matter.

The reports are presented and want to be understood as a mirror that (merely) reveals – with scientific clarity – the reality of school performance and the relationships with the socio-economic backgrounds and mediation factors (political intervention). They are not presented as a kind of magic Disney film mirror that judges the quality of the actors. The search for this condition is borne out of the diligence taken, throughout all of the reports, to enlighten the reader about how the data is and/or can be interpreted and which conclusions can be drawn from said data.

From the point of view of the PISA managers (and at the OECD level), what counts the most is to be recognized as a com-pulsory space for policy debates and policy decision-making founded on scientific evidence; the purpose being to uncover “new” problems and provide new information "based on evidence” – and not so much problem solving: “(…) I think that PISA is really about the fact that classroom reality and public policies often do not match; you know that public policy decisions often have fairly little do to with the real work that teachers carry out in the classroom. PISA allows to connect the two, because one can actually see what happens (intercourse) and in that sense, it generates the knowledge necessary to make the reality of education accessible (…)” (Interview, OECD’s PISA Secretariat).

This discourse about PISA reports reveals the presence of a vision of the OECD which brings to light and confirms its role as an ideational authority, “playing an idea game, through which it collects and handles data, knowledge, visions and ideas and distributes them to its member countries and, to a greater extent even, a number of non-member countries” (Marcussen, 200: 91). PISA surveys and reports are essential to display a constant capacity to produce innovation in national political discourses, breaking away from the stereotypes of public action and differentiation with regard to national policy ideas. As such, the question of the "independence‟ of the OECD and the independent nature of its products is also an issue that is raised when these participants discuss the identity of the organization ("I mean, the OECD is an independent organization, but now people understand that". "(…) [B]ecause PISA is an independent evaluation instrument (…)" (Interview, OECD’s PISA Secretariat).

In this perspective, the enrichment of the role of PISA is also championed by the dissemination of “evidence” and descriptions, rather than producing interpretations and prescriptions – “I think we should limit ourselves; we deal with descriptive analysis, which means more or less describing how the world is, without stating how to make the world better and, I think, that is something we must be very careful with” (…)» (Interview, OECD’s PISA Secretariat ) – and extending the analytical and interpretive work of others, as a standard of the quality to be pursued in the future [1].

As we advocated earlier, the “expert mirror” brings much more than a simple mapping of reality. “Indicators are not numbers: they are words; they are named” – thus, they define “what to look at”, “where to look”, and “how to look at it” (Mangez, 2008: 106). Furthermore, the image –knowledge – that the mirror projects is created in a theoretical and technical manner, under particular social circumstances and by specific actors (Nassehi et al., 2007:141). The reality shown by the mirror includes, articulates and disseminates assumptions about education, schooling, and their relationship with the economy and society (see Carvalho, 2009: 40-44). It also mobilizes, structures and disseminates models for their cognition (see Carvalho, 2009: 48-54) legitimated through statistical measurements, comparisons and various forms of substantive knowledge. Indeed, the image of the expert organizes and acts on “les catégories de perception et d’appréciation du monde social, les structures cognitives et évaluatives” (Bourdieu, 1987: 159).

One of the central aspects of this intervention on the categories of perception lies in the mirror showing each one (each country) in its relationship and position relative to others. As we wrote earlier, comparisons situate policy decision-makers on an imagined time-line leading from the industrial society into the knowledge society and simultaneously positions them in a competitive-cooperative worldwide space. The international comparison is a place where we imagine a transnational society (outlining for individuals a common background of performance and engagement expectations in social life based on the trust vested in scientific expertise) and in which a "transnational policy” is constructed (in which the states are voluntarily involved with other instances in the joint production of rules). In parallel, these comparisons comprise tools for the exercise of government authority and their scrutiny at national level. From this point of view, the image that the "mirror” returns to us is in itself a discourse of power, both through culpability and responsibility and through the hope, optimism and confidence in the possibility of reform brought by politicians and national policies.

Tools such as PISA are valuable resources to reflect over and act or to legitimize action, thanks to their reputed objectivity with regard to reality and the fact they are, at least prima facie, outside the interests that this type of knowledge entails in our societies. One can suppose that national politicians will find it difficult to disregard the studies produced by agencies that – taken as independent structures of expertise – provide them with resources they need to engage in political struggles and think and/or express their guidelines in the name of the independence and cosmopolitism of this specialized knowledge. As the reader knows, it is not this report’s responsibility to deal with this matter [2].

[1] “This report is very important and my dream is that in 10 years’ time, yes this report will exist but 50 others reports will exist, produced by your universities, produced by thinkers, produced by….that’s where we need to get. Today, the world listens to interpretation of data like this. I hope that in the future the world really listens to every little datum so that people will find information on the web and will draw their own conclusions. There may be controversies, they may not agree and with this report we always have to make sure that what we say is basically…you will not find any opinions in this book, we can’t do that, but I think I’ve nothing against preparing opinions, I think opinions will be great. If lots of people have lots of opinions about things, we should publish them and we’re working on it. I mean the online database that we’ve put out on the web is the first step, but we have realized that many people find it too difficult to use, so we’ve now developed manuals and I really hope that in the long term there will be a different balance in the use of the data and the interpretations that we have here (…)" (Interview, OECD’s PISA Secretariat).

[2] What comes out of this immense repository of results and analysis of results supplied by PISA when the audiences come into contact with them, and what happens as regards their use in the scope of policy action in education is a matter that will be analyzed in the reports about the active reception of PISA.

CARVALHO L.M. [with E. COSTA] (2009), Production of OECD’s - Programme for International Students Assessment" (PISA), KNOWandPOL Report.

Read document

  • BOURDIEU Pierre (1987), Choses Dites, Paris : Les Editions du Minuit.
  • CARVALHO Luis M. with E. Costa (2009), The production of OECD: Programme for International Student Assessment (PISA), KNOWandPOL Report (111-115).
  • MANGEZ Eric (2008), Knowledge economy, knowledge policy and knowledge regimes, in Delvaux Bernard & Mangez Eric, Towards a Sociology of the Knowledge-Policy Relation, KNOWandPOL Literature Review, Integrative Report, 98-118.
  • MARCUSSEN Martin (2004), The Organization for Economic Cooperation and Development as ideational artist and arbitrator, in: Reinalda Bob & Verbeek Bertjan (Eds), Decision Making Within International Organisations, London: Routledge, 90-105. NASSEHI Armin (2008), Making knowledge observable. Short considerations about the practice of ‘doing knowledge’, KNOWandPOl, unpublished paper.

© 2011 Knowandpol Designed and Powered by platanas