Quantitative Storytelling in the Making of a Composite Indicator

The reasons for and against composite indicators are briefly reviewed, as well as the available theories for their construction. After noting the strong normative dimension of these measures—which ultimately aim to ‘tell a story’, e.g. to promote the social discovery of a particular phenomenon, we inquire whether a less partisan use of a composite indicator can be proposed by allowing more latitude in the framing of its construction. We thus explore whether a composite indicator can be built to tell ‘more than one story’ and test this in practical contexts. These include measures used in convergence analysis in the field of cohesion policies and a recent case involving the World Bank’s Doing Business Index. Our experiments are built to imagine different constituencies and stakeholders who agree on the use of evidence and of statistical information while differing on the interpretation of what is relevant and vital.

     

Sensitivity Analysis for Importance Assessment

We review briefly some examples that would support an extended role for quantitative sensitivity analysis in the context of model-based analysis (Section 1). We then review what features a quantitative sensitivity analysis needs to have to play such a role (Section 2). The methods that meet these requirements are described in Section 3; an example is provided in Section 4. Some pointers to further research are set out in Section 5.     

Indicators for European Union Policies. Business as Usual?

This paper looks at the role of statistics-based knowledge in the making of EU policy. We highlight shortcomings in the use of statistical indicators made in the course of the Lisbon strategy, ended in 2010. In our opinion the shortcomings are: (i) The paradox of the coexistence within the same European Commission of two holistic frameworks: the Structural Indicators and the Sustainable Development Indicators. One does not understand which of these two systems is taken to measure the overall policy performance of the European Union. (ii) A communication issue whereby the Lisbon strategy and its offspring EU 2020 are not communicated (Lisbon is to the average citizen the capital of Portugal) and are especially not communicated in relation to existing statistical indicators of good quality, against the opinion of academicians that transparency and accountability based on sound statistics favour democracy and participation. We illustrate the reasons that lead us to see these points as problematic and offer suggestions on how these should be tackled in line with the practices developed in the Open Method of Coordination. The danger is that in the absence of a debate on the issue, these shortcomings be perpetuated in the EU 2020 strategy.     

Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators

Summary.  Composite indicators are increasingly used for bench-marking countries' performances. Yet doubts are often raised about the robustness of the resulting countries' rankings and about the significance of the associated policy message. We propose the use of uncertainty analysis and sensitivity analysis to gain useful insights during the process of building composite indicators, including a contribution to the indicators' definition of quality and an assessment of the reliability of countries' rankings. We discuss to what extent the use of uncertainty and sensitivity analysis may increase transparency or make policy inference more defensible by applying the methodology to a known composite indicator: the United Nations's technology achievement index.     

Composite Indicators between Analysis and Advocacy

We explore to what extent composite indicators, capable of aggregating multi-dimensional processes into simplified, stylised concepts, are up to the task of underpinning the development of data-based narratives for political advocacy. A recent OECD working paper (Nardo et al., 2005, Handbook on constructing composite indicators: methodology and user guide, OECD statistics working paper, STD/DOC(2005)3) offering ‘recommended practices’ for the construction of composite indicators is briefly illustrated, together with ‘pros’ and ‘cons’ associated with the use of aggregated statistical information. An attempt is made to summarise the terms of the controversy surrounding the use of composite indicators with practical and applied examples, as well as the mostly advocacy-driven spread of these measures in recent years. As an example, we focus on desirable narratives in support of the so-called Lisbon strategy and its ongoing revision, following one of the recommendations of a recent EU study [Kok: 2004, The High Level Group on Lisbon Strategy chaired by Wim Kok, Facing the Challenge, European Communities, Luxembourg, 2004 on how to streamline and reinvigorate the EU’s Lisbon Agenda. Finally we try to establish a link between the use of composite, even for analytic purposes, and the development of a robust culture of evaluation of policies based on information [Messerlin: 2005, 35th Wincott Lecture, October 3, 2005]. Of these, we try to offer stylised examples – also from the recent literature [Sapir: 2005, Globalisation and the Reform of European Social Models, 2005, http://www.bruegel.org/] where composite indicators are used.     

Winding Stairs: A sampling tool to compute sensitivity indices

Sensitivity analysis aims to ascertain how each model input factor influences the variation in the model output. In performing global sensitivity analysis, we often encounter the problem of selecting the required number of runs in order to estimate the first order and/or the total indices accurately at a reasonable computational cost. The Winding Stairs sampling scheme (Jansen M.J.W., Rossing W.A.H., and Daamen R.A. 1994. In: Gasman J. and van Straten G. (Eds.), Predictability and Nonlinear Modelling in Natural Sciences and Economics. pp. 334–343.) is designed to provide an economic way to compute these indices. The main advantage of it is the multiple use of model evaluations, hence reducing the total number of model evaluations by more than half. The scheme is used in three simulation studies to compare its performance with the classic Sobol' LP. Results suggest that the Jansen Winding Stairs method provides better estimates of the Total Sensitivity Indices at small sample sizes.     

Presenting Results from Model Based Studies to Decision-Makers: Can Sensitivity Analysis be a Defogging Agent?

The motivation of the present work is to provide an auxiliary tool for the decision-maker (DM) faced with predictive model uncertainty. The tool is especially suited for the allocation of R&Dresources. When taking decisions under uncertainties, making use of the output from mathematical or computational models, the DM might be helped if the uncertainty in model predictions be decomposed in a quantitative-rather than qualitativefashion, apportioning uncertainty according to source. This would allow optimal use of resources to reduce the imprecision in the prediction. For complex models, such a decomposition of the uncertainty into constituent elements could be impractical as such, due to the large number of parameters involved. If instead parameters could be grouped into logical subsets, then the analysis could be more useful, also because the decision maker might likely have different perceptions (and degrees of acceptance) for different kinds of uncertainty. For instance, the decomposition in groups could involve one subset of factors for each constituent module of the model; or one set for the weights, and one for the factors in a multicriteria analysis; or phenomenological parameters of the model vs. factors driving the model configuratiodstructure aggregation level, etc.); finally, one might imagine that a partition of the uncertainty could be sought between stochastic (or aleatory) and subjective (or epistemic) uncertainty. The present note shows how to compute rigorous decomposition of the output's variance with grouped parameters, and how this approach may be beneficial for the efficiency and transparency of the analysis.