Methodology of the InCiSE indicators
The sections of this chapter set out the methodology for each of the 12 indicators that make up the 2019 edition of the InCiSE Index. For each indicator the section outlines: the source data; the indicator structure and weighting; the nature and definition of the imported source data and any transformations; the approach to imputation of missing data; and, the rationale for any changes from the 2017 Pilot methodology.
The source data for InCiSE comes from a variety of sources which use different methodologies, we have applied the following taxonomy to describe the different types of data sources:
- Subjective data:
- Public opinion survey – a survey of the opinion/attitudes of the general population/households within a country (e.g. Transparency International’s Global Corruption Barometer)
- Business opinion survey – a survey of the opinion/attitudes of business owners/executives within a country (e.g. the World Economic Forum’s Executive Opinion Survey)
- Expert assessment – a survey/assessment of a country made by a small number of experts/researchers (e.g the Quality of Government Institute’s Expert Opinion Survey)
- Objective data:
- Analysis of published data – secondary analysis of information/data published by governments
- Social survey – studies that use scientific social survey methods to collect representative information about the population, but are not opinion surveys (e.g. the OECD’s Programme for the International Assessment of Adult Skills)
- Government assessments – official responses from governments to data collection exercises by international organisations (e.g. OECD surveys)
Each of these types has its strengths and limitations, and some types of data are more appropriate in certain cases than others. The InCiSE model places equal value on these different types of data and does not attempt to make ‘quality adjustments’, e.g. through weighting, to distinguish between the different types of data.
Critiques of subjective measures can include that they measure perceptions and other ‘subjective’ positions which may be influenced by considerations beyond just the specific item being measured – e.g. business perceptions of how effective the civil service is at delivering services may be influenced by their perceptions of how business-friendly the government’s political programme is. Another critique is through the use of expert assessments, which often rely on a small number of experts/researchers to assess government performance on a given topic or area. However, expert assessments often focus on niche areas which the general public/businesses may not be able to make a judgement about.
Objective data is also not without its own limitations. It can be argued that it is rare for any data to be truly ‘objective’ even if it is not directly ‘subjective’. Even if the data does not aim to measure perceptions or another form of subjective position, it is collected and analysed to fulfil a particular purpose, defined by a particular group of individuals, with a particular agenda. While efforts can be made to minimise biases and particular normative assumptions, in any study there are implicit or explicit subjective decisions made about the collection and analysis of data. The decisions a researcher or analyst makes, such as whether to collect one piece of data over another, which methods of collection and analysis to use, or what to consider in scope or out of scope, are all subjective and therefore will influence the results.
Each of the chapters in this part (3-14) list the data sources used to supply the input data for the InCiSE metrics of each indicator. For ease of reference in each section’s tables, the data sources are given an acronym. Figures in square brackets next to a data source indicate the reference year for the data (i.e. the year the data was collected/relates to) rather than the year of publication. A complete reference list of the data sources used for InCiSE is provided in the References chapter. Some metrics are calculated as aggregations of multiple data points, details of these calculations are provided in Appendix A.