Appendix B — Sensitivity analysis: detailed results

This Annex provides detailed results from the sensitivity analysis described in Chapter 16. Each table includes the index score and rank for each of the 38 countries included in the 2019 InCiSE Index results for each of the sensitivity tests carried out alongside the results of the 2019 index results.

Table B.1 shows the results of the sensitivity tests varying country coverage (Section 16.1):

• Using a data quality assessment threshold of 0.55 to determine country inclusion;
• Using a data quality assessment threshold of 0.6 to determine country inclusion;
• Using a threshold of 75% of the available data to determine country inclusion;
• Using only the countries included in the 2017 Pilot edition of the index.

Table B.2 shows the results of the sensitivity tests varying the reference date (Section 16.2):

• Excluding the capabilities indicator;
• Excluding the capabilities indicator and adjusting the weighting accordingly;
• Using only data with a reference year of 2015 or later;
• Using only data with a reference year of 2016 or later.

Table B.3 shows the results of the sensitivity tests using alternative weighting (Section 16.3):

• Using a 50:50 split for the equal-share and data-quality based weighting;
• Using only equal indicator weights(i.e. all indicator weights equal 1/12);
• Using only indicator weights based on the data quality assessment results;
• Not applying any within-indicator weights;
• Calculating the index as a sum of all metrics.

Table B.4 shows the results of the sensitivity tests adjusting the base data (Section 16.4):

• Ranking the metrics before imputation;
• Rescaling the metrics before imputation;
• Standardising the metrics before imputation.

Table B.5 shows the results of the sensitivity tests using alternative imputation methods (Section 16.5):

• Using an ‘all-in-one’ approach for imputation of missing data;
• Using the ‘midas touch’ method for imputation of missing data;
• Using the ‘random forests’ method for imputation of missing data;
• Replacing missing data with the mean of observed values.

Table B.1: Sensitivity tests varying country coverage

Country	Index score					Country rank
	2019 results	DQA ≥ 0.55	DQA ≥ 0.6	75% of data	2017 group	2019 results	DQA ≥ 0.55	DQA ≥ 0.6	75% of data	2017 group
GBR	1.000	1.000	1.000	1.000	1.000	1	1	1	1	1
NZL	0.980	0.987	0.987	0.992	0.985	2	2	2	2	2
CAN	0.916	0.907	0.902	0.906	0.898	3	3	3	3	3
FIN	0.883	0.887	0.881	0.886	0.879	4	4	4	4	4
AUS	0.863	0.859	0.858		0.859	5	5	5		5
DNK	0.832	0.854	0.847	0.850	0.843	6	6	6	5	6
NOR	0.830	0.832	0.830	0.831	0.828	7	7	7	6	7
NLD	0.794	0.792	0.790	0.790	0.786	8	8	8	7	8
KOR	0.785	0.781	0.779	0.777	0.773	9	9	9	8	10
SWE	0.785	0.773	0.775	0.775	0.775	10	10	10	9	9
USA	0.765	0.759			0.758	11	11			11
EST	0.674	0.675	0.674	0.672	0.671	12	12	11	10	12
CHE	0.650	0.639	0.635	0.641	0.635	13	13	12	11	13
IRL	0.625	0.623	0.617	0.619	0.611	14	16	15	14	16
FRA	0.619	0.626	0.627	0.627	0.628	15	15	13	12	14
AUT	0.617	0.626	0.621	0.624	0.620	16	14	14	13	15
ESP	0.599	0.596	0.589	0.586	0.590	17	17	16	15	17
MEX	0.507	0.500	0.504		0.505	18	19	17		18
DEU	0.505	0.502	0.503	0.504	0.502	19	18	18	16	19
LTU	0.487	0.490	0.483	0.484		20	20	19	18
BEL	0.485	0.483	0.475	0.482	0.473	21	22	21	19	21
JPN	0.472	0.485	0.481	0.488	0.479	22	21	20	17	20
LVA	0.466	0.471	0.463	0.462		23	23	22	20
CHL	0.454	0.451	0.450	0.452	0.446	24	24	23	21	22
ITA	0.419	0.431	0.428	0.424	0.427	25	25	24	22	23
SVN	0.369	0.359	0.358	0.360	0.359	26	26	25	23	24
ISR	0.315	0.322				27	27
POL	0.282	0.277	0.274	0.281	0.274	28	28	26	24	25
PRT	0.259	0.260	0.260	0.259	0.258	29	29	27	25	27
CZE	0.245	0.260	0.260	0.257	0.259	30	30	28	26	26
ISL	0.228	0.233				31	31
TUR	0.189	0.181	0.184	0.176	0.182	32	32	29	27	28
SVK	0.172	0.178	0.175	0.171	0.175	33	33	30	28	29
BGR	0.147					34
HRV	0.140					35
ROU	0.127					36
GRC	0.107	0.103	0.098	0.092	0.098	37	34	31	29	30
HUN	0.000	0.000	0.000	0.000	0.000	38	35	32	30	31
MAE	—	0.006	0.006	0.007	0.007
Table B.1 in the original PDF publication

Table B.2: Sensitivity tests varying reference year

Country	Index score					Country rank
	2019 results	Excl. CAP	Excl. CAP & reweight	2015-2018 data	2016-2018 data	2019 results	Excl. CAP	Excl. CAP & reweight	2015-2018 data	2016-2018 data
GBR	1.000	1.000	1.000	1.000	1.000	1	1	1	1	1
NZL	0.980	0.955	0.955	0.949	0.912	2	2	2	2	2
CAN	0.916	0.906	0.906	0.895	0.834	3	3	3	3	3
FIN	0.883	0.873	0.873	0.881	0.830	4	5	5	4	4
AUS	0.863	0.875	0.875	0.869	0.810	5	4	4	5	5
DNK	0.832	0.817	0.817	0.789	0.775	6	7	7	9	8
NOR	0.830	0.836	0.836	0.843	0.791	7	6	6	6	6
NLD	0.794	0.802	0.802	0.790	0.758	8	8	8	8	9
KOR	0.785	0.799	0.799	0.784	0.711	9	9	9	10	11
SWE	0.785	0.791	0.791	0.802	0.787	10	10	10	7	7
USA	0.765	0.740	0.741	0.735	0.731	11	11	11	11	10
EST	0.674	0.664	0.665	0.639	0.594	12	12	12	12	17
CHE	0.650	0.653	0.653	0.639	0.597	13	14	14	13	14
IRL	0.625	0.622	0.623	0.619	0.594	14	16	16	15	15
FRA	0.619	0.655	0.656	0.628	0.636	15	13	13	14	12
AUT	0.617	0.635	0.636	0.606	0.594	16	15	15	16	16
ESP	0.599	0.608	0.609	0.593	0.602	17	17	17	17	13
MEX	0.507	0.519	0.519	0.519	0.523	18	20	20	19	19
DEU	0.505	0.521	0.521	0.506	0.536	19	19	19	21	18
LTU	0.487	0.534	0.534	0.518	0.484	20	18	18	20	20
BEL	0.485	0.506	0.506	0.526	0.434	21	21	21	18	22
JPN	0.472	0.495	0.495	0.482	0.444	22	22	22	22	21
LVA	0.466	0.474	0.474	0.454	0.349	23	23	23	24	26
CHL	0.454	0.463	0.463	0.473	0.396	24	24	24	23	25
ITA	0.419	0.451	0.452	0.442	0.427	25	25	25	25	23
SVN	0.369	0.363	0.363	0.351	0.271	26	26	26	26	28
ISR	0.315	0.332	0.333	0.326	0.401	27	27	27	27	24
POL	0.282	0.268	0.269	0.237	0.083	28	28	28	30	36
PRT	0.259	0.256	0.256	0.238	0.220	29	30	30	29	29
CZE	0.245	0.240	0.240	0.221	0.309	30	31	31	32	27
ISL	0.228	0.238	0.238	0.233	0.218	31	32	32	31	30
TUR	0.189	0.261	0.262	0.244	0.141	32	29	29	28	32
SVK	0.172	0.156	0.156	0.159	0.150	33	34	34	33	31
BGR	0.147	0.153	0.153	0.135	0.110	34	35	35	34	34
HRV	0.140	0.144	0.144	0.118	0.073	35	36	36	36	37
ROU	0.127	0.128	0.128	0.105	0.104	36	37	37	37	35
GRC	0.107	0.158	0.158	0.129	0.113	37	33	33	35	33
HUN	0.000	0.000	0.000	0.000	0.000	38	38	38	38	38
MAE	—	0.015	0.015	0.017	0.045
Table B.2 in the original PDF publication

Table B.3: Sensitivity tests with alternative approaches to weighting

Country	Index score						Country rank
	2019 results	50:50	Equal weights	All DQA weights	No within indicator	Sum of metrics	2019 results	50:50	Equal weights	All DQA weights	No within indicator	Sum of metrics
GBR	1.000	1.000	0.999	1.000	1.000	0.949	1	1	2	1	1	2
NZL	0.980	0.970	1.000	0.941	0.941	1.000	2	2	1	2	2	1
CAN	0.916	0.911	0.927	0.895	0.896	0.928	3	3	3	3	3	3
FIN	0.883	0.876	0.895	0.858	0.824	0.897	4	4	4	4	5	4
AUS	0.863	0.859	0.869	0.850	0.864	0.864	5	5	5	5	4	5
DNK	0.832	0.823	0.851	0.797	0.774	0.855	6	7	6	7	8	6
NOR	0.830	0.825	0.839	0.812	0.805	0.851	7	6	7	6	7	7
NLD	0.794	0.787	0.809	0.765	0.745	0.783	8	8	8	10	10	10
KOR	0.785	0.786	0.784	0.786	0.809	0.787	9	9	11	8	6	9
SWE	0.785	0.782	0.790	0.775	0.763	0.805	10	10	9	9	9	8
USA	0.765	0.754	0.787	0.722	0.724	0.781	11	11	10	11	11	11
EST	0.674	0.667	0.687	0.649	0.657	0.740	12	12	12	12	12	12
CHE	0.650	0.647	0.656	0.637	0.613	0.639	13	13	13	13	14	14
IRL	0.625	0.616	0.641	0.593	0.598	0.601	14	15	14	15	16	17
FRA	0.619	0.618	0.620	0.615	0.635	0.609	15	14	16	14	13	16
AUT	0.617	0.608	0.635	0.583	0.607	0.649	16	16	15	16	15	13
ESP	0.599	0.593	0.611	0.576	0.589	0.611	17	17	17	17	17	15
MEX	0.507	0.511	0.497	0.524	0.497	0.412	18	18	19	18	19	24
DEU	0.505	0.503	0.507	0.499	0.514	0.525	19	19	18	19	18	18
LTU	0.487	0.489	0.483	0.494	0.482	0.470	20	20	21	20	20	22
BEL	0.485	0.482	0.491	0.473	0.477	0.488	21	21	20	21	21	21
JPN	0.472	0.471	0.475	0.467	0.438	0.503	22	22	23	22	24	19
LVA	0.466	0.460	0.479	0.442	0.447	0.498	23	23	22	24	23	20
CHL	0.454	0.451	0.459	0.444	0.461	0.440	24	24	24	23	22	23
ITA	0.419	0.419	0.418	0.420	0.436	0.368	25	25	25	25	25	26
SVN	0.369	0.360	0.388	0.333	0.338	0.394	26	26	26	26	26	25
ISR	0.315	0.309	0.326	0.294	0.314	0.334	27	27	27	27	27	27
POL	0.282	0.276	0.293	0.261	0.236	0.291	28	28	28	28	29	28
PRT	0.259	0.253	0.273	0.235	0.241	0.289	29	29	29	29	28	29
CZE	0.245	0.240	0.253	0.228	0.174	0.254	30	30	30	30	32	31
ISL	0.228	0.223	0.239	0.207	0.218	0.270	31	31	31	32	31	30
TUR	0.189	0.194	0.177	0.209	0.231	0.063	32	32	32	31	30	35
SVK	0.172	0.170	0.175	0.166	0.107	0.143	33	33	33	33	35	32
BGR	0.147	0.146	0.150	0.141	0.129	0.127	34	34	34	34	33	33
HRV	0.140	0.140	0.140	0.140	0.086	0.069	35	35	35	35	36	34
ROU	0.127	0.131	0.120	0.140	0.070	0.050	36	36	36	36	37	36
GRC	0.107	0.111	0.099	0.123	0.123	0.000	37	37	37	37	34	38
HUN	0.000	0.000	0.000	0.000	0.000	0.023	38	38	38	38	38	37
MAE	—	0.004	0.009	0.017	0.026	0.032
Table B.3 in the original PDF publication

Table B.4: Sensitivity tests adjusting the base data

Country	Index score				Country rank
	2019 results	Ranked data	Rescaled data	Standardise data	2019 results	Ranked data	Rescaled data	Standardise data
GBR	1.000	1.000	1.000	1.000	1	1	1	1
NZL	0.980	0.951	0.978	0.979	2	2	2	2
CAN	0.916	0.893	0.927	0.908	3	3	3	3
FIN	0.883	0.883	0.885	0.885	4	4	4	4
AUS	0.863	0.833	0.868	0.864	5	6	5	5
DNK	0.832	0.862	0.824	0.827	6	5	7	7
NOR	0.830	0.789	0.830	0.829	7	10	6	6
NLD	0.794	0.815	0.794	0.783	8	8	8	9
KOR	0.785	0.818	0.767	0.770	9	7	10	11
SWE	0.785	0.802	0.785	0.784	10	9	9	8
USA	0.765	0.775	0.737	0.781	11	11	11	10
EST	0.674	0.680	0.666	0.671	12	13	12	12
CHE	0.650	0.681	0.630	0.625	13	12	15	13
IRL	0.625	0.588	0.631	0.623	14	15	14	14
FRA	0.619	0.632	0.633	0.620	15	14	13	15
AUT	0.617	0.573	0.622	0.612	16	17	16	16
ESP	0.599	0.577	0.598	0.591	17	16	17	17
MEX	0.507	0.461	0.517	0.522	18	23	18	18
DEU	0.505	0.506	0.497	0.488	19	20	19	21
LTU	0.487	0.495	0.490	0.497	20	21	21	20
BEL	0.485	0.514	0.493	0.513	21	19	20	19
JPN	0.472	0.521	0.478	0.484	22	18	23	22
LVA	0.466	0.432	0.461	0.453	23	24	24	24
CHL	0.454	0.465	0.484	0.474	24	22	22	23
ITA	0.419	0.402	0.415	0.412	25	25	25	25
SVN	0.369	0.386	0.366	0.361	26	26	26	26
ISR	0.315	0.322	0.324	0.327	27	27	27	27
POL	0.282	0.242	0.276	0.273	28	29	28	28
PRT	0.259	0.223	0.252	0.238	29	31	31	31
CZE	0.245	0.275	0.257	0.251	30	28	29	30
ISL	0.228	0.230	0.254	0.263	31	30	30	29
TUR	0.189	0.185	0.179	0.190	32	32	32	32
SVK	0.172	0.170	0.177	0.162	33	33	33	33
BGR	0.147	0.122	0.135	0.113	34	35	35	35
HRV	0.140	0.131	0.151	0.139	35	34	34	34
ROU	0.127	0.107	0.102	0.090	36	36	36	36
GRC	0.107	0.072	0.066	0.075	37	37	37	37
HUN	0.000	0.000	0.000	0.000	38	38	38	38
MAE	—	0.021	0.010	0.011
Table B.4 in the original PDF publication

Table B.5: Sensitivity tests adjusting the base data

Country	Index score					Country rank
	2019 results	All-in-one	Midas touch	Random forests	Mean value	2019 results	All-in-one	Midas touch	Random forests	Mean value
GBR	1.000	0.989	1.000	1.000	1.000	1	2	1	1	1
NZL	0.980	1.000	0.975	0.981	0.966	2	1	2	2	2
CAN	0.916	0.862	0.902	0.891	0.867	3	5	3	3	3
FIN	0.883	0.881	0.884	0.874	0.850	4	3	4	4	4
AUS	0.863	0.872	0.807	0.854	0.821	5	4	7	5	6
DNK	0.832	0.833	0.838	0.837	0.808	6	7	5	6	7
NOR	0.830	0.834	0.831	0.831	0.824	7	6	6	7	5
NLD	0.794	0.791	0.794	0.766	0.762	8	9	8	9	9
KOR	0.785	0.738	0.764	0.755	0.728	9	10	10	10	10
SWE	0.785	0.799	0.792	0.785	0.788	10	8	9	8	8
USA	0.765	0.691	0.747	0.721	0.699	11	11	11	11	11
EST	0.674	0.639	0.627	0.665	0.590	12	12	13	12	15
CHE	0.650	0.621	0.646	0.613	0.595	13	15	12	14	13
IRL	0.625	0.562	0.608	0.601	0.534	14	17	16	16	17
FRA	0.619	0.629	0.619	0.603	0.605	15	13	14	15	12
AUT	0.617	0.621	0.618	0.614	0.593	16	14	15	13	14
ESP	0.599	0.609	0.598	0.598	0.574	17	16	17	17	16
MEX	0.507	0.460	0.498	0.485	0.482	18	20	18	19	19
DEU	0.505	0.488	0.494	0.484	0.490	19	18	19	20	18
LTU	0.487	0.431	0.472	0.474	0.427	20	22	20	21	22
BEL	0.485	0.465	0.471	0.501	0.447	21	19	21	18	20
JPN	0.472	0.421	0.460	0.452	0.429	22	24	22	23	21
LVA	0.466	0.367	0.414	0.423	0.365	23	26	24	24	25
CHL	0.454	0.447	0.458	0.469	0.419	24	21	23	22	23
ITA	0.419	0.426	0.412	0.405	0.397	25	23	25	25	24
SVN	0.369	0.395	0.368	0.363	0.358	26	25	26	26	26
ISR	0.315	0.246	0.294	0.322	0.269	27	29	27	27	27
POL	0.282	0.302	0.277	0.262	0.260	28	27	28	29	28
PRT	0.259	0.294	0.266	0.241	0.252	29	28	29	31	30
CZE	0.245	0.219	0.227	0.241	0.243	30	30	31	30	31
ISL	0.228	0.207	0.235	0.297	0.254	31	31	30	28	29
TUR	0.189	0.184	0.195	0.197	0.180	32	33	32	32	32
SVK	0.172	0.203	0.170	0.153	0.132	33	32	33	33	34
BGR	0.147	0.138	0.144	0.095	0.129	34	35	35	36	35
HRV	0.140	0.144	0.159	0.145	0.169	35	34	34	34	33
ROU	0.127	0.095	0.128	0.101	0.121	36	37	36	35	36
GRC	0.107	0.119	0.081	0.075	0.066	37	36	37	37	37
HUN	0.000	0.000	0.000	0.000	0.000	38	38	38	38	38
MAE	—	0.026	0.011	0.018	0.032
Table B.5 in the original PDF publication