Appendix B: The Executive Opinion Survey: The Voice of the Business Community
For almost 40 years, the Executive Opinion Survey (the Survey) has been a key ingredient of the Global Competitiveness Report series. The Survey provides a yearly evaluation of critical aspects of competitiveness for which statistical data is missing because it is either impossible or extremely difficult to measure on a global scale. The aim of the Survey is to capture reality as best as possible, and business leaders are arguably the best positioned to assess these aspects.
The indicators derived from the Survey are used in the calculation of the Global Competitiveness Index 4.0 (GCI), as well as a number of other World Economic Forum indexes, such as the Networked Readiness Index, the Enabling Trade Index, the Travel & Tourism Competitiveness Index, the Gender Gap Index, and the Human Capital Index, as well as several other reports, including The Inclusive Economic Growth and Development Report, The Global Risks Report and a number of regional competitiveness studies. A truly unique source of data, the Survey has also long been used by a number of international and nongovernmental organizations, think tanks and academia for empirical and policy work.
The Survey 2018 in numbers
The 2018 edition captured the views of 16,658 business executives in 140 economies between January and April 2018. Following the data editing process described below, a total of 12,274 responses were retained. This year half of the retained surveys (50.7%) were completed online. In 52 economies over 90% of respondents complete the Survey online, while in a further 21 economies, at least 50% of respondents completed the Survey online (see Figure 1). The 2018 edition of the Survey was made available in 42 languages (see Table 1).
Survey structure, administration and methodology
The Survey comprises 148 questions divided into 15 sections. Most questions ask respondents to evaluate on a scale of 1 (considered among the worst in the world) to 7 (considered among the best in the world) specific aspects of the business environment in the country where the respondent operates. The 2018 edition of the Survey instrument is available in the Downloads section of the Global Competitiveness Report.
The administration of the Survey is supervised by the World Economic Forum and conducted at the national level by the Forum’s network of Partner Institutes. Partner Institutes are typically universities or other research organizations, business associations, competitiveness councils, or survey companies. These organizations have the private sector network for reaching out to leading business executives and a firm commitment to improving the competitiveness of their respective economies (for the full list, see the Contributors and Acknowledgments section of this report).1
In administering the Survey and in order to gather the strongest dataset, Partner Institutes are asked to follow detailed sampling guidelines and collect the data in a specific timeframe. The collection process is based on best practices in the field of Survey administration and on discussions with survey experts. It is put in place to ensure that the sample of respondents is the most representative possible and comparable across the globe.
The sampling guidelines specify that the Partner Institutes create a sample frame (Figure 2) that is a large list of potential business executives from companies of various sizes and from the various sectors of activity, as detailed below. The Partner Institutes separate the frame into two lists: one that includes only large firms, and a second that includes all other firms (both lists representing the various economic sectors). To reduce bias, Partner Institutes randomly select firms from each list to receive the Survey.
The sample frame should reflect the structure of the country/economy:
- In proportion to the share of GDP by sector: agriculture, manufacturing industry, non-manufacturing industry (mining and quarrying, electricity, gas and water supply, construction), and services.
- Ensuring the representation of both large- (more than 250 employees) and small-sized companies (249 employees or fewer), again reflecting each sector. At least one-third of companies are large and one-third are small, and the remaining one-third are determined by the structure of the economy in proportion to the share of GDP by company size.
- Ensuring the chosen companies also have a sufficiently wide geographical coverage.
The Survey is administered in a variety of formats, including face-to-face or telephone interviews with business executives, mailed paper forms and online surveys. For energy, time and cost considerations, the Forum encourages the use of a dedicated online Survey tool.
The Partner Institutes also play an active and essential role in disseminating the findings of The Global Competitiveness Report and other reports published by the World Economic Forum by holding press events and workshops to highlight the results at the national level to the business community, the public sector and other stakeholders.
Data treatment and score computation
This section details the process whereby individual responses are edited and aggregated in order to produce the Survey question scores of each economy. These are the results that then feed into the GCI other indices and projects listed above.
Data editing
Prior to aggregation, the respondent-level data are subjected to a careful editing process. The following observations are excluded from the dataset: Surveys where the respondent gives the same answer to at least 80% of the questions; Surveys with a completion rate inferior to 50%; respondents who are not based in the same country as the Partner Institute; respondents who are not in a leadership position within their firm; and finally, duplicate Surveys—which can occur, for example, when a Survey is both completed online and mailed in.
In a second step, a multivariate test is applied to the data using the Mahalanobis distance method. This test estimates the probability that an individual Survey in a specific country “belongs” to the sample of that country by comparing the pattern of answers of that Survey against the average pattern of answers in the country sample.
A univariate outlier test is then applied at the country level for each question of each survey. We use the standardized score—or “z-score”—method, which indicates by how many standard deviations any one individual answer deviates from the mean of the country sample. Individual answers with a standardized score greater than 3 are dropped.2
Aggregation and computation of country averages
We use a simple average to compute scores at the economy level. As the sample frame aims to replicate an economy’s sectoral composition and includes companies of different sizes, the country-level score of each Executive Opinion Survey question is the arithmetic mean of all answers in each country. That is, for a given question, all individual answers carry the same weight.
Formally, the average of a Survey indicator i for country c, qi,c , is computed as follows:
where
qi,c,j is the answer to question i in country c from respondent j; and
Ni,c is the number of respondents to question i in country c.
Once responses have been aggregated at the country level, a test to detect statistical outliers is run. We leverage the strong relationship between the indicators derived from the Survey and some 50 statistical indicators included in the GCI: countries doing well on these indicators tend to do well in the Survey. A univariate linear regression is used to predict the expected average score of Survey indicators based on the average performance in the other indicators. Average Survey scores that lie outside the 90% confidence interval around the predicted values are considered “outliers”. The scores of individual Survey indicators are systematically corrected by a factor corresponding to the distance between the observed average Survey score and the predicted Survey average at the limit of the confidence interval.
In addition, an analysis to assess the reliability and consistency of the Survey data over time is carried out. As part of this analysis, an inter-quartile range (IQR) test is performed to identify large swings—positive and negative— between two editions. For each country, we compute the year-on-year difference, d, in the average score of a core set of 66 Survey questions. We then compute the inter-quartile range (i.e. the difference between the 25th percentile and the 75th percentile). Any value d outside the range bounded by the 25th percentile minus 1.5 times the IQR and the 75th percentile plus 1.5 times the IQR is identified as a potential “outlier”. This test is complemented by a series of additional empirical tests, including an analysis of five-year trends and a comparison of changes in the Survey results with changes in other indicators capturing similar concepts. We interview local experts and consider the latest developments in a country in order to assess the plausibility of the Survey results. Based on the result of this test and the complementary qualitative analysis, the data collected in 2018 for Azerbaijan, Bahrain, Burundi, China, Ethiopia and Guinea were not used. Instead, Survey results from the previous editions were used (for details see Table 2).
Moving average and computation of country scores
We then proceed to compute moving averages of country scores. The moving average technique consists of taking a weighted average of the most recent year’s Survey results, together with a discounted average of the previous year. There are several reasons for doing this. First, it makes results less sensitive to the specific point in time when the Survey is administered. Second, it increases the amount of available information by providing a larger sample size. Additionally, because the Survey is carried out during the first quarter of the year, the average of the responses in the first quarter of 2017 and the first quarter of 2018 better aligns the Survey data with many of the data indicators from sources other than the Survey, which are often annual-averages data.
To calculate the moving average, we use a weighting scheme composed of two overlapping elements. We place more weight on the year with the larger sample size to attribute equal weight to each response. At the same time, we attribute greater weight to the most recent sample because it contains most up to date information. That is, we also “discount the past.” Table 2 reports the exact weights used in the computation of the scores of each country.
Economy score calculation
The details of the method applied to compute the country scores for the vast majority of economies included in The Global Competitiveness Report 2018 are as follows.
In Equation (3), the first component of the weighting scheme is the discounted-past weighted average. The second component is the sample-size weighted average. The two components are given half-weight each. One additional characteristic of this approach is that it prevents a country sample that is much larger in one year from overwhelming the smaller sample from the other year.
In the case of Survey questions that were introduced in 2018 for which, by definition, no past data exist, full weight is given to the 2018 score. For newly covered economies, this treatment is applied to all questions. For countries whose 2018 data were discarded, the results from the previous editions of the report are used instead. Box 1 provides a clarifying example of the methodology.
Box 1: Example of score computation
For this example, we compute the score of Denmark for the indicator Hiring and firing practices, which is included in the Global Competitiveness Index (indicator 8.02) and derived from the following Survey question: “In your country, to what extent do regulations allow for the flexible hiring and firing of workers? (1 = not at all, 7 = to a great extent).” This question is not a new Survey question and therefore the normal treatment applies, using Equation (1). Denmark’s Survey score was 4.93 in 2017 and 5.15 in 2018. The weighting scheme described above indicates how the two scores are combined. In Denmark, the size of the sample was 63 in 2017 and 85 in 2018. Using a = 0.6 and applying Equations (2a) and (2b) yields weights of 48.7% for 2017 and 51.3% for 2018 (see Table 2). The final country score for this question is therefore:
This is the final score used in the computation of the GCI. Although numbers are rounded to two decimal places in this example and to one decimal place in the Denmark country profile, exact figures are used in all calculations.
References
- Browne, Ciara and Thierry Geiger, “The Executive Opinion Survey: Capturing the Views of the Business Community”, The Global Competitiveness Report 2009–2010, World Economic Forum, 2009.
- Chandra, Prasanta, “On the generalised distance in statistics”, Proceedings of the National Institute of Sciences of India, vol. 2, no. 1, 1936, pp. 49–55, https://insa.nic.in/writereaddata/UpLoadedFiles/PINSA/Vol02_1936_1_Art05.pdf, retrieved 27 September 2016.