Measures to prevent data falsification

A working paper that assesses ways to address possible data falsification in the European Social Survey (ESS) has been published on our website.

Several measures initiated by the Core Scientific Team (CST) are already in place to ensure the best possible interviewing is undertaken in all participating countries.

These include extensive pretesting of survey questions, guidelines for briefing interviewers, for selecting and recruiting respondents, and for monitoring fieldwork, analyses of the process of obtaining interviews and of the answering process during interviews.

The working paper explores how data falsification can occur and evaluates efforts to combat false data being collected during ESS fieldwork periods.

Data falsification is described by the American Association for Public Opinion Research (AAPOR) as the ‘intentional departure from the designed interviewer guidelines or instructions, which could result in the contamination of the data’.

The ESS Round 9 specifications prescribe that 10% of respondents and 5% of ineligible respondents should be contacted in back-checks conducted by the survey agency in each country.

The working paper confirms that the CST will strengthen the back-checks procedure ahead of Round 10 (2020/21) of the survey, to ensure that the process is more consistent across different countries.

Using only computer-assisted personal interviewing (CAPI) during Round 9 fieldwork allows the ESS to collect time stamps during the interview. This means that the speed at which the questionnaire is answered can give an indication of undesirable interviewer behaviour (speeding, skipping introductions) or data fabrication.

After fieldwork, analyses are also conducted to identify data quality issues – such as straightlining or odd patterns.

An analysis of partial duplicates is also conducted by the ESS Data Archive after fieldwork and, based on the evidence, all cases from interviewers who produced suspicious cases may be removed from the dataset (this has happened in three countries in Round 8).

Looking to the future, the paper suggests that audio recording, GPS tracking of interviewers and continued transparency from the ESS could all help prevent data falsification.

A new work package included in the ESS Work Programme 2019-21 aims to promote desirable interviewer behaviour, to minimise undesirable interviewer behaviour, to prevent falsification as much as possible and to detect it in an early stage.

The paper – Data falsification in the European Social Survey? – was written by ESS Deputy Director – Methodological, Dr. Ineke Stoop (SCP) and CST colleagues, Roberto Briceño-Rosas (GESIS), Achim Koch (GESIS) and Dr. Caroline Vandenplas (KU Leuven).