An annual mixed-mode survey to measure victimisation in France from 2022: The chronicle of an overhaul in 5 acts

1.Introduction

The “Cadre de vie et sécurité” (CVS) survey was conducted annually between 2007 and 2021 under the joint management of INSEE and the French National Observatory of Crime and Criminal Justice (ONDRP) until its disbandment in 2020 and the Ministerial Statistical Department for Internal Security (SSMSI, created in 2014). Known as the “victimisation” survey, it aims to identify the crimes that households and individuals may have fallen victim to during the two years preceding the survey, regardless of whether or not those crimes were reported to the police or gendarmerie. The information gathered through the victimisation survey is separate from and complements the data recorded by the national police and gendarmerie, since victims do not always file reports. When combined, this information and data offer valuable tools to assess and analyse both crime and feelings of insecurity. The CVS survey involved an average of 15,000 households each year with face-to-face interviews. As is the case for other Official Statistics surveys, the CVS survey has evolved over the years, seeing changes to the questions, the addition or removal of thematic modules, occasional territorial extensions to include the overseas territories and even methodological revisions.

In April 2018, INSEE announced to the SSMSI and ONDRP that, due to severe budgetary constraints, it would no longer be able to conduct the CVS survey in its current form (conducted annually, face-to-face collection) with effect from 2022. The Ministry of the Interior is committed to providing the financial and human resources required in order to guarantee, through the SSMSI, the sustainability of a victimisation survey scheme.11 Therefore, in accordance with the commitments made before the National Council for Statistical Information (CNIS), in spring 2019, with the support of INSEE, the SSMSI began considering the changes that could be made to the overall scheme of the CVS survey by 2022, exploring in particular the possibilities offered by combining the various modes of collection (Internet, telephone, paper, face-to-face, etc.) being developed within the Official Statistical Service.

The aim of this article is to present, in five acts, the broad outlines of the project that led to the design of the new “Vécu et ressenti en matière de sécurité” survey, which was conducted for the first time in 2022.

2.Act I: Take stock of the existing situation and draw up a roadmap for the new scheme

3.Act II: Focusing on an innovative protocol against a backdrop of prolific growth of mixed-mode collection

3.1Mixed-mode collection or the sense of history

The increase in the use of mixed-mode collection for INSEE’s household surveys has become a necessity that has been incorporated into INSEE’s 2025 strategic areas,55 as well as those of the main foreign national statistical institutes. In the context of the total survey error paradigm [11], the use of a mixed-mode scheme, i.e. one that combines different collection methods (Internet, paper, telephone, face-to-face) can be considered for a number of reasons: to improve coverage, to encourage participation in the survey and therefore to reduce the total non-response error or even to reduce measurement error. The development of these schemes is also motivated by cost control. Collection methods could, for example, be combined with the aim of improving the quality of the survey at constant cost or to reduce costs without any adverse effect on quality [12]. These various dimensions must therefore be included in the analysis when seeking out the relevant combination of the various collection methods. The sequential proposal of the various collection methods typically starts with the cheapest collection method and works towards the most expensive.

All of the household surveys conducted by the SSP are involved in the transition to mixed-mode collection. The “Entrée dans la vie adulte” (Entry Into Adult Life, EVA) and “Technologies de l’information et de la communication” (Information and Communication Technologies, ICT) surveys and the “enquête sur les salariés de l’État” (Survey of Government Employees, FPE) are already based on mixed collection methods, including a web component. The “enquête annuelle de recensement” (annual census survey, EAR) has been offering Internet collection since 2015. Since the beginning of 2020, it has been the iconic Labor force survey (LFS) that has partially switched, as part of a more general overhaul, to mixed-mode collection (Internet collection for repeat interviews [13, 14]). Finally, by 2023, other surveys will have conducted pilots with a view to implementing the switch to mixed-mode collection (“enquête Logement” (Housing survey), “enquête Emploi du temps” (Time Use survey), etc.).

Taking a similar approach to other national statistical institutes, INSEE has developed a programme aimed at pooling the developments required for the design of the collection media. Working groups have been in place since 2017 with the aim of providing practical solutions for the design of collection media suitable for online and self-administered collection methods and for measuring the adjustment of collection method effects. Finally, an electronic newsletter informs those involved at the Official Statistical Service of the latest developments with regard to mixed-mode collection (Dépêche Mode newsletter).

Figure 2.

General structure of the new victimisation survey scheme. Definitions: CAWI (Computer-assisted web interviewing) refers to online responses; CATI (Computer-assisted telephone interviewing) refers to telephone responses; NR: non-respondents. Reading note: 30,000 NR as at D40 means that 30,000 individuals who had not responded to the survey after six weeks (D40) were selected at random from among the non-respondents with a mobile or landline telephone number to conduct a telephone interview.

It was in this fertile environment that the SSMSI prepared to overhaul the CVS survey protocol with the aim of establishing innovative mixed-mode collection. Generally speaking, when designing a quality victimisation survey, the statistician is faced with two key difficulties upstream of the design of the questionnaire. On the one hand, they need to manage the low frequency of the crimes being measured – in particular the most serious violence, such as rape – which requires large sample sizes. On the other hand, they must take account of the “sensitive” nature of certain types of victimisation (sexual and domestic violence), which makes it especially difficult to weigh up the advantages and disadvantages of the various collection methods.

Too small a sample leads to estimates for which the relative accuracy is mediocre and limits the possibilities of describing the phenomena studied. The EPCV (which relied on around 6,500 respondent households and a total of 11,000 to 12,000 individuals on average) has often been criticised for this, as has the CVS survey, as we have previously discussed.

The introduction of cheaper and more modern collection methods than interviewing respondents face-to-face is a good way to increase the sample size and improve the accuracy of estimates. Nevertheless, these protocol changes raise the question of the degree of confidentiality and security offered to respondents revealing sensitive information, as well as the continuity of historic series.

4.Act III: Experimentation and evaluation to ensure a reliable switch to mixed-mode collection and preparation to link the series

The mass use of self-administered online questionnaires poses a number of methodological questions when the usual and validated paradigm involves collection in the presence of an interviewer. In particular, a key question addresses the collection method effects, particularly measurement biases. In other words, how similar are the answers given by a respondent to a question asked via two different collection methods [15, 16, 17, 12]?

Therefore, before rolling out mixed-mode collection to household surveys, with the Internet as the preferred collection method, INSEE first conducted a vast array of experiments in the early 2010s. At the same time, the use of this new collection method for the census annual surveys (EAR) and the “enquêtes entreprises” (company surveys) has grown considerably. Since the findings in the literature are not always easy to apply more generally, the direction INSEE has chosen for the establishment of mixed-mode collection for household surveys is to focus on defining, for each survey, the best possible use of the Internet.

As regards the victimisation surveys, four trials were conducted between 2013 and 2021. Their conclusions, including those of the two following trials, contributed to the debate concerning the switch of the CVS survey to mixed-mode collection.

The “CVS mixed-mode panel” trial, which involved conducting repeat interviews of individuals who responded to the 2018 CVS survey in 2019 online and by telephone using a simplified questionnaire (10,271 questionnaires were completed, 5,490 of which were conducted by telephone and 4,781 online), was the subject of a paper presented at the 11th International Francophone Conference on Surveys [18]. It gave rise to a number of interesting conclusions:

The preliminary results of the “CVS mixed-mode panel” trial, which looked at panelization, appear to indicate that its interest is not proven: little selection was observed on the variables collected during the first interview and, during the trial, the attrition of rare profiles of victims of sensitive violence was higher than for the other profiles.

Given its high demand on the subject of violence against women, in November 2019, the SSMSI applied for and received EU funding (EUR 1.5 million) to conduct the GENESE survey on gender-based violence at national level in 2021 based on the Gender-Based Violence questionnaire developed by Eurostat.

The GENESE survey was designed to meet two key ambitions of the SSMSI: i) to inform the public debate at European level by providing new data on patterns of gender-based and sexual violence, and; ii) to conduct a full-scale mixed-mode trial to measure victimisation and perceptions of safety according to the two-phase protocol designed by the SSMSI in collaboration with INSEE experts (see Fig. 2). In January 2021, it obtained the mandatory general interest and statistical quality label from CNIS.

The first phase of the survey took place over a period of 11 weeks from 1 March to 16 May 2021 and targeted 169,060 individuals aged between 18 and 74 residing in metropolitan France. In total, almost 109,000 individuals (64%) responded, with 51% responding online, 3% by telephone and 10% on paper. The follow-up survey (phase 2), based on the questionnaire designed by Eurostat, was conducted among 15,000 phase 1 respondents, among whom victims of gender-based and sexual violence were over-represented. More than 10,000 people (68%) responded: 53% online and 15% by telephone.

Finally, at the end of phase 1, a face-to-face methodological component took place between 1 June and 24 July 2021 involving more than 3,000 people: the 1,000 total non-respondents in Gironde plus a random selection of 2,000 total non-respondents in Île-de-France. More than 1,000 completed questionnaires were collected (32%).

The conclusions of the studies looking into the effectiveness of the GENESE survey protocol [19] have allowed for the overall validation of the survey scheme designed by the SSMSI in collaboration with INSEE experts and for this protocol to be kept as a frame of reference for the new “Vécu et Ressenti en matière de Sécurité” survey (survey on experiences and feelings about safety, VRS survey). The decisions made by the designers with regard to the nature and frequency of follow-ups has enabled the achievement of high response rates, particularly online. The repeated sending of emails appears to be a key lever for participation, particularly at the start of collection. The introduction of telephone follow-ups also helps to encourage participation, not only by telephone, but also online, which militates for deferred rather than “pure” sequential competitive protocols. In terms of representativeness, telephone collection has resulted in significant gains with regard to profiles unlikely to respond at the start of collection, particularly among socially disadvantaged populations. Finally, the original link between phase 1 and phase 2 proved to be relevant: the attrition of rare profiles was particularly well managed thanks to the smoothed reminders and the sequencing of questionnaires, which was widely acclaimed by eligible individuals. In accordance with the objectives of the survey, the data collected in phase 1 allow for the production of more precise and localised prevalence indicators.66 In addition, the targeted selection of the phase 2 sample allows unprecedented insight into gender-based and sexual violence: 1,200 female victims of physical or sexual violence during the reference period responded to the Eurostat questionnaire compared with an average of 200 for an annual edition of the CVS survey. The initial findings were published in autumn 2022 in the form of a general overview of violence, paving the way for many more in-depth studies. This publication describes three types of violence: violence suffered in childhood, violence committed “by a partner” within couples and violence committed by somebody other than a partner [20] (see Table 1). Beyond measuring the number and proportion of men and women who have fallen victim to these types of violence during their life or more recently, the experiences of victims throughout their life is explored according to their gender with a view to precisely characterising the violence suffered (psychological, physical or sexual) and to provide certain information regarding the characteristics of the victims and the perpetrators.

As has already been mentioned above, the GENESE survey scheme was designed in advance in order to offer a framework that is, in theory, relatively close to random experimentation [21], allowing for the comparison of responses received online and by telephone. Therefore, people who responded online who are eligible for telephone follow-up (40 days after the start of collection) but who are not selected (by random draw) are compared with telephone respondents (eligible and selected). The idea is to have two sub-populations that are as comparable as possible, including from the point of view of their non-observable characteristics: the collection method effect purged of selection differences on the basis of observable characteristics can then be considered a collection method effect that is intrinsically linked to the measurement.

In order to compare the responses of these two populations to the main variables of interest present in the CVS survey and intended to be carried over into the new victimisation survey, two selection control methods were implemented: regression models and matching models [22]. The observation made on the basis of these initial analyses is rather reassuring: the collection method effects linked to the measurement are generally limited when it comes to the victimisation indicators [23]. Therefore, of all of the indicators tested, only two present statistically significant effects that withstand both control methods selected: acts of vandalism (under-reported via telephone when compared with online) and physical violence (over-reported via telephone). The common feature of these indicators is the relative ambiguity of the wording of the questions, which points to the possibility of improving the wording in order to limit these effects. However, the collection method effects are more pronounced for questions concerning opinions of security. These findings, which are consistent with the literature, are linked to social desirability: the social interactions with the interviewer bring about a degree of conformity with normal expectations. As a result, responses received by telephone allude to greater satisfaction with the actions of the police/gendarmes and the judicial system, while feelings of insecurity are less likely to be declared. The collection method effects are also significant for some of the questions relating to the observation of crime, but are less pronounced.

5.Act IV: Consultation for the design of questionnaires to respond to social demand

Figure 3.

Content of the “core” (phase 1) questionnaire of the 2022 Vécu et ressenti en matière de sécurité survey.

The SSMSI organised the official meeting to launch the consultation for the overhaul of the victimisation survey scheme on 14 December 2020. Attendees included INSEE, Ministerial Statistical Offices, government bodies, research centres and trade union representatives, professional organisations and local assemblies and CNIS members. During this meeting, the SSMSI presented the conclusions of the consultations carried out, together with the committees associated with the project, which is based around a steering committee tasked with ensuring compliance with the objectives, monitoring the progress of the design phases and preparing and conducting the survey, and a consultation committee tasked with collaborating with other stakeholders to design the statistical methodology, the coverage of the population of interest, the coverage of the crimes, the questionnaires and the documents linked to the survey collections. In order to ensure the quality and the scope of the studies carried out by the SSMSI, it also appeared important to provide the project to overhaul the survey with a multi-disciplinary scientific council in order to scientifically evaluate the studies carried out and to decide how the data produced will be used.

In order to determine the topic to be explored in the follow-up survey (phase 2) in 2022, a call for proposals was launched by the SSMSI within the consultation committee. The only topic put forward concerned the relationship between public security services and the population. Therefore, the questionnaire for the 2022 VRS follow-up survey provides for the collection of detailed information regarding feedback, provided by individuals who have had interactions with the police and gendarmerie services, regarding the expectations and satisfaction of the general population with regard to digital procedures and looks deeper into the opinions of the respondents with regard to the actions of the police and gendarmerie services in their neighbourhood/village.

In accordance with the provisional schedule, within the scope of the consultation, the SSMSI designed test questionnaires for the core survey and the thematic survey and organised:

Figure 4.

Content of the “thematic” (phase 2) questionnaire of the 2022 Vécu et ressenti en matière de sécurité survey.

Upon the completion of these tests and based on feedback from the scientific council and the consultation committee, the SSMSI drew up the final survey questionnaires (see Figs 3 and 4). The VRS 2022 survey provided coverage for the population aged 18 and over residing in France, excluding Mayotte and French Guiana. The sample of 200,000 individuals was selected by INSEE from the tax source (Fidéli) sampling frame.

In December 2021, the SSMSI submitted the application for the mandatory general interest and statistical quality label to the Label Committee. The Committee issued its positive opinion in January 2022.

The first phase of the survey took place over a period of 11 weeks from 26 February to 15 May 2022 and involved 200,622 individuals aged 18 and over residing in metropolitan France, Martinique, Guadeloupe and La Réunion. In total, around 107,000 individuals (53%) responded, 42% online, 2% by telephone and 9% on paper. The follow-up survey (phase 2) was conducted between 26 February and 17 July 2022 and involved 20,000 of the respondents from phase 1 with over-representation of those who had been in contact with the public security services. More than 13,000 individuals (66%) responded: 48% online and 18% by telephone. When compared with the GENESE survey, on the comparable field of persons aged 18–74 residing in metropolitan France, the collection rate recorded for the VRS 2022 is 9 points lower for phase 1 and very similar for phase 2. The longer stated duration of the phase 1 questionnaire of 30 minutes in the notice letter for the VRS survey, compared with 20 minutes for that of the GENESE survey, is the main reason for this lower participation rate. The context may also have had a significant impact: the GENESE survey took place in the middle of a regionalised semi-lockdown period, whereas the VRS survey was conducted post-lockdown in the middle of the electoral period. This may have had an impact on the availability of respondents and, potentially, on the participation of certain respondents due to the high media profile of security-related issues, as well as on the longer delivery times for post and the more complex monitoring of its distribution.

6.Act V: Consolidation of the survey and ensuring the dissemination of data

The second edition of the annual VRS survey will take place during Q2 2023. Although the bulk of the work for the overhaul project therefore appears to have been completed, the second phase, which is just as ambitious and complex, is just beginning: continue to lead the consultation and address outstanding issues (inclusion of minors, other overseas departments and regions, translations of questionnaires, etc.); design subsequent editions of the thematic questionnaires exploring new security-related topics and ensure the dissemination of data.

The committees in place for the first edition of the VRS survey are retained with feedback of the work carried out by the working groups during two plenary sessions per year: i) a spring meeting, the primary objective of which is to present an initial version of the thematic questionnaire for year N, as well as to provide an update on the topics currently being investigated for future VRS surveys and in particular the selection of the topic for year N+ 1; ii) an autumn meeting, during which the final versions of the core and thematic surveys for year N are put forward for approval with a view to their submission to the Label Committee.

If, during the second edition of the VRS, the coverage of the survey remains unchanged, the SSMSI looks into the possibilities of extending the coverage to minors and to other territories (French Guiana and Mayotte) that are not currently surveyed. Work will commence in 2023, once the study into the associated legal constraints and the necessary adjustments with regard to the questionnaires and the collection protocol has begun.

To ensure communication within the scope of the consultation, the SSMSI provides a documentary area dedicated to the VRS survey on Osmose, a collaborative platform for the State’s professional communities. All of the documentation associated with the survey, as well as the schedule of events, is shared on this platform, which has more than a hundred registered members.

In 2023 and for future editions, the pilot – a kind of mini general rehearsal for the survey – conducted in 2022 and during the first edition of the VRS is being replaced with a qualitative test. It is based on a hundred semi-structured face-to-face or telephone interviews with panellists. Its primary objective is to test the understanding of the thematic questionnaire, which is different for each edition. Since the panellists are also interviewed with regard to the core questionnaire (phase 1), the test also allows the difficulties that may be encountered with this questionnaire to be updated and contributes to its improvement.

Finally, the methodological work is prioritised to ensure that the initial publication of the results of the VRS 2022 survey takes place as quickly as possible, along with the dissemination of a database that can be used by researchers. In the summer of 2022, the SSMSI therefore commenced data reconciliation and adjustment work (calculation of weights and precisions at departmental level). A processing chain has been set up and will be consolidated in future editions.

7.Epilogue or end of part 1

In 2022, the SSMSI conducted the first edition of the “Vécu et ressenti en matière de sécurité” (VRS) among 200,000 individuals in France (excluding Mayotte and French Guiana). The main objective of this mandatory victimization survey is to provide prevalence of a wide range of offenses (burglary, theft, fraud, physical and sexual violence, harassment, threats, insults, discrimination, corruption), the corresponding complaint rates, as well as indicators of perception and opinion regarding security. This new survey follows on the “Cadre de vie et sécurité” (CVS) survey carried out annually by INSEE between 2007 and 2021. To carry out the overhaul project from 2019 to 2022, the SSMSI completed the 5 following main stages to ensure the switch from CVS to VRS: take stock of the existing system to develop a roadmap, design an innovative protocol in a context of multimode development, experiment and evaluate to make mixed-mode switchover and series connection more reliable, establish a framework for consultation to structure the design work and consolidate the survey and ensure data dissemination.

The second edition of the annual “Vécu et ressenti en matière de sécurité” survey will be conducted during the first semester of 2023. The methodological evaluation still needs to continue, looking in particular at the thorny issue of the linking of the series from the old and new schemes. Indeed, temporal comparison over a long period is the key interest of a survey repeated over time. As part of a major overhaul, this question of comparability over time finds itself at the crossroads of a number of issues. Should we focus on the “right” measurement for a given period or comparability with the past? Should we guard against comparability issues in the future?

To accompany the overhaul of the CVS survey, it will be crucial for the SSMSI to document, understand, explain and interpret the possible breaks observed in the indicators historically monitored over time in advance of the work aimed at potentially linking the series. Concerning the estimates of victimisation and opinions with regard to security, the first results of the VRS 2022 survey will be disseminated in the last half of 2023.