Implementation of online questionnaires in the General Household Survey in Hong Kong

1.Introduction

The General Household Survey (GHS) of Hong Kong is a monthly household survey on the characteristics of the labour force and household income. In each month, a random sample of about 9,000 households is enumerated, of which about half are new cases sampled for the first time and half are repeated cases that had been sampled 3 months ago. In the past, data collection for the GHS dealt with repeated cases without telephone numbers (including cases which could not be enumerated and cases which were enumerated but the respondents concerned were unwilling to provide telephone numbers in the first round) and new cases (all without telephone numbers) via face-to-face interviews through field visits, while repeated cases with telephone numbers were enumerated through computer-assisted telephone interviews (CATI).

With the wider penetration and advancement of information technology, adopting online questionnaires (OQ) in statistical surveys has become more feasible in recent years. The successful experience of the Hong Kong 2016 Population By-census in adopting an electronic questionnaire demonstrated that society was ready to respond to statistical surveys via the online mode. In addition, households were increasingly difficult to reach by field visits as people were returning home later from their daily activities and many preferred not to be disturbed at home. Tighter security control in many residential buildings also increased the cost and difficulty in reaching households by field visits.

These challenges prompted the Census and Statistics Department (C&SD) to consider introducing OQ as a new self-reporting channel in the GHS to cater for the living styles and preferences of modern households. The C&SD started to introduce the OQ on a trial basis in July 2017 as an additional mode of data collection in the GHS, on top of its existing channels via face-to-face interviews and CATI. This paper summarises the implementation experience of the OQ as a new data collection mode in a labour force household survey, with Section 2 discussing the review of literatures and precedent relevant activities undertaken prior to the implementation, and Section 3 presenting our approach to implementation. Section 4 describes the methodology of evaluation of the trial in terms of levels of adoption of the OQ and data quality. Section 5 presents the evaluation results and Section 6 provides conclusions and recommendations.

2.Literature review

There are many literatures discussing methodologies and precedent cases of OQ implementation in National Statistical Offices (NSOs), particularly in labour force surveys. In the course of planning for the OQ implementation in the GHS, desktop research had been conducted and references had been drawn from the experiences of other NSOs on the stages of data collection methodology and design, system and questionnaire design, and implementation approach.

As there are already two data collection modes in place in the GHS, a sequential mixed-mode design was adopted by the C&SD in which the OQ is provided to respondents as the first option introduced in the invitation letter that is mailed to the sampled households. If they do not respond to the OQ by a certain date, then the field officers will follow up with field visits or CATI (for repeated cases with telephone number available). Such design has been widely studied and adopted. Bethlehem and Bifflgnandi explained that mixed-mode surveys are increasingly common to reduce survey costs and maintain response rates [1]. Dillman et al. noted that offering multiple data collection modes can attract different types of respondents to respond, which in turn can improve timeliness, reduce coverage errors, improve response rates and reduce nonresponse errors [2]. While some previous case studies and literature noted that response rate may not go up with the introduction of an online mode in a mixed-mode design, other cases suggested that a sequential mixed-mode design could perform better and bring improvement to the response rate [1, 3, 4, 5].

In the GHS, such sequential mixed-mode design fits the C&SD’s goal of increasing the likelihood of receiving responses from those households that could not be normally reached or interviewed by field visits, hence reducing coverage error and nonresponse error. In addition, starting off with OQ as the first data collection option which is relatively less costly than face-to-face interviews, data collection costs can be reduced with increased participation from sampled households, in line with other preceding cases [4, 6]. Also, studies have suggested that allowing the respondents to choose the mode of responding could bring the benefits of higher data quality with less rounded numerical responses and higher completion [7].

References can also be drawn to the system and questionnaire design of similar implementations in other NSOs that aimed to ensure that the computer system interface is user-friendly and the question wordings and instructions are suitably tuned to the perspectives of the respondents. With the experiences from NSOs such as the United Kingdom Office for National Statistics, Statistics Canada, the Federal Statistical Office of Germany and Statistics Netherlands, and examples in literatures, the C&SD paid particular attention to the visual design, questionnaire flow, error checks and ease of navigation of the online questionnaire system [8, 9, 10, 11, 12]. In particular, in order not to overburden the respondents to the extent that they would drop out in the middle of completing the OQ, only essential routing and edit checks that are less sophisticated than interviewer-administered modes are included in the OQ for checking on the spot [3, 13]. After receiving the completed OQ, further rounds of validation checks are conducted at the office and field officers will follow up on those cases that do not pass the checks.

Another important aspect to consider in adopting mixed-mode design is the mode effect. Broadly speaking, mode effect refers to the differences in results compiled from data collected via different modes in a survey [10]. Mode effects can affect various types of errors, including coverage error, nonresponse error and measurement error [6]. These different types of errors tend to be confounded with one another, since the impact of sampling, questionnaire construction and implementation are difficult to disentangle from measurement errors [3, 4, 13, 14]. In order to accurately measure the “pure” mode effects on measurement errors with other confounding factors removed, dedicated experiments are required [4, 10, 14, 15]. Tourangeau et al. noted that in case such experiments cannot be conducted, indirect measures can be used to evaluate data quality in different modes, such as the missing data and breakoff rates [10]. De Leeuw et al. also noted that matching could be an option if no separate experiments could be set up to measure mode effects [4]. Specifically, matched subjects based on variables independent of mode can be studied to see if the results collected from different modes are different. Although this is less robust than randomised experiments, it can still provide insights regarding potential mode effects.

In planning for testing and implementation, it was noted that many NSOs put in much time to test out the system, questionnaire, and workflows. For instance, in United Kingdom Office for National Statistics, a limited-scale testing was conducted in 2015, and further testing was conducted in 2018 before the planned live-run in 2020 [8, 16]. Statistics Canada also phased out the roll-out of their online electronic questionnaire in LFS by one rotational group at a time, and completed the roll-out to all groups in 6 months [17]. It seems prudent to adopt a longer implementation time frame to allow time for testing, fixing any system or questionnaire problems, and examine the data quality before full implementation.

3.Implementation approach

The GHS is a monthly survey. Statistics of moving 3-month periods compiled from the GHS, including size of labour force, unemployment, underemployment, and household income, are vital statistical indicators of economic conditions and are published monthly. It is thus imperative to ensure that data collection remains uninterrupted such that these data series will continue to be available as scheduled without affecting its continuity. On the other hand, adding a new data collection mode to existing ones is operationally and technically complex. If a full-blown implementation approach was adopted, it would take a long time to get all testing completed, operational procedures well planned and all staff trained up at the same time, and it would be prone to service disruption when unexpected problems with the new system or operations occurred. In particular, in view of the short data collection period of the survey, any hiccups in the new operations might pose high risks to the timely dissemination of the survey results.

In view of the above considerations and making references to the implementation experiences in other NSOs, it was decided to roll out the OQ on a trial basis first, starting with a small number of cases and then gradually increasing the scale of the trial month by month. The trial was divided into 2 stages. The first stage of trial implementation focused on testing the adequacy of the OQ computer systems and operational arrangements, the effectiveness of the OQ questionnaire design, and the receptiveness of GHS respondents. The second stage involved a gradual rollout of the OQ to representative sub-samples of the GHS sampled cases of increasing size month by month, focusing on studying the take-up rate and impacts on key survey estimates and manpower resources requirement.

The first stage of trial implementation lasted from July 2017 to April 2018 for 10 months. It began with only 81 cases (less than 1% of all sampled cases in a month) invited to use the OQ in the first month of trial; more cases were invited in each subsequent month, reaching 364 cases in the 10th month (about 4% of all sampled cases in a month). The selection of cases was mainly based on the ability of field officers in handling the new operational procedures involving the OQ. Only cases assigned to well-trained field officers were selected at first, while training was extended to the rest of the field officers progressively. A small scale in the first stage also allowed prompt support to be provided to fix any technical or operational problems encountered. Based on the experience and feedback received from both field officers and respondents, operational procedures and system functionalities were refined, such as simplifying the OQ activation page, modifying the question flow, and deriving some data fields automatically based on answers to other questions rather than asking directly for the data.

The second stage of trial implementation commenced in May 2018 and lasted for 8 months up to December 2018. The monthly sample of the GHS consists of 18 replicates of quarters (housing units), each being a representative sample of quarters in Hong Kong. In the first month of the second stage of trial implementation, one replicate was invited to participate in the GHS via the OQ. Introducing the OQ to one out of 18 replicates allowed expanding OQ adoption in a controlled manner. In case something went wrong, the remaining 17 replicates could still be used to compile representative survey results. This replicate-by-replicate approach also allowed comparison of the OQ replicates and non-OQ replicates to understand the impacts of introducing the OQ on key survey estimates. The number of replicates invited to complete the OQ then gradually increased every month, until all 18 replicates were invited in the last month of trial implementation in December 2018 (Fig. 1).

Figure 1.

Number of GHS sampled cases invited to use OQ during trial implementation (July 2017–December 2018).

It is important to ensure that the data collected via this new data collection mode is of as high quality as those collected via the usual modes (i.e. face-to-face interview and CATI). During the first stage of trial implementation, all OQ cases were selected for quality checking in order to fully assess their data quality by verifying with the respondents the key information (such as economic activity status) provided on the OQ, and to identify potential areas for improvement in the computer system. Towards the second stage of trial implementation, a sampling plan was formulated to select a certain proportion of the OQ cases for quality checking. The plan took into consideration which data items in the submitted OQs are subject to higher risk of reporting errors and thus which categories of the submitted OQs should be subject to a higher rate of checking, response burden of OQ respondents as well as time and manpower required for the checking.

4.Methodology of evaluation of trial

The evaluation of trial implementation involved both qualitative evaluation with descriptive measures, and quantitative assessment on take-up and data quality. The questionnaire, system design and operational flows were mainly tested and evaluated qualitatively based on the feedback of field officers and respondents who had participated in the trial. The trial began with only a handful of respondents invited to use the OQ, and their comments provided valuable feedback for the team to review and fine-tune the questionnaire and workflows. The number and contents of all enquiries were duly monitored throughout the trial period as a measure of system performance and ease of use from respondents’ perspective.

Figure 2.

Number of OQ-related enquiries as a percentage of cases invited to use OQ during trial implementation in GHS (October 2017–December 2018).

The level of take-up can be measured and analysed by looking at the number of enumerated cases using the OQ out of the number of households invited. Since the sampling scheme of the GHS is based on replicates (each being an independent and representative sub-sample) and the OQ was introduced gradually by replicates in the trial period, replicates that were invited to fill out the OQ (OQ replicates) were compared with those not invited to fill out the OQ (non-OQ replicates). The demographic profiles of respondents in the OQ and non-OQ replicates can be compared by chi-square tests to study if they differ significantly. The mode effect (in the sense of measurement errors and nonresponse error) between the existing data collection channels and the newly introduced OQ would be difficult to measure precisely without setting up experiments to collect additional data. Due to resources and operational considerations, no embedded experiments were set up in the trial for testing the mode effect. A method similar to the “matching” method proposed by de Leeuw et al. and similar testing done in Statistics Canada was used instead, in which the key statistics compiled from the OQ replicates and non-OQ replicates were compared to assess if the introduction of this new data collection mode have any impact on the key statistics [4, 15].

5.Results

5.1Error rate and enquiries

The error rate in the OQ data, as measured by the number of OQ cases with errors identified in quality checks over the total number of cases responded by the OQ (including partially responded cases), was in general low (< 5%) and were largely attributed to misinterpretation of the survey concepts involved and were later remedied by fine-tuning the wording of questions and explanatory notes in the OQ.

Enquiries received during the trial implementation period also provided valuable information on users’ experiences. Throughout the trial implementation, the number of OQ-related enquiries (as a percentage of cases invited to use OQ) had been dropping to a very low level as seen in Fig. 2. The mild increase in May and June 2018 as the trial entered the second stage was due to system problems, which had been quickly rectified. Most of the enquiries were related to account and login questions (39%) and problems in completing the questionnaire (33%). Detailed comments were requested from these enquirers, so that enhancements could be made promptly on the questions, system functionalities and performance.

Figure 3.

OQ take-up rate in GHS during trial implementation and after full implementation (July 2017–June 2020).

6.Conclusions and recommendations

As the take-up rate was satisfactory and no impact to major labour force statistics was observed, the OQ was eventually fully implemented in January 2019. The OQ implementation in the GHS was successful, with favourable responses from respondents and minimal disruption to monthly survey operations. A number of key success factors in the implementation have been identified.