Error analysis for hybrid estimates of proportions using big data¹

1.Introduction

High quality traditional probability surveys are becoming more difficult and expensive to conduct. Non-response is a growing problem especially among some segments of the population and therefore there is an increasing risk of bias in estimates derived from sample surveys. At the same time, there is a plethora of data from other sources as a result of the digital revolution. These data can have their own quality problems e.g. important parts of the population may not be covered, the data generation process is unknown and self-selection biases often exist, or the concepts measured do not align with the target measures. However, they do have advantages over probability samples such as a greater number of observations, lower data collection costs and no additional reporting burden. These types of data are often referred to as big data. For the purposes of this paper, we are including administrative data as one example of big data.

How can big data be used in official statistics? There are four possibilities as described below and Option 4 is the subject of this paper.

Probability surveys and big data have different strengths and weaknesses. As they are different, it does raise the question of whether they can be used in combination to build on their respective strengths and to compensate for their different weaknesses. Estimates combining both data sources are referred to as hybrid estimates.

The layout of this paper is as follows: Section 2 provides an Error Frameworks for Big Data extended to cover Hybrid Estimates. Sections 3 to 6 provide a mathematical exposition of hybrid estimates for finite population proportions, discuss when they are more accurate than either the big data themselves or survey data, and consider the case where there are measurement errors in the big data. Section 7 provides a Case Study. Section 8 discusses a possible epidemiological application with concluding remarks outlined in Section 9.

2.Total error framework for big data and hybrid estimates

For this paper, we define hybrid data as survey and big data sets that complement each other and are combined in some way to provide more robust estimates. Estimates derived in the way outlined later in this paper from hybrid data are called hybrid estimates. For example, a sample survey could be used to complement a big data source to address its weaknesses e.g. under-coverage, self-selection errors of certain populations or providing the data to enable the adjustments for any validity or measurement errors in the big data. Of course, one can have more complex hybrid estimates that combine more than two data sets.

The starting point for the consideration of hybrid estimates is understanding the sources of error in both the probability survey and the big data source. There is a lot of literature on Total Survey Error (TSE) that covers the former, e.g. [1]. A common representation of TSE is to categorise Errors of Measurement as specification error, measurement error, processing error or model/estimation error and Errors of Representation as frame error, sampling error or non-response error. Total Error is a similar concept to Total Survey Error but recognises that the term ‘survey’ is misleading in the case of big data. Furthermore, the sources of error will be different, requiring a modified framework for specifying the errors.

There have been some recent approaches to describe Total Error for Big Data [2]. We have chosen to adapt the work of [3]. Their work was targeted at administrative data but can be easily extended to big data. An important initiative in their work is to describe errors separately for each of three phases of the production process – (1) single input data, (2) integrated data sources, and (3) output. We will use the same structure except that the integration aspects of phase 2 are considered as part of input phase and the output phase comprises the hybrid estimates discussed in this paper. Only high-level detail is shown in Table 1. Also, the explanations are most relevant to applications of hybrid estimates for official statistics.

The framework focusses on the residual errors after the hybrid estimates have been created by combining the survey and big data sources. The residual errors will contain both systematic (bias) and random components. It is important to understand both. It is almost certain that some random components will still exist even when there have been adjustments for systematic bias.

Having defined the relevant frameworks, how do we use them to make decisions on the choice of survey, big data or hybrid data source for estimation? The first step is to understand the errors in the data sources irrespective of whether they are survey or big data based. Wherever possible, this should be based on quantitative information, but we appreciate this is not always possible. Informed guesses may be necessary. Independent and expert third party advice can also be especially useful. The aim here is to identify the most important sources of error, not every source of error, and the extent to which they can be mitigated. These are the errors that need to be considered in determining whether the big data can be used or not, the design of any complementary data collections, and the development of the hybrid estimates.

The focus should then be on understanding how these errors might be measured, controlled or mitigated. The framework described in Table 1 is relevant to the error analysis of the hybrid estimates. If it is decided to consider hybrid estimates, the next step is to design the hybrid estimates themselves. This will include consideration of important residual errors and how they might be further mitigated. It may be necessary to commission some special studies to support this analysis. For adjustment of coverage errors, as seen below, representivity ratios of both the big data and the survey data are very important considerations. For the adjustment of stochastic measurement errors, statistical models will also be necessary.

3.How to determine when to use hybrid estimates?

If one has a probability sample, A, with partial response, and a big data set B with under-coverage/self-selection errors and, possibly measurement errors as well, which estimate out of the three, from the partially responding sample, big data set or a combination of the two, would be preferred?

[4] showed that, for simple random sampling with negligible finite population correction and full response, a hybrid estimator of the finite population total is preferred as it generally has smaller mean squared error (MSE) than either the big data estimator or the survey estimator. The hybrid estimator will have a smaller variance than that of the estimator from the probability sample, A, if (1-NBN)⁢SC2<SU2 where NB,N,SU and SC are the size of the big data set, size of the finite population, and the standard deviation of the population, and of the population segment missed by the big data set, C, respectively. The inequality is almost always true when NB is large in comparison with N. The methodology can also be examined from a dual sampling frame perspective [5].

In this paper, we extend the idea of [4] in the following ways. First, we extend the comparison of the efficiency (in MSE terms) of the hybrid estimator with the Horvitz-Thompson estimator of the finite population proportion from a responding probability sample adjusted for non-response errors, and the big data estimator of the proportion adjusted for under-coverage/self-selection errors. Second, the adjustment is carried out using “representivity ratios”. Third, when the big data is subject to measurement errors, instead of using the erroneous big data as benchmarks as in [4] for the hybrid estimate, we use a statistical method to adjust for the measurement errors before we combine the big data with survey data.

To illustrate ideas, let AR and B denote the responding sample and big data respectively. We consider the simple case of estimating the finite population proportion, PU=∑i∈UYiN, where Yi= 1 or 0. We also assume initially that there are no measurement errors in the big data, and then extend our results to the measurement error case.

For this set up, we are interested in the choice between estimators from the big data sample, B, PB=∑i∈BYiNB, the partially responding probability sample, AR, P^AR=∑i∈ARYinAR, or a hybrid estimate based on B and AR.

We use a simple example to illustrate the concept of “representivity ratios”. Suppose the population comprises 5 million males and 5 million females, and the big data set consists of 5 million males and 4 million females. The population proportion of males is 0.5 but the corresponding proportion in the big data set is 0.556, giving an estimation error of 0.056. Why does this happen? It happens because whilst the propensity of males included in the big data set is 100%, the inclusion propensity for females is only 80%. If we let rB denote the representivity ratio of B, i.e. the ratio of the propensity of Yi′⁢s=1 to the propensity of Yi′⁢s=0 to be included in B, it gives us an indication of how balanced the representation of the population of Yi′⁢s=1 and Yi′⁢s=0 is in B. Where this ratio is not equal to 1, there is an imbalance and the proportion computed from B would be biased. In addition, [6] showed that PU can be recovered from the expression PU=(rB⁢PB-1-rB+1)-1. In the hypothetical example above, rB=1.25 and this gives us an indication that males are over-represented in the big data set. Using this information and putting PB=0.556 into the “recovery” formula gives us back PU=0.5. Similarly, if we let rR denote the representivity ratio for AR, we can also use PU=(rR⁢PAR-1-rR+1)-1 to recover the finite population proportion based on the partially responding sample.

From PU=(rB⁢PB-1-rB+1)-1, we have PB=rB⁢PU⁢{1+(rB-1)⁢PU}-1, hence 𝑀𝑆𝐸⁢(PB)=(PB-PU)2={(rB-1)⁢PU⁢(1-PU)1+(rB-1)⁢PU}2 which is just the bias squared. Likewise, the bias squared for p^AR is {(rR-1)⁢PU⁢(1-PU)1+(rR-1)⁢PU}2 and its sampling variance, ignoring finite population corrections, is p^AR⁢(1-p^AR)nAR noting p^AR=rR⁢PU⁢{1+(rR-1)⁢PU}-1. Hence, 𝑀𝑆𝐸⁢(p^AR)≐1nAR⁢rR⁢PU⁢(1-PU){1+(rR-1)⁢PU}2+{(rR-1)⁢PU⁢(1-PU)1+(rR-1)⁢PU}2 comprising a sampling variance term and a bias squared term.

To construct the hybrid estimator, we can combine the proportion from B, with the estimated proportion for C, through the information from AR∩C. This assumes that there is sufficient information available for the analyst to identify the units in the responding sample that are not in the big data set, using methods of deterministic or probabilistic matching. Let p^AR∩C be the computed proportion of Yi=1 in AR∩C. Then the hybrid estimator of PU is given by p^H=WB⁢PB+WC⁢p^AR∩C, where WB=NBN and WC=1-WB. We know that p^H~=WB⁢PB+WC⁢p^A∩C is approximately unbiased because p^A∩C is an approximately unbiased estimator of the proportion of Y=1 in C. From p^A∩C=R⁢p^AR∩C+(1-R)⁢p^AN⁢R∩C, where p^AN⁢R∩C is the unobserved proportion from the non-responding sample in C, we therefore have |p^H-p^H~|=WC⁢(1-R)⁢|p^AR∩C-p^AN⁢R∩C|⩽WC⁢(1-R) where R is the response rate, and hence p^His almost unbiased to O⁢{WC⁢(1-R)}. Therefore, the MSE of p^H is bounded by: 𝑀𝑆𝐸⁢(p^H)⩽WC2nAR∩C⁢rR⁢PC⁢(1-PC){1+(rR-1)⁢PC}2+WC2⁢(1-R)2, where PC denotes the proportion of Yi=1 in C. Using arguments similar to deriving 𝑀𝑆𝐸⁢(p^AR) above, we can show that 𝑀𝑆𝐸⁢(p^H)≐WC2nAR∩C⁢rR⁢PC⁢(1-PC){1+(rR-1)⁢PC}2+WC2⁢(rR-1)2⁢PC2⁢(1-PC)2{1+(rR-1)⁢PC}2. Noting that from PC=1WC⁢(PU-WB⁢PB) and PB=rB⁢PU⁢{1+(rB-1)⁢PU}-1, we can also express 𝑀𝑆𝐸⁢(p^H) in terms of PU.

4.Which estimator is better under what conditions?

The choice of estimators is determined by comparing their MSEs. The following provides sufficient conditions for one estimator to be better than another. It is easily seen that 𝑀𝑆𝐸⁢(PB)⩽𝑀𝑆𝐸⁢(p^AR) provided that |(rB-1)|1+(rB-1)⁢PU⩽|(rR-1)|1+(rR-1)⁢PU, i.e. the absolute bias of PB is smaller than that of p^AR as the latter also has a sampling variance in its MSE. Otherwise, we have to assess the sign of: 𝑀𝑆𝐸⁢(PB)-𝑀𝑆𝐸⁢(p^AR)=[(rB-1)2⁢PU2⁢(1-PU)2{1+(rB-1)⁢PU}2-(rR-1)2⁢PU2⁢(1-PU)2{1+(rR-1)⁢PU}2]-rR⁢PU⁢(1-PU)nAR⁢{1+(rR-1)⁢PU}2 which is data dependent and can only resolved numerically.

Provided that WC=(1-WB) is sufficiently small such that WC2≈0, we have 𝑀𝑆𝐸⁢(p^H)≈0⩽𝑀𝑆𝐸⁢(PB) or 𝑀𝑆𝐸⁢(p^AR) and the hybrid estimates is always preferred. Otherwise, for WB not close to one, the choice will have to be determined numerically by comparing their MSEs.

5. How to determine the representivity ratios?

Let θA=∑i∈A(1-Yi)∑i∈AYi,θB=∑i∈B(1-Yi)∑i∈BYi and θAR=∑i∈AR(1-Yi)∑i∈ARYi, then it can be shown using the Bayes Theorem ([6]) that: r^B=θAθB and r^R=θAθAR. As both θB and θAR are observed, we only need θA which can be estimated by well-established non-response adjustments methodology used in statistical offices [7, 8], e.g. θ^A=∑i∈AR(1-Yi)/ρ^i∑i∈ARYi//ρ^i, where ρi is the propensity of the ith sampling unit to respond to the survey, with ρi to be estimated using e.g. a logistic regression model. For the application described in Section 7, we used the two-step adjustment of [9] to determine the non-response adjustment weights. This is to ensure that the initial weight 1ρ^i is further adjusted to make the weighted auxiliary variables in the sample equal to their benchmarks, subject to the constraint that the weighted (by ρ^i) sum of the squared differences between the initial and final weights is a minimum. Note that we can use the plug-ins P^U=(1+θ^A)-1 and P^C=1WC⁢(P^U-WB⁢PB) to estimate the MSEs of the different proportion estimators.

6. What if the big data is subject to measurement errors?

We model the random variable, Z, to be Y observed with errors, such that the false negative rate, i.e. observing a “0” when the actual value should be “1”, Pr⁡(Z=0|Y=1), is α0; and the false positive rate, Pr⁡(Z=1|Y=0), is α1. We assume that random variables are mutually statistically independent, i.e., Zi⊥Zj,Yi⊥Yj,Yi⊥Zj where i≠j. Let Z¯B=∑i=1NBZiNB be used to estimate PU. Because there are false positive and false negative values in Z, there will be increased uncertainty in using Z¯B to estimate PU.

Heuristically, the number of false negatives in the observed counts would be α0⁢NB⁢PB and the number of false positives would be α1⁢NB⁢(1-PB). Hence the average net error in counting the number of Y=1 cases in B would be 1NB⁢|α1⁢NB⁢(1-PB)-α0⁢NB⁢PB|=|α1⁢(1-PB)-α0⁢PB|.

It can be shown (refer to the Appendix) that: 𝑀𝑆𝐸⁢(Z¯B)=E⁢(Z¯B-PU)2≐{α1⁢(1-PB)-α0⁢PB}2+𝑀𝑆𝐸⁢(PB). This result shows that the additional term added to the MSE is the square of the net counting error, and the approximation is correct to O⁢(NB-1).

Alternatively, a better approach (see Section 7 below) will be to adjust the big data for measurement errors first before constructing the hybrid estimates, as follows: p^H𝑎𝑑𝑗=WB⁢{P^B}+WC⁢p^AR∩C, where α^0 and α^1 are consistent estimators of α0 and α1 respectively. To compute P^B, we use the following iterative approach:

We call p^H𝑎𝑑𝑗 the measurement bias-adjusted hybrid estimator. Using equation A1 of Appendix, it can be seen that:

Hence

For simple random sampling and ignoring the finite population correction, we have

Hence,

Furthermore, to estimate αo and α1, one can take a random sample of B, compare the actual values with the observed values and use the information to tally the “confusion” matrix as in Table 2.

Then α^0=n01n11+n01 and α^1=n10n10+n00. In addition, E⁢(α^0)≐α0, and E⁢(α^1)≐α1. As a special case, one can use the linked AR and B sample used to help compute the hybrid estimate to estimate α^0 and α^1. This is an efficient approach provided that the cost of verifying the observed value against the actual value for the linked sample is less than total cost of selecting a smaller sample and verification. We used the linked sample to compute α^0 and α^1 in the case study below.

Finally, as θB=∑i∈B(1-Yi)∑i∈BYi is not observed, we can estimate it by the following “plug-in” estimator: NB-∑i=1NBZi-NB⁢{α^0⁢PB-α^1⁢(1-PB)}∑i=1NBZi+NB⁢{α^0⁢PB-α^1⁢(1-PB)} because, from (A1) of the appendix, we have

7.Case study

We tested the methods outlined in the previous section through an empirical study, using data from the Australian Bureau of Statistics (ABS) 2015/2016 Agriculture Census (Ag Census) and 2014/2015 Rural Environment and Agricultural Commodities Survey (REACS). The set of responding businesses on the 2015/2016 Ag Census is considered as the big data B while the 2014/2015 REACS is our probability sample A. For the purposes of the empirical study, the agriculture population in 2014/2015 was deemed to be the population of interest.

There were a number of changes in the agriculture population between 2014/2015 and 2015/2016. These had an impact on the coverage of the 2015/2016 Ag Census relative to the population of interest. The changes included:

Points 1 and 3, combined with non-response in the Ag Census, introduce some under-coverage error relative to the 2014/2015 population. We end up with an overall coverage rate for the Ag Census of about 53% relative to the 2014/2015 population.

Point 2 introduces some over-coverage error in the Ag Census dataset – we addressed this by removing the new farms from the Ag Census dataset to produce estimates.

Point 4, reflecting the changes in farming activity that may occur from year to year, introduces some measurement error in the 2015/2016 Ag Census data relative to the ‘true’ 2014/2015 population. This will have an impact on the proportion estimates and the resulting MSEs produced from the Ag Census data.

There is some non-response error present in the 2014/2015 REACS data. For that collection, the response rate was around 80%. Given that it is a sample survey rather than a census of the population, there are also sample errors that will arise from using this dataset to form estimates.

For this case study, linkage error is not a concern, as there is a consistent ABS unit identifier across the two datasets that we were able to use to form the hybrid estimator.

In terms of the Error Frameworks outlined in Section 2, the most important errors for the big data were coverage, missing units (because of non-response) and measurement. The hybrid estimate mitigates the first two error sources but inherits sampling error and potentially non-response error. The third error source, measurement error, can be corrected for by using the bias-adjusted hybrid estimator.

For the case study, two versions of the Ag Census dataset were prepared. The first version replaced the original Ag Census data with reported values from the 2014/2015 REACS where we could. This was possible for farms that were part of the responding REACS sample, AR, and were also in B. This dataset represented a “no-measurement-error” scenario for the Big Data.

The “no-measurement-error” dataset was amended by artificially introducing some measurement error (false negatives and false positives) into the reported data for four data items of interest:

This second dataset represented the “with-measurement-error” scenario. Table 3 contains the simulated rate of false negatives and false positives for the dataset.

Estimates and MSEs were produced for four proportions of interest using the approaches discussed in Sections 3 to 6:

8.Application

The Case Study illustrates how the REACS data, in conjunction with the data from the Ag Census can be used to develop hybrid estimates, when the scope of the Census was narrower than the survey (e.g. the exclusion of smaller agriculture units). It shows that big data can be extremely inaccurate if the big data is not representative and, for this data set at least, the hybrid estimates are more accurate than those estimates based on the big data or responding sample only, with the exception of estimator for beef because the big data is representative for this commodity. The case study also shows that measurement error in the big data set may impact on the effectiveness of the hybrid estimates and, where this happens, it is advisable to remove these errors before compiling the hybrid estimators. Where this is done, the case study shows that, with the exception of beef, the measurement error bias-adjusted hybrid estimates are still better than survey or big data estimates.

Another potential application of hybrid estimates is to obtain more reliable estimates of Covid-19 prevalence. Many countries have a surveillance testing data base which is certainly a big data set. However, it is not representative of the population. It is based on a self-selected sample of persons who have reported for testing because they are concerned for some reason and there are testing facilities they can access. It will also include persons who have been asked to test through Covid-19 surveillance systems e.g. close contacts of someone who has tested positive or required to test to enable travel etc.. This sample will under-represent certain classes of persons especially those who are asymptomatic or mildly symptomatic. The sample will also over-represent those that are more seriously ill including those requiring hospitalisation. Trends can also be misleading as cases will increase as tests increase. Nevertheless, this data base contains a lot of detailed data of potential value. It could be supplemented with a probability survey similar to that conducted successfully by the UK Office of National Statistics (https://www.ons.gov.uk/cis). Hybrid estimates could then be produced that use the strengths of both data sets to provide estimates of population prevalence by population group.

It may help overcome a tendency for an upward bias in forecasts derived from epidemiological models over a long period of time (see [11]). We believe a contributing factor is the non-adjustment for selection bias in the testing data bases that are used to estimate the model parameters.

In this application, the big data includes coverage errors in part induced by self-selection bias. Validity and measurement errors should be low if the testing protocols are satisfactory. The probability survey should not have coverage errors but will have sampling and non-response errors and will have larger measurement errors if testing is of lower quality. It is likely that hybrid estimates will be effective if any non-response bias can be mitigated in the survey (e.g. through weighting) and quantitative information is available on measurement errors that can be used in the estimation process.

Linkage error may be a concern. It should not be so important if a population register is used to identify those persons missing from the scope of the surveillance system. However, linkage error may be important if there is reliance on a question such as “Have you undertaken a Covid-19 test over the last x weeks?” where there would be measurement errors in the survey data set. [10] explores the use of a combination of surveillance testing and probability survey data and propose something like a hybrid estimate.

9.Discussion and conclusions

Hybrid estimates play an important role in potentially making greater use of big data especially when it is subject to coverage error. The first step should be to assess the error structure of the big data. Are the errors sufficiently unimportant for it to be used in isolation to produce official statistics perhaps with the type of adjustments described in the first three options outlined in Section 1.

Alternatively, does the big data need to be supplemented by a survey data source to adjust for the most important error sources? For estimating the finite population proportion, a good indicator is to check the representivity ratio against 1, or the relative size of the off-diagonals in a confusion matrix. Where required, hybrid estimates may be used, for example, to adjust for coverage error or self-selection error, or to adjust for validity error where the data concept available in the big data source is different to the target concept.

The paper provides an error framework that can be used to make decisions among these alternatives. They help to identify the most important potential error sources so consideration can be given to how they might be best mitigated. In this analysis, it is important to understand both the systematic and random components of these errors. It may be necessary to commission some special studies to support this analysis.

This paper also provides a method for helping make these decisions under restrictive (but realistic) assumptions on representational and measurement errors. It assumes the most important sources of error are non-response for surveys and coverage, self-selection bias or measurement errors in big data. It also assumes that survey statistician can determine which of the survey units is included in the big data set and can construct a “validity” sample for measurement error adjustment, where required. Representivity ratios are the key statistic in determining the “representativeness” of the big data and survey data when estimating the finite population proportion and should be used as a guide to determine if the efficiency of the estimation can be improved by hybrid estimates. As well, the confusion matrix can be used to determine the extent of measurement errors and can be used to provide estimates of false positive and false negative rates for measurement error adjustment.

It is also clear in the above exposition that sample surveys are going to stay in the statistician’s tool kit in spite of the emergence and popularity of big data.