Big Data ethics and selection-bias: An official statistician’s perspective

3.1Boundary between public good and private good

Unlike censuses and surveys which are created and owned by NSOs, and administrative data owned by government agencies, which are often shared with NSOs, most of the newer Big Data sets are created by commercial organisations who we shall describe as data custodians for the purpose of this paper.

In spite of having custodianship of these data sets, an interesting question arises as to who has ownership for the data. Are they the data custodians, or the individuals who provided these data in their transaction with these commercial organisations, i.e. data subjects? For example, whether the information provided to a commercial company when creating an account with the company, and the subsequent activities carried out by the account holder and are logged by the companies, belongs to the company or the individual. Clarity on this issue may, however, but not necessarily always, be provided by referring to the terms and conditions of use of the facilities for transacting with the commercial organisations agreed by the data subjects. If the data is considered to be not owned by the data custodian, can the private company provide such data to an NSO for the production of official statistics? In Australia, a Federal Court in 2017 ruled the meta data related to customers using telecommunication services is not personal data and therefore the telecommunication services provider is not obliged to provide the data to the customer [16].

Provided that commercial organisations have ownership of the Big Data, what is the obligation for these organisations to provide the data to NSOs for public good purposes? Given the commercial value of the data sets, what is the boundary between public good and private good?

For decades, NSOs have been getting the cooperation of these organisations to provide information on their operations, e.g. inventories held, sales information, number of employees, industry etc. Are NSOs empowered by statistics legislation to require commercial organisations to provide information on their customers and their activities, and should they?

If they are prepared to release their data to the NSO for the compilation of official statistics, how does the official statistician ensure:

Provided that there is more than one data custodians to provide the data for the production of a particular field of official statistics, we argue that the second ethical challenge is not new and NSOs have developed and applied sophisticated statistical disclosure avoidance techniques in their data products, which can equally be applied to data provided by Big Data custodians. However, the first challenge is relatively new and requires good judgement in the development of data products by the NSO.

3.2Privacy and confidentiality

Whilst in ordinary usage, the terms privacy and confidentiality are used interchangeably, they have different statistical meaning and different obligation on an NSO. In general terms, privacy is the right of an individual to control the information related to the person and be freed from intrusion. Confidentiality on the other hand is the obligation on the custodian of the private information to keep it secret and from being disclosed.

In deciding on the information to be collected in a census or survey, the NSO has to balance between respecting one’s privacy, and the need for information for society’s decision making, and public good. This balance is generally informed by consultation with the relevant stakeholders, including privacy commissioners, affected individuals and users of the statistical information. In the case of Australia, impact assessments on privacy are normally conducted on Australian Bureau of Statistics (ABS) collections as a matter of course, and for high profile collections, they are conducted by independent consultants. This will inform the NSO if the collection processes are consistent with the Information Privacy Principles and whether any privacy impact from the proposed statistical enquiry is within or beyond community expectation. As well, the statistics legislation requires the ABS to table in Parliament all proposed topics to be asked in any compulsory collections, which provides another check on whether the right balance between privacy and public good has been struck.

With the extensive development of statistical disclosure avoidance methods over the past decades, it can be argued that NSOs are well equipped in protecting the confidentiality of individual’s private information in its data releases, whether they are in the form of aggregate statistics, or unit record files. As a matter of fact, there is a contemporary view that NSOs are too conservative in their policy on the privacy stance for releasing unit record files, which led to a number of NSOs, including the ABS, looking at beyond safe data protections, e.g. safe projects, safe users, safe setting and safe output [17, 18], in more recent data release practices.

In the Big Data space, consideration on the privacy of the information provided by account holders or related to the account holder’s activities will be different from that of a statistical collection. Unlike statistical collections which, backed by statistical legislation, oblige respondents to provide the information to the NSO, information from Big Data is either voluntarily provided by account holders, or a by-product of their activities with the account. However, as mentioned before, there are questions on ownership of this information, whether the data custodians have the authority to release this information for use by others including NSOs, and if the information is released, whether is it done in a way that the privacy of the account holders is protected.

In deciding whether it is ethical to use a particular Big Data source in the production of official statistics, it is prudent for NSOs to consider whether it is legitimate to use the source, undertaking a privacy impact assessment to consider privacy issues arising from its use (e.g. integrating a Big Data source with NSO’s census or survey data), and applying statistical disclosure avoidance techniques in its data releases involving the source.

3.3Transparency

By transparency, we mean openness in the processes and methods used in the collection, processing, compilation and dissemination of the statistics. Transparency is important as it provides the information needed to allow users of official statistics to determine if valid statistical inferences can be made from the statistics, and also if the statistics are fit for the purposes to which the statistics are to be put.

This challenge is generally met by NSOs through the publication of methodologies used in the production of the statistics, including collection instruments, sampling methods, where applicable, non-response follow up or adjustment methods for assessing measures of uncertainty, and data visualisation.

When Big Data are used in the production of official statistics, selection bias correction such that those described in Section 4 below will be required. The transparency challenge for Big Data will be met if the NSO publications on methodologies will be extended to include selection bias correction methods.

3.4Equity of access

With the advent of the Internet, governments are increasingly adopting a policy of open data and open access – see for example data.gov, data.gov.uk and data.gov.au. Increasingly too, NSOs are also making their data freely available to all, thus removing the financial barrier to access, and making statistical available to, and accessible by, all [23].

Because some statistics produced by the NSO are market sensitive, and owing to the need to ensure that the statistics are not seen to be subject to political interference, many NSOs have a policy of making statistics available to users only after official release. This ensures “a level playing field” for users, and no one will have an advantage, financially or otherwise, from prior access to the information.

In some NSOs, however, limited access to official statistics prior to official release is allowed under “lock up” arrangements, where users are not allowed to communicate with people outside the lock up, or leave it, until official release of the statistics has occurred. The benefit of such arrangement is to allow the users to prepare briefings on the statistics in time to be used after the lock up.

Where the statistics are compiled using Big Data, it is logical for existing policy on equity of access to be extended to these statistics, and where needed, lock ups to be arranged for pre-embargo access to official statistics compiled using Big Data sources.

3.5Informed use of information

Informed use of the statistics requires, amongst other things, the provision of meta data describing the quality dimensions of the collection in accordance with quality frameworks – see for example, the ABS Data Quality Framework [24]. Quality Declarations can also be used to describe certain class of statistics [25].

Providing this information to facilitate informed use of the statistics is now common practice amongst NSOs and, provided the same practice is extended to Big Data sources, we do not see any new ethical challenges in this area with the use of Big Data sources in producing official statistics.

4.1Fundamental theorem for estimation error

In a key note speech to the 2016 Royal Statistical Society conference, Meng [20] gave the following fundamental theorem for estimation error:

where

g is a function of the the N values of the finite population, y1,….,yN,Ii=1 if yi is included in the Big Data set (i.e. observed), or 0 otherwise, for i=1,…,N , ρI,g is the correlation between I and g⁢(Y) and σg2=Var⁢(g⁢(Y)) , where the expectation is over a uniform distribution, unif{1,N}, f=NB/N , and NB= the size of the Big Data set and is assumed to be fixed throughout this paper.

Assuming the sample of NB observations of yi is selected by a probability mechanism, ζ , which in the Big Data case, would generally be unknown to the analyst, then

where the Defect Index [18], Eζ⁢(ρI,g2) , is simply 1N-1 if ζ is the probability distribution associated with simple random sampling. If the observations are recorded from Big Data, NB will be very large (but still not equal to N so that f<1 ), and if the selection bias is ignored, the variance of μg will be incorrectly assumed to be (1-f)⁢σg2/NB , resulting in very small confidence interval and leading to incorrect inference about μg . This is what Meng [20] coined as the Paradox of Big Data i.e. the larger the Big Data size (but with f<1 ), the more misleading it is for valid statistical inference. For proper inference, Meng [20] showed that the effective sample size, n𝑒𝑓𝑓 , for a Big Data with size of NB is approximately f(1-f)⁢Eζ⁢(ρI,g2) .

Consider the special case of g⁢(yi)=yi , and yi= 0 or 1, i.e. Yi is a binary variable. Suppose further

Then it can be shown [21, 22] that:

given that N is large and provided that b> 0. The bias, B , of the estimator of p derived from Big Data is B=-p⁢(1-p)⁢(1-r)1-(1-r)⁢p .

Note that both B and the approximate n𝑒𝑓𝑓 are both independent of NB .

4.2An example on effective sample sizes and selection bias

To illustrate the power of Meng’s Fundamental Theorem, assume that we want to estimate the proportion of Australians who speak English at home from a “Big Data” set which comprises between 10% to 50% of the Australian population (estimated to be over 23 million from the 2016 Census of Population). The proportion derived from the Census was 73%. Tables 1 and 2 provide values of the effective sample size, and the estimation bias, for different value of the Big Data size, b and r respectively.

It can be seen that the inferential value of Big Data is limited by the extent of selection (absolute) bias, b.

Similarly, the bias in estimating the proportion of English speakers at home depends on the relative selection bias, b.

4.3Selection bias correction for proportions

How do we adjust for selection bias in Big Data? In general, we can use consider the use of pseudo weights [12]. Let

Then

In the sequel, we will write E(.) instead of Eς(.) to simplify notations.

In the special case of g(yi)=yi,yi= 0 or 1, μg=p ,

denotes the estimate of p based on the Big Data set, B etc. as above, from

or

where

Noting p=(1+θ)-1 and p^B=(1+θ^B)-1 , where

is an estimate of θB , an estimate of p , p^ , can be provided by the following:

or

where

is an estimate of r .

Puza and O’Neill [21] derived the same result by showing that

and thus

and hence

To estimate the variance of p^ , 𝑉𝑎𝑟⁢(p^) , note that using Taylor expansion:

Hence

noting that

as θ^B≐θB given 𝑉𝑎𝑟⁢(θ^B)≐0 when NB is large.

To obtain the approximately unbiased estimate, p^ of p and associated uncertainty, S^⁢E⁢(p^) , the question remains on the estimation of r and its standard error. Using a random sample of the target population, A, available from the NSO and assuming that matching of the units in the random sample with the Big Data set is possible, after which, we can observe, for each of the n units in the random sample, the value of the variables, Ii and Yi , to indicate if the unit is in the Big Data set ( I= 1) or not ( I= 0) and if Y= 1 or 0. Using these data, the following Table can be constructed from which r can be estimated using maximum likelihood by

Using Taylor expansion, assuming the random sample, A, is drawn by simple random sampling without replacement and ignoring the finite population correction, it can be shown (see Appendix 1) the variance of r^ can be estimated by:

where

which leads to

where

Thus, noting that

In other words, the correction factor, κ , to be applied to the standard formulae for estimating the variance of p^ is:

giving S⁢E^⁢(p^)≐κ⁢p^⁢(1-p^)n .

4.4An optimal choice of selection-bias adjusted estimator

Which estimator, p^ or p^𝑆𝑅𝑆 , should one use to estimate p ? If φ^1⩾1/2 and φ^0⩾1/2 , from

i.e. the estimator from the Big Data is preferred, recalling that p^𝑆𝑅𝑆 is an estimator of p .

The strong requirement that min⁡{φ^0,φ^1}⩾1/2 for the estimator from Big Data to perform better than that from a simple random sample is another demonstration of the corollary of the Fundamental Theorem of Estimation Error, namely, that the effective sample size of Big Data is not as good as one imagines, even with selection bias adjustment. In fact, as pointed out by a referee, unless the requirement min⁡{φ^0,φ^1}⩾1/2 is met, the random sample plays the main role for estimation, with information being supplemented by the Big Data source, rather than the other way around.

However, noting that 𝐶𝑜𝑣⁢(p^,p^𝑆𝑅𝑆)≅0 (Appendix 2), and from the Gauss-Markov theorem, the best linear combination, in terms of minimal variance, of p^ and p^𝑆𝑅𝑆 is give by p^ , where

with

being smaller than V⁢a^⁢r⁢(p^) or V⁢a^⁢r⁢(p^𝑆𝑅𝑆) .

In other words, one can always get a better estimator by borrowing strength from both the biased-adjusted estimator from Big Data, and the estimator from the random sample.

4.5An alternative method for selection bias correction

Alternatively, if matching of the Big Data units to the random sample is not possible, but there is auxiliary information available from both the Big Data set and the random sample, say xi=k , where k=1,…,K , and provided that:

where χk denotes the propensity of unit i with auxiliary value xi to be included in the Big Data set, then this propensity can be approximated by dividing the number of Big Data units that have a value of k by the estimated number of units in the target population that have the same value of k . Mathematically this can be denoted by:

where wA⁢i denotes the weight of the ith unit in the random sample, and recalling B and A denotes the Big Data set and random sample respectively. In this case, a different estimate of p is possible, namely,

where χ^i=χ^k , if xi=k .

Assuming the sample is drawn using simple random sampling without replacement, then

where

Let

be the number of yi′ s with yi= 1, and xi=k , then p^ may be rewritten as

Then

given 𝑉𝑎𝑟⁢(NBj⁢1NBj)≐0 for large NBj⁢1 and NBj , j=1,…,K . It can be shown (Appendix 3) that an approximately unbiased estimator of 𝑉𝑎𝑟⁢(p^) is given by:

It is easy to see that the above method can be extended from binary to multi-nominal variables, which we shall not further discuss in this paper.

4.6 Relaxing the assumption of a constant r

In deriving the estimator of p in Section 4.3, we made the assumption of a constant r , that is,

for i=1,…,N . Where this assumption cannot be made, but when auxiliary information xi is available for all the units, we can extend the idea of Section 4.5, when matching of the random sample to the Big Data set is possible, by using

where λ^i=λ^k , if xi=k , and

for simple random sampling.

Let nBk⁢1=∑i∈Ayi∗I(xi=k)∗Ii , nk⁢1=∑i∈Ayi∗I(xi=k) then λ^k=nBk/nk ,

and

given 𝑉𝑎𝑟⁢(NBj⁢1N)≐0 for large NBj⁢1 , j=1,…,K . Using Taylor expansion, njnBj=1λ^j=1λj-1λj2⁢(λ^j-λj) , it can be shown, using arguments similar to those of Appendix 2, that:

and

Hence