COVID-19 Trends and Impact Survey

• Source name: fb-survey
• Earliest issue available: April 29, 2020
• Number of data revisions since May 19, 2020: 1
• Date of last change: June 3, 2020
• Available for: county, hrr, msa, state, nation (see geography coding docs)
• Time type: day (see date format docs)
• License: CC BY

Overview

This data source is based on the COVID-19 Trends and Impact Survey (CTIS) run by the Delphi group at Carnegie Mellon. Facebook directed a random sample of its users to these surveys, which were voluntary. Users age 18 or older were eligible to complete the surveys, and their survey responses are held by CMU and are sharable with other health researchers under a data use agreement. No individual survey responses are shared back to Facebook. See our surveys page for more detail about how the surveys work and how they are used outside the COVIDcast API.

As of November 2021, the average number of Facebook survey responses we received each day was about 40,000. The survey ran from April 6, 2020 to June 25, 2022, collecting roughly 29.5 million responses in the United States in that time.

We produce several sets of signals based on the survey data, listed and described in the sections below:

1. Influenza-like and COVID-like illness indicators, based on reported symptoms
2. Behavior indicators, including mask-wearing, traveling, in-person schooling, and other activities outside the home
3. Testing indicators based on respondent reporting of their COVID test results
4. Vaccination indicators, based on respondent reporting of COVID vaccinations, whether they would accept a vaccine, and reasons for any hesitancy to accept a vaccine
5. Mental health indicators, based on self-reports of anxiety, depression, isolation, and worry about COVID
6. Belief, experience, and information indicators, about the respondent’s beliefs about COVID-19, their experiences of health care, their sources of information about COVID-19, and their degree of trust in different sources

Many of these signals can also be browsed on our survey dashboard at any selected location.

Additionally, contingency tables containing demographic breakdowns of survey data are available for download. Researchers can request access to (fully de-identified) individual survey responses for research purposes.

Table of Contents

1. Overview
2. Survey Text and Questions
3. ILI and CLI Indicators
   1. Defining Household ILI and CLI
   2. Estimating Percent ILI and CLI
   3. Estimating “Community CLI”
   4. Smoothing
4. Behavior Indicators
   1. Mask Use
   2. Social Distancing and Travel
   3. Schooling Indicators
      1. Schooling Type
      2. School Safety Measures
5. Testing Indicators
6. Vaccination Indicators
   1. Vaccine Uptake and Acceptance
   2. Vaccine Uptake and Acceptance for Children
   3. Vaccine Doses
   4. Barriers to Accessing Vaccination
   5. Reasons for Vaccine Hesitancy
   6. Reasons for Believing Vaccine is Unnecessary
   7. Outreach and Image
7. Mental Health Indicators
8. Belief, Experience, and Information Indicators
   1. Beliefs About COVID-19
   2. Medical Care Experiences
   3. Sources of News
   4. Trusted Sources of Information
   5. Desired Information
9. Limitations
   1. Privacy Restrictions
   2. Declining Response Rate
   3. Target Region
10. Survey Weighting and Estimation
    1. Geographic Weighting and Mixing
    2. Adjusting Household ILI and CLI
    3. Adjusting Other Percentage Estimators
11. Appendix

Survey Text and Questions

The survey starts with the following 5 questions:

1. In the past 24 hours, have you or anyone in your household had any of the following (yes/no for each):
   • (a) Fever (100 °F or higher)
   • (b) Sore throat
   • (c) Cough
   • (d) Shortness of breath
   • (e) Difficulty breathing
2. How many people in your household (including yourself) are sick (fever, along with at least one other symptom from the above list)?
3. How many people are there in your household in total (including yourself)? [Beginning in wave 4, this question asks respondents to break the number down into three age categories.]
4. What is your current ZIP code?
5. How many additional people in your local community that you know personally are sick (fever, along with at least one other symptom from the above list)?

Beyond these 5 questions, there are also many other questions that follow in the survey, which go into more detail on symptoms, contacts, risk factors, and demographics. These are used for many of our behavior and testing indicators below. The full text of the survey (including all deployed versions) can be found on our questions and coding page.

ILI and CLI Indicators

We define COVID-like illness (fever, along with cough, or shortness of breath, or difficulty breathing) or influenza-like illness (fever, along with cough or sore throat) for use in forecasting and modeling. Using this survey data, we estimate the percentage of people (age 18 or older) who have a COVID-like illness, or influenza-like illness, in a given location, on a given day.

Signals beginning raw_w or smoothed_w are adjusted using survey weights to be demographically representative as described below. Weighted signals have 1-2 days of lag, so if low latency is paramount, unweighted signals are also available. These begin smoothed_ or raw_, such as raw_cli instead of raw_wcli.

Note that for raw_whh_cmnty_cli and raw_wnohh_cmnty_cli, the illnesses included are broader: a respondent is included if they know someone in their household (for raw_whh_cmnty_cli) or community with fever, along with sore throat, cough, shortness of breath, or difficulty breathing. This does not attempt to distinguish between COVID-like and influenza-like illness.

Influenza-like illness or ILI is a standard indicator, and is defined by the CDC as: fever along with sore throat or cough. From the list of symptoms from Q1 on our survey, this means a and (b or c).

COVID-like illness or CLI is not a standard indicator. Through our discussions with the CDC, we chose to define it as: fever along with cough or shortness of breath or difficulty breathing.

Symptoms alone are not sufficient to diagnose influenza or coronavirus infections, and so these ILI and CLI indicators are not expected to be unbiased estimates of the true rate of influenza or coronavirus infections. These symptoms can be caused by many other conditions, and many true infections can be asymptomatic. Instead, we expect these indicators to be useful for comparison across the United States and across time, to determine where symptoms appear to be increasing.

Smoothing. The signals beginning with smoothed estimate the same quantities as their raw partners, but are smoothed in time to reduce day-to-day sampling noise; see details below. Crucially, because the smoothed signals combine information across multiple days, they have larger sample sizes and hence are available for more counties and MSAs than the raw signals.

Defining Household ILI and CLI

For a single survey, we are interested in the quantities:

• \(X =\) the number of people in the household with ILI;
• \(Y =\) the number of people in the household with CLI;
• \(N =\) the number of people in the household.

Note that \(N\) comes directly from the answer to Q3, but neither \(X\) nor \(Y\) can be computed directly (because Q2 does not give an answer to the precise symptomatic profile of all individuals in the household, it only asks how many individuals have fever and at least one other symptom from the list).

We hence estimate \(X\) and \(Y\) with the following simple strategy. Consider ILI, without a loss of generality (we apply the same strategy to CLI). Let \(Z\) be the answer to Q2.

• If the answer to Q1 does not meet the ILI definition, then we report \(X=0\).
• If the answer to Q1 does meet the ILI definition, then we report \(X = Z\).

This can only “over count” (result in too large estimates of) the true \(X\) and \(Y\). For example, this happens when some members of the household experience ILI that does not also qualify as CLI, while others experience CLI that does not also qualify as ILI. In this case, for both \(X\) and \(Y\), our simple strategy would return the sum of both types of cases. However, given the extreme degree of overlap between the definitions of ILI and CLI, it is reasonable to believe that, if symptoms across all household members qualified as both ILI and CLI, each individual would have both, or neither—with neither being more common. Therefore we do not consider this “over counting” phenomenon practically problematic.

Estimating Percent ILI and CLI

Let \(x\) and \(y\) be the number of people with ILI and CLI, respectively, over a given time period, and in a given location (for example, the time period being a particular day, and a location being a particular county). Let \(n\) be the total number of people in this location. We are interested in estimating the true ILI and CLI percentages, which we denote by \(p\) and \(q\), respectively:

\[p = 100 \cdot \frac{x}{n} \quad\text{and}\quad q = 100 \cdot \frac{y}{n}.\]

In a given aggregation unit (for example, daily-county), let \(X_i\) and \(Y_i\) denote number of ILI and CLI cases in the household, respectively (computed according to the simple strategy described above), and let \(N_i\) denote the total number of people in the household, in survey \(i\), out of \(m\) surveys we collected. Then our unweighted estimates of \(p\) and \(q\) are:

\[\hat{p} = 100 \cdot \frac{1}{m}\sum_{i=1}^m \frac{X_i}{N_i} \quad\text{and}\quad \hat{q} = 100 \cdot \frac{1}{m}\sum_{i=1}^m \frac{Y_i}{N_i}.\]

See below for details on weighting and standard errors for these estimates.

Estimating “Community CLI”

Over a given time period, and in a given location, let \(u\) be the number of people who know someone in their community with CLI, and let \(v\) be the number of people who know someone in their community, outside of their household, with CLI. With \(n\) denoting the number of people total in this location, we are interested in the percentages:

\[a = 100 \cdot \frac{u}{n} \quad\text{and}\quad b = 100 \cdot \frac{y}{n}.\]

For a single survey, let:

• \(U = 1\) if and only if a positive number is reported for Q2 or Q5;
• \(V = 1\) if and only if a positive number is reported for Q2.

Let \(U_i\) and \(V_i\) denote these quantities for survey \(i\), and \(m\) denote the number of surveys total. We report the percentage of surveys where \(U_i = 1\) as in the hh_cmnty_cli signals and the percentage where \(V_i = 1\) in the nohh_cmnty_cli signals. The exact estimators are described below.

Note that \(\sum_{i=1}^m U_i\) is the number of survey respondents who know someone in their community with either ILI or CLI, and not CLI alone; and similarly for \(V\). Hence \(\hat{a}\) and \(\hat{b}\) will generally overestimate \(a\) and \(b\). However, given the extremely high overlap between the definitions of ILI and CLI, we do not consider this to be practically very problematic.

Smoothing

The smoothed versions of all fb-survey signals (with smoothed prefix) are calculated using seven day pooling. For example, the estimate reported for June 7 in a specific geographical area (such as county or MSA) is formed by collecting all surveys completed between June 1 and 7 (inclusive) and using that data in the estimation procedures described above.

Behavior Indicators

Signals beginning smoothed_w are adjusted using survey weights to be demographically representative as described below. Weighted signals have 1-2 days of lag, so if low latency is paramount, unweighted signals are also available. These begin smoothed_, such as smoothed_wearing_mask instead of smoothed_wwearing_mask.

Mask Use

Social Distancing and Travel

Schooling Indicators

Schooling Type

School Safety Measures

Testing Indicators

Signals beginning smoothed_w are adjusted using survey weights to be demographically representative as described below. Weighted signals have 1-2 days of lag, so if low latency is paramount, unweighted signals are also available. These begin smoothed_, such as smoothed_tested_14d instead of smoothed_wtested_14d.

These indicators are based on questions in Wave 4 of the survey, introduced on September 8, 2020.

Vaccination Indicators

Signals beginning smoothed_w are adjusted using survey weights to be demographically representative as described below. Weighted signals have 1-2 days of lag, so if low latency is paramount, unweighted signals are also available. These begin smoothed_, such as smoothed_covid_vaccinated instead of smoothed_wcovid_vaccinated.

Vaccine Uptake and Acceptance

Vaccine Uptake and Acceptance for Children

Vaccine Doses

Barriers to Accessing Vaccination

Reasons for Vaccine Hesitancy

Reasons for Believing Vaccine is Unnecessary

Respondents who indicate that “I don’t believe I need a COVID-19 vaccine” (in items V5a, V5b, V5c, or, prior to Wave 11, V5d) are asked a follow-up item asking why they don’t believe they need the vaccine. These signals summarize the reasons selected. Respondents who do not select any reason (including “Other”) are treated as missing.

Note: Item V5d was removed in Wave 11, thus these indicators no longer include respondents who indicate in V5d that “I don’t believe I need a COVID-19 vaccine”. Item V5d was shown to those who received one dose of a COVID-19 vaccine, but are not planning to get all recommended doses.

Outreach and Image

The “vaccine_likely_*” indicators are based on questions added in Wave 6 of the survey, introduced on December 19, 2020; however, Delphi only enabled item V1 beginning January 6, 2021.

Mental Health Indicators

Signals beginning smoothed_w are adjusted using survey weights to be demographically representative as described below. Weighted signals have 1-2 days of lag, so if low latency is paramount, unweighted signals are also available. These begin smoothed_, such as smoothed_anxious_5d instead of smoothed_wanxious_5d.

Some of these questions were present in the earliest waves of the survey, but only in Wave 4 did respondents consent to our use of aggregate data to study other impacts of COVID, such as mental health. Hence, these aggregates only include respondents to Wave 4 and later waves, beginning September 8, 2020.

Belief, Experience, and Information Indicators

Signals beginning smoothed_w are adjusted using survey weights to be demographically representative as described below. Weighted signals have 1-2 days of lag, so if low latency is paramount, unweighted signals are also available. These begin smoothed_, such as smoothed_belief_children_immune instead of smoothed_wbelief_children_immune.

Beliefs About COVID-19

Medical Care Experiences

Sources of News

Trusted Sources of Information

Desired Information

Limitations

When interpreting the signals above, it is important to keep in mind several limitations of this survey data.

• Survey population. People are eligible to participate in the survey if they are age 18 or older, they are currently located in the USA, and they are an active user of Facebook. The survey data does not report on children under age 18, and the Facebook adult user population may differ from the United States population generally in important ways. We use our survey weighting to adjust the estimates to match age and gender demographics by state, but this process doesn’t adjust for other demographic biases we may not be aware of.
• Non-response bias. The survey is voluntary, and people who accept the invitation when it is presented to them on Facebook may be different from those who do not. The survey weights provided by Facebook attempt to model the probability of response for each user and hence adjust for this, but it is difficult to tell if these weights account for all possible non-response bias.
• Social desirability. Previous survey research has shown that people’s responses to surveys are often biased by what responses they believe are socially desirable or acceptable. For example, if it there is widespread pressure to wear masks, respondents who do not wear masks may feel pressured to answer that they do. This survey is anonymous and online, meaning we expect the social desirability effect to be smaller, but it may still be present.
• False responses. As with anything on the Internet, a small percentage of users give deliberately incorrect responses. We discard a small number of responses that are obviously false, but do not perform extensive filtering. However, the large size of the study, and our procedure for ensuring that each respondent can only be counted once when they are invited to take the survey, prevents individual respondents from having a large effect on results.
• Repeat invitations. Individual respondents can be invited by Facebook to take the survey several times. Usually Facebook only re-invites a respondent after one month. Hence estimates of values on a single day are calculated using independent survey responses from unique respondents (or, at least, unique Facebook accounts), whereas estimates from different months may involve the same respondents.

Whenever possible, you should compare this data to other independent sources. We believe that while these biases may affect point estimates – that is, they may bias estimates on a specific day up or down – the biases should not change strongly over time. This means that changes in signals, such as increases or decreases, are likely to represent true changes in the underlying population, even if point estimates are biased.

Privacy Restrictions

To protect respondent privacy, we discard any estimate (whether at a county, MSA, HRR, or state level) that is based on fewer than 100 survey responses. For signals reported using a 7-day average (those beginning with smoothed_), this means a geographic area must have at least 100 responses in 7 days to be reported.

This affects some items more than others. For instance, items about vaccine hesitancy reasons are only asked of respondents who are unvaccinated and hesitant, not to all survey respondents. It also affects some geographic areas more than others, particularly rural areas with low population densities. When doing analysis of county-level data, one should be aware that missing counties are typically more rural and less populous than those present in the data, which may introduce bias into the analysis.

Declining Response Rate

We have noted a steady decrease in the number of daily survey responses, beginning no later than January 2021. As the number of survey responses declines, some indicators will become unavailable once they no longer meet the privacy limit for sample size. This affects some signals, such as those based on a subset of responses, more than others, with finer geographic resolutions becoming unavailable first.

Target Region

Facebook only invites users to take the survey if they appear, based on attributes in their Facebook profiles, to reside in the 50 states or Washington, DC. Puerto Rico is sampled separately as part of the international version of the survey. If Facebook believes a user qualifies for the survey, but the user then replies that they live in Puerto Rico or another US territory, we do not include their response in the aggregations.

Survey Weighting and Estimation

When Facebook sends a user to our survey, it generates a random ID number and sends this to us as well. Once the user completes the survey, we pass this ID number back to Facebook to confirm completion, and in return receive a weight. (The random ID number is completely meaningless for any other purpose than receiving this weight, and does not allow us to access any information about the user’s Facebook profile. Nor does it provide Facebook any information about the survey responses.)

We can use these weights to adjust our estimates so that they are representative of the US population—adjusting both for the differences between the US population and US Facebook users (according to a state-by-age-gender stratification of the US population from the 2018 Census March Supplement) and for the propensity of a Facebook user to take our survey in the first place.

In more detail, we receive a participation weight

\[w^{\text{part}}_i \propto \frac{1}{\pi_i},\]

where \(\pi_i\) is an estimated probability (produced by Facebook) that an individual with the same state-by-age-gender profile as user \(i\) would be a Facebook user and take our survey. The adjustment we make follows a standard inverse probability weighting strategy.

Detailed documentation on how Facebook calculates these weights is available in our survey weight documentation.

For unweighted survey signals, we set \(w^\text{part}_i = 1\) for all respondents.

Geographic Weighting and Mixing

Besides the participation weight \(w^\text{part}_i\), each survey response receives a geographical-division weight \(w^{\text{geodiv}}_i\) describing how much a participant’s ZIP code “belongs” in the spatial unit of interest. For example, a ZIP code may overlap with multiple counties, so the weight describes what proportion of the ZIP code’s population is in each county.

Each survey’s weight is hence \(w^{\text{init}}_i = w^{\text{part}}_i w^{\text{geodiv}}_i\). When a ZIP code spans multiple counties or states, a single survey may have different weights when used to calculate different geographic aggregates.

Adjusting Household ILI and CLI

For a given aggregation unit (for example, daily-county), let \(X_i\) and \(Y_i\) denote the numbers of ILI and CLI cases in household \(i\), respectively (computed according to the simple strategy above), and let \(N_i\) denote the total number of people in the household. Let \(i = 1, \dots, m\) denote the surveys started during the time period of interest and reported in a ZIP code intersecting the spatial unit of interest.

First, we adjust the initial weights \(w^\text{init}\) to reduce sensitivity to any individual survey by “mixing” them with a uniform weighting across all relevant surveys. This prevents specific survey respondents with high survey weights having disproportionate influence on the weighted estimates.

Specifically, we select the smallest value of \(a \in [0.05, 1]\) such that

\[w_i = a\cdot\frac1m + (1-a)\cdot w^{\text{init}}_i \leq 0.01\]

for all \(i\). If such a selection is impossible, then we have insufficient survey responses (less than 100), and do not produce an estimate for the given aggregation unit.

Next, we rescale the weights \(w_i\) over all \(i\) so that \(\sum_{i=1}^m w_i=1\). Then our adjusted estimates of \(p\) and \(q\) are:

\[\begin{aligned} \hat{p}_w &= 100 \cdot \sum_{i=1}^m w_i \frac{X_i}{N_i} \\ \hat{q}_w &= 100 \cdot \sum_{i=1}^m w_i \frac{Y_i}{N_i}, \end{aligned}\]

with estimated standard errors:

\[\begin{aligned} \widehat{\mathrm{se}}(\hat{p}_w) &= 100 \cdot \frac{1}{1 + n_e} \sqrt{ \left(\frac12 - \frac{\hat{p}_w}{100}\right)^2 + n_e^2 \hat{s}_p^2 }\\ \widehat{\mathrm{se}}(\hat{q}_w) &= 100 \cdot \frac{1}{1 + n_e} \sqrt{ \left(\frac12 - \frac{\hat{q}_w}{100}\right)^2 + n_e^2 \hat{s}_q^2 }, \end{aligned}\]

where

\[\begin{aligned} \hat{s}_p^2 &= \sum_{i=1}^m w_i^2 \left(\frac{X_i}{N_i} - \frac{\hat{p}_w}{100}\right)^2 \\ \hat{s}_q^2 &= \sum_{i=1}^m w_i^2 \left(\frac{Y_i}{N_i} - \frac{\hat{q}_w}{100}\right)^2 \\ n_e &= \frac1{\sum_{i=1}^m w_i^2}, \end{aligned}\]

which are the delta method estimates of variance associated with self-normalized importance sampling estimators above, after combining with a pseudo-observation of 1/2 with weight \(1/n_e\), assigned to appear like a single effective observation. The use of the pseudo-observation prevents standard error estimates of zero, and in simulations improves the quality of the standard error estimates. See the Appendix for further motivation for these estimators.

The pseudo-observation is not used in \(\hat{p}\) and \(\hat{q}\) themselves, to avoid potentially large amounts of estimation bias, as \(p\) and \(q\) are expected to be small.

The sample size reported is calculated by rounding down \(\sum_{i=1}^{m} w^{\text{geodiv}}_i\) before adding the pseudo-observations. When ZIP codes do not overlap multiple spatial units of interest, these weights are all one, and this expression simplifies to \(m\). When estimates are available for all spatial units of a given type over some time period, the sum of the associated sample sizes under this definition is consistent with the number of surveys used to prepare the estimate. (This notion of sample size is distinct from “effective” sample sizes based on variance of the importance sampling estimators which were used above.)

Adjusting Other Percentage Estimators

The household ILI and CLI estimates are complex to weight, as shown in the previous subsection, because they use an estimator based on the survey respondent and their household. All other estimates reported in the API are simply based on percentages of respondents, such as the percentage who report knowing someone in their community who is sick. In this subsection we will describe how survey weights are used to construct weighted estimates for these indicators, using community CLI as an example.

In a given aggregation unit (for example, daily-county), let \(U_i\) denote the indicator that the survey respondent knows someone in their community with CLI, including their household, for survey \(i\), out of \(m\) surveys collected. Also let \(w_i\) be the weight that accompanies survey \(i\), normalized to sum to 1 as above. Then our initial weighted estimate of the population proportion \(a\) is:

\[\hat{a}_{w, \text{init}} = 100 \cdot \sum_{i=1}^m w_i U_i\]

To prevent observations and standard errors from being zero, we add a pseudo-observation of 1/2 with weight \(1/n_e\). (This psuedo-observation can be thought of as equivalent to using a Bayesian estimate of the proportion, with a Jeffreys prior.) The estimate is hence:

\[\hat{a}_w = 100 \cdot \frac{n_e \frac{\hat{a}_{w, \text{init}}}{100} + \frac12}{1 + n_e},\]

with estimated standard error:

\[\widehat{\mathrm{se}}(\hat{a}_w) = 100 \cdot \sqrt{\frac{\frac{\hat{a}_w}{100}(1-\frac{\hat{a}_w}{100})}{1 + n_e}}\]

which is the plug-in estimate of the standard error of the binomial proportion.

Appendix

Here are some details behind the choice of estimators for percent ILI and percent CLI.

Suppose there are \(h\) households total in the underlying population, and for household \(i\), denote \(\theta_i=N_i/n\). Then note that the quantities of interest, \(p\) and \(q\), are

\[p = \sum_{i=1}^h \frac{X_i}{N_i} \theta_i \quad\text{and}\quad q = \sum_{i=1}^h \frac{Y_i}{N_i} \theta_i.\]

Let \(S \subseteq \{1,\dots,h\}\) denote sampled households, with \(m=|S|\), and suppose we sampled household \(i\) with probability \(\theta_i=N_i/n\) proportional to the household size. Then unbiased estimates of \(p\) and \(q\) are simply

\[\hat{p} = \frac{1}{m} \sum_{i \in S} \frac{X_i}{N_i} \quad\text{and}\quad \hat{q} = \frac{1}{m} \sum_{i \in S} \frac{Y_i}{N_i},\]

which are an equivalent way of writing our previously-defined estimates.

Note that we can again rewrite our quantities of interest as

\[p = \frac{\mu_x}{\mu_n} \quad\text{and}\quad q = \frac{\mu_y}{\mu_n},\]

where \(\mu_x=x/h\), \(\mu_y=y/h\), \(\mu_n=n/h\) denote the expected number people with ILI per household, expected number of people with CLI per household, and expected number of people total per household, respectively, and \(h\) denotes the total number of households in the population.

Suppose that instead of proportional sampling, we sampled households uniformly, resulting in \(S \subseteq \{1,\dots,h\}\) denote sampled households, with \(m=|S|\). Then the natural estimates of \(p\) and \(q\) are instead plug-in estimates of the numerators and denominators in the above,

\[\tilde{p} = \frac{\bar{X}}{\bar{N}} \quad\text{and}\quad \tilde{q} = \frac{\bar{X}}{\bar{N}}\]

where \(\bar{X}=\sum_{i \in S} X_i/m\), \(\bar{Y}=\sum_{i \in S} Y_i/m\), and \(\bar{N}=\sum_{i \in S} N_i/m\) denote the sample means of \(\{X_i\}_{i \in S}\), \(\{Y_i\}_{i \in S}\), and \(\{N_i\}_{i \in S}\), respectively.

Whether we consider \(\hat{p}\) and \(\hat{q}\), or \(\tilde{p}\) and \(\tilde{q}\), to be more natural—mean of fractions, or fraction of means, respectively—depends on the sampling model: if we are sampling households proportional to household size, then it is \(\hat{p}\) and \(\hat{q}\); if we are sampling households uniformly, then it is \(\tilde{p}\) and \(\tilde{q}\). We settled on the former, based on both conceptual and empirical supporting evidence:

• Conceptually, though we do not know the details, we have reason to believe that Facebook offers an essentially uniform random draw of eligible users—those 18 years or older—to take our survey. In this sense, the sampling is done proportional to the number of “Facebook adults” in a household: individuals 18 years or older, who have a Facebook account. Hence if we posit that the number of “Facebook adults” scales linearly with the household size, which seems to us like a reasonable assumption, then sampling would still be proportional to household size. (Notice that this would remain true no matter how small the linear coefficient is, that is, it would even be true if Facebook did not have good coverage over the US.)

• Empirically, we have computed the distribution of household sizes (proportion of households of size 1, size 2, size 3, etc.) in the Facebook survey data thus far, and compared it to the distribution of household sizes from the Census. These align quite closely, also suggesting that sampling is likely done proportional to household size.

---