Understanding and accessing versioned data • epidatr

The Epidata API records not just each signal’s estimate for a given location on a given day, but also when that estimate was made, and all updates to that estimate. This is particularly relevant for data sources that have updates or additional information coming in, due to reporting and data flow processes.

For example, let’s look at the emergency department visits signal from the nssp source, which estimates the percentage of emergency department visits that are influenza-related. Consider a result row with reference_time 2024-12-07 for geo_values = "pa". This is an estimate for Pennsylvania on December 7, 2024. That estimate was first reported (published) on December 13, 2024, the delay being due to the aggregation of data by our source and the time taken by the Epidata API to ingest the data provided. Later, the estimate for December 7th was updated, as additional visit data from December 7th arrived at our source and was reported to us. This constitutes a new version of the data. Let’s walk through how to look at that below.

Data known “as of” a specific date

By default, endpoint functions fetch the most recent data available. This is the best option for users who simply want to graph the latest data or construct dashboards. But if we are interested in knowing when data was reported, we can request specific data versions using snapshot_date in epidata_snapshot().

First, we can request the data that was available as of a specific date, using the snapshot_date argument with epidata_snapshot():

# The percent of ED visits due to influenza from the NSSP source, for
# 2024-12-07, as of 2024-12-14
past_snapshot <- epidata_snapshot(
  source = "nssp",
  signals = "pct_ed_visits_influenza",
  geo_type = "state",
  geo_values = "pa",
  reference_time = "2024-12-07",
  snapshot_date = "2024-12-14"
)
knitr::kable(past_snapshot)

signal	report_time	geo_type	geo_value	fill_method	reference_time	value
pct_ed_visits_influenza	2024-12-13	state	pa	source	2024-12-07	0.55

This shows that an estimate of about 0.55% was known as of December 14. If we don’t specify snapshot_date, we get the most recent estimate available:

latest_snapshot <- epidata_snapshot(
  source = "nssp",
  signals = "pct_ed_visits_influenza",
  geo_type = "state",
  geo_values = "pa",
  reference_time = "2024-12-07"
)
knitr::kable(latest_snapshot)

signal	report_time	geo_type	geo_value	fill_method	reference_time	value
pct_ed_visits_influenza	2026-06-26	state	pa	source	2024-12-07	0.57

Note the change in the estimate, from 0.55% to 0.57%, reflecting new data that became available after December 14 about visits occurring on December 7. This illustrates the importance of version tracking, particularly for forecasting tasks. To backtest a forecasting model on past data, it is important to use the data that would have been available at the time the model was or would have been fit, not data that arrived much later.

Multiple versions and revision histories

Unlike epidata_snapshot(), epidata_archive() does not default to the latest data: by using it with the report_time argument, we can request all versions reported in a certain time period, and if report_time is omitted entirely, we get every version ever reported.

# All versions of the percent of ED visits due to influenza from the NSSP
# source, for 2024-12-07, reported between 2024-12-01 and 2025-01-15
archive_data <- epidata_archive(
  source = "nssp",
  signals = "pct_ed_visits_influenza",
  geo_type = "state",
  geo_values = "pa",
  reference_time = "2024-12-07",
  report_time = epirange("2024-12-01", "2025-01-15")
)
knitr::kable(archive_data)

signal	report_time	geo_type	geo_value	fill_method	reference_time	value
pct_ed_visits_influenza	2024-12-13	state	pa	source	2024-12-07	0.55
pct_ed_visits_influenza	2024-12-20	state	pa	source	2024-12-07	0.56
pct_ed_visits_influenza	2024-12-27	state	pa	source	2024-12-07	0.56
pct_ed_visits_influenza	2025-01-03	state	pa	source	2024-12-07	0.56
pct_ed_visits_influenza	2025-01-10	state	pa	source	2024-12-07	0.56

This estimate was updated several times as new data for December 7th arrived.

Note that these results include only data reported between (inclusive) 2024-12-01 and 2025-01-15. If a value was first reported outside this range, a query for reports between 2024-12-01 and 2025-01-15 will not include that value among its results.

The report_time parameter also accepts comparison operators (< and >, each strictly exclusive of the given date) or a range with epirange(). (Note: exact dates are not supported on epidata_archive(); use epidata_snapshot(snapshot_date = ...) if you need data as of a single date.)

# Versions reported strictly before 2025-01-01
epidata_archive(
  source = "nssp",
  signals = "pct_ed_visits_influenza",
  geo_type = "state",
  geo_values = "pa",
  reference_time = "2024-12-07",
  report_time = "<2025-01-01"
)
#> # A tibble: 3 × 7
#>   signal         report_time geo_type geo_value fill_method reference_time value
#>   <chr>          <date>      <chr>    <chr>     <chr>       <date>         <dbl>
#> 1 pct_ed_visits… 2024-12-13  state    pa        source      2024-12-07     0.550
#> 2 pct_ed_visits… 2024-12-20  state    pa        source      2024-12-07     0.560
#> 3 pct_ed_visits… 2024-12-27  state    pa        source      2024-12-07     0.560

# Versions reported strictly after 2024-12-13
epidata_archive(
  source = "nssp",
  signals = "pct_ed_visits_influenza",
  geo_type = "state",
  geo_values = "pa",
  reference_time = "2024-12-07",
  report_time = ">2024-12-13"
)
#> # A tibble: 116 × 7
#>   signal         report_time geo_type geo_value fill_method reference_time value
#>   <chr>          <date>      <chr>    <chr>     <chr>       <date>         <dbl>
#> 1 pct_ed_visits… 2024-12-20  state    pa        source      2024-12-07     0.560
#> 2 pct_ed_visits… 2024-12-27  state    pa        source      2024-12-07     0.560
#> 3 pct_ed_visits… 2025-01-03  state    pa        source      2024-12-07     0.560
#> 4 pct_ed_visits… 2025-01-10  state    pa        source      2024-12-07     0.560
#> # ℹ 112 more rows

Calculating reporting lag

In the V5 API, reporting lag is computed directly as the difference between report_time (the publication date) and reference_time (the observation date):

archive_with_lag <- archive_data %>%
  mutate(lag_days = as.integer(report_time - reference_time)) %>%
  select(signal, reference_time, report_time, lag_days, value)

knitr::kable(archive_with_lag)

signal	reference_time	report_time	lag_days	value
pct_ed_visits_influenza	2024-12-07	2024-12-13	6	0.55
pct_ed_visits_influenza	2024-12-07	2024-12-20	13	0.56
pct_ed_visits_influenza	2024-12-07	2024-12-27	20	0.56
pct_ed_visits_influenza	2024-12-07	2025-01-03	27	0.56
pct_ed_visits_influenza	2024-12-07	2025-01-10	34	0.56

If your analysis requires filtering data to a specific maximum reporting lag (for example, simulating data available with at most a 14-day delay), filter by lag_days:

archive_with_lag %>%
  filter(lag_days <= 14)
#> # A tibble: 2 × 5
#>   signal                  reference_time report_time lag_days value
#>   <chr>                   <date>         <date>         <int> <dbl>
#> 1 pct_ed_visits_influenza 2024-12-07     2024-12-13         6 0.550
#> 2 pct_ed_visits_influenza 2024-12-07     2024-12-20        13 0.560

For sources that have not yet transitioned from the legacy V4 API, versioning is specified differently; see vignette("migration-guide") for the argument and column mappings between V4 and V5, including worked before-and-after examples of revision history queries.