Compute calibration of a quantile forecaster

Compute calibration of a quantile forecaster

compute_actual_vs_nominal_prob(
  predictions_cards,
  geo_type,
  backfill_buffer = 10,
  grp_vars = c("forecaster", "forecast_date", "ahead"),
  avg_vars = c("geo_value")
)

Arguments

predictions_cards	tibble containing at least columns actual, quantile, value and any grouping or averaging variables named in the next arguments
geo_type	String indicating geographical type, such as "county", or "state". See the COVIDcast Geographic Coding documentation for available options.
backfill_buffer	How many days until response is deemed trustworthy enough to be taken as correct?
grp_vars	character vector of named columns in the score_card at which average performance will be returned
avg_vars	character vector of named columns in the score_card over which averaging performance will be computed

Value

A tibble containing grp_vars, nominal_prob (the claimed interval coverage), prop_below (the proportion of actual values falling below the forecast quantile) and prop_above (the proportion of actual values falling above the forecast quantile). All proportions are calculated for each available probability at each combination of grouping variables by averaging over any variables listed in avg_vars.

Details

Note that no checks are performed to ensure that averaging variables all contain the same predicted quantiles. avg_vars and grp_vars must have an empty intersection.