{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Migrating from pub_covidcast to the new Epidata API\n",
    "\n",
    "The Delphi Epidata API is moving from its V4 endpoints (`pub_covidcast()`\n",
    "and other `{pub/pvt}_*` endpoints, such as `pub_fluview()`, `pub_flusurv()`,\n",
    "and `pvt_quidel()`) to a new set of V5 endpoints, served by\n",
    "`epidata_snapshot()`, `epidata_archive()`, and `epidata_meta()`. The\n",
    "transition is in progress: sources are moving to the new API one at a time,\n",
    "and the V4 functions still work for sources that have not moved yet. New\n",
    "analyses should start with the new functions and fall back to a V4 function\n",
    "only when a source is not yet available there.\n",
    "\n",
    "Starting in October 2026, the V4 functions are tentatively deprecated in\n",
    "favor of the V5 API, and calling them raises a `UserWarning` pointing back\n",
    "to this guide.\n",
    "\n",
    "For the current list of sources and indicators available on the new API, see the\n",
    "[V5 signals documentation](https://cmu-delphi.github.io/delphi-epidata/api/v5_signals.html).\n",
    "\n",
    "This guide walks through `pub_covidcast()`'s arguments and columns in\n",
    "detail, since it's the most widely used V4 endpoint, but the mapping is the\n",
    "same for the other `{pub/pvt}_*` endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# Hidden cell (set in the metadata for this cell)\n",
    "import pandas as pd\n",
    "\n",
    "# Set common options and context\n",
    "pd.set_option(\"display.max_columns\", None)\n",
    "pd.set_option(\"display.max_rows\", 10)\n",
    "pd.set_option(\"display.width\", 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from epidatpy import EpiDataContext, EpiRange\n",
    "\n",
    "epidata = EpiDataContext()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function mapping\n",
    "\n",
    "| Old | New | Purpose |\n",
    "|---|---|---|\n",
    "| `pub_covidcast()` | `epidata_snapshot()` | Data as it appeared on a given date (or the latest) |\n",
    "| `pub_covidcast(issues=...)` | `epidata_archive()` | Full revision history of a signal |\n",
    "| `pub_covidcast_meta()`, `CovidcastEpidata()` | `epidata_meta()` | Discover sources, signals, geo types, and date ranges |\n",
    "\n",
    "`epidata()` is a convenience wrapper that routes to `epidata_snapshot()` or\n",
    "`epidata_archive()` based on which versioning argument you pass.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Argument changes\n",
    "\n",
    "| `pub_covidcast()` argument | New argument | Notes |\n",
    "|---|---|---|\n",
    "| `data_source`, `signals`, `geo_type`, `geo_values` | same | |\n",
    "| `time_type` | none | Dropped. Times in the new API are always dates. |\n",
    "| `time_values` | `reference_time` | Accepts dates or `EpiRange`. Filtered locally after the fetch. |\n",
    "| `as_of` | `snapshot_date` | `epidata_snapshot()` only. `None` returns the latest data. |\n",
    "| `issues` | `report_time` | `epidata_archive()` only. Accepts exact dates, operators like `\"<2025-10-16\"`, or `EpiRange`. |\n",
    "| `lag` | none | Compute it yourself: `report_time - reference_time`. |\n",
    "\n",
    "The new functions also add `fill_method`, which has no covidcast equivalent.\n",
    "Some sources publish several variants of the same signal that differ in how\n",
    "nulls were handled during geographic aggregation: `\"source\"` (raw source\n",
    "data, no imputation), `\"fill_ave\"` (nulls filled with the average of\n",
    "neighboring values), and `\"fill_zero\"` (nulls filled with zero). The default\n",
    "`None` returns all variants, so filter on this column (or pass the argument)\n",
    "if you want exactly one time series per location.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Column changes\n",
    "\n",
    "| `pub_covidcast()` column | New column | Notes |\n",
    "|---|---|---|\n",
    "| `geo_value`, `geo_type`, `signal`, `value` | same | |\n",
    "| `time_value` | `reference_time` | The date the value describes. Always a date. |\n",
    "| `issue` | `report_time` | The date the value was published. Present in both snapshot and archive output. |\n",
    "| `source` | dropped | You queried by source; add it back with `.assign()` if you concatenate results across sources. |\n",
    "| `time_type` | dropped | No longer needed since times are dates. |\n",
    "| `lag` | dropped | Compute as `report_time - reference_time`. |\n",
    "| `direction` | dropped | Was already deprecated in the covidcast API. |\n",
    "| `stderr`, `sample_size` | `ci_lower`, `ci_upper` | Uncertainty is now expressed as confidence interval bounds on `value` instead of a standard error. Populated only for sources that publish them. See below. |\n",
    "| `missing_value`, `missing_stderr`, `missing_sample_size` | dropped | Missingness is now expressed through `fill_method` variants and plain `NaN`s. |\n",
    "| none | `fill_method` | Which null-handling variant of the signal this row belongs to. See above. |\n",
    "\n",
    "Some sources also carry extra columns in the new API, for example\n",
    "`age_group` (pophive) and `nwss_source`, `sample_index`, `pcr_target`\n",
    "(nwss).\n",
    "\n",
    "### Uncertainty columns\n",
    "\n",
    "The covidcast columns `stderr` and `sample_size` have no fixed replacement.\n",
    "The shared schema carries only `value`; a source that quantifies\n",
    "uncertainty adds its own columns, such as `ci_lower` and `ci_upper`. Use\n",
    "the metadata or the [documentation](https://cmu-delphi.github.io/delphi-epidata/api/v5_signals.html)\n",
    "to see which value columns a source returns:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta_sleepcycle = epidata.epidata_meta(source=\"sleepcycle\")\n",
    "meta_sleepcycle[\"sleepcycle\"][\"value_columns\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A query, before and after\n",
    "\n",
    "Fetching NSSP influenza ED visit percentages for two states, as the data\n",
    "looked on January 1, 2025:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "old = epidata.pub_covidcast(\n",
    "    data_source=\"nssp\",\n",
    "    signals=\"pct_ed_visits_influenza\",\n",
    "    geo_type=\"state\",\n",
    "    time_type=\"week\",\n",
    "    geo_values=[\"pa\", \"ca\"],\n",
    "    time_values=EpiRange(202440, 202501),\n",
    "    as_of=20250101,\n",
    ").df()\n",
    "old.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new = epidata.epidata_snapshot(\n",
    "    source=\"nssp\",\n",
    "    signals=\"pct_ed_visits_influenza\",\n",
    "    geo_type=\"state\",\n",
    "    geo_values=[\"pa\", \"ca\"],\n",
    "    reference_time=EpiRange(\"2024-10-01\", \"2025-01-01\"),\n",
    "    snapshot_date=\"2025-01-01\",\n",
    ").df()\n",
    "new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Revision history queries\n",
    "\n",
    "Where you used to pass `issues` to `pub_covidcast()`, use\n",
    "`epidata_archive()` with `report_time`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "revisions = epidata.epidata_archive(\n",
    "    source=\"nssp\",\n",
    "    signals=\"pct_ed_visits_influenza\",\n",
    "    geo_type=\"state\",\n",
    "    geo_values=\"pa\",\n",
    "    reference_time=EpiRange(\"2024-10-01\", \"2025-01-01\"),\n",
    "    report_time=\"<2025-06-01\",\n",
    ").df()\n",
    "revisions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you filtered by `lag`, fetch the archive and filter afterwards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "revisions[(revisions[\"report_time\"] - revisions[\"reference_time\"]) <= pd.Timedelta(days=7)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking whether a source has moved\n",
    "\n",
    "Use `epidata_meta()` to see what a source offers in the new API. It\n",
    "returns signals, geo types, and the available `reference_time` and\n",
    "`report_time` ranges:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta = epidata.epidata_meta(source=\"nssp\")\n",
    "meta[\"nssp\"][\"signals\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta[\"nssp\"][\"reference_time_range\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `epidata_meta()` does not know the source yet, keep using\n",
    "`pub_covidcast()` for it and check back after package updates. The [API\n",
    "mailing\n",
    "list](https://lists.andrew.cmu.edu/mailman/listinfo/delphi-covidcast-api)\n",
    "announces sources as they move."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Endpoints kept for historical reference\n",
    "\n",
    "Not every V4 endpoint is moving to V5. The functions below cover data\n",
    "sources whose collection has already ended (e.g. Google Flu Trends, the\n",
    "HealthTweets signal, the various nowcasts). They are not part of the\n",
    "V4-to-V5 transition, so they are not deprecated and will keep working. The\n",
    "historical data they return is frozen and will remain available. They will\n",
    "just no longer receive new data.\n",
    "\n",
    "| Function | Data source |\n",
    "|---|---|\n",
    "| `pvt_cdc()` | CDC total and by-topic webpage visits |\n",
    "| `pub_covid_hosp_facility_lookup()` | COVID hospitalization facility lookup |\n",
    "| `pub_covid_hosp_facility()` | COVID hospitalizations by facility |\n",
    "| `pub_covid_hosp_state_timeseries()` | COVID hospitalizations by state |\n",
    "| `pub_delphi()` | Delphi's ILINet outpatient doctor visits forecasts |\n",
    "| `pub_dengue_nowcast()` | Delphi's PAHO dengue nowcasts (Americas) |\n",
    "| `pvt_dengue_sensors()` | PAHO dengue digital surveillance sensors (Americas) |\n",
    "| `pub_ecdc_ili()` | ECDC ILI incidence (Europe) |\n",
    "| `pub_gft()` | Google Flu Trends flu search volume |\n",
    "| `pvt_ght()` | Google Health Trends health topics search volume |\n",
    "| `pub_kcdc_ili()` | KCDC ILI incidence (Korea) |\n",
    "| `pvt_meta_norostat()` | Metadata for the NoroSTAT endpoint |\n",
    "| `pub_nidss_dengue()` | NIDSS dengue cases (Taiwan) |\n",
    "| `pub_nidss_flu()` | NIDSS flu doctor visits (Taiwan) |\n",
    "| `pvt_norostat()` | CDC NoroSTAT norovirus outbreaks |\n",
    "| `pub_nowcast()` | Delphi's wILI nowcasts |\n",
    "| `pub_paho_dengue()` | PAHO dengue data (Americas) |\n",
    "| `pvt_sensors()` | Influenza and dengue digital surveillance sensors |\n",
    "| `pvt_twitter()` | HealthTweets total and influenza-related tweets |\n",
    "| `pub_wiki()` | Wikipedia webpage counts by article |"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}