Skip to content

/education-data-explorer

Read-only discovery reference that maps a research question to the Education Data mirror files that answer it, giving path, row grain, year coverage, key variables and join keys for 14 sources at a pinned mirror vintage. Used before education-data-query, which handles retrieval.

Category: data-handling
Field: social-sciences
License: LGPL-3.0-or-later (as declared in the README, badge and CITATION.cff; the repo ships the GPL-3.0 text as LICENSE plus COPYING.LESSER)
Updated: 2026-08-09
Stages: data-acquisition

Education Data Explorer

Discovers which Urban Institute Education Data mirror files answer a research question and returns them as acquisition-ready routing facts: the canonical mirror path, what one row represents (grain), year coverage, curated key variables, and join keys. This is the discovery half of the education-data workflow — it owns question→file routing judgment, grain semantics, variable/code meaning, and cross-source joins. It does not own fetch mechanics: retrieval code, mirror configuration, codebook-URL construction, and the curated path catalog live in education-data-query. Load this skill first (routing), then education-data-query (retrieval). Every inventory fact here — years, variables, coded values — is keyed to mirror vintage 0.26.1 (pinned); the discovery path is pure markdown reading, with no instruction to fetch, probe, or execute anything.

Why mirror-first (and why no API)

DAAF's education pipeline never touches Urban's live REST API. All acquisition is mirror-based: a revision-pinned HuggingFace parquet mirror (Portal vintage 0.26.1, revision 0ad00ce0e232c96b0642459e4e7326607a8d26aa), with an unpinned urban_csv bulk-CSV fallback that uses the identical canonical paths. Acquisition consumes canonical mirror paths of the form {source}/{filename} (extensionless — the mirror layer appends .parquet or .csv). So discovery's job is to emit those paths, not URL routes. Portal endpoint routes appear here only as small citation/maintenance annotations, never as the thing you hand downstream.

The live REST API moves quarterly (endpoints rename, variables drift); a prior version of this skill hand-documented that API and went badly stale. The mirror is a frozen, reproducible snapshot — so the facts below are stable for vintage 0.26.1 and can be read as reference rather than re-probed.

Standing caveat (vintage + fallback). All year coverage, variables, and coded values below are exact for the pinned HuggingFace mirror at vintage 0.26.1. The urban_csv fallback is unpinned/current-Portal, so if a fetch falls through to it, vintage guarantees no longer hold exactly. Cite Portal version 0.26.1 for data fetched from the pinned mirror.

Reference File Structure

File Purpose When to Read
references/schools-datasets.md K-12 school-level mirror families (CCD, CRDC, EDFacts, MEPS, NHGIS) Researching individual schools
references/districts-datasets.md District/LEA-level mirror families (CCD, SAIPE, EDFacts) Researching school districts
references/colleges-datasets.md College-level mirror families (IPEDS, Scorecard, FSA, PSEO, EADA, NACUBO, NCCS, NHGIS, Campus Safety) Researching higher education
references/variable-dictionary-{source}.md Complete per-source variable inventory (2,235 per-source variables, deduplicated from 2,994 endpoint-variable rows; verbatim labels/types/filter flags/coded values) Confirming a real variable name or its coded values
references/variable-codes.md Curated, analytically load-bearing coded-value schemes with cross-source populated-subset notes Interpreting or filtering categorical codes
references/maintenance-live-api.md Live-API catalog surfaces + regeneration mechanics — maintenance sessions only Rebuilding the mirror/dictionaries at a new vintage

The 14 variable-dictionary-{source}.md files are mechanically generated from the Portal varlist at vintage 0.26.1. The routing files curate 5-12 key variables per family and point to the full dictionary entry for the rest. Never invent a variable name — if it is not in the dictionary, it is not in the mirror.

Decision Trees

What data level (and file) do I need?

Text Only
What entity is one row of my analysis?
├─ An individual K-12 school (ncessch) → schools level
│   └─ references/schools-datasets.md → pick a CCD/CRDC/EDFacts/MEPS/NHGIS family
├─ A school district / LEA (leaid) → school-districts level
│   └─ references/districts-datasets.md → pick a CCD/SAIPE/EDFacts family
└─ A college / university (unitid) → college-university level
    └─ references/colleges-datasets.md → pick an IPEDS/Scorecard/FSA/PSEO/EADA/NACUBO/NCCS/csafety family

What topic → which mirror family?

Text Only
Research topic?
├─ K-12 enrollment / demographics → ccd/schools_ccd_enrollment_{year} (or ccd/schools_ccd_lea_enrollment_{year} at district level)
├─ School/district directory & staffing → ccd/schools_ccd_directory | ccd/school-districts_lea_directory
├─ District finance → ccd/districts_ccd_finance
├─ District poverty → saipe/districts_saipe
├─ Civil rights / discipline / course access → crdc/schools_crdc_* (see schools-datasets.md — one file per topic)
├─ K-12 assessments / grad rates → edfacts/{schools,districts}_edfacts_assessments_{year} | _grad_rates_{year}
├─ School poverty measure → meps/schools_meps
├─ College directory / characteristics → ipeds/colleges_ipeds_directory | _institutional-characteristics
├─ College enrollment → ipeds/colleges_ipeds_fall-enrollment-race_{year} | _enrollment-fte | _headcount
├─ College completions (by CIP) → ipeds/colleges_ipeds_completions-2digcip_{year} | -6digcip_{year}
├─ College graduation rates → ipeds/colleges_ipeds_grad-rates | -200pct | -pell
├─ College finance / endowment → ipeds/colleges_ipeds_finance | nacubo/colleges_nacubo_endow | nccs/colleges_nccs_all
├─ Student aid / loans → ipeds/colleges_ipeds_sfa_* | fsa/colleges_fsa_grants | _loans | _campus_based_volume
├─ Post-college earnings → scorecard/colleges_scorecard_earnings | pseo/colleges_pseo_{year}
├─ Athletics equity → eada/colleges_eada_inst_characteristics
├─ Campus crime (hate crimes) → csafety/colleges_csafety_hate_crimes
└─ Census geography crosswalk → nhgis/{schools,colleges}_nhgis_geog_{census-year}

How do I confirm a variable name or code?

Text Only
Need a variable?
├─ Curated key variables → the family entry in the *-datasets.md routing file
├─ Every variable in a family → references/variable-dictionary-{source}.md (§ that family)
└─ What a categorical code means → references/variable-codes.md

Quick Reference: Data Levels

Level Key Sources Primary ID ID Format
schools CCD, CRDC, EDFacts, MEPS, NHGIS ncessch 12-char zero-padded string
school-districts CCD, SAIPE, EDFacts leaid 7-char zero-padded string
college-university IPEDS, Scorecard, FSA, PSEO, EADA, NACUBO, NCCS, NHGIS, Campus Safety unitid 6-digit integer

Zero-padded IDs: ncessch (12) and leaid (7) are zero-padded numeric strings — leading zeros are load-bearing (FIPS 01-09 states). education-data-query handles the force-string/pad-and-assert read recipe; discovery just needs to name the join key.

Quick Reference: Join Keys

Join across Key Example use
School sources (CCD ↔ CRDC ↔ EDFacts ↔ MEPS ↔ NHGIS) ncessch (+ year) Enrollment + discipline + poverty
District sources (CCD ↔ SAIPE ↔ EDFacts) leaid (+ year) Directory + poverty + outcomes
College sources (IPEDS ↔ Scorecard ↔ FSA ↔ PSEO ↔ EADA ↔ NACUBO ↔ NCCS ↔ csafety) unitid (+ year) Characteristics + earnings + aid
School → district rollup leaid present on school files Aggregate schools to their LEA
Any level → census geography ncessch/unitid via NHGIS crosswalk Link to tract/CBSA/region

Match on year when joining — coverage windows differ across families (see below), so align years before merging.

Year Coverage Overview (vintage 0.26.1)

Verified ranges from the endpoint-file bridge and route reconciliation. Non-contiguous families are flagged; see the routing files for exact year lists.

Family (representative) Coverage Note
CCD directory / enrollment 1986-2024 Annual
CCD district finance 1991, 1994-2020 Non-contiguous early years
CRDC (most topics) 2011, 2013, 2015, 2017, 2020, 2021 Biennial; chronic-absenteeism reaches 2022
EDFacts assessments 2009-2018, 2020 2019 absent (COVID testing waivers)
EDFacts grad rates 2010-2019 —
IPEDS directory / inst-characteristics 1980, 1984-2024 —
IPEDS finance 1979, 1983-2017 Mirror ends 2017 (API reaches further back, differently packaged)
IPEDS grad-rates (150%) 1996-2023 Wholesale-revalued in v0.26.1 — re-run any pre-2026q3 analysis
IPEDS completions (2-/6-dig CIP) 1991-2023 / 1983-2023 Yearly shards
Scorecard earnings 2003-2014, 2018 Sparse; other Scorecard families 1996-2020
SAIPE 1995, 1997, 1999-2024 Poverty pct on a 0-1 scale despite _pct
FSA 1999-2021 90/10 from 2014; campus-based from 2001
MEPS 2009-2022 2023 empty on 2026-08-06 probe
NHGIS (schools / colleges) 1986-2023 / 1980-2023 Crosswalk file-years; census vintages 1990-2020
PSEO 2001-2021 Yearly shards
NACUBO / NCCS / EADA / Campus Safety 2012-2022 / 1993-2016 / 2002-2021 / 2005-2021 —

Grain Logic: What a Row Represents

A mirror file's grain is the set of columns that uniquely identify a row. Reading it correctly is the difference between a clean join and a silent fan-out.

  • Base grain is the entity ID + year (e.g., one row per ncessch × year in CCD directory).
  • Disaggregation dimensions multiply rows. In the live API these appear as URL path segments (.../enrollment/{year}/{grade}/race/sex/); in the mirror file the same dimensions are columns (grade, race, sex) and each combination is its own row. So CCD enrollment is one row per ncessch × year × grade × race × sex. To get a school total, filter to the 99/total code on each dimension rather than summing (summing double-counts the totals rows).
  • CRDC files carry disability and lep dimensions in addition to race/sex; a discipline file is one row per school × year × the disaggregation the file name advertises.
  • Yearly vs Single files. "Yearly" families have one file per year with {year} in the path (fetch each year, concatenate). "Single" families pack all years into one file (filter on year locally). The routing files mark each family Single or Yearly.
  • Coded totals. Categorical dimensions use 99 = Total and negative sentinels for missing (-1/-2/-3/-99) — see variable-codes.md. A "total" row and its component rows coexist in the same file.

What Discovery Should Emit (the acquisition currency)

Discovery output feeds planning, then acquisition. For each dataset a research question needs, emit:

  1. Canonical mirror path(s) — {source}/{filename}, extensionless, with {year} placeholder where the family is Yearly. This is the primary currency.
  2. Grain — what one row represents, including disaggregation dimensions.
  3. Years — the specific years needed, within the family's coverage window.
  4. Key variables — the real column names (from the routing file / dictionary) the analysis will use.
  5. Join keys — the ID(s) linking this file to the others in the plan.
  6. Caveats — load-bearing warnings (suppression, revaluation, scale, coverage gaps).

Portal endpoint routes are optional citation annotations only — never the hand-off artifact. Do not instruct downstream to probe, fetch, or call an API; naming the path and years is sufficient for education-data-query to construct the fetch.

Skill Purpose When
education-data-query Fetch code, mirror config, codebook URLs, path catalog After identifying paths/years/variables here
education-data-context Portal-wide interpretation (missing codes, grade encoding, licensing, joins) Before analyzing retrieved data
education-data-source-{ccd,ipeds,crdc,scorecard,saipe,edfacts,fsa,meps,nhgis,nccs,nacubo,eada,campus-safety,pseo} Deep source context: methodology, limitations, historical changes When you need collection methodology or source-specific caveats beyond routing

Load a education-data-source-* skill when a question needs collection methodology, known limitations, or historical coding changes for a specific source. Each source skill carries a skill-last-updated provenance date — flag stale ones to the orchestrator.

Worked Example: Planning a Discovery Output

Question: "Relationship between school poverty and AP course offerings in California high schools?"

  • Level / grain: schools — one row per ncessch × year.
  • Files:
  • crdc/schools_crdc_apib_enroll (Single; 2011-2021) — AP/IB enrollment. Key vars: enrl_ap, enrl_ap_math, enrl_ap_science, enrl_ib, enrl_gifted_talented. Grain adds race/sex/disability/lep columns — filter to totals for a school-level measure.
  • meps/schools_meps (Single; 2009-2022) — modeled poverty. Key var: meps_poverty_pct.
  • Join: ncessch + year; overlap window 2011-2021.
  • Filter: fips == 6 (California), high schools via CCD school_level == 3 (join ccd/schools_ccd_directory).
  • Emit: the three paths, the year window, the key variables, the ncessch join key, and the CRDC disaggregation-totals caveat. Then load education-data-query to fetch.

Common Pitfalls

  • Emitting a route instead of a path. Downstream needs crdc/schools_crdc_discipline_k12_{year}, not /schools/crdc/discipline/{year}/....
  • Summing disaggregated rows. Total rows (99) coexist with component rows — filter, don't sum.
  • Assuming a variable exists. If it is not in variable-dictionary-{source}.md, it is not in the mirror.
  • Ignoring the coverage window. Align year across families before joining; several have gaps (EDFacts 2019, IPEDS finance ends 2017).
  • Trusting fallback vintage. Facts are exact for the pinned mirror only; the urban_csv fallback is current-Portal.