# Public-Data Evidence Engineering Method

Version: `1.0.0`  
Schema: `f01-public-data-v1.0.0`  
Use case: decide whether a public-data claim, table, chart, or model is safe to cite and can be regenerated from frozen evidence.

## The core rule

A public number is not identified by its value alone. Its identity is the combination of:

`publisher × dataset/series × concept × universe × geography × observation period × frequency × unit × adjustment/price basis × vintage/release state`

Two values conflict only after this concept signature has been shown to match. If the signature differs, the pair is either transformable with an explicit loss profile or not comparable. “Both are official” is not a reconciliation rule.

## Evidence classes

Keep the epistemic status of each statement visible:

| class | meaning | permitted use |
| --- | --- | --- |
| `official_disclosure` | a publisher explains its release, revision, method, or scenario policy | mechanism and interpretation claims within the page's scope and date |
| `primary_technical` | a first-party data file, workbook, dictionary, handbook, or methodology | observations, definitions, and transformations directly represented there |
| `direct_observation` | a reproducible measurement made from the scoped primary source | descriptive claims, with locators and collection constraints |
| `derived_analysis` | a deterministic transform of saved inputs | claims supported by the transform registry and parent lineage |
| `controlled_fixture` | a declared synthetic case designed to test a rule | method behavior, never empirical prevalence |
| `inference` | an interpretation that combines evidence | only with assumptions, counterevidence, confidence, and falsifier |
| `hypothesis` | a falsifiable claim not yet established | experiment planning, never a finding |
| `unknown` | not disclosed or not resolved | must remain missing; do not fill with a plausible story |

Current source rows are in `SOURCE_LEDGER.csv`; material conclusions and their epistemic fields are in `CLAIM_LEDGER.ndjson`.

## End-to-end workflow

### 1. Freeze the question before the outcome

Write the target concept, decision rule, observation grain, inclusion/exclusion rules, time window, planned comparisons, stopping rule, missingness policy, confounders, and access constraints. Assign a schema ID and version the sampling frame. If collection forces a change, record the change before inspecting the replacement outcome distribution.

This run froze four bundles: BLS QCEW national revisions, Census state/DC population vintages, BEA real-GDP estimate stages, and EIA AEO2026 scenario cases. The BLS frame changed once, from an inaccessible linked CSV to source-located official HTML tables, without spoofing a user agent or bypassing the 403 response. The change is recorded in `RESEARCH_PLAN.md`.

### 2. Freeze and verify inputs

For every native file record its URL, publisher, publication/update date, retrieval time, byte size, response metadata where useful, and SHA-256 hash. Analysis must stop on a hash mismatch. Never silently replace a frozen input with a newer download.

`ANALYSIS/processed/RAW_MANIFEST.csv` contains the six frozen-file checks. Large primary files remain in `OBSERVATIONS/raw/`; rights-safe excerpts retain source table and row locators.

### 3. Profile before interpretation

Check row and column counts, key uniqueness, types, missing states, ranges, source-hash coverage, overlap, and schema drift. This is where the run found that the two Census files contain 75 versus 97 columns even though the named `POPESTIMATE2020`–`POPESTIMATE2024` fields and 51 state/DC keys align. It also found that EIA metadata declares `start=2024`, while each selected data vector contains only 2025–2050.

A quality failure is an analytical result, not an invitation to repair the source invisibly. The EIA pipeline therefore contains no fabricated 2024 observation.

### 4. Apply the semantic compatibility gate

Compare these fields before joining values:

1. concept and population/universe;
2. geography and boundary version;
3. reference instant or interval;
4. frequency and aggregation role (stock, flow, average, end-of-period);
5. unit and scale;
6. seasonal adjustment and annualization;
7. nominal/real price basis and reference year;
8. source method/base and release state.

Classify each proposed comparison:

- `semantically_compatible`: the required fields match or an explicit reversible transform resolves them;
- `compatible_with_loss`: reconciliation is useful but discards information or requires a declared approximation;
- `not_comparable`: a material difference remains.

The executable decision examples are in `semantic_compatibility_matrix.csv`. Examples: Census V2024 and V2025 July 1 state estimates are comparable as vintages; population and covered employment are not one measure; BEA real GDP growth and nominal GDP levels are not interchangeable; an EIA model output must not be spliced to history without a handoff rule.

### 5. Reconcile conflicts without erasing them

Use this precedence sequence, in order:

1. **Target-concept fit.** Choose the source that measures the declared target, not the source with the most prestige or newest webpage.
2. **Same-grain comparability.** Reject or transform mismatched universe, timing, unit, adjustment, or geography.
3. **Publisher authority for that concept.** Prefer the originating first-party series over an aggregator when both claim the same field.
4. **Release role.** For a real-time question use the release available at that date; for a best-current-estimate question use the latest eligible vintage. Never substitute one for the other without changing the question.
5. **Lineage and quality.** Prefer a value with verifiable source row, hash, method, and transformation path; unresolved conflicts stay side by side.

Retain every legitimate release as a separate canonical observation. Link matched releases with a deterministic `pair_id`. A later value does not overwrite the earlier value, and a difference is not automatically an error: it may reflect later records, benchmarking, or a revised method.

### 6. Register transformations

Every derived value needs a versioned transform ID, formula, input/output units, guardrail, and parent IDs. The current registry includes raw identity, revision deltas, percentage-point revision to a rate, percent-of-level revision, period growth, rebasing, deterministic ranking, missingness bounds, scenario compounding, and scenario spread.

Important guardrails:

- subtract rates in percentage points; do not divide by a near-zero earlier rate;
- compute growth only from comparable adjacent periods and a nonzero denominator;
- rank only inside a frozen frame, with an explicit tie rule;
- do not add chain-type real components unless the source method permits it;
- retain the separately published BLS delta when rounded displayed levels differ by $1 million;
- never zero-fill simply to make a column numeric.

`lineage_edges.csv` makes parentage queryable. Canonical observation IDs and pair/model IDs use stable SHA-256 prefixes over canonical key fields.

### 7. Measure revision risk at the decision layer

Report both magnitude and conclusion sensitivity. For matched earlier/later values `x0` and `x1`:

- absolute revision: `x1 - x0`;
- level-relative revision: `100 × (x1 - x0) / x0` when `x0 != 0`;
- growth revision: `growth1 - growth0` in percentage points;
- revision-to-change ratio: `abs(growth1 - growth0) / max(abs(growth0), declared floor)`;
- sign flip: the earlier and later changes have different signs;
- threshold flip: a declared decision threshold changes truth value;
- rank flip: membership in a declared rank set changes.

Use medians, quartiles, maxima, named outliers, and denominators. Do not infer population frequencies or print p-values from a convenience case-study frame. `revision_summary.csv`, `conclusion_fragility.csv`, and the bundle-specific metrics contain these diagnostics.

### 8. Preserve missingness as data

Valid states include `observed`, `publisher_missing`, `suppressed`, `not_applicable`, `not_yet_published`, `unavailable_at_retrieval`, and `parse_error`. Zero is an observed numeric value, never a generic sentinel.

For an aggregate, declare the estimand and denominator. Compare at least:

- complete-case treatment;
- any defensible lower/upper bounds;
- zero-fill only as an explicitly labeled stress test, never the default.

The controlled fixture shows why: the complete-case mean is 43.75, the documented bounded mean is 26.43–59.29, and naive zero-fill produces 25—below the documented lower bound. `not_applicable` is excluded from the denominator.

### 9. Keep uncertainty layers separate

Do not collapse unlike uncertainty into one unlabeled ribbon:

| layer | question | representation |
| --- | --- | --- |
| observation status | was a value published and usable? | explicit state/coverage table |
| vintage/revision | how did legitimate releases of the same observation differ? | matched deltas and empirical revision distribution |
| statistical measurement | what sampling/model uncertainty did the publisher quantify? | publisher-defined interval, if available |
| scenario/assumption | what changes under named assumptions? | separate conditional paths or sensitivity grid |
| model form/structural | what plausible mechanisms are omitted? | alternative model specification or explicit unknown |

An empirical revision range is not automatically a confidence interval. Scenario extremes are not quantiles unless probabilities exist.

### 10. Build scenarios, not disguised forecasts

State the starting vintage, every varied assumption, constant assumptions, calculation horizon, and decision threshold. Provide low/base/high only when the labels correspond to named input changes. Report the conditional result as “if these assumptions hold,” not “this will happen.”

The population fixture combines two starting vintages with rates equal to V2025's published 2023–2024 national growth and plus/minus 0.5 percentage point. It is deliberately labeled `derived_analysis_scenario_not_forecast`. EIA's Counterfactual Baseline is preserved as an experimental control; its high and low economic-growth cases have no published probability and are not a confidence band.

### 11. Contract every chart

Before plotting, write the question, takeaway, grain, fields, scale, unit, sufficiency rule, and output. Every exported figure must have:

- a visible title, subtitle, unit, source/method note, and honest zero/reference line where material;
- a backing processed table and source lineage;
- non-color encoding or direct labels in addition to color;
- deterministic dimensions and output bytes;
- an SVG `<title>` and `<desc>` for accessibility;
- full-size and reduced-width visual inspection for clipping, collision, and legibility.

`FIGURES/CHART_CONTRACTS.md` records the preregistered contracts. `FIGURE_MANIFEST.csv` binds each figure to its table and SHA-256 hash.

## Reproducibility

From the artifact root:

```bash
python3 ANALYSIS/reproduce.py
python3 ANALYSIS/determinism_check.py
python3 ANALYSIS/verify.py
```

`reproduce.py` performs the audited XLSX extraction and then runs the network-free standard-library analysis. It uses `node` on `PATH`, the bundled runtime path recorded in the worklog, or `F01_NODE_BIN`. The JavaScript extraction requires `@oai/artifact-tool`; this run used the prebundled workspace dependency exposed through the isolated run-root `node_modules` link. It does not download a package.

`run_pipeline.py` independently verifies every frozen raw hash before reading outcomes. `determinism_check.py` runs the complete extraction and analysis twice and compares every processed CSV/JSON and SVG byte-for-byte. `verify.py` checks schema, counts, keys, lineage, transforms, source/claim ledgers, article packets, figures, hypotheses, and acceptance artifacts.

## Limits of the demonstrated evidence

The four bundles are purposive case studies, not a random sample of public statistics. Their measured flip rates describe these exact frames and rules, not the prevalence of revision risk across government data. BLS evidence is a 36-row national excerpt because the linked CSV returned 403; it is not a state-level census of revisions. Census vintages combine new inputs and method changes, so the run does not causally attribute each state revision. BEA “latest available” is not always final. EIA scenarios do not cover every structural uncertainty and their bulk-record metadata disagrees with the data vector's first year. The population fixture tests scenario mechanics, not demographic forecast accuracy.

The method is reusable precisely because these limits remain part of the evidence product rather than disappearing from the chart.