Staggered adoption of a reform
Where jurisdictions adopt a rule at different times, difference-in-differences designs with estimators robust to staggered timing compare adopters with those not yet treated, after testing for pre-existing trends.
GRF · Policy Nexus
Causal inference for public policy
Causal inference and impact evaluation ask whether a policy changed outcomes, and by how much relative to what would have happened without it. In this field they separate the effect of laws, programmes and institutional reforms from everything else that changed, and state the assumptions each estimate depends on.
01Contents
Causal inference for public policy in five parts.
02Application
What causal inference and impact evaluation does here.
Causal inference and impact evaluation develop methods to determine whether and how interventions or exposures cause outcomes. They separate genuine effects from correlation. In policy, governance and institutions and risk management and systemic risk, causal inference shows which risk reduction measures actually work. It supports allocating resources toward interventions with demonstrated effect across connected risks. The canonical description is on the STEM Commons.
Eligibility thresholds in programmes
Where a programme uses a score or population cut-off, regression discontinuity compares units just either side of the threshold, where they are otherwise similar.
Synthesis across evaluations
Systematic reviews collect evaluations of a policy type, appraise their designs and combine compatible estimates, reporting variation by context rather than a single average.
03Monitor
Impact evaluation.
The screen a team would use to follow this method: its measures, states and decisions. Names are examples and values are illustrative.
Impact evaluation
Policy · Causal inference for public policy · Run R-4547
Parallel trends holdGRF · Policy Nexus
Event study
Method card
- Method
- Causal inference and impact evaluation
- Design
- difference in differences
- Units
- 291
- Label
- MODELLED
Robustness checks
| Main specification | 0.13 | Yes |
| Drop early adopters | 0.23 | Yes |
| Synthetic control | 0.16 | Yes |
04Data
What the method consumes.
Datasets and records used by causal inference and impact evaluation on this platform, with the role each plays and the label its values carry. Sources are listed in the source register.
| Data | Role | Label |
|---|---|---|
| Administrative records | Outcomes and timing of treatment | Published statistics |
| Panel surveys | Household and firm outcomes | ESTIMATED |
| Policy adoption databases | Dates and design of reforms | Published statistics |
| Evaluation repositories | Prior estimates for synthesis | REPORTED |
05Policy scenarios
Compare instruments: rules, incentives and information.
Three families of instruments side by side against the baseline, each with its own lag and adoption speed, under the coverage and compliance you set; effect sizes are teaching values to replace with evidence. Scenario: named assumptions about the future, not a forecast.
Controls
- Baseline trend of the outcome
- 2.0% a year
- Coverage of the instrument
- 70%
- Compliance
- 75%
- Effect size, against the teaching value
- 100%
- Years ahead
- 20 years
Readouts
- Largest reduction at the horizon
- Regulation
- Its reduction against the baseline
- 21%
Numbers, methods and sources
| Year | Baseline | Regulation | Economic incentive | Information and capacity |
|---|---|---|---|---|
| 0 | 100.0 | 99.5 | 98.8 | 99.1 |
| 1 | 102.0 | 101.2 | 100.1 | 99.8 |
| 2 | 104.0 | 102.5 | 101.1 | 99.9 |
| 3 | 106.1 | 103.5 | 101.6 | 100.0 |
| 4 | 108.2 | 103.7 | 101.8 | 100.7 |
| 5 | 110.4 | 103.2 | 101.7 | 102.1 |
| 6 | 112.6 | 102.0 | 101.6 | 103.9 |
| 7 | 114.9 | 100.4 | 101.6 | 105.9 |
| 8 | 117.2 | 99.2 | 102.1 | 108.0 |
| 9 | 119.5 | 98.6 | 102.9 | 110.1 |
| 10 | 121.9 | 98.9 | 104.2 | 112.3 |
| 11 | 124.3 | 99.7 | 105.7 | 114.5 |
| 12 | 126.8 | 101.1 | 107.4 | 116.8 |
| 13 | 129.4 | 102.7 | 109.4 | 119.2 |
| 14 | 131.9 | 104.5 | 111.4 | 121.6 |
| 15 | 134.6 | 106.5 | 113.5 | 124.0 |
| 16 | 137.3 | 108.5 | 115.7 | 126.5 |
| 17 | 140.0 | 110.7 | 118.0 | 129.0 |
| 18 | 142.8 | 112.9 | 120.4 | 131.6 |
| 19 | 145.7 | 115.1 | 122.8 | 134.2 |
| 20 | 148.6 | 117.4 | 125.2 | 136.9 |
Methods
- Tools of government Hood (1983), The Tools of Government, Macmillan
- Carrots, sticks and sermons Bemelmans-Videc, Rist and Vedung (1998), Transaction
- Diffusion of innovations Rogers (2003), Diffusion of Innovations, Free Press
- Uncertainty fan (parameter Monte Carlo) Spiegelhalter, Pearson and Short (2011), Visualising uncertainty about the future, Science 333
- One-at-a-time sensitivity (tornado) Saltelli and others (2008), Global Sensitivity Analysis: The Primer, Wiley
Data
The model runs on the teaching values set by the controls. To calibrate it for a place, read it against these admitted layers and series, which this page's map can draw:
- Borders Natural Earth
- Cities Natural Earth
Across the six pillars, led by Nexus Consortium
- Experiments Labs
- Built in Policy Nexus in Foundry
- Tested at Policy Nexus in Universe
- Codified in Foundations and Governance in Standards
- Published in Policy Nexus in Reports
- Taught in Faculty of Governance, Policy and International Affairs in Academy
06Validation
How results are checked.
Identification assumptions are probed through placebo tests, falsification checks and sensitivity analysis. Replication across settings tests external validity.
| Check | How it is done |
|---|---|
| Pre-trend and placebo tests | Outcomes before adoption, and outcomes the policy cannot affect, are tested for spurious effects |
| Sensitivity to unobserved confounding | Bounds show how strong a hidden confounder would need to be to remove the estimated effect |
| Pre-registration | Hypotheses and analysis plans are registered before outcome data are examined, where feasible |
| External validity review | Context, population and implementation are described so that transfer to another setting can be judged |
Unobserved confounding and spillovers can bias estimates. Effects found in one context may not transfer to another.
07Implementation
From mandate to evaluation, by pillar.
The steps an instrument takes from mandate to evaluation, each with the pillar that runs it, and the decisions that stay with the authority. Teaching model with stated illustrative values, not a prediction.
- Mandate and problem framing Competent authority (outside the programme)
A competent authority frames the problem and the mandate under its own rules. Howlett, Ramesh and Perl (2009) - Evidence and system model Foundry
The system is modelled with its feedbacks, data and stated limits. OECD (2014), principle 2 - Options appraised with scenarios Foundry
Instruments are compared under scenarios, with uncertainty shown. Hood (1983) - Exercise with stakeholders Universe
The options are tested in an exercise with those who would implement and bear them. OECD (2014), principle 3 - Standard or guidance drafted Nexus Standards
What works is codified as a standard or guidance, open to comment. Nexus Standards process - Decision by the authority Competent authority (outside the programme)
The authority decides, outside the programme, with the evidence on record. Howlett, Ramesh and Perl (2009) - Capacity and training Risk Academy
Those who implement are trained on the decided instrument and its data. Sendai Framework (2015), priority 2 - Monitoring and public report Reports
Indicators are published with sources, limits and corrections. OECD (2014), principle 5 - Evaluation and revision Nexus Standards
Results return to the standard and the model: the cycle closes. Howlett, Ramesh and Perl (2009)
Methods
- Policy cycle Howlett, Ramesh and Perl (2009), Studying Public Policy, Oxford University Press
- Governance of critical risks OECD (2014), Recommendation of the Council on the Governance of Critical Risks
- Layered drawing of directed graphs Sugiyama, Tagawa and Toda (1981), IEEE SMC 11
Across the six pillars, led by Nexus Consortium
- Experiments Labs
- Built in Policy Nexus in Foundry
- Tested at Policy Nexus in Universe
- Codified in Foundations and Governance in Standards
- Published in Policy Nexus in Reports
- Taught in Faculty of Governance, Policy and International Affairs in Academy
09Instruments and commitments
How causal inference for public policy is governed: instruments, mandates and commitments.
The instruments, mandates, actors and accountability arrangements a policymaker, diplomat or development agency works with, citing the agreements that set them. Structures follow the frameworks named. Any state or progress drawn is illustrative, not a record.
Policy instrument matrix
The kinds of instrument available on causal inference for public policy, by level of government and cooperation.
| Instrument type | Local | National | Regional | Global |
|---|---|---|---|---|
| Regulatory | Zoning, building codes | Laws, standards, licensing | Harmonised rules | Conventions and treaties |
| Economic | Local levies, grants | Budget lines, taxes, insurance schemes | Risk pools, regional funds | Financing mechanisms |
| Information | Local warnings | Disclosure, early warning, statistics | Data sharing arrangements | Global monitoring and reporting |
| Institutional | Local committees | Lead agencies, coordination councils | Regional bodies and secretariats | Multilateral mandates |
| Cooperative | Community agreements | Partnership compacts | Memoranda and joint programmes | Agreements and joint commitments |
Sendai Framework for Disaster Risk Reduction · UNDRRParis Agreement · UNFCCC2030 Agenda and the Sustainable Development Goals · United NationsInternational Health Regulations · WHO
10Stewardship
Who maintains the method.
Economic Development stewards causal inference and impact evaluation for the Consortium and reviews how it is applied across platforms.
11Connected
Where this page leads across the Consortium.
12Questions
Questions and answers.
What are the limits of causal inference and impact evaluation in this field?
Unobserved confounding and spillovers can bias estimates. Effects found in one context may not transfer to another.
How are results checked?
Identification assumptions are probed through placebo tests, falsification checks and sensitivity analysis. Replication across settings tests external validity.
Are the values on this page real?
The data names, roles and checks are real descriptions of practice. The canvas shows a design of the method's use, not a run; no result is reported here.
Where else is this method applied?
The canonical method page lists every platform that applies it.
ReportsContribute
Contribute evidence from this page.
Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
Global Risks ForumPolicy NexusEvidence BriefPortfolio and Readiness
- EmailGateway GW-GRFSend it from this pageEvidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen depositDeposit it and submit to the communityPublish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsGlobal Risks ForumSee every way to contributeThe contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.
Policy Nexus
Bring a governance question to the Forum.
Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.
Legal
Legal documents.
Show all 18 legal documents
Maps, data and figures on this site are shown for context and general information. They are not advice, forecasts or warnings; the limits of the information are set out in the Disclaimers and Data Notices.