The Global Risks Forum (GRF)
Risk evidence comes from experiments, observational studies, models, case studies, expert judgement and lived experience, and each answers some questions well and others poorly. Methods dialogue is the comparison of those designs: what each assumes, where each fails and how evidence from different designs can be weighed together. This page sets out why methods choices carry governance consequences, how their quality can be checked, and who decides which evidence counts.
GRF · Established · service level R0
Research methods dialogue is the comparison of study designs, from experiments and observational studies to models, case studies and expert judgement: what each assumes, where each fails and how evidence from different designs can be weighed together. Choices of method carry governance consequences, because they decide which evidence counts.
For research methods dialogue, in GRF's situational-awareness reading, the map opens on true-colour imagery, population density and flood water with clouds now from NASA GIBS, framed on Europe; 3 further NASA layers wait in the list.
The interactive map needs JavaScript; the layer list and sources are below.
Layers 3
Hazards and exposure, for this page
Keyboard, with the map focused
Hold Ctrl or ⌘ and scroll to zoom
Latest available
NASA GIBS / EOSDIS · ASTER GDEM (METI, NASA) · Blue Marble (NASA) · © OpenStreetMap contributors (ODbL) · Natural Earth
Loading the latest NASA imagery.
Drag to move. Ctrl or ⌘ and scroll, pinch or the buttons to zoom; hover for the values under the pointer; reset returns to the region.
For research methods dialogue, in GRF's situational-awareness reading, the map opens on true-colour imagery, population density and flood water with clouds now from NASA GIBS, framed on Europe; 3 further NASA layers wait in the list.Satellite and model products as NASA publishes them, each with its own date; indicators, not assessments of any place; claim ceiling INTEGRATED.
Layers and dates
The working screen behind this subject: its structure, terms and states. Names are placeholders and values illustrative.
Research methods dialogue · Research network A · Year 2
nodes: institutions · placeholder
| Compound risk methods | 4 | Active |
| Local knowledge in warnings | 5 | Proposal |
| Open data for risk | 10 | Reporting |
Comparative methods and limitations. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Evidence hierarchies built for clinical questions are applied to policy problems where trials are impossible or unethical. Relevant evidence from other designs is discounted, and decisions wait for evidence that cannot exist.
Every model and estimate rests on assumptions about data, mechanisms and context. When these go unstated, users cannot judge transfer, and disputes about results are really unacknowledged disputes about assumptions.
Many analytic choices remain open after data are seen. Choosing among them by result produces findings that do not replicate, and the problem compounds across a literature that favours positive results.
An effect estimated in one setting depends on its institutions, population and timing. Transferring it without stating those conditions misleads decisions elsewhere, especially when evidence moves between very different settings.
Each term as this page uses it. Where a standard-setting body defines the term, the wording follows that body.
| Term | Definition |
|---|---|
| Preregistration | Recording a study's hypotheses, design and analysis plan in a public registry before data are collected or analysed. |
| Reproducibility | Obtaining the same results from the same data and code; replicability means obtaining consistent results from new data. |
| Certainty of evidence | A judgement of how confident one can be in a body of evidence, made with a stated framework such as GRADE. |
| Sensitivity analysis | Repeating an analysis under different assumptions to show how much the result depends on them. |
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Label |
|---|---|---|---|---|
| Studies with preregistered protocols | percent of studies on a theme | annual | Registry and publication records | OBSERVED |
| Replication and reproduction attempts | count per theme | periodic | Replication databases and journals | REPORTED |
| Outputs sharing analysis code | percent of outputs | annual | Publication and repository metadata | OBSERVED |
| Syntheses rated with a stated certainty framework | percent of syntheses | annual | Synthesis libraries | OBSERVED |
| Mix of study designs on a theme | count by design type | periodic | Evidence and gap maps | ESTIMATED |
| Studies reporting sensitivity analysis | percent of studies reviewed | periodic | Content review | ESTIMATED |
The topic as a computational graph: named sources feed the methods, an evidence review gates the outputs, and the decisions stay with the authorities that hold them.
Fig. 02Nodes carry their type, two metric rows and a state; connectors show what feeds what; dashed frames name the owners; the timeline is the cadence. Design of the pipeline, not a record of a run.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Source | What it gives | Licence class |
|---|---|---|
| Campbell Collaboration evidence and gap maps | Study designs and coverage across social policy questions | Open access, attribution |
| Cochrane Handbook and methods guidance | Standard methods for systematic reviews and their known limitations | Free to read; reuse terms per item |
| GRADE working group guidance | A framework for rating the certainty of evidence and the strength of recommendations | Free to read, working group terms |
| OSF Registries | Preregistered protocols and registered reports | Open, terms of service |
| FORRT Replication Database | Records of replication attempts in the social and behavioural sciences | Open, terms per release |
| EQUATOR Network reporting guidelines | Reporting standards by study design, such as CONSORT and STROBE | Free to read, terms per guideline |
| OECD Global Science Forum reports | Policy analysis on reproducibility and research infrastructure | Free to read, OECD terms |
Research Nexus prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Research funders | Whether to require preregistration, data and code sharing, or to fund replication | Comparative policies; evaluations of their effects with stated limits |
| Evidence synthesis and guideline bodies | Which study designs and certainty frameworks to use for a policy question | Methods comparisons; known failures of design hierarchies in policy settings |
| Government evaluation units and statistical offices | Which evaluation designs to require for public programmes | Methods dialogue records; feasibility and ethics of experimental and quasi-experimental designs |
| Journal editors | Whether to adopt registered reports and reporting guidelines | EQUATOR guidance; replication evidence by field |
Analyses of the same data by different expert teams often reach different conclusions, because each team makes defensible choices about variables, models and assumptions. This variation is a real source of uncertainty that single studies hide. Comparing methods openly reveals it and helps users judge how robust a finding is.
Different methods also answer different questions. A randomized evaluation can show whether an intervention worked on average in one setting; a case study can show how and why it worked; a simulation can explore what might happen elsewhere. Confusion arises when one kind of evidence is asked to answer another kind of question.
Methods comparison is most useful when it is routine rather than reserved for disputes. Several practices make it so, each with its own cost.
Sessions where researchers present and challenge each other's methods. Limits: depend on openness and good faith.
Several teams analyse the same question independently. Limits: resource-intensive.
Explicit causal assumptions and identification strategies, from causal inference and impact evaluation. Limits: assumptions cannot always be tested.
Combining statistical and qualitative evidence, with probability and statistics. Limits: no single agreed approach.
The current service level is R0: information and enquiries. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.