GRF · Research Nexus
Research methods dialogue
Risk evidence comes from experiments, observational studies, models, case studies, expert judgement and lived experience, and each answers some questions well and others poorly. Methods dialogue is the comparison of those designs: what each assumes, where each fails and how evidence from different designs can be weighed together. This page sets out why methods choices carry governance consequences, how their quality can be checked, and who decides which evidence counts.
Research methods dialogue on the record.
Research methods dialogue: how the risk develops.
Comparative methods and limitations. The dossier below sets out how the risk develops, what is measured, where the data come from, how it is modelled and who decides.
Hierarchies are misapplied
Evidence hierarchies built for clinical questions are applied to policy problems where trials are impossible or unethical. Relevant evidence from other designs is discounted, and decisions wait for evidence that cannot exist.
Assumptions stay hidden
Every model and estimate rests on assumptions about data, mechanisms and context. When these go unstated, users cannot judge transfer, and disputes about results are really unacknowledged disputes about assumptions.
Flexibility inflates findings
Many analytic choices remain open after data are seen. Choosing among them by result produces findings that do not replicate, and the problem compounds across a literature that favours positive results.
Context does not travel
An effect estimated in one setting depends on its institutions, population and timing. Transferring it without stating those conditions misleads decisions elsewhere, especially when evidence moves between very different settings.
Research network directory.
The screen a team would use to follow this subject: its measures, states and decisions. Names are examples and values are illustrative.
Research network directory
Research methods dialogue · Research network A · Year 2
Collaboration graph
nodes: institutions · illustrative
Joint projects
| Compound risk methods | 4 | Active |
| Local knowledge in warnings | 5 | Proposal |
| Open data for risk | 10 | Reporting |
What is measured, and how well.
Each indicator carries its unit, its cadence, the kind of source behind it and the evidence label a value would carry. No values are shown: a programme supplies them under its own authority.
| Indicator | Unit | Cadence | Source type | Label |
|---|---|---|---|---|
| Studies with preregistered protocols | percent of studies on a theme | annual | Registry and publication records | OBSERVED |
| Replication and reproduction attempts | count per theme | periodic | Replication databases and journals | REPORTED |
| Outputs sharing analysis code | percent of outputs | annual | Publication and repository metadata | OBSERVED |
| Syntheses rated with a stated certainty framework | percent of syntheses | annual | Synthesis libraries | OBSERVED |
| Mix of study designs on a theme | count by design type | periodic | Evidence and gap maps | ESTIMATED |
| Studies reporting sensitivity analysis | percent of studies reviewed | periodic | Content review | ESTIMATED |
How research methods dialogue is governed: instruments, mandates and commitments.
The instruments, mandates, actors and accountability arrangements a policymaker, diplomat or development agency works with, citing the agreements that set them. Structures follow the frameworks named. Any state or progress drawn is illustrative, not a record.
Who holds which role
The actors around research methods dialogue: who decides, who implements, who finances and who brings knowledge and voice.
Where the evidence comes from.
Named public datasets and institutional records, each with its licence class. Sources by agreement are used only under their terms and never republished.
| Source | What it gives | Licence class |
|---|---|---|
| Campbell Collaboration evidence and gap maps | Study designs and coverage across social policy questions | Open access, attribution |
| Cochrane Handbook and methods guidance | Standard methods for systematic reviews and their known limitations | Free to read; reuse terms per item |
| GRADE working group guidance | A framework for rating the certainty of evidence and the strength of recommendations | Free to read, working group terms |
| OSF Registries | Preregistered protocols and registered reports | Open, terms of service |
| FORRT Replication Database | Records of replication attempts in the social and behavioural sciences | Open, terms per release |
| EQUATOR Network reporting guidelines | Reporting standards by study design, such as CONSORT and STROBE | Free to read, terms per guideline |
| OECD Global Science Forum reports | Policy analysis on reproducibility and research infrastructure | Free to read, OECD terms |
Who decides, and what the evidence serves.
Research Nexus prepares evidence. The decisions below are taken by the authorities named, under their own rules, and never by the platform.
| Who decides | The question | Evidence this page serves |
|---|---|---|
| Research funders | Whether to require preregistration, data and code sharing, or to fund replication | Comparative policies; evaluations of their effects with stated limits |
| Evidence synthesis and guideline bodies | Which study designs and certainty frameworks to use for a policy question | Methods comparisons; known failures of design hierarchies in policy settings |
| Government evaluation units and statistical offices | Which evaluation designs to require for public programmes | Methods dialogue records; feasibility and ethics of experimental and quasi-experimental designs |
| Journal editors | Whether to adopt registered reports and reporting guidelines | EQUATOR guidance; replication evidence by field |
Outputs this topic produces
- Methods seminars comparing designs on a shared risk question
- Peer learning briefs on weighing mixed evidence for decisions
- Dialogue records of methodological disagreements and their sources
- Plain-language notes on the assumptions behind common methods
What methods dialogue asks.
- What kind of claim can this method support: correlation, causation, mechanism, possibility?
- How do results change when a different but defensible method is used?
- How can quantitative and qualitative evidence be combined without losing the strengths of either?
- What are the known failure modes of the methods most used in this field?
Why methods comparison matters.
Analyses of the same data by different expert teams often reach different conclusions, because each team makes defensible choices about variables, models and assumptions. This variation is a real source of uncertainty that single studies hide. Comparing methods openly reveals it and helps users judge how robust a finding is.
Different methods also answer different questions. A randomised evaluation can show whether an intervention worked on average in one setting; a case study can show how and why it worked; a simulation can explore what might happen elsewhere. Confusion arises when one kind of evidence is asked to answer another kind of question.
Why now: disagreements about risk often turn on method: models, definitions and uncertainty treatments that differ between disciplines. Research Nexus convenes method dialogues that make those differences explicit, records them with their consequences for decisions, and can propose shared definitions to the Nexus Standards.
Practices for comparing methods.
Methods comparison is most useful when it is routine rather than reserved for disputes. Several practices make it so, each with its own cost.
- 01
Methods seminars
Sessions where researchers present and challenge each other's methods. Limits: depend on openness and good faith.
- 02
Multi-team analyses
Several teams analyse the same question independently. Limits: resource-intensive.
- 03
Causal inference frameworks
Explicit causal assumptions and identification strategies, from causal inference and impact evaluation. Limits: assumptions cannot always be tested.
- 04
Mixed-methods synthesis
Combining statistical and qualitative evidence, with probability and statistics. Limits: no single agreed approach.
How it is studied, and what the methods cannot say.
- 01Causal inference and impact evaluationEvaluating funder policy changes · Assessing knowledge brokerage schemesApplied here
- 02Probability and statisticsBuilding bibliometric indicators · Detecting selective reportingApplied here
- 03Policy, institutional and decision scienceComparing science advice systems · Mapping integrity proceduresApplied here
Questions and answers.
Is a randomised trial always the strongest evidence?
What is preregistration?
Why do methods disagree?
Is one method best for studying risk?
Why do expert teams reach different conclusions from the same data?
Can I present my methods at a seminar?
Are seminar records published?
Contribute evidence from this page.
Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
- EmailGateway GW-GRFSend it from this pageEvidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen depositDeposit it and submit to the communityPublish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsGlobal Risks ForumSee every way to contributeThe contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.
Bring a governance question to the Forum.
Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.
Legal documents.
Show all 18 legal documents
Maps, data and figures on this site are shown for context and general information. They are not advice, forecasts or warnings; the limits of the information are set out in the Disclaimers and Data Notices.