Building bibliometric indicators
Counts of outputs, co-authorship and retractions are normalised by field, year and output volume, with the errors from author and institution disambiguation estimated and reported beside each indicator.
GRF · Research Nexus
Statistics for research governance
Probability and statistics give research governance its measurement discipline: how bibliometric and survey indicators are built, how uncertainty is stated in advice, how selective reporting is detected, and how confident a synthesis can be. In this field they are applied to evidence about research systems, and the indicators carry their construction with them.
01Register
Statistics for research governance on the record.
Method recordSTEM method
- MethodProbability and statistics
- Applied inResearch Nexus
- Data4 inputs
- Checks4 validation steps
- Canonical pageSTEM Commons
02Application
What probability and statistics does here.
Probability and statistics provide the mathematical framework for describing uncertainty and learning from data. They underpin estimation, testing and prediction in every risk discipline. In research and evidence and governance and institutions, statistical methods quantify the likelihood and uncertainty of losses across interconnected systems. They make the evidence behind risk decisions explicit and reproducible. The canonical description is on the STEM Commons.
Detecting selective reporting
Distributions of reported test statistics across a literature are examined for patterns consistent with selective reporting, and outcomes registered in advance are compared with outcomes published.
Calibrated language for uncertainty
Verbal likelihood and confidence scales, such as those in IPCC guidance, are compared with how readers interpret them, to inform how advisers state probability in plain words.
03Monitor
Tail fit.
The screen a team would use to follow this method: its measures, states and decisions. Names are examples and values are illustrative.
Tail fit
Research · Statistics for research governance · Run R-0211
Shape parameter uncertainGRF · Research Nexus
Return level curve
Method card
- Method
- Probability and statistics
- Approach
- peaks over threshold
- Threshold
- 90th percentile
- Label
- MODELLED
Parameters
generalised Pareto fit
| Shape ξ | 0.27 | 0.00 | 0.49 | heavy tail if above zero |
| Scale σ | 4.7 | 1.0 | 5.8 | units of the variable |
| Exceedances | 45 | declustered |
04Data
What the method consumes.
Datasets and records used by probability and statistics on this platform, with the role each plays and the label its values carry. Sources are listed in the source register.
| Data | Role | Label |
|---|---|---|
| OpenAlex and Crossref metadata | Bibliometric counts and citation links | OBSERVED |
| R&D surveys under the Frascati Manual | Expenditure and personnel series | Published statistics |
| Trial and study registries | Planned outcomes against reported outcomes | OBSERVED |
| Surveys of researchers and institutions | Measures of practice and perception | REPORTED |
05Policy scenarios
Compare instruments: rules, incentives and information.
Three families of instruments side by side against the baseline, each with its own lag and adoption speed, under the coverage and compliance you set; effect sizes are teaching values to replace with evidence. Scenario: named assumptions about the future, not a forecast.
Controls
- Baseline trend of the outcome
- 2.0% a year
- Coverage of the instrument
- 70%
- Compliance
- 75%
- Effect size, against the teaching value
- 100%
- Years ahead
- 20 years
Readouts
- Largest reduction at the horizon
- Regulation
- Its reduction against the baseline
- 21%
Numbers, methods and sources
| Year | Baseline | Regulation | Economic incentive | Information and capacity |
|---|---|---|---|---|
| 0 | 100.0 | 99.5 | 98.8 | 99.1 |
| 1 | 102.0 | 101.2 | 100.1 | 99.8 |
| 2 | 104.0 | 102.5 | 101.1 | 99.9 |
| 3 | 106.1 | 103.5 | 101.6 | 100.0 |
| 4 | 108.2 | 103.7 | 101.8 | 100.7 |
| 5 | 110.4 | 103.2 | 101.7 | 102.1 |
| 6 | 112.6 | 102.0 | 101.6 | 103.9 |
| 7 | 114.9 | 100.4 | 101.6 | 105.9 |
| 8 | 117.2 | 99.2 | 102.1 | 108.0 |
| 9 | 119.5 | 98.6 | 102.9 | 110.1 |
| 10 | 121.9 | 98.9 | 104.2 | 112.3 |
| 11 | 124.3 | 99.7 | 105.7 | 114.5 |
| 12 | 126.8 | 101.1 | 107.4 | 116.8 |
| 13 | 129.4 | 102.7 | 109.4 | 119.2 |
| 14 | 131.9 | 104.5 | 111.4 | 121.6 |
| 15 | 134.6 | 106.5 | 113.5 | 124.0 |
| 16 | 137.3 | 108.5 | 115.7 | 126.5 |
| 17 | 140.0 | 110.7 | 118.0 | 129.0 |
| 18 | 142.8 | 112.9 | 120.4 | 131.6 |
| 19 | 145.7 | 115.1 | 122.8 | 134.2 |
| 20 | 148.6 | 117.4 | 125.2 | 136.9 |
Methods
- Tools of government Hood (1983), The Tools of Government, Macmillan
- Carrots, sticks and sermons Bemelmans-Videc, Rist and Vedung (1998), Transaction
- Diffusion of innovations Rogers (2003), Diffusion of Innovations, Free Press
- Uncertainty fan (parameter Monte Carlo) Spiegelhalter, Pearson and Short (2011), Visualising uncertainty about the future, Science 333
- One-at-a-time sensitivity (tornado) Saltelli and others (2008), Global Sensitivity Analysis: The Primer, Wiley
Data
The model runs on the teaching values set by the controls. To calibrate it for a place, read it against these admitted layers and series, which this page's map can draw:
- Borders Natural Earth
- Cities Natural Earth
Across the six pillars, led by Nexus Consortium
- Experiments Labs
- Built in Research Nexus in Foundry
- Tested at Research Nexus in Universe
- Codified in Data and Evidence in Standards
- Published in Research Nexus in Reports
- Taught in Faculty of Risk Science and Methods in Academy
06Validation
How results are checked.
Results are checked through diagnostics, cross-validation and sensitivity to assumptions. Independent replication confirms findings.
| Check | How it is done |
|---|---|
| Normalisation stated | Every comparative indicator states its field and time normalisation; raw counts are not compared across fields |
| Coverage audit | Index coverage by language, region and output type is reported before any comparison is made |
| Uncertainty reported | Estimates carry intervals from resampling or a stated sampling error |
| Multiple comparisons | Analyses that test many indicators adjust for multiplicity or say that they did not |
Inference depends on assumptions about independence, distribution and sampling that may not hold. Small or biased samples limit reliability.
07Implementation
From mandate to evaluation, by pillar.
The steps an instrument takes from mandate to evaluation, each with the pillar that runs it, and the decisions that stay with the authority. Teaching model with stated illustrative values, not a prediction.
- Mandate and problem framing Competent authority (outside the programme)
A competent authority frames the problem and the mandate under its own rules. Howlett, Ramesh and Perl (2009) - Evidence and system model Foundry
The system is modelled with its feedbacks, data and stated limits. OECD (2014), principle 2 - Options appraised with scenarios Foundry
Instruments are compared under scenarios, with uncertainty shown. Hood (1983) - Exercise with stakeholders Universe
The options are tested in an exercise with those who would implement and bear them. OECD (2014), principle 3 - Standard or guidance drafted Nexus Standards
What works is codified as a standard or guidance, open to comment. Nexus Standards process - Decision by the authority Competent authority (outside the programme)
The authority decides, outside the programme, with the evidence on record. Howlett, Ramesh and Perl (2009) - Capacity and training Risk Academy
Those who implement are trained on the decided instrument and its data. Sendai Framework (2015), priority 2 - Monitoring and public report Reports
Indicators are published with sources, limits and corrections. OECD (2014), principle 5 - Evaluation and revision Nexus Standards
Results return to the standard and the model: the cycle closes. Howlett, Ramesh and Perl (2009)
Methods
- Policy cycle Howlett, Ramesh and Perl (2009), Studying Public Policy, Oxford University Press
- Governance of critical risks OECD (2014), Recommendation of the Council on the Governance of Critical Risks
- Layered drawing of directed graphs Sugiyama, Tagawa and Toda (1981), IEEE SMC 11
Across the six pillars, led by Nexus Consortium
- Experiments Labs
- Built in Research Nexus in Foundry
- Tested at Research Nexus in Universe
- Codified in Data and Evidence in Standards
- Published in Research Nexus in Reports
- Taught in Faculty of Risk Science and Methods in Academy
09Instruments and commitments
How statistics for research governance is governed: instruments, mandates and commitments.
The instruments, mandates, actors and accountability arrangements a policymaker, diplomat or development agency works with, citing the agreements that set them. Structures follow the frameworks named. Any state or progress drawn is illustrative, not a record.
Who holds which role
The actors around statistics for research governance: who decides, who implements, who finances and who brings knowledge and voice.
Sendai Framework for Disaster Risk Reduction · UNDRRParis Agreement · UNFCCC2030 Agenda and the Sustainable Development Goals · United NationsInternational Health Regulations · WHO
10Stewardship
Who maintains the method.
Systems Science and Modelling stewards probability and statistics for the Consortium and reviews how it is applied across platforms.
11Connected
Where this page leads across the Consortium.
12Questions
Questions and answers.
What are the limits of probability and statistics in this field?
Inference depends on assumptions about independence, distribution and sampling that may not hold. Small or biased samples limit reliability.
How are results checked?
Results are checked through diagnostics, cross-validation and sensitivity to assumptions. Independent replication confirms findings.
Are the values on this page real?
The data names, roles and checks are real descriptions of practice. The canvas shows a design of the method's use, not a run; no result is reported here.
Where else is this method applied?
The canonical method page lists every platform that applies it.
ReportsContribute
Contribute evidence from this page.
Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
Global Risks ForumResearch NexusEvidence BriefUniverse Annual Synthesis
- EmailGateway GW-GRFSend it from this pageEvidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen depositDeposit it and submit to the communityPublish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsGlobal Risks ForumSee every way to contributeThe contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.
Research Nexus
Bring a governance question to the Forum.
Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.
Legal
Legal documents.
Show all 18 legal documents
Maps, data and figures on this site are shown for context and general information. They are not advice, forecasts or warnings; the limits of the information are set out in the Disclaimers and Data Notices.