Classifier error in moderation
Moderation classifiers are evaluated for false positive and false negative rates by language and group, showing who is over- or under-moderated.
GRF · Information Integrity and Public Trust
Artificial intelligence and machine learning for information integrity and public trust
Machine learning powers recommender systems, content moderation, generative AI and detection tools. In information integrity it is both a cause of risk and a method for studying it, so its limits, errors and biases are made explicit wherever it is used.
01Register
Key facts.
Method recordSTEM method
- MethodArtificial intelligence and machine learning
- Applied inInformation Integrity and Public Trust
- Data4 inputs
- Checks3 validation steps
- Canonical pageSTEM Commons
02Application
What artificial intelligence and machine learning does here.
Artificial intelligence and machine learning develop algorithms that learn patterns from data to classify, predict or generate outputs. They are increasingly used for hazard detection, forecasting and decision support, and are themselves a source of risk. In information and public trust and society and human development, machine learning can scale detection and forecasting across interconnected systems. Its own failure modes must be weighed when automated outputs inform consequential decisions. The canonical description is on the STEM Commons.
Synthetic media detection limits
Detection tools are tested on edited and compressed media to show where they fail, before any reliance on their outputs.
Recommender audits
Sock-puppet and data-donation audits estimate how ranking systems expose users to different sources over time.
03Monitor
Model training.
The screen a team would use to follow this method: its measures, states and decisions. Names are examples and values are illustrative.
Model training
Information integrity · Artificial intelligence · Run R-2811
Drift on one sliceGRF · Information Integrity and Public Trust
Evaluation by slice
| All | 0.78 | 0.02 | 8592 | Pass |
| Region A | 0.75 | 0.06 | 1645 | Pass |
| Region B | 0.78 | 0.10 | 843 | Review |
| Rare class | 0.58 | 0.07 | 78 | Below bar |
Method card
- Method
- Artificial intelligence and machine learning
- Split
- time-based, no leakage
- Features
- 22
- Label
- MODELLED
Population stability
04Data
What the method consumes.
Datasets and records used by artificial intelligence and machine learning on this platform, with the role each plays and the label its values carry. Sources are listed in the source register.
| Data | Role | Label |
|---|---|---|
| Labelled content datasets | Training and evaluation | REPORTED |
| Platform transparency data | Moderation outcomes | REPORTED |
| Audit and data donation records | Exposure evidence | OBSERVED |
| Benchmark results | Detector accuracy | ESTIMATED |
05Decision model
What tips a population into acting.
How the share of early movers and the thresholds people hold decide whether a protective behaviour, or a false narrative, stays local or spreads to most people. Teaching model with stated illustrative values, not a prediction.
Controls
- Mean threshold (share of neighbours)
- 18%
- Spread of thresholds
- 8 points
- Early movers
- 2.0% of people
- Contacts per person
- 8
Readouts
- Share who acted in the end
- 100%
- Global cascade
- Yes
- Steps to settle
- 9
Numbers, methods and sources
| Step | Share who have acted (%) |
|---|---|
| 0 | 2.0 |
| 1 | 8.3 |
| 2 | 19.3 |
| 3 | 35.5 |
| 4 | 55.8 |
| 5 | 76.8 |
| 6 | 92.0 |
| 7 | 98.8 |
| 8 | 100.0 |
| 9 | 100.0 |
Methods
- Threshold models of collective behaviour Granovetter (1978), American Journal of Sociology 83
- Global cascades on random networks Watts (2002), PNAS 99
- Complex contagions Centola and Macy (2007), American Journal of Sociology 113
- One-at-a-time sensitivity (tornado) Saltelli and others (2008), Global Sensitivity Analysis: The Primer, Wiley
Data
No observed data enters this model: it runs on the teaching values set by the controls, to be replaced with a programme's own evidence.
Across the six pillars, led by Nexus Consortium
- Experiments Labs
- Built in Information Integrity and Public Trust in Foundry
- Tested at Information Integrity and Public Trust in Universe
- Codified in Governance, Security and Society in Standards
- Published in Information Integrity and Public Trust in Reports
- Taught in Faculty of Governance, Policy and International Affairs in Academy
06Validation
How results are checked.
Models are evaluated on held-out data that respects spatial, temporal and group structure. Robustness, calibration and fairness are tested alongside accuracy.
| Check | How it is done |
|---|---|
| Error by group and language | Errors are reported across languages and groups, not only overall |
| Temporal validity | Models are tested on newer data than they were trained on |
| Human review | Automated labels used in research are checked against human coding |
Performance degrades when deployment conditions differ from training data. Models can encode biases and fail without clear warning on rare events.
07Implementation
From mandate to evaluation, by pillar.
The steps an instrument takes from mandate to evaluation, each with the pillar that runs it, and the decisions that stay with the authority. Teaching model with stated illustrative values, not a prediction.
- Mandate and problem framing Competent authority (outside the programme)
A competent authority frames the problem and the mandate under its own rules. Howlett, Ramesh and Perl (2009) - Evidence and system model Foundry
The system is modelled with its feedbacks, data and stated limits. OECD (2014), principle 2 - Options appraised with scenarios Foundry
Instruments are compared under scenarios, with uncertainty shown. Hood (1983) - Exercise with stakeholders Universe
The options are tested in an exercise with those who would implement and bear them. OECD (2014), principle 3 - Standard or guidance drafted Nexus Standards
What works is codified as a standard or guidance, open to comment. Nexus Standards process - Decision by the authority Competent authority (outside the programme)
The authority decides, outside the programme, with the evidence on record. Howlett, Ramesh and Perl (2009) - Capacity and training Risk Academy
Those who implement are trained on the decided instrument and its data. Sendai Framework (2015), priority 2 - Monitoring and public report Reports
Indicators are published with sources, limits and corrections. OECD (2014), principle 5 - Evaluation and revision Nexus Standards
Results return to the standard and the model: the cycle closes. Howlett, Ramesh and Perl (2009)
Methods
- Policy cycle Howlett, Ramesh and Perl (2009), Studying Public Policy, Oxford University Press
- Governance of critical risks OECD (2014), Recommendation of the Council on the Governance of Critical Risks
- Layered drawing of directed graphs Sugiyama, Tagawa and Toda (1981), IEEE SMC 11
Across the six pillars, led by Nexus Consortium
- Experiments Labs
- Built in Information Integrity and Public Trust in Foundry
- Tested at Information Integrity and Public Trust in Universe
- Codified in Governance, Security and Society in Standards
- Published in Information Integrity and Public Trust in Reports
- Taught in Faculty of Governance, Policy and International Affairs in Academy
09Instruments and commitments
Instruments, mandates and commitments.
The instruments, mandates, actors and accountability arrangements a policymaker, diplomat or development agency works with, citing the agreements that set them. Structures follow the frameworks named. Any state or progress drawn is illustrative, not a record.
Accountability cycle
How commitments on artificial intelligence and machine learning for information integrity and public trust are held to account, from commitment to correction.
- CommitGovernmentA commitment with a mandate and a legal basis.
- PlanLead agencyA national plan with owners and milestones.
- ResourceMinistry of financeA budget line or financing on record.
- ImplementAgencies and local governmentDelivery against the plan.
- ReportStatistics officeIndicators reported in agreed definitions.
- ReviewLegislature, audit, civil societyIndependent review of results and gaps.
- CorrectGovernmentChanges to the plan, published with reasons.
Sendai Framework for Disaster Risk Reduction · UNDRRParis Agreement · UNFCCC2030 Agenda and the Sustainable Development Goals · United NationsInternational Health Regulations · WHO
10Stewardship
Who maintains the method.
Artificial Intelligence and Autonomous Systems stewards artificial intelligence and machine learning for the Consortium and reviews how it is applied across platforms.
11Connected
Where this page leads across the Consortium.
12Questions
Questions and answers.
What are the limits of artificial intelligence and machine learning in this field?
Performance degrades when deployment conditions differ from training data. Models can encode biases and fail without clear warning on rare events.
How are results checked?
Models are evaluated on held-out data that respects spatial, temporal and group structure. Robustness, calibration and fairness are tested alongside accuracy.
Are the values on this page real?
The data names, roles and checks are real descriptions of practice. The canvas shows a design of the method's use, not a run; no result is reported here.
Where else is this method applied?
The canonical method page lists every platform that applies it.
ReportsContribute
Contribute evidence from this page.
Reports is the Consortium's quarterly publication. What you know about this subject can become evidence, a case or a correction in the next edition, credited to you and published under CC BY 4.0 where it is your own work.
Global Risks ForumInformation Integrity and Public TrustEvidence Brief
- EmailGateway GW-GRFSend it from this pageEvidence, a correction, a case, a method, a review, a translation or profile input. It arrives tagged with this page, so the editor who owns it receives it.
- ZenodoOpen depositDeposit it and submit to the communityPublish your own record on Zenodo with your ORCID iD, then submit it to the Reports community there. Curators review it; the DOI stays yours.
- ReportsGlobal Risks ForumSee every way to contributeThe contribution types, how each is reviewed, what credit you receive, the licence and what is never accepted.
Never send personal data, sensitive infrastructure detail or anything you have no right to share. A contribution is reviewed before use; sending it confers no authorship or editorial rights, and sponsorship never buys a finding.
Information Integrity and Public Trust
Bring a governance question to the Forum.
Information and enquiries are open today. An enquiry receives a reference and names the next step; applications, payments, compute allocations and registrations open only when their team, terms and review capacity exist.
Legal
Legal documents.
Show all 18 legal documents
Maps, data and figures on this site are shown for context and general information. They are not advice, forecasts or warnings; the limits of the information are set out in the Disclaimers and Data Notices.