# Envisioning · Hindsight | Envisioning Source: https://www.envisioning.com/hindsight/envisioning --- --- 1. [Hindsight](/hindsight) 2. ›Envisioning --- # Envisioning Every publication series from Envisioning in Hindsight, and how its predictions turned out. --- [Publishers](/hindsight)[Forecast vs actual](/hindsight/forecasts)[Side by side](/hindsight/comparisons) --- [Envisioning the future of education technology](/hindsight/envisioning/education)Study posters Validated: two blind graders, an adjudicator on every disagreement, and an agent audit of a fixed sample ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)). 1 publications, 2012 to 2012 · 42 predictions 75% 050%100% hit rate, 21 of 28, 95% interval 53% to 91% with the audit error applied ([D28](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d28-the-audit-error-rate-widens-every-published-interval-d11-issue-61)) Light band: with the audit error. Dark bar: count-only interval 57% to 87%. - hit21 - partial7 - contested1 1. 29claims each graded by two agents who cannot see each other's verdict ([D7](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d7-agentic-verdicts-with-independent-agreement)) 2. → 25agreed agreement beyond chance, kappa 0.63 3. → 4settled the graders disagreed; a third agent decided ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)) 4. → 25audited an agent re-checked a fixed sample of agreed verdicts: 0 corrected, 1 left unsettled ([D11](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d11-validation-thresholds)) 5. → 28published 1 contested: no verdict publishes [Envisioning the future of health technology](/hindsight/envisioning/health)Study posters Validated: two blind graders, an adjudicator on every disagreement, and an agent audit of a fixed sample ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)). 1 publications, 2012 to 2012 · 55 predictions 43% 050%100% hit rate, 12 of 28, 95% interval 23% to 65% with the audit error applied ([D28](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d28-the-audit-error-rate-widens-every-published-interval-d11-issue-61)) Light band: with the audit error. Dark bar: count-only interval 27% to 61%. - hit12 - partial13 - miss3 1. 28claims each graded by two agents who cannot see each other's verdict ([D7](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d7-agentic-verdicts-with-independent-agreement)) 2. → 26agreed agreement beyond chance, kappa 0.88 3. → 2settled the graders disagreed; a third agent decided ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)) 4. → 26audited an agent re-checked a fixed sample of agreed verdicts: 1 corrected, 0 left unsettled ([D11](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d11-validation-thresholds)) 5. → 28published every claim has a verdict [Envisioning Emerging Technologies (Policy Horizons Canada)](/hindsight/envisioning/emerging-technologies)Study posters Validated: two blind graders, an adjudicator on every disagreement, and an agent audit of a fixed sample ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)). 1 publications, 2014 to 2014 · 88 predictions 75% 050%100% hit rate, 41 of 55, 95% interval 60% to 86% with the audit error applied ([D28](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d28-the-audit-error-rate-widens-every-published-interval-d11-issue-61)) Light band: with the audit error. Dark bar: count-only interval 62% to 84%. - hit41 - partial6 - miss8 - contested1 1. 56claims each graded by two agents who cannot see each other's verdict ([D7](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d7-agentic-verdicts-with-independent-agreement)) 2. → 54agreed agreement beyond chance, kappa 0.92 3. → 2settled the graders disagreed; a third agent decided ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)) 4. → 50audited an agent re-checked a fixed sample of agreed verdicts: 0 corrected, 1 left unsettled ([D11](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d11-validation-thresholds)) 5. → 55published 1 contested: no verdict publishes [Envisioning Technology posters](/hindsight/envisioning/technology)Technology placements Validated: two blind graders, an adjudicator on every disagreement, and an agent audit of a fixed sample ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)). 2 publications, 2011 to 2012 · 229 predictions 45% 050%100% hit rate, 49 of 108, 95% interval 28% to 63% with the audit error applied ([D28](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d28-the-audit-error-rate-widens-every-published-interval-d11-issue-61)) Light band: with the audit error. Dark bar: count-only interval 36% to 55%. - hit49 - partial13 - miss46 - unfalsifiable2 - contested3 1. 113claims each graded by two agents who cannot see each other's verdict ([D7](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d7-agentic-verdicts-with-independent-agreement)) 2. → 95agreed agreement beyond chance, kappa 0.75 3. → 18settled the graders disagreed; a third agent decided ([D20](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d20-agents-do-all-grading-adjudication-and-audit-amends-d7-d11)) 4. → 50audited an agent re-checked a fixed sample of agreed verdicts: 1 corrected, 3 left unsettled ([D11](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md#d11-validation-thresholds)) 5. → 110published 3 contested: no verdict publishes How testable the claims were 2% too vague to test 2 of 113. unfalsifiable: the plausible readings give different verdicts. --- Data, code and grading rules are public at [github.com/envisioning/hindsight](https://github.com/envisioning/hindsight). Data under CC BY 4.0 (the BCB Focus survey under ODbL), code under MIT. Every rule is in [the decision log](https://github.com/envisioning/hindsight/blob/main/docs/DECISIONS.md). Think a verdict is wrong? [Open a dispute](https://github.com/envisioning/hindsight/issues/new?template=dispute-verdict.yml). ---