Most catalog entries map to exactly one category and one insurance exposure. Ranking by how many distinct categories and exposures an eval suite’s entries span surfaces the rare cases where the field leans on one underlying suite across many fronts, rather than the hundreds of standalone, single-purpose tools and benchmarks that make up the rest of the catalog.
| # | Eval suite | Catalog entries | Categories | Exposures |
|---|---|---|---|---|
| 1 | inspect_evals | 19 | 10 | 7 |
| 2 | 0DIN AI Scanner | 1 | 1 | 1 |
| 3 | 3CB (Catastrophic Cyber Capabilities Benchmark) | 1 | 1 | 1 |
| 4 | A.I.G (AI-Infra-Guard) | 1 | 1 | 1 |
| 5 | A.S.E (AICGSecEval) | 1 | 1 | 1 |
| 6 | A2A Scanner | 1 | 1 | 1 |
| 7 | ACES (Agent Capability Evaluation Suite) | 1 | 1 | 1 |
| 8 | Adversarial Robustness Toolbox (ART) | 1 | 1 | 1 |
| 9 | Aegis 2.0 / Nemotron Content Safety V2 | 1 | 1 | 1 |
| 10 | Agent Governance Toolkit | 1 | 1 | 1 |
Beyond the top one or two suites, entries and counts tie at the minimum (one category, one exposure) — that tie is the honest shape of the data, not a ranking we smoothed over.
Possible duplicates
The same benchmark sometimes appears more than once under a different name or repo — most often an original implementation and a port into another harness (e.g. an Inspect port). These are grouped below as related, not merged or deleted: when we weren’t confident two entries were the same benchmark, we left them as separate, unrelated rows.
- 3CB (Catastrophic Cyber Capabilities Benchmark)Cyber offense capability · Breach/cyber · apartresearch/3cb
- 3CB (Inspect port)Cyber offense capability · Breach/cyber · UKGovernmentBEIS/inspect_evals
- Awesome-AI-SecurityMeta-lists & catalogs · Unmapped · TalEliyahu/Awesome-AI-Security
- awesome-ai-securityMeta-lists & catalogs · Unmapped · ottosulin/awesome-ai-security
- RE-BenchAutonomy, R&D & self-replication · Unmapped · METR/RE-Bench
- RE-Bench (Inspect port)Autonomy, R&D & self-replication · Unmapped · METR/inspect-tasks-public
Wanich Risk Lab, 2026.