Evaluation and benchmarking harnesses

Agentic eval systems, reasoning benchmarks, and open agent benchmarks.

#	Project	Stars	Tier	OSS	Description
1	Agent Lightning evalstrainingpython	18.4k	complex	✅	Microsoft's training-oriented harness: optimization loops for agent behavior—when you need to improve policies over rollouts, not only score a fixed prompt.
2	SWE-bench evalssandboxpython	5.9k	slightly complex	✅	LMs resolve real GitHub issues; Docker harness, instance IDs; standard for code-agent evals.
3	AgentBench evalssandboxragworkflowpython	3.7k	complex	✅	ICLR'24 benchmark: agents across AlfWorld, DB, knowledge graphs, OS, webshop; Docker Compose, function-calling interface.
4	inspect_ai evalssandboxpython	2.8k	complex	✅	Inspect AI core: composable eval tasks, sandboxes, scorers, and multi-model runs; the framework behind inspect_evals, not just the task bundle.
5	WebArena python	1.6k	complex	✅	Realistic web env (e.g. e‑commerce, CMS, dev tools); 812 tasks; measures end-to-end web agent success.
6	WebVoyager evalsvision	1.1k	slightly complex	✅	End-to-end web agent with LMMs: screenshots + actions on real sites; benchmark on 15 sites, GPT-4V for automatic eval.
7	agent-qa mcpmemorysandboxclitypescript	887	slightly complex	⚠️ FSL-1.1-ALv2	Self-improving QA harness for web and mobile apps: natural-language tests, memory-backed self-healing, dashboard/CLI, MCP and skills support, plus sandboxed hooks for production regression checks.
8	ClawBench evalsvisionsandboxpython	801	complex	✅	Open web-agent evaluation harness: runs selectable agents in isolated Docker containers across 153 live-site tasks (plus 130 in V2), intercepts irreversible requests, and records video, screenshots, HTTP traffic, actions, and agent messages for replayable scoring.
9	swe-smith trainingpython	775	slightly complex	✅	Data generation for SWE agents; 50k+ instances across 128 repos; used for SWE-agent-LM training.
10	SWE-Gym evalstrainingpython	742	slightly complex	✅	Training and evaluation for SWE agents and verifiers (ICML 2025).
11	ARC-AGI-2	738	super simple	✅	ARC Prize task set: grid-based abstraction/reasoning; public and private splits for generalization.
12	Terminal-Bench evalsclipython	738	slightly complex	✅	The terminal-task benchmark coding agents now cite next to SWE-bench: hard, containerized terminal tasks scored end to end. Terminal-Bench 2.0 runs on the harbor evaluation framework; the 1.0 tasks live on in the org's terminal-bench-1 repo.
13	inspect_evals evalssandbox	675	slightly complex	✅	UK AISI/Arcadia/Vector: GAIA and other evals in Inspect AI; level 1–3, sandboxed, tool-calling solvers.
14	arc-agi-benchmarking evalsprovider-agnosticpython	362	mostly simple	✅	Runner for ARC-AGI: multi-provider (OpenAI, Anthropic, Gemini, etc.), rate limits, retries, and scoring.
15	VitaBench	177	complex	✅	ICLR'26: 66 tools, real-world apps (delivery, travel, retail); 100 cross-scenario + 300 single-scenario tasks; adopted by Qwen/Seed.
16	AgencyBench evalssandboxpython	100	complex	✅	Long-horizon agent benchmark: 32 scenarios, 138 tasks, ~1M tokens and ~90 tool calls; Docker sandbox and rubric-based + LLM judges.
17	letta-evals memorypython	82	mostly simple	✅	Eval harness for stateful Letta agents; configurable suites and grading (LLM or rule-based) so you can measure what you ship.
18	SUPER sandboxpython	58	slightly complex	✅	Agents that set up and run ML/NLP from GitHub repos; 45 expert problems, 152 masked tasks, 602 AutoGen tasks; Docker-based.
19	TRAIL	24	mostly simple	✅	Trace reasoning and agentic issue localization; 148 long-context traces, 841 errors, 20+ error types; Hugging Face dataset.