Open source evaluation orchestration for AI systems.

EvalHub is a platform for running systematic evaluations of models, agents, and AI systems across multiple frameworks, without locking you into any single one.

Run evaluations against any registered benchmark, whether built-in or one you create yourself, track experiments in MLflow, and store immutable results as OCI artefacts.

It works locally for development and scales on Kubernetes for production.

• Versioned REST API (v1) with OpenAPI specification and interactive docs
• Provider registry with benchmark discovery and category filtering
• Benchmark collections with weighted scoring and compliance requirements
• Bring Your Own Framework implement a single method to add any evaluation framework
• Kubernetes-native job orchestration with resource isolation
• MLflow integration for experiment tracking, lineage, and result comparison
• OCI artefact persistence for reproducible, immutable evaluation results
• Multi-provider batching groups compatible benchmarks to reduce execution time
• Prometheus metrics and OpenTelemetry tracing

Full documentation is available at eval-hub.github.io:

• Overview — architecture and core concepts
• Installation — install server and SDK
• Quick start — run your first evaluation
• Python SDK reference — client and adapter API
• Building adapters — create your own framework adapter

Apache 2.0 — see LICENCE.