# Trajekt

> Trajekt is an open-source, self-hosted flight recorder for AI agents. It records every run of an agent (steps, tool calls with arguments and results, tokens, latency, cost), replays runs step by step, compares versions (app, agent, prompt, model) and detects regressions. One Postgres, one Docker Compose file.

## Key facts

- Ingestion: `POST /v1/traces` with an ingestion key (`Authorization: Bearer tk_live_…`). Accepts OTLP over HTTP, JSON or protobuf (OpenTelemetry GenAI semantic conventions, Vercel AI SDK telemetry, OpenInference — LangChain, LlamaIndex, CrewAI — and OpenLLMetry), Langfuse trace exports, Vercel AI SDK `generateText` results and Trajekt trajectories. Not supported: gRPC. Existing Langfuse projects can be imported with `pnpm import-langfuse`.
- Each run carries an agent name and four comparison dimensions: app version (`service.version`), agent version (`trajekt.agent.version`, defaults to the app version), prompt version (`trajekt.prompt.version`) and model (read from LLM calls).
- Per-run metrics: status, duration, cost (from the source or estimated from tokens), steps, tool calls, failed tool calls, repeated tool calls (same tool, same arguments as an earlier successful call, with no state-changing call in between), retries, tool path.
- Quality scores (higher is better, usually 0–1): `trajekt.score.<name>` span attributes, Langfuse scores, or `POST /v1/scores` with a traceId after the fact. A drop is a regression, in production and in eval suites.
- Eval runs: runs tagged with `trajekt.scenario` come from an eval suite (mocked tools and data, run in dev or CI). They are excluded from production metrics and alerts, and compared scenario by scenario between two versions (paired Wilcoxon test across the suite, p < 0.05, Holm-corrected, +15% / +5 pts / −0.05 score minimum, 8 scenarios minimum; per scenario: starts failing, score −0.25, +50%, new repeated calls, new tool path). Result: pass or fail. `GET /api/agents/:name/evals?dimension=agent&a=&b=` returns it for CI; the GitHub Action `LeoLR/Trajekt/actions/eval-suite@main` comments it on pull requests and fails the job on a regression.
- `@trajekt-ai/core` also provides `assertTrajectory(run, expectations)` for tests (tools called, order, arguments, budgets) and `compareScenarios(a, b)` to compare two suites without a server.
- In production, a regression is flagged only when the change is statistically significant (one-sided test, p < 0.01, Holm-corrected across metrics), large enough (+15% on a mean, +3 points on a rate, −0.05 on a score), with at least 20 runs on each side.
- Sensitive data is masked before storage: secrets by default; emails, IBANs, cards, phones, IPs and custom patterns with `TRAJEKT_REDACT` / `TRAJEKT_REDACT_PATTERNS`.
- Several projects per instance, each with its own keys; retention (`TRAJEKT_RETENTION_DAYS`) and ingestion limits are configurable.

## MCP server (read-only)

Endpoint: `POST /api/mcp` (Streamable HTTP, JSON responses), with a read key (`Authorization: Bearer tk_read_…`). The public demo at https://trajekt-demo.vercel.app/api/mcp needs no key.

Tools:
- `list_agents` — agents, 24 h activity, health, current regressions. Start here.
- `list_versions(agent, dimension)` — values of app | agent | prompt | model with their metrics.
- `compare(agent, dimension?, a?, b?, mode?)` — B vs A on eight metrics, one verdict each; without a/b, latest vs previous; mode "window" = last 24 h vs previous 7 days.
- `compare_eval_suite(agent, dimension?, a?, b?)` — eval runs only: PASS/FAIL between two versions, suite-level verdicts, scenarios that got worse and what changed.
- `get_regressions(agent?)` — regressions detected now and alerts sent.
- `find_runs(agent?, status?, app_version?, agent_version?, prompt_version?, model?, scenario?, kind?, repeated_tool_calls_only?, limit?)` — kind: live (production) or eval
- `explain_run(run_id)` — tool path and the failed, retried or repeated steps.
- `diff_runs(run_id, with_run_id?)` — two runs aligned step by step, and the configuration that changed.

Report the numbers the tools return; do not recompute them.

## Docs

- [Documentation](https://trajekt-eight.vercel.app/docs/): install, send traces, dashboard, regressions, MCP, @trajekt-ai/core, testing, API
- [API contract](https://github.com/LeoLR/trajekt/blob/main/docs/api-contract.md)
- [Live demo](https://trajekt-demo.vercel.app)