Install, record, replay.

Everything you need to run Trajekt next to your agents, send it traces, read what they did and get told when a new version gets worse.

Overview

Trajekt has two parts:

• The server (apps/dashboard): a Next.js app with one Postgres. It receives traces on POST /v1/traces, stores one row per run, and serves the dashboard.
• The library (@trajekt-ai/core): reads a trace from any supported format into one model, computes its metrics and compares groups of runs. The server uses it; you can use it in your own tests.

Your agent does not need a Trajekt SDK. It sends standard OpenTelemetry spans, or you post a trace file.

Quickstart

Docker and Git are the only requirements.

git clone https://github.com/LeoLR/trajekt.git
cd trajekt
cp .env.example .env     # set DASHBOARD_PASSWORD and DASHBOARD_SESSION_SECRET
docker compose up -d --build

docker compose exec dashboard pnpm create-project my-project

docker compose exec dashboard pnpm seed

Open http://localhost:3000 and sign in with DASHBOARD_PASSWORD. With the demo data you will see an agent, sentry-triage, whose version 1.5.0 starts looping on a tool call, costs more and fails more often than 1.4.0 — that is what a regression looks like in Trajekt.

Then send your own traces.

Installation

Docker Compose

The compose file starts two containers: Postgres 16 and the dashboard. Migrations run when the dashboard container starts. The dashboard and the ingestion endpoint share port 3000.

docker compose up -d --build     # start
docker compose logs -f dashboard # follow logs
docker compose down              # stop, keep data
docker compose down -v           # stop and delete the database volume

From source

You need Node.js 22, pnpm 10 and a Postgres database.

pnpm install
pnpm --filter @trajekt-ai/core build    # the dashboard imports the built package

cd apps/dashboard
cp .env.example .env                 # DATABASE_URL, DASHBOARD_PASSWORD, DASHBOARD_SESSION_SECRET
pnpm migrate
pnpm create-project my-project
pnpm dev                             # http://localhost:3000

Configuration

Sending traces

Every request to POST /v1/traces needs the project API key: Authorization: Bearer tk_live_…. Only its hash is stored, so a lost key cannot be recovered — create a new project.

OpenTelemetry

Point any OTLP/HTTP exporter at the root URL of Trajekt; the exporter adds /v1/traces itself. JSON and protobuf encodings both work, so the default settings of the Python, Java or Go SDKs are fine.

OTEL_SERVICE_NAME=sentry-triage
OTEL_RESOURCE_ATTRIBUTES=service.version=3.2.0,trajekt.agent.version=1.5.0,trajekt.prompt.version=v8
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3000
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf   # or http/json
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer tk_live_..."

Or in code, with the Node SDK:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

new NodeSDK({
  serviceName: 'sentry-triage',
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:3000/v1/traces',
    headers: { Authorization: `Bearer ${process.env.TRAJEKT_API_KEY}` },
  }),
}).start();

Trajekt reads spans that follow the OpenTelemetry GenAI semantic conventions: gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens, gen_ai.tool.name, gen_ai.tool.call.arguments / result, gen_ai.input.messages / output.messages. It also reads OpenInference and OpenLLMetry spans (see below). A span is a failure when its status is ERROR or when it records an exception.

An exporter sends a trace in several batches, children first. Trajekt merges every batch with the same traceId into one run; sending a batch twice does not duplicate steps.

Instrument your code

If your agent is hand-written, @trajekt-ai/core has two helpers that create the spans Trajekt expects, with any OpenTelemetry tracer. Tool arguments matter: without them, repeated calls cannot be detected.

import { trace } from '@opentelemetry/api';
import { traceRun, traceTool } from '@trajekt-ai/core';

const tracer = trace.getTracer('sentry-triage');

await traceRun(tracer, { agent: 'sentry-triage', agentVersion: '1.5.0', promptVersion: 'v8' }, async (run) => {
  const issue = await traceTool(tracer, 'sentry.getIssue', { id }, () => sentry.getIssue(id));
  const ticket = await traceTool(tracer, 'jira.createIssue', { project, summary }, () => jira.createIssue(project, summary));
  run.score('correct', ticket.project === expectedProject); // optional quality score
  return ticket;
});

Calls to the model: wrap the Anthropic client once and every messages.create is recorded — model, tokens including the prompt cache (priced as such), stop reason, and the messages unless { captureContent: false }. Other SDKs: traceLlm(tracer, { model, input }, () => call()) reads Anthropic and OpenAI responses.

import Anthropic from '@anthropic-ai/sdk';
import { traceAnthropic } from '@trajekt-ai/core';

const anthropic = traceAnthropic(new Anthropic(), tracer);
// unchanged agent code:
const msg = await anthropic.messages.create({ model: 'claude-sonnet-5-5', max_tokens: 1024, system, messages, tools });

traceRun also takes appVersion, environment and scenario (for eval runs), and exposes run.traceId to attach a score later. An exception that escapes the run marks it as failed; one thrown by a tool marks only that step.

Vercel AI SDK

With an OpenTelemetry exporter set up as above, turn telemetry on for each call. functionId becomes the agent name.

const result = await generateText({
  model,
  prompt,
  tools,
  experimental_telemetry: {
    isEnabled: true,
    functionId: 'trip-planner',
    metadata: { appVersion: '3.2.0', agentVersion: '2.3.0', promptVersion: 'v8' },
  },
});

Without OpenTelemetry, you can post the result object itself (the value with steps) to /v1/traces. It carries no timestamps and no trace id, so each post is a separate run, and the agent name must come from a header.

LangChain, LlamaIndex, CrewAI…

Python frameworks are usually traced with OpenInference or OpenLLMetry. Trajekt reads both: model, tokens, cost and messages of LLM spans; name, arguments and result of tool spans; the agent name from the AGENT span or the workflow name. Point their OTLP exporter at Trajekt:

from openinference.instrumentation.langchain import LangChainInstrumentor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="http://localhost:3000/v1/traces",
    headers={"Authorization": "Bearer tk_live_..."},
)))
trace.set_tracer_provider(provider)
LangChainInstrumentor().instrument()

Set the versions with resource attributes (OTEL_RESOURCE_ATTRIBUTES=service.version=3.2.0,trajekt.agent.version=1.5.0,trajekt.prompt.version=v8) so you can compare them.

Langfuse exports and trace files

Already on Langfuse? Import its traces, scores included, without touching your agent. Run it on a schedule: it only adds what is new and never duplicates a run.

LANGFUSE_HOST=https://cloud.langfuse.com \
LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-... \
pnpm --filter dashboard import-langfuse --since 7d --name sentry-triage

The versions come from the trace: release becomes the app version, version the agent version, metadata.promptVersion the prompt version. A single Langfuse trace — the response of GET /api/public/traces/:traceId — can also be posted as-is, as can an OTLP/JSON file.

curl -X POST http://localhost:3000/v1/traces \
  -H "Authorization: Bearer tk_live_..." \
  -H "Content-Type: application/json" \
  -H "X-Trajekt-Agent: support-bot" \
  --data @trace.json

Example files for each format: OTLP, Langfuse, Vercel AI SDK.

Agent, versions and model

Each run carries five labels. The agent name groups runs together; the four others are the dimensions you can compare on:

• App version — the release of the application that embeds the agent (your service, your web app).
• Agent version — the agent itself: its code, tools and orchestration. It defaults to the app version when you do not set it.
• Prompt version — the version of the system prompt or prompt template.
• Model — read from the LLM calls of the run (the most used one), nothing to set.

For each field the first value found wins.

Headers are used only when the trace carries no value. A run without any agent name is filed under unknown.

Quality scores

Cost, duration and failures say how an agent ran, not whether it did the right thing. A version that is cheaper but files tickets in the wrong project looks better on every other metric. Scores close that gap: one number per run and per name, higher is better, ideally between 0 and 1 — correct or not, share of expectations met, an LLM judge's grade, a thumbs up.

• In the trace: an attribute trajekt.score.correct = 1 on any span, or run.score('correct', true) with traceRun. Langfuse scores come along with the trace.
• After the fact, when you know: the user's feedback, the merge request that got merged, a review. Send it with the trace id your app already has. A score sent this way is never overwritten by the trace.

curl -X POST https://trajekt.example.com/v1/scores \
  -H "Authorization: Bearer tk_live_..." -H "Content-Type: application/json" \
  -d '{ "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "name": "correct", "value": 0, "comment": "wrong Jira project" }'

It also takes a list ({ "scores": [ … ] }, up to 500) and a Trajekt runId instead of a traceId. In tests, scoreTrajectory(run, expectations) turns assertions into a score (the share of expectations met).

Scores are compared like the other metrics, but a drop is the regression: in production (at least −0.05, significant, 20 scored runs on each side — alerts included), and in eval suites, where they come first and a scenario whose score drops by 0.5 or more fails the suite. They show on each run, in the runs list when below 1, and per version on the agent page.

Using the dashboard

Comparing versions

On an agent page, Compare by picks the dimension — app version, agent version, prompt version or model — and the comparison follows it:

• Latest vs previous: the most recent value against the one before it.
• Pick two: any A and B, e.g. prompt v7 against v8, or app 3.1.0 against 3.2.0. For models, A defaults to the most used one, so a cheaper model under test shows up as B.

The URL keeps the selection (/agents/sentry-triage?dim=prompt&compare=ab&a=v7&b=v8), so a comparison can be shared as a link. The versions table, the tool paths and the run filters follow the same dimension.

How the run metrics are computed:

• Repeated tool call: the same tool with the same arguments (key order ignored) as an earlier call that succeeded, with no state-changing call in between. Reading a file again after editing it, or re-running the tests after a fix, is not a loop; doing the same write twice in a row is. A tool counts as state-changing from its name (write, edit, create, update, post, run, bash… but not get, read, search, list…); setMutatingTools() in @trajekt-ai/core overrides the rule. Calls without captured arguments are never counted.
• Retry: a call to the same tool or model right after that call failed. A retry is not counted as a repeat.
• Status: with a single root span, the root decides — errors caught inside the run do not fail it. Otherwise the last step that is not a system span decides.
• Cost: the cost from the source when there is one (Langfuse), otherwise estimated from tokens and shown with ≈.
• Tool path: tools in order; retries are dropped and consecutive repeats become ×n, e.g. sentry.getIssue → gitlab.searchCode ×3 → jira.createIssue.

Regression alerts

Two comparisons run for each agent that had runs in the last 24 h:

• the last 24 h against the 7 days before;
• the latest agent version, prompt version and app version against the previous one, each while it is less than 7 days old — a prompt change alone can raise an alert.

Eight metrics are compared — failed runs, failed tool calls, mean duration, mean cost, steps per run, tokens per run, repeated tool calls, share of runs on a tool path never seen before — plus every quality score present on both sides. A metric is flagged only when all three hold:

• the change is statistically significant (one-sided test, p < 0.01), and still is after correcting for the number of metrics tested at once (Holm) — otherwise one metric in ten would cry wolf by chance;
• it is large enough to matter: +15% on a mean, +3 points on a rate, −0.05 on a score;
• there are at least 20 runs on each side.

Run the check on a schedule. It covers every project; each new regression is stored and posted to Slack once per agent and metric every 24 h. The same call purges old runs when TRAJEKT_RETENTION_DAYS is set.

0 * * * * curl -fsS -X POST https://trajekt.example.com/api/alerts/check -H "Authorization: Bearer $CRON_SECRET"

docker compose exec dashboard pnpm check-regressions

Projects & operations

A project groups runs and has its own API keys: one per team, product or environment. Create and open projects on the /projects page of the dashboard — the keys are shown once, at creation — or from the command line with pnpm --filter dashboard create-project <name>. There is still one sign-in password for the whole instance.

Sensitive data

Prompts, tool arguments and tool results often carry customer data. Trajekt masks it before anything is stored — messages, arguments, results, errors and the original payload alike. By default it masks secrets: API keys and tokens (Anthropic, OpenAI, GitHub, GitLab, Slack, AWS, Stripe, Google, Trajekt), bearer headers, JWTs, and the value of keys such as password, token, api_key or authorization. Add what your domain needs:

TRAJEKT_REDACT=secrets,email,iban,card,phone
TRAJEKT_REDACT_PATTERNS=[{"name":"customer","pattern":"CUST-\\d{6}"},{"name":"contract","pattern":"CTR-[A-Z0-9]{8}"}]

A match becomes a marker — [email], [iban], [card] (only numbers that pass the Luhn check), [customer] — always the same for the same kind of value, so metrics and repeated-call detection still work. If the configuration is invalid, ingestion refuses the request (503) rather than storing data you thought was masked. In your own code, createRedactor and redactTrajectory from @trajekt-ai/core apply the same rules; with traceAnthropic, { captureContent: false } keeps prompts out of the traces entirely.

• Retention: TRAJEKT_RETENTION_DAYS=90 deletes older runs (and their scores) each time the cron runs; alerts are kept at least 90 days.
• Storage: a run stores every step with its original payload. TRAJEKT_STORE_RAW=false drops the payload and roughly halves the size; Postgres already compresses the rest.
• Ingestion limits: 600 requests per minute and per project (TRAJEKT_INGEST_RATE_PER_MIN), 20 MB per request after gzip (TRAJEKT_MAX_BODY_MB). OpenTelemetry exporters retry on 429 and honour Retry-After. The rate limit is kept in memory, per server instance: a safety net, not a quota.

Ask your agents (MCP)

Trajekt is an MCP server. Connect Claude, Cursor or your own agent to it and ask in plain language: "did sentry-triage get worse since prompt v8?", "why did this run fail?", "is the cheaper model good enough?".

The tools return answers Trajekt has already computed — statistical verdicts, run summaries, differences — as short text plus structured JSON. Your agent never has to read thousands of spans, so its answers stay short and grounded in the numbers. All tools are read-only.

claude mcp add --transport http trajekt http://localhost:3000/api/mcp \
  --header "Authorization: Bearer tk_read_..."

{
  "mcpServers": {
    "trajekt": {
      "type": "http",
      "url": "http://localhost:3000/api/mcp",
      "headers": { "Authorization": "Bearer tk_read_..." }
    }
  }
}

Try it without installing anything: the live demo exposes its MCP endpoint without a key at https://trajekt-demo.vercel.app/api/mcp.

Read API keys

Keys have a scope. An ingestion key (tk_live_…) can only send traces; a read key (tk_read_…) can only read — the MCP server and every GET /api/* route. Give agents a read key, never your ingestion key.

docker compose exec dashboard pnpm create-read-key claude-desktop

Tools

There is deliberately no free-form query tool: traces contain customer data, and an agent writing its own SQL is both fragile and risky. A plain-text summary of these docs for agents is at /llms.txt.

@trajekt-ai/core

Cost estimation. Spans rarely carry a cost, so it is estimated from tokens with a table of indicative public prices for common Anthropic (including Claude 5.x), OpenAI, Google and Mistral models. Prompt-cache reads and writes are priced separately when the spans report them. The longest matching model prefix wins; an unknown model gets no estimate rather than a wrong one. Prices change: override them.

import { setPricing } from '@trajekt-ai/core';
setPricing({ 'my-fine-tune': { input: 1.2, output: 4.8 } }); // USD per million tokens

TRAJEKT_PRICING={"my-fine-tune":{"input":1.2,"output":4.8}}

Testing your agent

Production tells you an agent got worse after the fact. While you develop it, replay a fixed set of scenarios — mocked tools and data — on every change, and check that nothing regressed before it ships.

Assert on a run

Assert on what the agent did, not only on its final answer. Works with node:test, vitest or jest; the error lists every expectation that failed, with the tool path of the run.

import { test } from 'node:test';
import { readFileSync } from 'node:fs';
import { normalize, assertTrajectory } from '@trajekt-ai/core';

test('npe-checkout: opens a PAY ticket, no loop, within budget', () => {
  const run = normalize(JSON.parse(readFileSync('runs/npe-checkout.json', 'utf8')));
  assertTrajectory(run, {
    status: 'ok',
    order: ['sentry.getIssue', 'gitlab.readFile', 'jira.createIssue'],
    calledWith: [{ tool: 'jira.createIssue', args: { project: 'PAY' } }],
    notCalls: ['gitlab.createMergeRequest'],
    maxRepeatedToolCalls: 0,
    maxSteps: 12,
    maxCostUsd: 0.08,
  });
});

Also available: calls, maxCallsPerTool, maxToolCalls, maxFailedToolCalls, maxRetries, maxDurationMs, maxTokens. calledWith matches arguments partially (only the keys you give), and reads arguments serialized as JSON.

Eval suites

One run is not enough to judge a change: the model is not deterministic. Run each scenario a few times, on each version, and send the runs to Trajekt with a trajekt.scenario attribute:

import { trace } from '@opentelemetry/api';

for (const scenario of scenarios) {          // e.g. 10–30 mocked Sentry issues
  for (let i = 0; i < 3; i++) {              // a few repeats per scenario
    await trace.getTracer('eval').startActiveSpan('invoke_agent sentry-triage', async (span) => {
      span.setAttributes({
        'gen_ai.agent.name': 'sentry-triage',
        'trajekt.scenario': scenario.name,
        'trajekt.agent.version': process.env.GIT_SHA ?? 'dev',
        'trajekt.prompt.version': PROMPT_VERSION,
      });
      await runAgent(scenario.input, { tools: mockedTools(scenario) });
      span.end();
    });
  }
}

Runs with a scenario are eval runs: they are kept out of production metrics, trends and alerts, and compared on /agents/:name/evals. Each scenario is compared with itself across the two versions, which is far more sensitive than comparing averages:

• Across the suite: a paired test (Wilcoxon signed-rank, one-sided, p < 0.05, Holm-corrected across metrics) on each metric and each quality score. Duration, cost, steps and tokens are compared as ratios, so a large scenario does not outweigh a small one. A metric is flagged if the typical change is also large enough: +15%, +5 points on a rate, −0.05 on a score. It needs at least 8 scenarios.
• Scenario by scenario: no statistics with three runs, plain thresholds instead — a scenario that starts failing (e.g. 0/3 → 2/3), a score down by 0.25 or more, +50% duration, cost, steps or tokens, a tool call repeated that was not before, a tool path never seen in the baseline.
• Result: fail if a metric regresses across the suite, if a scenario starts failing, or if a scenario's score drops by 0.5 or more. A new tool path is shown but never fails the suite.

Each scenario links to a step-by-step diff of the two versions. Compare on any dimension: agent version, prompt version, app version or model.

In CI

GitHub Actions. Record the suite on main under the commit SHA, run it again on each pull request, and let the action compare the two. It writes the result to the job summary, posts one comment on the pull request (updated on each push) with the scenarios that got worse and a link to their diff, and fails the job on a regression.

name: Agent evals
on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  evals:
    runs-on: ubuntu-latest
    env:
      AGENT_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
    steps:
      - uses: actions/checkout@v4
      # Your suite: mocked tools, a few runs per scenario, each traced with
      # trajekt.scenario and trajekt.agent.version=$AGENT_SHA.
      - run: npm ci && npm run eval
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: ${{ vars.TRAJEKT_URL }}
          OTEL_EXPORTER_OTLP_PROTOCOL: http/json
          OTEL_EXPORTER_OTLP_HEADERS: Authorization=Bearer ${{ secrets.TRAJEKT_INGEST_KEY }}
      - if: github.event_name == 'pull_request'
        uses: LeoLR/Trajekt/actions/eval-suite@main
        with:
          url: ${{ vars.TRAJEKT_URL }}
          read-key: ${{ secrets.TRAJEKT_READ_KEY }}
          agent: sentry-triage
          baseline: ${{ github.event.pull_request.base.sha }}
          candidate: ${{ env.AGENT_SHA }}

Inputs: dimension (agent by default; prompt, app or model to compare on another label), fail-on-regression and comment (both true). Outputs: result and report-url. The baseline commit must have been evaluated — that is what the push trigger on main is for. All options are in the action's README.

Any other CI calls the same endpoint:

curl -s -H "Authorization: Bearer $TRAJEKT_READ_KEY" \
  "$TRAJEKT_URL/api/agents/sentry-triage/evals?dimension=agent&a=$BASE_SHA&b=$GIT_SHA" \
  | jq -e '.comparison.result == "pass"' || { echo "eval suite regressed"; exit 1; }

Without a server, compare two saved runs of the suite in code:

import { normalize, scenarioRunOf, compareScenarios } from '@trajekt-ai/core';

const baseline = loadRuns('eval/baseline/').map((t) => scenarioRunOf(normalize(t)));
const current = loadRuns('eval/current/').map((t) => scenarioRunOf(normalize(t)));
const r = compareScenarios(baseline, current);
if (r.result === 'fail') {
  console.error(r.reasons.join('\n'));
  process.exit(1);
}

A runnable version of all this — a toy agent, mocked scenarios, behaviour tests and the comparison — is in examples/eval-suite. An agent can ask the same question through MCP with compare_eval_suite: "did my branch make sentry-triage worse?".

Test Trajekt itself

pnpm --filter @trajekt-ai/core test

The dashboard tests run the real SQL against a throwaway database, created and dropped by each test file. They need a Postgres that accepts the postgres user without a password:

docker run -d --name trajekt-test -p 54329:5432 -e POSTGRES_HOST_AUTH_METHOD=trust postgres:16-alpine
pnpm --filter dashboard test

TEST_PG_HOST, TEST_PG_PORT and TEST_PG_USER change the target (defaults: 127.0.0.1, 54329, postgres). pnpm --filter dashboard preview renders every dashboard page with simulated data into static HTML files, without Next.js.

End-to-end tests run the built app (next start) in a real browser with Playwright: sign-in, every page, project creation, ingestion in JSON and protobuf, scores, redaction and MCP. They live in e2e/; the CI workflow shows how to start the app before running npm test there.

API reference

Read routes accept the dashboard session or a read key: Authorization: Bearer tk_read_….

Errors always have the shape { "error": { "code", "message" } }. The full contract is in docs/api-contract.md.

Troubleshooting

The exporter cannot connect, or uses port 4317

That is gRPC, which Trajekt does not accept. Use OTLP over HTTP: OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf (or http/json) and the dashboard's URL.

429 RATE_LIMITED or 413 PAYLOAD_TOO_LARGE

Too many requests per minute for this project, or a batch too large. Exporters retry on 429; for 413 lower the batch size (OTEL_BSP_MAX_EXPORT_BATCH_SIZE) or raise TRAJEKT_MAX_BODY_MB.

422 UNKNOWN_FORMAT

The JSON is valid but is not one of the supported shapes. The message lists the top-level keys that were found.

401 UNAUTHORIZED

The Authorization header is missing, or the key belongs to no project. Keys start with tk_live_.

Runs appear under unknown

No agent name was found. Set OTEL_SERVICE_NAME (the SDK default, unknown_service, is ignored) or send X-Trajekt-Agent.

Cost shows —

The model is not in the price table, or the spans carry no token counts. Add the model with TRAJEKT_PRICING.

Every metric says "not enough data"

A production comparison needs at least 20 runs on each side, an eval suite at least 8 scenarios.

A metric moved a lot but stays "stable"

The change is not significant once corrected for the number of metrics tested, or it is below the minimum effect. Hover the verdict to see why.

The sign-in page says sign-in is disabled

Set DASHBOARD_PASSWORD and DASHBOARD_SESSION_SECRET (16 characters or more) and restart.

Missing something? Open an issue.